Skip to main content

5.4 Cost and token management

Track it from usage​

Every response carries the numbers. Log all of them.

FieldMeaningRelative cost
input_tokensUncached input1×
cache_creation_input_tokensWritten to cache~1.25×
cache_read_input_tokensServed from cache~0.1×
output_tokensGenerated, including thinkingSeveral × input

Cost per request is therefore:

cost = input_tokens × input_rate
+ cache_creation_tokens × input_rate × 1.25
+ cache_read_tokens × input_rate × 0.10
+ output_tokens × output_rate

Estimate before sending with client.messages.count_tokens(...) — same request shape, no generation. Never a third-party tokenizer: it does not match Claude's.

Cost per completed task, not per request

A cheaper model that needs two retries and a fallback costs more than one correct call. Instrument the task, counting every request it took, including retries, validation failures, and agent loop turns. This is also why unbounded agent loops are a cost bug: each turn resends the whole history.

Prompt caching​

A prefix match. Render order is tools → system → messages; any byte change invalidates everything after it.

MechanicDetail
Breakpointcache_control: {"type": "ephemeral"} on a content block, or top-level on the request to auto-cache the last cacheable block
Max breakpoints4 per request
TTL5 minutes by default; ttl: "1h" available
Minimum cacheable prefixModel-dependent (roughly 512–4096 tokens) — shorter prefixes silently do not cache
Verificationusage.cache_read_input_tokens above zero on the second identical-prefix request
ScopeCaches are model-scoped — switching models starts cold

Placement​

[ tools ] ← deterministic order, frozen
[ system: rules ] ← frozen wording
[ system: corpus ] ← ← cache_control breakpoint here
[ messages: history] ← grows; a later breakpoint can check-point it
[ messages: latest ] ← volatile: the question, timestamps, per-request IDs

Check-pointing a long conversation: put a breakpoint at a stable point in the history so each turn re-reads the prefix from cache instead of paying full price for the whole transcript. With up to four breakpoints you can hold tools+system, an early history segment, and a later one — while the newest turn stays outside the cache.

Silent invalidators​

InvalidatorFix
Date.now() / datetime.now() in the system promptMove it after the last breakpoint, or drop it
A request ID, session ID, or user name in the prefixMove to the user message
Non-deterministic JSON key order in tool definitions or serialized resultsSort keys
A tool list that varies per requestFreeze the set, or split stable and volatile tools around the breakpoint
Editing the top-level system mid-conversationAppend a mid-conversation {"role": "system", ...} message instead — model-gated, but it preserves the prefix
Prefix shorter than the model's minimumConsolidate, or accept no caching
A model or effort change mid-conversationExpect a cold cache — an effort change invalidates the messages cache

The lever order​

Free wins before tradeoffs. Every lever below the line costs you something.

#LeverTypical effect
1Prompt caching on a large stable prefixUp to ~90% off the cached portion
2Input hygiene — prune tool output, drop dead instructions, trim examples that stopped helpingProportional, permanent
3Loop hygiene — bound history, cap turns, stop resending what the model no longer needsCompounding on agents
4Output hygiene — ask for the fields you need, not prose around them; cap max_tokens sensiblyOutput is the expensive side
5Batch for latency-tolerant work50%
—— tradeoffs below this line —
6Lower effort on routes where quality holdsLarge, measurable
7Task budgets on agentic runsBounds the worst case
8Smaller model on well-specified routesLarge, but re-run the evals
9Multi-model cascadeMeasure single-model-lower-effort first; a cascade forfeits cross-model cache reuse

Budgeting a workload​

  1. Sample real inputs and measure tokens with count_tokens — do not estimate from character counts.
  2. Separate the stable prefix from the per-request portion; only the second scales with volume once caching works.
  3. Multiply by volume for input, cached read, and output separately.
  4. Add the retry and validation-failure rate you actually observe.
  5. Add agent loop turns — an agent's cost is turns × growing history, not one request.
  6. Compare against the ceiling, then walk the lever table until it fits.
  7. Instrument in production and alert on cost per task, not just total spend.

Quiz​

  1. Question 1

    An 80,000-token policy corpus sits in the system prompt with a cache_control breakpoint after it, but usage.cache_read_input_tokens stays at zero across repeated requests made seconds apart. What is the most likely cause?

  2. Question 2

    A workload resends the same large prefix on every request, is not latency sensitive, and must get cheaper without re-running the quality evals. Which change do you make first?