5.4 Cost and token management
Track it from usage
Every response carries the numbers. Log all of them.
| Field | Meaning | Relative cost |
|---|---|---|
input_tokens | Uncached input | 1× |
cache_creation_input_tokens | Written to cache | ~1.25× |
cache_read_input_tokens | Served from cache | ~0.1× |
output_tokens | Generated, including thinking | Several × input |
Cost per request is therefore:
cost = input_tokens × input_rate
+ cache_creation_tokens × input_rate × 1.25
+ cache_read_tokens × input_rate × 0.10
+ output_tokens × output_rate
Estimate before sending with client.messages.count_tokens(...) — same request shape, no generation. Never a third-party tokenizer: it does not match Claude's.
A cheaper model that needs two retries and a fallback costs more than one correct call. Instrument the task, counting every request it took, including retries, validation failures, and agent loop turns. This is also why unbounded agent loops are a cost bug: each turn resends the whole history.
Prompt caching
A prefix match. Render order is tools → system → messages; any byte change invalidates everything after it.
| Mechanic | Detail |
|---|---|
| Breakpoint | cache_control: {"type": "ephemeral"} on a content block, or top-level on the request to auto-cache the last cacheable block |
| Max breakpoints | 4 per request |
| TTL | 5 minutes by default; ttl: "1h" available |
| Minimum cacheable prefix | Model-dependent (roughly 512–4096 tokens) — shorter prefixes silently do not cache |
| Verification | usage.cache_read_input_tokens above zero on the second identical-prefix request |
| Scope | Caches are model-scoped — switching models starts cold |
Placement
[ tools ] ← deterministic order, frozen
[ system: rules ] ← frozen wording
[ system: corpus ] ← ← cache_control breakpoint here
[ messages: history] ← grows; a later breakpoint can check-point it
[ messages: latest ] ← volatile: the question, timestamps, per-request IDs
Check-pointing a long conversation: put a breakpoint at a stable point in the history so each turn re-reads the prefix from cache instead of paying full price for the whole transcript. With up to four breakpoints you can hold tools+system, an early history segment, and a later one — while the newest turn stays outside the cache.
Silent invalidators
| Invalidator | Fix |
|---|---|
Date.now() / datetime.now() in the system prompt | Move it after the last breakpoint, or drop it |
| A request ID, session ID, or user name in the prefix | Move to the user message |
| Non-deterministic JSON key order in tool definitions or serialized results | Sort keys |
| A tool list that varies per request | Freeze the set, or split stable and volatile tools around the breakpoint |
Editing the top-level system mid-conversation | Append a mid-conversation {"role": "system", ...} message instead — model-gated, but it preserves the prefix |
| Prefix shorter than the model's minimum | Consolidate, or accept no caching |
| A model or effort change mid-conversation | Expect a cold cache — an effort change invalidates the messages cache |
The lever order
Free wins before tradeoffs. Every lever below the line costs you something.
| # | Lever | Typical effect |
|---|---|---|
| 1 | Prompt caching on a large stable prefix | Up to ~90% off the cached portion |
| 2 | Input hygiene — prune tool output, drop dead instructions, trim examples that stopped helping | Proportional, permanent |
| 3 | Loop hygiene — bound history, cap turns, stop resending what the model no longer needs | Compounding on agents |
| 4 | Output hygiene — ask for the fields you need, not prose around them; cap max_tokens sensibly | Output is the expensive side |
| 5 | Batch for latency-tolerant work | 50% |
| — | — tradeoffs below this line — | |
| 6 | Lower effort on routes where quality holds | Large, measurable |
| 7 | Task budgets on agentic runs | Bounds the worst case |
| 8 | Smaller model on well-specified routes | Large, but re-run the evals |
| 9 | Multi-model cascade | Measure single-model-lower-effort first; a cascade forfeits cross-model cache reuse |
Budgeting a workload
- Sample real inputs and measure tokens with
count_tokens— do not estimate from character counts. - Separate the stable prefix from the per-request portion; only the second scales with volume once caching works.
- Multiply by volume for input, cached read, and output separately.
- Add the retry and validation-failure rate you actually observe.
- Add agent loop turns — an agent's cost is turns × growing history, not one request.
- Compare against the ceiling, then walk the lever table until it fits.
- Instrument in production and alert on cost per task, not just total spend.
Quiz
- Question 1
An 80,000-token policy corpus sits in the system prompt with a cache_control breakpoint after it, but usage.cache_read_input_tokens stays at zero across repeated requests made seconds apart. What is the most likely cause?
- Question 2
A workload resends the same large prefix on every request, is not latency sensitive, and must get cheaper without re-running the quality evals. Which change do you make first?
Links
- https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- https://platform.claude.com/docs/en/build-with-claude/token-counting
- https://platform.claude.com/docs/en/build-with-claude/batch-processing
- https://platform.claude.com/docs/en/about-claude/pricing
- https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents