5.1 LLM fundamentals
Tokens and context windows
| Concept | Detail |
|---|---|
| Token | The unit the model reads and writes — roughly a word fragment. Not characters, not words |
| Context window | The maximum input tokens for one request. 1M on current Claude models; 200K on Claude Haiku 4.5 |
| Max output | A separate ceiling, set per request by max_tokens (up to 128K on current models; streaming required for very large values) |
| Tokenizer | Model-family specific. It changes between model generations — re-baseline with count_tokens after a migration |
| Counting | client.messages.count_tokens(...). Never a third-party tokenizer library — it will be wrong for Claude |
| Billing | Input and output are priced separately, output several times higher. Cached reads are much cheaper than fresh input |
The window holds everything: system prompt, tool definitions, full conversation history, tool results, documents, and the reasoning the model produces. History is resent every turn, so a long conversation is a growing bill even when the last question is short.
More context means more competition for attention. Long contexts show measurable degradation — relevant facts buried in the middle lose to material at the edges, and contradictory instructions spread across 400K tokens do not resolve themselves. The answer to "output is inconsistent across many inputs" is to decompose into focused passes, not to raise the window. See 6.1.
Next-token generation and non-determinism
The model produces a probability distribution over the next token, samples one, appends it, and repeats. Consequences that appear as items:
- Output is not deterministic by default. Identical inputs can produce different text. Design for it: schemas, validation, explicit criteria, and evals measured as rates rather than pass/fail on one run.
- Sampling parameters are gone on current models.
temperature,top_p, andtop_kare removed on Claude Opus 5 / 4.8 / 4.7, Sonnet 5, and Fable 5 — sending them returns a 400. "Raise the temperature to make it more/less X" is a distractor, always. - You constrain shape, not sampling. Use
output_config.format(structured outputs) orstrict: trueon the tool schema to make the form reliable; use explicit criteria and few-shot examples to make the judgment reliable. - Truncation is not completion.
stop_reason: "max_tokens"means generation was cut mid-thought. - Streaming does not change the result — only when you see it.
Thinking and effort
| Option | What it is | How to set it |
|---|---|---|
| Adaptive thinking | The model decides when and how much to think, per request | thinking: {"type": "adaptive"} — the recommended default on current models |
| Thinking display | Whether a summary of the reasoning is returned | display: "summarized"; the default on current models is "omitted" (empty thinking text). Billing is unchanged either way |
| Effort levels | Overall depth and token spend: low, medium, high, xhigh, max | output_config: {"effort": "high"} — inside output_config, not top-level. Default is high |
| Fast mode | The same model at up to ~2.5× output tokens/second, premium pricing | Research preview, Claude Opus 5 / Opus 4.8 only, first-party only: client.beta.messages, beta fast-mode-2026-02-01, and speed: "fast" as a top-level parameter |
| Task budget | An advisory token ceiling the model is aware of, so it paces an agentic run | Beta task-budgets-2026-03-13, output_config.task_budget (minimum 20,000) |
budget_tokens is not a current parameterThe fixed thinking-token-budget concept is deprecated. It is removed on Claude Opus 5 / 4.8 / 4.7, Sonnet 5, and Fable 5 — a request carrying it returns 400 — and deprecated on the 4.6 family. When a requirement says "cap the thinking", the answer is a lower effort level (and, for an agentic run, a task budget), not budget_tokens.
Effort selection, briefly: low for subagents and simple high-volume routes; high for intelligence-sensitive work; xhigh for coding and long-horizon agentic work; max only when correctness outweighs cost and measurement shows headroom at the level below. Measure on real requests before raising a default, and tune per route rather than globally.
Prompting techniques by number of examples
| Technique | Shape | Use when |
|---|---|---|
| Zero-shot | Instructions only | The task is common and the output format is simple or schema-constrained |
| Single-shot (one-shot) | One worked example | The output format is unusual and one demonstration pins it down |
| Multi-shot (few-shot) | Typically 2–5 diverse examples | The judgment boundary is fuzzy — escalate vs resolve, flag vs pass — or the format is intricate |
Rules that decide items:
- Examples must cover the boundary, including the near-miss and the negative case. Five examples of the obvious case teach nothing.
- Examples are input tokens on every request — put them in the cached prefix, and stop adding once accuracy plateaus.
- Few-shot fixes judgment, not compliance. A rule that must hold every time needs a deterministic gate; a rule about when to use a tool belongs in the tool description.
Quiz
- Question 1
A requirement says "make the classifier less erratic". A developer proposes setting temperature to 0. What is wrong with that on current Claude models?
- Question 2
A requirement asks you to cap how much reasoning the model spends per request on current models. Which mechanism applies?
- Question 3
An agent must decide between resolving a ticket and escalating it. The boundary is genuinely fuzzy, and the agent currently escalates easy cases while attempting hard ones. What is the most effective first change?
Links
- https://platform.claude.com/docs/en/build-with-claude/context-windows
- https://platform.claude.com/docs/en/build-with-claude/token-counting
- https://platform.claude.com/docs/en/build-with-claude/extended-thinking
- https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/multishot-prompting
- https://platform.claude.com/docs/en/about-claude/models/overview
- https://en.wikipedia.org/wiki/Large_language_model
- https://en.wikipedia.org/wiki/Lexical_analysis