4.1 Debugging and error handling
The whole of Domain 4 (2.6%) — roughly one item, but the reasoning transfers to Domains 2, 6, and 8.
Error taxonomy and recovery
| Class | Examples | Recovery |
|---|---|---|
| Transient / capacity | 429 rate limited, 529 overloaded, 5xx, connection reset, timeout | Retry with exponential backoff and jitter, honouring retry-after; bounded attempts |
| Request invalid | 400 malformed body, unsupported parameter for this model, 413 too large | Fix the request. Retrying is pure waste — this is where stale parameters surface |
| Auth / permission | 401, 403 | Fix credentials or scope; never retry blindly |
| Not found | 404 (bad model ID, missing file or batch) | Fix the identifier |
| Model output invalid | Schema validation failure, missing field, unparseable JSON | Retry once with the validation error fed back; then fall back or escalate |
| Truncation | stop_reason: "max_tokens" | Raise max_tokens, stream, or ask for less output. Not a retry-as-is |
| Refusal | stop_reason: "refusal" | Read stop_details.category; a retry of the identical request will refuse again |
| Tool execution failure | Backend down, bad argument, permission denied | Return tool_result with is_error: true and a structured, actionable message so the agent can adapt |
| Business-rule rejection | "Refund exceeds policy limit" | Not an error to retry — explain it to the user or escalate |
Two rules the exam leans on:
- Retry only what is retryable. A bounded, backed-off retry on transient classes; an immediate fix-or-fail on the rest. And retries must be idempotent, or a timeout on a successful call double-charges.
- Never swallow the class distinction. One broad
except APIError/catch (error)loses the difference between "try again in 2 s" and "your request is wrong". Write a most-specific-first chain using the SDK's typed exception classes.
Structured errors for the agent
An error the model reads should tell it what to do next.
{
"error": {
"category": "transient",
"retryable": true,
"retry_after_seconds": 2,
"message": "Inventory service timed out after 5s.",
"next_step": "Retry once; if it fails again, tell the customer stock cannot be confirmed right now."
}
}
"Error: 500" gives the agent nothing to act on, so it either loops or gives up. Category plus retryability plus a human-readable message is the shape that works — and it is the same shape you want in your own logs.
Trace analysis
To debug a Claude application you need the trace, and the trace has to be captured before it is needed.
| Log per request | Why |
|---|---|
| Model ID, prompt version, effort, beta headers | Attribute a regression to a version |
| Request ID from the response headers | Support escalation |
stop_reason (and stop_details when refusal) | Truncation and refusal detection |
Full usage — input, output, cache read, cache creation | Cost and cache diagnosis |
Every tool_use name, its parsed input, and the result (or is_error) | Reconstruct the decision path |
| Validation outcome | Distinguish "model was wrong" from "we mis-parsed" |
| Latency per turn | Find the slow step, not the slow request |
Reading a trace, in order:
- Where did it diverge? Find the first turn whose output is wrong. Everything after it is downstream noise.
- Was the input right? If the prompt or tool result at that turn was already wrong, the model is not the bug.
- What did the model actually receive? Truncated context, a pruned-away fact, a stale cached prefix.
- Which layer produced the failure? See the isolation table below.
- Is it systematic or stochastic? Re-run the same input several times. Consistent failure means the prompt, schema, or tool description; intermittent failure means sampling, ordering, or a race.
Isolating the integration layer from model output
| Symptom | Integration layer | Model output |
|---|---|---|
| JSON will not parse | Wrong block extracted; concatenated text blocks; encoding | Model produced prose around the JSON, or was truncated |
| A field is missing | Schema not sent; mapping code drops it | Model omitted it — often because the schema said required on something absent from the source |
| Tool never runs | tools not passed; name mismatch; dispatch table gap; tool_use block ignored | Model chose not to call it — a description problem |
| Wrong tool chosen | Two tools registered with the same name | Descriptions do not differentiate them |
| Loop never ends | Termination not keyed on stop_reason | — |
| Loop ends early | max_tokens treated as completion; tool_use mishandled | — |
| Answer ignores a fact you supplied | It was pruned, compacted away, or never appended | It was present and the model missed it — a placement or prompt problem |
| Output changes run to run | Nondeterministic serialization, cache state | Sampling — expected; constrain with schema and criteria |
| Cost spiked | History unbounded; cache invalidated by a volatile prefix | Longer outputs, higher effort |
Replay the exact request — same model, same system prompt, same messages, same tools — outside your application. If it reproduces, it is the model or the prompt. If it does not, it is your integration layer. Skipping this step is how teams spend a week prompt-tuning around a mapping bug.
Quiz
- Question 1
A field is missing from an extraction result in production. What is the most efficient way to decide whether the fault is in the integration layer or in the model output?