Skip to main content

4.1 Debugging and error handling

The whole of Domain 4 (2.6%) — roughly one item, but the reasoning transfers to Domains 2, 6, and 8.

Error taxonomy and recovery​

ClassExamplesRecovery
Transient / capacity429 rate limited, 529 overloaded, 5xx, connection reset, timeoutRetry with exponential backoff and jitter, honouring retry-after; bounded attempts
Request invalid400 malformed body, unsupported parameter for this model, 413 too largeFix the request. Retrying is pure waste — this is where stale parameters surface
Auth / permission401, 403Fix credentials or scope; never retry blindly
Not found404 (bad model ID, missing file or batch)Fix the identifier
Model output invalidSchema validation failure, missing field, unparseable JSONRetry once with the validation error fed back; then fall back or escalate
Truncationstop_reason: "max_tokens"Raise max_tokens, stream, or ask for less output. Not a retry-as-is
Refusalstop_reason: "refusal"Read stop_details.category; a retry of the identical request will refuse again
Tool execution failureBackend down, bad argument, permission deniedReturn tool_result with is_error: true and a structured, actionable message so the agent can adapt
Business-rule rejection"Refund exceeds policy limit"Not an error to retry — explain it to the user or escalate

Two rules the exam leans on:

  1. Retry only what is retryable. A bounded, backed-off retry on transient classes; an immediate fix-or-fail on the rest. And retries must be idempotent, or a timeout on a successful call double-charges.
  2. Never swallow the class distinction. One broad except APIError / catch (error) loses the difference between "try again in 2 s" and "your request is wrong". Write a most-specific-first chain using the SDK's typed exception classes.

Structured errors for the agent​

An error the model reads should tell it what to do next.

{
"error": {
"category": "transient",
"retryable": true,
"retry_after_seconds": 2,
"message": "Inventory service timed out after 5s.",
"next_step": "Retry once; if it fails again, tell the customer stock cannot be confirmed right now."
}
}

"Error: 500" gives the agent nothing to act on, so it either loops or gives up. Category plus retryability plus a human-readable message is the shape that works — and it is the same shape you want in your own logs.

Trace analysis​

To debug a Claude application you need the trace, and the trace has to be captured before it is needed.

Log per requestWhy
Model ID, prompt version, effort, beta headersAttribute a regression to a version
Request ID from the response headersSupport escalation
stop_reason (and stop_details when refusal)Truncation and refusal detection
Full usage — input, output, cache read, cache creationCost and cache diagnosis
Every tool_use name, its parsed input, and the result (or is_error)Reconstruct the decision path
Validation outcomeDistinguish "model was wrong" from "we mis-parsed"
Latency per turnFind the slow step, not the slow request

Reading a trace, in order:

  1. Where did it diverge? Find the first turn whose output is wrong. Everything after it is downstream noise.
  2. Was the input right? If the prompt or tool result at that turn was already wrong, the model is not the bug.
  3. What did the model actually receive? Truncated context, a pruned-away fact, a stale cached prefix.
  4. Which layer produced the failure? See the isolation table below.
  5. Is it systematic or stochastic? Re-run the same input several times. Consistent failure means the prompt, schema, or tool description; intermittent failure means sampling, ordering, or a race.

Isolating the integration layer from model output​

SymptomIntegration layerModel output
JSON will not parseWrong block extracted; concatenated text blocks; encodingModel produced prose around the JSON, or was truncated
A field is missingSchema not sent; mapping code drops itModel omitted it — often because the schema said required on something absent from the source
Tool never runstools not passed; name mismatch; dispatch table gap; tool_use block ignoredModel chose not to call it — a description problem
Wrong tool chosenTwo tools registered with the same nameDescriptions do not differentiate them
Loop never endsTermination not keyed on stop_reason—
Loop ends earlymax_tokens treated as completion; tool_use mishandled—
Answer ignores a fact you suppliedIt was pruned, compacted away, or never appendedIt was present and the model missed it — a placement or prompt problem
Output changes run to runNondeterministic serialization, cache stateSampling — expected; constrain with schema and criteria
Cost spikedHistory unbounded; cache invalidated by a volatile prefixLonger outputs, higher effort
The isolation test

Replay the exact request — same model, same system prompt, same messages, same tools — outside your application. If it reproduces, it is the model or the prompt. If it does not, it is your integration layer. Skipping this step is how teams spend a week prompt-tuning around a mapping bug.

Quiz​

  1. Question 1

    A field is missing from an extraction result in production. What is the most efficient way to decide whether the fault is in the integration layer or in the model output?