4.4 Validation, retry, and feedback loops
Retry with error feedback
Append the specific validation error on retry — with the original document and the failed extraction. Schema validation is typically done with Pydantic in Python or a JSON Schema validator elsewhere:
- TypeScript
- Python
messages.push(
{ role: "assistant", content: response.content },
{
role: "user",
content: [
{ type: "tool_result", tool_use_id: block.id, is_error: true,
content: JSON.stringify({
validation_errors: [
"line_items sum to 1284.00 but stated_total is 1248.00",
"invoice_date '2026-13-04' is not a valid ISO 8601 date",
],
}) },
{ type: "text", text: "Re-extract, correcting the errors above. The document is unchanged." },
],
},
);
messages += [
{"role": "assistant", "content": response.content},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": block.id,
"is_error": True,
"content": json.dumps({
"validation_errors": [
"line_items sum to 1284.00 but stated_total is 1248.00",
"invoice_date '2026-13-04' is not a valid ISO 8601 date",
]
}),
},
{
"type": "text",
"text": "Re-extract, correcting the errors above. The document is unchanged.",
},
],
},
]
When retry cannot help
Retries fix format mismatches and structural output errors. They are ineffective when the required information is simply absent from the source document — for example a purchase-order number that lives only in an external document you never supplied. More retries produce fabrication, not correction. The fix there is a nullable field plus a human review route.
| Failure | Retry helps? |
|---|---|
| Wrong date format | Yes |
| Values in the wrong field | Yes |
| Totals do not sum | Yes |
| Field not present in the document | No |
Self-correcting validation design
Build the check into the schema:
- Extract
calculated_totalalongsidestated_totaland flag discrepancies. - Add a
conflict_detectedboolean for inconsistent source data.
Feedback loops on false positives
Add a detected_pattern field to each structured finding. When developers dismiss findings, you can group dismissals by pattern and see systematically which constructs generate noise — rather than guessing from anecdotes.
{
"file": "src/orders/refund.ts",
"line": 84,
"severity": "major",
"summary": "Refund issued without an idempotency key.",
"detected_pattern": "money-mutation-without-idempotency-key",
"suggested_fix": "Pass `idempotencyKey: order.id + attempt`."
}
Quiz
- Question 1
Validation rejects 6% of extractions. Investigation splits the failures: about two-thirds have line items that do not sum to the stated total or a date landing in the wrong field, and about one-third are missing a purchase order number that does not appear anywhere in the source document. The current retry resubmits the document with the message "extraction failed, try again". How should the retry be redesigned?
- Question 2
Some invoices state a total that does not equal the sum of their own line items, because the source document is internally inconsistent. The extractor currently returns whichever figure it judges correct, and downstream reconciliation fails unpredictably days later. Which schema change surfaces the inconsistency instead of absorbing it?
- Question 3
A validation-retry loop currently resubmits with "the previous extraction was invalid, please correct it". The retry succeeds about 35% of the time. What raises that most?
- Question 4
Developers dismiss about 30% of extraction QA findings, and nobody can say which kinds — the arguments about it are all anecdotal. You want to identify the false-positive patterns systematically. What change enables that?