Skip to main content

4.4 Validation, retry, and feedback loops

Retry with error feedback​

Append the specific validation error on retry — with the original document and the failed extraction. Schema validation is typically done with Pydantic in Python or a JSON Schema validator elsewhere:

messages.push(
{ role: "assistant", content: response.content },
{
role: "user",
content: [
{ type: "tool_result", tool_use_id: block.id, is_error: true,
content: JSON.stringify({
validation_errors: [
"line_items sum to 1284.00 but stated_total is 1248.00",
"invoice_date '2026-13-04' is not a valid ISO 8601 date",
],
}) },
{ type: "text", text: "Re-extract, correcting the errors above. The document is unchanged." },
],
},
);

When retry cannot help​

Limits of retry

Retries fix format mismatches and structural output errors. They are ineffective when the required information is simply absent from the source document — for example a purchase-order number that lives only in an external document you never supplied. More retries produce fabrication, not correction. The fix there is a nullable field plus a human review route.

FailureRetry helps?
Wrong date formatYes
Values in the wrong fieldYes
Totals do not sumYes
Field not present in the documentNo

Self-correcting validation design​

Build the check into the schema:

  • Extract calculated_total alongside stated_total and flag discrepancies.
  • Add a conflict_detected boolean for inconsistent source data.

Feedback loops on false positives​

Add a detected_pattern field to each structured finding. When developers dismiss findings, you can group dismissals by pattern and see systematically which constructs generate noise — rather than guessing from anecdotes.

{
"file": "src/orders/refund.ts",
"line": 84,
"severity": "major",
"summary": "Refund issued without an idempotency key.",
"detected_pattern": "money-mutation-without-idempotency-key",
"suggested_fix": "Pass `idempotencyKey: order.id + attempt`."
}

Quiz​

  1. Question 1

    Validation rejects 6% of extractions. Investigation splits the failures: about two-thirds have line items that do not sum to the stated total or a date landing in the wrong field, and about one-third are missing a purchase order number that does not appear anywhere in the source document. The current retry resubmits the document with the message "extraction failed, try again". How should the retry be redesigned?

  2. Question 2

    Some invoices state a total that does not equal the sum of their own line items, because the source document is internally inconsistent. The extractor currently returns whichever figure it judges correct, and downstream reconciliation fails unpredictably days later. Which schema change surfaces the inconsistency instead of absorbing it?

  3. Question 3

    A validation-retry loop currently resubmits with "the previous extraction was invalid, please correct it". The retry succeeds about 35% of the time. What raises that most?

  4. Question 4

    Developers dismiss about 30% of extraction QA findings, and nobody can say which kinds — the arguments about it are all anecdotal. You want to identify the false-positive patterns systematically. What change enables that?