Skip to main content

4.5 Batch processing strategies

Message Batches API properties​

Full reference: batch processing.

PropertyValue
Cost50% savings vs synchronous
Processing windowUp to 24 hours
Latency SLANone guaranteed
Multi-turn tool calling within a requestNot supported — cannot execute tools mid-request and return results
Correlationcustom_id on each request/response pair
Result orderingArbitrary — key by custom_id, never by position

Appropriateness​

SuitableNot suitable
Overnight technical-debt reportsBlocking pre-merge checks
Weekly auditsAnything a developer waits on
Nightly test generationInteractive agent turns
The proposal-evaluation item

"Switch both the blocking pre-merge check and the overnight debt report to batch for 50% savings." The answer is batch for the overnight report only. "Batches are often faster than 24 h" is not acceptable for a blocking workflow, and a timeout-fallback-to-realtime design adds complexity when matching each API to its use case is simpler. Batch result ordering is a non-issue — that is what custom_id is for.

SLA arithmetic​

With a 24-hour worst-case batch window and a 30-hour end-to-end SLA, you have 6 hours of slack — so submit at least every 4 hours to keep a comfortable margin. Expect an item that asks you to compute submission cadence from a stated SLA.

const batch = await client.messages.batches.create({
requests: documents.map((doc) => ({
custom_id: doc.id, // the correlation key
params: {
model: "claude-opus-5",
max_tokens: 16000,
tools: [extractInvoice],
tool_choice: { type: "tool", name: "extract_invoice" },
messages: [{ role: "user", content: doc.content }],
},
})),
});

// Poll, then stream results and key by custom_id.
for await (const result of await client.messages.batches.results(batch.id)) {
if (result.result.type === "succeeded") {
store(result.custom_id, result.result.message);
} else {
failed.push(result.custom_id); // resubmit only these
}
}

Failure handling​

Resubmit only the failed documents, identified by custom_id, with appropriate modifications — for example chunking documents that exceeded context limits.

Refine before you scale​

Run prompt refinement on a sample set before batch-processing large volumes. First-pass success rate is the lever that keeps resubmission costs down.

Quiz​

  1. Question 1

    Real-time Claude calls power two workflows: a blocking pre-merge check developers wait on, and a technical debt report generated overnight for the next morning. Your manager proposes switching both to the Message Batches API for its 50% cost savings. How should you evaluate this?

  2. Question 2

    A contract requires every submitted document to be processed within 30 hours of arrival, and documents arrive continuously through the day. The Message Batches API completes a batch within 24 hours with no guaranteed latency inside that window. How often should batches be submitted?

  3. Question 3

    Three CI workloads are proposed for the Message Batches API: a pre-merge security scan that developers wait on before merging, a weekly dependency audit, and a nightly technical debt report. Which should move to batch?

  4. Question 4

    A 5,000-document batch returns with 78 failures. All 78 are documents whose content exceeded the context limit for the request. What is the correct recovery?