Skip to main content

2.2 Systems life cycle for Claude applications

The phases, with the Claude-specific work in each​

PhaseOrdinary workThe Claude-specific addition
RequirementsFunctional and non-functional specsAccuracy target as a measurable rate; failure behaviour for wrong output (2.1)
DesignInterfaces, data flow, schemasModel tier, sync vs batch, tool surface, context strategy, trust boundaries
BuildCode, unit tests, reviewPrompts and tool descriptions are artifacts under version control, reviewed like code
TestUnit, integration, regressionAn eval set with a pass threshold, run in CI — prompts have no compiler
ReleaseVersioning, deploy, rollbackPinned model IDs, pinned prompt versions, canary or shadow rollout
OperateMonitoring, alerting, on-callTrack usage, stop_reason distribution, validation-failure rate, cost per request, latency percentiles
MaintainPatching, refactoring, deprecationModel deprecation and migration; prompt drift as the product changes

Environments and configuration​

The same code path must be able to point at different models and different limits per environment.

ConcernDevStagingProduction
ModelMay float to try new releasesPinned to the production candidatePinned
PromptsIterated freelyFrozen at the release candidateVersioned, changed only through release
KeysDeveloper key, low limits, separate workspaceService account scoped to stagingService account, least privilege, rotated
DataSynthetic or redactedRedacted production-shaped dataReal
Cost controlsHard low ceilingRealisticBudget alarms

Config belongs in the environment, not the code: model ID, effort level, max tokens, timeouts, retry policy, and feature flags for beta headers.

Never float the model ID in production

An unpinned or aliased model choice means the behaviour of your system can change without a deploy. Pin the exact model ID string, treat a model bump as a code change with its own eval run and its own rollback, and record the model ID on every stored output so you can attribute a regression later. See 2.6 Configuration management.

Evals as the release gate​

Prompts and tool descriptions have no type checker; an eval set is the only thing standing between a one-word prompt edit and a production regression.

PracticeDetail
Golden setReal inputs with expected outputs, including the hard and adversarial cases that once broke
Deterministic checks firstSchema validity, required fields present, no forbidden content — cheap and unambiguous
Graded checks secondAn LLM-as-judge or human rubric for the parts that are genuinely judgment
Threshold in CIThe build fails below the bar; the bar is committed alongside the prompt
Regression on changeAny change to prompt, tool description, schema, model ID, effort, or temperature triggers the run

Rollout and rollback​

StrategyMechanismWhen
ShadowRun the new configuration on live traffic, log both, serve the oldHighest-confidence comparison before any user sees it
CanaryRoute a small share of traffic to the new configuration, watch the metricsStandard for a model or prompt bump
Feature flagModel ID, effort, and beta headers behind flagsInstant rollback with no deploy
Blue/greenTwo full stacksWhen the change spans more than the model

Rollback must be a config change, not a revert-and-redeploy — which is another reason model IDs and prompt versions belong in configuration.

Operating: what to monitor​

SignalWhy it matters
usage.input_tokens / output_tokensCost, and the early warning for prompt bloat
usage.cache_read_input_tokensCaching actually working; zero across identical prefixes means a silent invalidator
stop_reason distributionRising max_tokens means truncation; any refusal needs stop_details inspection
Validation-failure rateSchema drift, prompt drift, or a model change
Retry and 429/529 rateCapacity and backoff tuning
Latency p50/p95, time-to-first-tokenThe number users feel
Cost per completed taskThe only cost metric that survives a retry loop

Maintenance: deprecation and migration​

Models are released and retired on a cadence the application must absorb:

  1. Read the migration notes for breaking API changes, not just quality changes — removed parameters, changed defaults, new stop reasons.
  2. Run the eval set on the new model before touching prompts.
  3. Re-baseline token counts with count_tokens when the tokenizer changed — cost models built on the old tokenizer are wrong.
  4. Audit prompts for instructions written for an older model's quirks; they often degrade newer ones.
  5. Re-tune effort and thinking settings; the old settings were optimised for a different model.
  6. Canary, then cut over, then delete the dead prompt scaffolding.

Quiz​

  1. Question 1

    A one-word change to a production prompt degraded extraction quality, and nobody noticed for a week. Which practice would have caught it before release?

  2. Question 2

    You are moving a production workload to a newer Claude model. All four of the following are genuine steps in the migration. Which one comes first?