2.2 Systems life cycle for Claude applications
The phases, with the Claude-specific work in each
| Phase | Ordinary work | The Claude-specific addition |
|---|---|---|
| Requirements | Functional and non-functional specs | Accuracy target as a measurable rate; failure behaviour for wrong output (2.1) |
| Design | Interfaces, data flow, schemas | Model tier, sync vs batch, tool surface, context strategy, trust boundaries |
| Build | Code, unit tests, review | Prompts and tool descriptions are artifacts under version control, reviewed like code |
| Test | Unit, integration, regression | An eval set with a pass threshold, run in CI — prompts have no compiler |
| Release | Versioning, deploy, rollback | Pinned model IDs, pinned prompt versions, canary or shadow rollout |
| Operate | Monitoring, alerting, on-call | Track usage, stop_reason distribution, validation-failure rate, cost per request, latency percentiles |
| Maintain | Patching, refactoring, deprecation | Model deprecation and migration; prompt drift as the product changes |
Environments and configuration
The same code path must be able to point at different models and different limits per environment.
| Concern | Dev | Staging | Production |
|---|---|---|---|
| Model | May float to try new releases | Pinned to the production candidate | Pinned |
| Prompts | Iterated freely | Frozen at the release candidate | Versioned, changed only through release |
| Keys | Developer key, low limits, separate workspace | Service account scoped to staging | Service account, least privilege, rotated |
| Data | Synthetic or redacted | Redacted production-shaped data | Real |
| Cost controls | Hard low ceiling | Realistic | Budget alarms |
Config belongs in the environment, not the code: model ID, effort level, max tokens, timeouts, retry policy, and feature flags for beta headers.
An unpinned or aliased model choice means the behaviour of your system can change without a deploy. Pin the exact model ID string, treat a model bump as a code change with its own eval run and its own rollback, and record the model ID on every stored output so you can attribute a regression later. See 2.6 Configuration management.
Evals as the release gate
Prompts and tool descriptions have no type checker; an eval set is the only thing standing between a one-word prompt edit and a production regression.
| Practice | Detail |
|---|---|
| Golden set | Real inputs with expected outputs, including the hard and adversarial cases that once broke |
| Deterministic checks first | Schema validity, required fields present, no forbidden content — cheap and unambiguous |
| Graded checks second | An LLM-as-judge or human rubric for the parts that are genuinely judgment |
| Threshold in CI | The build fails below the bar; the bar is committed alongside the prompt |
| Regression on change | Any change to prompt, tool description, schema, model ID, effort, or temperature triggers the run |
Rollout and rollback
| Strategy | Mechanism | When |
|---|---|---|
| Shadow | Run the new configuration on live traffic, log both, serve the old | Highest-confidence comparison before any user sees it |
| Canary | Route a small share of traffic to the new configuration, watch the metrics | Standard for a model or prompt bump |
| Feature flag | Model ID, effort, and beta headers behind flags | Instant rollback with no deploy |
| Blue/green | Two full stacks | When the change spans more than the model |
Rollback must be a config change, not a revert-and-redeploy — which is another reason model IDs and prompt versions belong in configuration.
Operating: what to monitor
| Signal | Why it matters |
|---|---|
usage.input_tokens / output_tokens | Cost, and the early warning for prompt bloat |
usage.cache_read_input_tokens | Caching actually working; zero across identical prefixes means a silent invalidator |
stop_reason distribution | Rising max_tokens means truncation; any refusal needs stop_details inspection |
| Validation-failure rate | Schema drift, prompt drift, or a model change |
| Retry and 429/529 rate | Capacity and backoff tuning |
| Latency p50/p95, time-to-first-token | The number users feel |
| Cost per completed task | The only cost metric that survives a retry loop |
Maintenance: deprecation and migration
Models are released and retired on a cadence the application must absorb:
- Read the migration notes for breaking API changes, not just quality changes — removed parameters, changed defaults, new stop reasons.
- Run the eval set on the new model before touching prompts.
- Re-baseline token counts with
count_tokenswhen the tokenizer changed — cost models built on the old tokenizer are wrong. - Audit prompts for instructions written for an older model's quirks; they often degrade newer ones.
- Re-tune effort and thinking settings; the old settings were optimised for a different model.
- Canary, then cut over, then delete the dead prompt scaffolding.
Quiz
- Question 1
A one-word change to a production prompt degraded extraction quality, and nobody noticed for a week. Which practice would have caught it before release?
- Question 2
You are moving a production workload to a newer Claude model. All four of the following are genuine steps in the migration. Which one comes first?