7.2 Guardrails and safe deployment
Guardrail layering
A guardrail is a control that constrains what the system can do or say. One layer is a preference; layers are a defence.
| Layer | Runs | Enforces | Example |
|---|---|---|---|
| Input validation | Before the model | Shape, size, type, source | Reject a 40 MB upload; strip hidden characters; require a known media type |
| Input classification | Before the model | Policy on the request | Route an out-of-scope or abusive request away from the main prompt |
| Prompt-level | In the model | Scope, tone, refusal behaviour | "You answer only questions about this product's documentation" |
| Tool scoping | At tool selection | Capability | The read-only tool set on turns that process untrusted content |
| Pre-action gate | Before an effect | Invariants | A hook blocking a refund above a threshold, or a write outside an allowed path |
| Server-side authorization | In your backend | The caller's rights | The API rejects the operation regardless of what the model asked for |
| Output validation | After the model | Correctness and safety of the result | Schema plus semantic checks before a database write |
| Output classification | After the model | Policy on the response | Block or redact disallowed content before display |
| Monitoring | Continuously | Detection | Alert on refusal spikes, anomalous tool sequences, cost anomalies |
Which layer answers the item
"Must never happen" → a pre-action gate plus server-side authorization. "Should usually happen" → prompt plus criteria. "Must be caught before it is stored or shown" → output validation. Ranking these correctly is most of Domain 7.
Content policy
| Practice | Detail |
|---|---|
| Write the policy down | An enumerated list of what the application must refuse, in the system prompt and in the eval set |
| Scope positively | Defining what is in scope is easier to enforce than enumerating everything out of scope |
| Give a refusal path | Say what to do instead — a canned message, an escalation, a handoff |
| Test it | Adversarial cases in the eval suite, run in CI like any other regression |
| Layer it | Prompt-level refusal plus an output classifier for the classes that matter most |
| Handle platform refusals | stop_reason: "refusal" is a normal path — surface something sensible, do not retry blindly |
Secure by design
| Principle | Implementation |
|---|---|
| Least privilege | Each agent, subagent, skill, and tool gets the narrowest capability that works. Read-only unless writing is the job. Scope credentials per tool, not one god key |
| Privacy by default | Send the minimum data; redact before logging; do not persist what you do not need; set retention deliberately |
| Identity and access management | Authenticate every request; authorize every tool call against the caller server-side; never trust a model-supplied identity or role claim |
| Fail closed | If the gate cannot verify, deny. An unavailable authorization service is not an approval |
| Defence in depth | No single control is assumed to hold |
| Complete mediation | Check on every call, not once per session — permissions change mid-session |
| Auditability | Log who asked, what the model decided, which tools ran with which arguments, and what changed |
| Reversibility | Prefer reversible operations; require approval for irreversible ones; keep a rollback path |
| Blast-radius limits | Rate limits, spend caps, per-operation thresholds, sandboxed execution, network egress restrictions |
Deploying an agent safely
| Stage | Control |
|---|---|
| Pre-launch | Adversarial evals including injection attempts; a written list of irreversible operations and the gate on each |
| Launch | Narrow scope first — fewer tools, lower thresholds, human approval on the risky classes |
| Ramp | Widen only with evidence: measured accuracy, reviewed audit logs, no gate violations |
| Steady state | Monitoring on refusals, gate blocks, tool-error rates, cost; periodic re-review of the tool surface |
| Incident | A kill switch that disables the agent or a tool without a deploy, plus an audit trail good enough to reconstruct what happened |
Human-in-the-loop belongs where the cost of error is unrecoverable, not everywhere: approval on every action trains reviewers to click through. Gate the classes that matter, and let the rest run.
Quiz
- Question 1
An agent can delete cloud resources. Which control prevents an unauthorised deletion, rather than recording or limiting one?
Links
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks
- https://code.claude.com/docs/en/hooks
- https://www.anthropic.com/news/anthropics-responsible-scaling-policy
- https://en.wikipedia.org/wiki/Principle_of_least_privilege
- https://en.wikipedia.org/wiki/Defense_in_depth_(computing)