Skip to main content

7.2 Guardrails and safe deployment

Guardrail layering​

A guardrail is a control that constrains what the system can do or say. One layer is a preference; layers are a defence.

LayerRunsEnforcesExample
Input validationBefore the modelShape, size, type, sourceReject a 40 MB upload; strip hidden characters; require a known media type
Input classificationBefore the modelPolicy on the requestRoute an out-of-scope or abusive request away from the main prompt
Prompt-levelIn the modelScope, tone, refusal behaviour"You answer only questions about this product's documentation"
Tool scopingAt tool selectionCapabilityThe read-only tool set on turns that process untrusted content
Pre-action gateBefore an effectInvariantsA hook blocking a refund above a threshold, or a write outside an allowed path
Server-side authorizationIn your backendThe caller's rightsThe API rejects the operation regardless of what the model asked for
Output validationAfter the modelCorrectness and safety of the resultSchema plus semantic checks before a database write
Output classificationAfter the modelPolicy on the responseBlock or redact disallowed content before display
MonitoringContinuouslyDetectionAlert on refusal spikes, anomalous tool sequences, cost anomalies
Which layer answers the item

"Must never happen" → a pre-action gate plus server-side authorization. "Should usually happen" → prompt plus criteria. "Must be caught before it is stored or shown" → output validation. Ranking these correctly is most of Domain 7.

Content policy​

PracticeDetail
Write the policy downAn enumerated list of what the application must refuse, in the system prompt and in the eval set
Scope positivelyDefining what is in scope is easier to enforce than enumerating everything out of scope
Give a refusal pathSay what to do instead — a canned message, an escalation, a handoff
Test itAdversarial cases in the eval suite, run in CI like any other regression
Layer itPrompt-level refusal plus an output classifier for the classes that matter most
Handle platform refusalsstop_reason: "refusal" is a normal path — surface something sensible, do not retry blindly

Secure by design​

PrincipleImplementation
Least privilegeEach agent, subagent, skill, and tool gets the narrowest capability that works. Read-only unless writing is the job. Scope credentials per tool, not one god key
Privacy by defaultSend the minimum data; redact before logging; do not persist what you do not need; set retention deliberately
Identity and access managementAuthenticate every request; authorize every tool call against the caller server-side; never trust a model-supplied identity or role claim
Fail closedIf the gate cannot verify, deny. An unavailable authorization service is not an approval
Defence in depthNo single control is assumed to hold
Complete mediationCheck on every call, not once per session — permissions change mid-session
AuditabilityLog who asked, what the model decided, which tools ran with which arguments, and what changed
ReversibilityPrefer reversible operations; require approval for irreversible ones; keep a rollback path
Blast-radius limitsRate limits, spend caps, per-operation thresholds, sandboxed execution, network egress restrictions

Deploying an agent safely​

StageControl
Pre-launchAdversarial evals including injection attempts; a written list of irreversible operations and the gate on each
LaunchNarrow scope first — fewer tools, lower thresholds, human approval on the risky classes
RampWiden only with evidence: measured accuracy, reviewed audit logs, no gate violations
Steady stateMonitoring on refusals, gate blocks, tool-error rates, cost; periodic re-review of the tool surface
IncidentA kill switch that disables the agent or a tool without a deploy, plus an audit trail good enough to reconstruct what happened

Human-in-the-loop belongs where the cost of error is unrecoverable, not everywhere: approval on every action trains reviewers to click through. Gate the classes that matter, and let the rest run.

Quiz​

  1. Question 1

    An agent can delete cloud resources. Which control prevents an unauthorised deletion, rather than recording or limiting one?