Last reviewed: July 2026
Safety is about what your system produces when nobody is attacking it. A support assistant that invents a refund policy, a summariser that leaks a customer's home address into a shared ticket, a sales bot that promises a discount your billing system cannot honour: no attacker involved, just a probabilistic system doing what probabilistic systems do.
Security is about what your system produces when someone is trying to make it misbehave. Prompt injection through a retrieved document, an agent tricked into calling a tool it should not, data exfiltration through a crafted output. That is covered in Security for AI systems.
The distinction matters because it changes what you buy and who owns it. Safety controls are largely product and quality engineering: validation, policy, fallbacks, evaluation. Security controls are largely adversarial engineering: threat modelling, isolation, least privilege. Teams that conflate the two buy a content moderation API, tick a box, and remain wide open to injection. Or they run a penetration test and still ship a bot that confidently states the wrong price. You need both, and they are different budgets.
This page is about the first one: the engineering of not shipping the answer that ends up on the news.
flowchart TD
A["User request"] --> B["Input checks"]
B -->|"Blocked"| C["Refusal message plus human route"]
B -->|"Redact PII"| D["Model call"]
B -->|"Pass"| D
D --> E["Output checks"]
E -->|"Pass"| F["Response to user"]
E -->|"Schema or rule failure"| G["Retry once with error"]
G --> E
E -->|"Policy violation"| C
E -->|"High risk action"| H["Human approval queue"]
G -->|"Retry exhausted"| I["Deterministic fallback"]
I --> F
C --> J["Log and alert"]
E --> J
B --> J
The shape is deliberately boring. Checks before the model, checks after the model, and a defined path for every branch that is not "pass". The interesting engineering is not in the model call; it is in what happens on the edges. Most teams build the happy path in a fortnight and then spend six months discovering the edges one incident at a time.
Two things to notice in the diagram. First, every branch terminates somewhere real: a message, a fallback, a queue. There is no arrow that just stops. Second, the log path is fed from everywhere, including from checks that pass. If you only log failures you cannot compute a block rate, and if you cannot compute a block rate you cannot tell the difference between a guardrail that works and a guardrail that is quietly destroying your conversion funnel.
Input checks run before you spend a token. They are cheap, they are fast, and they are the only place you can control cost before it is incurred.
What to check:
Policy violations and abuse. Obvious category. Use a classifier, not a word list. Word lists block "Scunthorpe" and miss every paraphrase.
Prompt injection heuristics. Pattern matching for instruction-like strings ("ignore previous", "you are now"), unusual encoding, or suspicious structure. This is a speed bump, not a wall. Treat it as noise reduction and read the security page for the real controls.
PII detection and redaction. Detect names, emails, card numbers, national IDs, and either redact them before the model sees them or refuse. This matters most when your provider is a third party and your data processing agreement is narrower than your users' typing habits. Tools like Microsoft Presidio do this pattern-and-NER hybrid reasonably well; the reason to redact rather than refuse is that most PII in a support ticket is incidental, and refusing the whole request punishes the user for including their own email address.
Off-topic or out-of-scope requests. A cheap intent classifier that routes anything outside the feature's remit to a canned response. The reason: a model asked about something outside its brief will still answer, and that answer is unowned by anyone in your organisation.
Length and cost bounds. Hard token ceilings per request. The reason: a single pasted PDF can cost more than a day of normal traffic.
Rate limits per user. Per-user, not just per-IP, and ideally per-cost rather than per-request, because one long request costs what fifty short ones do.
Now the honest part. Input classifiers have false positives, and every false positive is a real user you just blocked. A safety filter that blocks a small percentage of legitimate traffic is invisible on your dashboards and extremely visible in your support inbox. Instrument the block rate as a first-class metric. Sample blocked requests weekly and have a human read them. If nobody has read a sample of blocks in the last month, you do not know what your filter does; you know what you hoped it would do.
Output checks are where most of the value is, because this is the last point before a human sees something.
Schema validation. The cheapest and most effective control by a distance. Force structured output (JSON schema, tool call arguments, a Pydantic or Zod model) and validate it. The reason it wins: a parse failure is a deterministic signal. You do not need judgement, a classifier, or a second model call to know that {"refund_amount": "as much as you like"} failed. It either parses into your type or it does not, and you can branch on that in an if statement. Every guardrail you can convert from "a model judges the output" to "a parser accepts or rejects the output" is a guardrail that gets cheaper, faster, and more reliable at the same time.
Groundedness checks. For RAG, verify that claims in the output are supported by the retrieved context. Simplest useful version: require citations, then check that each cited span actually exists in the retrieved chunks. Harder version: a model-based entailment check per claim. Start with the simple one.
PII and secret leakage checks. Run the same detectors outbound as inbound. Models repeat what is in their context, and their context includes your retrieved documents.
Policy classification. A classifier over the output for the categories your product cannot emit.
Competitor and legal-risk term checks. Blunt but genuinely useful. If your legal team has a list of things you must never say (comparative claims, guarantees, medical or financial advice phrasing), a term check on output is a five-line control with a named owner.
Business rule validation. The discount the model offered must be one the system can actually honour. The delivery date must be one the logistics API supports. The plan tier must exist.
That last category deserves emphasis. Validate against data, not vibes. If the model states a price, look the price up in the database and compare before the user sees it. If it names a product, check the catalogue. If it quotes a policy, check the policy store. This is unglamorous integration work, and it is worth more than any amount of prompt engineering, because it converts a probabilistic claim into a deterministic assertion. The general rule: for any factual claim your system makes in a domain where you own the source of truth, you should be checking the claim against the source of truth, not asking the model to be careful.
A policy is a product decision, not a model setting. It belongs written down, owned by a named person, versioned in a repository, and testable. "The model refuses that" is not a policy. It is an observation about a vendor's training run that may change on their next release.
| Layer | Enforced by | Owner | Example |
|---|---|---|---|
| Provider policy | The model's own training and safety filters | The model vendor | The model declines to give instructions for synthesising a nerve agent |
| Platform policy | Your gateway or middleware rules | Platform or infrastructure team | No request over 20k tokens; PII redacted before egress; all calls logged |
| Product policy | Feature-level rules you define | The product owner for that feature | This assistant discusses billing only, never gives tax advice, never quotes a price not in the catalogue |
| Regulatory policy | Law in the markets you serve | Legal and compliance | Automated decisions with legal effect require a human review route |
These layers compose, and the strictest wins. Your product policy cannot loosen the provider's; the provider's does not satisfy your regulator's. Most teams only think about the first layer, ship, and are then surprised by two things: that the provider's policy has gaps exactly where their product is most sensitive (the vendor never heard of your billing rules), and that the provider's policy also blocks things their product legitimately needs (a security tool that must discuss malware, a medical product that must name drugs).
Write the product policy as a document. Version it. Put its rules in the eval suite. When someone asks "can the bot say X", the answer should be a line in a file, not a discussion.
This is the section engineering managers most need, and the one most often skipped.
The design rule: every AI path needs a defined non-AI answer. If you cannot describe what the user sees when the model is unavailable, wrong, or unsure, you have not designed the feature. You have designed the demo.
| Failure | Bad response | Graceful response |
|---|---|---|
| Model API down or rate-limited | Spinner, then a 500, then a support ticket | Fall back to a smaller model, a secondary provider, or a cached/deterministic answer; tell the user the mode changed if it matters |
| Output fails validation | Return the malformed output and let the UI break | Retry once with the validation error fed back into the prompt, then fall back to a template response |
| Retrieval returns nothing relevant | Model answers from parametric memory and invents a policy | Say plainly that nothing relevant was found, offer keyword search or a human. Never guess |
| Low model confidence | Confident prose regardless | Hedge explicitly, show sources, offer escalation to a human |
| Guardrail blocks the request | "Something went wrong" | A specific, honest message naming the category, plus a route to a human who can help |
| Timeout | Hang until the browser gives up | Return the partial answer with a clear marker, or queue the job and notify on completion |
The principle underneath the table: the worst outcome is a confident wrong answer, and the second worst is a dead end. A confident wrong answer costs you trust, and possibly money or a regulator's attention. A dead end costs you the user. Between those, an honest "I do not have a reliable answer, here is a human" is dramatically better than both, and it is not something you get for free. It is a feature. Somebody has to build the confidence signal, the message, the routing, and the handover context. That means it belongs in the PRD with acceptance criteria, not in the backlog as a nice-to-have. See AI feature PRD for the shape of that.
Test the fallback path on a schedule. A fallback that has never been exercised is not a fallback; it is a comment.
Tie guardrails to blast radius. The cost of a control should be proportional to the cost of the mistake it prevents.
| Blast radius | Example | Control |
|---|---|---|
| Read-only informational | Internal doc search, meeting summary | Sampled review, user feedback button, basic output logging |
| User-visible content | Support reply drafted for a customer, marketing copy | Full output guardrails, schema and policy checks, feedback loop into evals |
| Action-taking | Creates a ticket, updates a CRM record, sends an email | Human approval, or strict validation plus guaranteed reversibility and an undo path |
| Money, legal, or medical | Issues a refund, files a claim, suggests a dosage | Human in the loop on every decision, full audit trail, hard-coded limits that the model cannot argue past |
The principle: autonomy should be proportional to reversibility. If a mistake can be undone in one click by the user, let the system act. If undoing it requires a phone call to a bank, a human approves it first. Hard limits belong in code, not in the prompt. A prompt that says "never refund more than 50 pounds" is a suggestion. A check in the refund service that rejects amounts over 50 is a limit.
Your guardrails are software with a false positive rate and a false negative rate, and you do not know either one until you measure them. Build a dedicated eval suite for the guardrail layer, separate from your model eval suite:
- A set of clearly-should-block cases, for recall.
- A set of clearly-should-pass cases that look superficially risky, for false positive rate. These are the ones teams forget, and they are the ones that cost you real users.
- Edge cases from production blocks, added every time you review a sample.
Run it in CI. Treat a regression in false positive rate as a build failure, the same as a broken test. Otherwise you discover in production that you block a meaningful slice of legitimate users, and you discover it from a churn report rather than a dashboard. See Evaluation fundamentals for how to build the suite.
Guardrails added the week before launch. They then get designed under time pressure by whoever is free, tuned by vibes, and shipped untested. Guardrail design belongs in the design phase, because it changes the architecture (where validation sits, what the fallback is, what the schema looks like).
A blocklist of words as a safety strategy. It blocks legitimate speech and misses every rephrasing. It exists because it is easy to build and easy to show a stakeholder.
A filter that blocks legitimate traffic and nobody measures it. No block rate metric, no sampled review. The filter is now a silent tax on your funnel with no owner.
A fallback path that has never been tested. It fails the first time it is needed, which is by definition during an incident, which is the worst moment to discover that the fallback also calls the model.
The model that apologises and continues anyway. You told it not to give financial advice. It says "I am not a financial adviser, but..." and then gives financial advice. Prompt instructions are not enforcement. If a rule matters, enforce it outside the model.
No logging of blocks, so you cannot tune them. You know the filter fired. You do not know on what, so you cannot tell whether to loosen it. Log the input (redacted), the rule that fired, and the score.
- What does the user see when the model has no good answer, and have we tested that path this month?
- What is our block rate on input guardrails, and who read a sample of blocked requests most recently?
- Which of our output checks validate against data we own, and which are just a model judging another model?
- If our primary model provider is down for two hours, what happens, and has anyone run that drill?
- Where is our product policy written down, who owns it, and is it in the eval suite?
If the answer to any of these is a shrug or a name of a person who left, that is your next sprint.
Guardrails are a reliable way to separate candidates who have shipped from candidates who have prototyped. Expect a question shaped like "how would you make sure this assistant does not say something harmful or wrong?"
The weak answer is a single layer: "we would add a content filter" or "we would prompt it carefully". The strong answer does four things. It separates safety from security in the first sentence. It describes layered controls with input and output checks and names schema validation as the highest-value one. It talks about failure paths and reversibility, because that is where the engineering judgement lives. And it mentions measurement: block rate, false positives, guardrail evals in CI.
If you want to score well, bring a trade-off. Something like: "we validate prices against the database rather than asking the model to be careful, which costs a lookup per response and about 30ms of latency, and we take that because a wrong price is a contractual problem and 30ms is not." Real numbers from your own system, honestly caveated, beat a list of best practices every time.
The follow-up is usually "what if the guardrail blocks legitimate users?" The answer they want: you measure it, you sample it, you have a false positive budget agreed with product, and you have a route to a human for the ones you get wrong.
Next: Security for AI systems
Related: Responsible AI in practice | Evaluation fundamentals | What AI can and cannot do