Last reviewed: July 2026
An eval is a test for a system that is allowed to give different answers to the same question. That single sentence is the whole reframing. In normal software you write assertEqual(add(2, 2), 4) and the test is either green or red. In an AI system the correct output for "summarise this support ticket" is not one string. It is a large set of acceptable strings, and the set boundary is fuzzy. You cannot assert equality on an output that legitimately varies.
So you replace binary pass/fail on a string with measured quality on a distribution. You collect a set of representative inputs, run the system over all of them, score each output against criteria you defined in advance, and look at the aggregate plus the tails. The unit of truth stops being "this test passed" and becomes "this suite scored 0.81 on 120 cases, up from 0.78, with zero regressions on the 14 cases we said must never fail".
That shift is what makes an AI team an engineering team. Without it, the loop looks like this: someone tweaks a prompt, eyeballs three outputs, declares it better, and ships. Someone else disagrees. The disagreement resolves in favour of whoever is more senior, more confident, or more recently burned. A team without evals is not shipping AI, it is shipping vibes, and every argument becomes seniority-weighted opinion.
The insight that makes it click for engineers: evals are not a research activity. They are your test suite, your CI gate, and your regression net, adapted to a probabilistic component. You already believe in tests. You already refuse to merge without them. Evals are the same commitment applied to the part of your system that cannot be asserted on.
This is why evals repay more than anything else an AI team invests in. They convert opinion into evidence. Everything downstream (model selection, prompt iteration, retrieval tuning, vendor negotiation, the decision to fine-tune) becomes a measurable experiment instead of a debate. A team with a decent eval suite can try ten ideas a week and keep the two that worked. A team without one tries ten ideas and keeps the ones proposed by the loudest person.
Use evals whenever an LLM output reaches a user, feeds another system, or influences a decision. That is nearly every production use. Specifically:
- You are changing prompts, models, retrieval, or tools, and need to know if it got better or worse.
- You are choosing between vendors or model tiers and want a defensible comparison rather than a demo.
- You have a quality bar you must not fall below (legal, safety, brand, accuracy).
- You are being asked "is it good enough to ship?" and want an answer that survives the room.
Honestly, when not to. If you are building a two-week internal prototype for five people who will tell you directly when it is wrong, a full eval harness is overhead. Ship it, watch it, and build the suite when you decide to keep it. If the output is purely creative with no correctness notion and no downstream consumer (a brainstorming toy), heavy scoring will measure the wrong thing. And if you have zero real traffic and zero domain expertise available, do not invent a golden dataset from imagination; get five real users first, because a dataset built from your assumptions will measure your assumptions.
The trap is using "we are still exploring" as a permanent excuse. Most teams are past the prototype stage long before they admit it.
flowchart TD
A["Golden dataset"] --> B["Eval runner"]
C["System under test"] --> B
B --> D["Assertions (cheap)"]
B --> E["Scorers (similarity, retrieval)"]
B --> F["LLM judge (rubric)"]
D --> G["Results store"]
E --> G
F --> G
G --> H["Baseline comparison"]
H --> I["Report on pull request"]
H --> J["Dashboard over time"]
K["Production traces"] --> L["Triage and labelling"]
L --> A
Walk it component by component. The golden dataset is a versioned set of inputs, each with an expected answer or a set of grading criteria. It lives in your repository or a dataset store, not in someone's spreadsheet. The system under test is the whole pipeline (prompt, retrieval, model, post-processing), not just the model call, because users experience the pipeline.
The eval runner executes the system over every case and dispatches each output to one or more scorers. Assertions are deterministic code: does it parse as JSON, is the customer_id field present, is it under 400 characters, does it contain no email addresses. These cost nothing and catch the dumbest failures. Scorers are programmatic metrics: string similarity, retrieval recall against known-relevant documents. The LLM judge applies a rubric to outputs that need semantic assessment.
Everything lands in a results store keyed by commit, model version, and dataset version. That keying matters; without it you cannot tell whether last Tuesday's score drop was the prompt or the dataset growing. Baseline comparison diffs the current run against the last known-good run and produces the number that gates the merge. Results go to the pull request for the person making the change and to a dashboard for the trend, because a single run tells you where you are and the trend tells you where you are going.
The loop at the bottom is the part teams skip. Production traces get sampled, triaged, and labelled, and the interesting failures get promoted into the golden dataset. That is the flywheel. Without it the suite ages into irrelevance while reality moves on.
The suite scores 0.95 while users complain. Symptom: green dashboard, angry support queue. Cause: your dataset is easier than reality. You built it from clean, well-formed examples you wrote yourself, and real users send typos, ambiguity, three questions in one message, and pasted screenshots of tables. Fix: rebuild the dataset from sampled production traffic, stratified so the hard categories are represented at least as often as they occur. Then check that your suite score moves when you deliberately break the system; if you can degrade the prompt and the score barely drops, the suite is not measuring anything.
The average hides a catastrophic subgroup. Symptom: aggregate score improves, one customer segment gets nonsense. Cause: 200 easy cases drown out 8 hard ones. Fix: report per-slice scores as a matter of course (by input type, by language, by document source, by customer tier) and gate on the worst slice, not just the mean. The mean is a summary for executives. The slices are how you find the problem.
A judge nobody calibrated. Symptom: judge scores are stable, confident, and uncorrelated with what humans think. Fix: label a sample by hand, measure agreement between judge and humans, and do not use the judge until agreement is good enough to be worth acting on.
Evals written after the feature shipped. Symptom: the suite exists, but it was constructed to explain the current behaviour rather than to define correct behaviour, so it passes everything the system already does. Fix: write the criteria before the change, from the product intent. If the eval was written by reading the outputs, it encodes the bugs.
The team overfits to the dataset. Symptom: score climbs steadily, user metrics do not move. Cause: the golden set is also the development set, so people tune until the cases pass. Fix: hold out a slice that nobody looks at during development and only run it before release. Rotate what is held out over time.
Evals owned by nobody, so they rot. Symptom: a broken eval job that has been red for six weeks and everybody scrolls past. Cause: it was a project, not a system with an owner. Fix: name an owner, put suite health on the same footing as CI health, and treat a broken eval like a broken build.
Measuring what is easy instead of what matters. Symptom: BLEU score on a creative writing task, or exact match on a summarisation feature where there are twenty good answers. Cause: the metric was available, so it was used. Fix: start from the question "what would make a user say this is wrong?" and build the metric backwards from that. A crude judge on the right question beats a precise metric on the wrong one.
| Kind | What it answers | Cost | Speed | When to run |
|---|---|---|---|---|
| Unit-style assertions | Is the output structurally legal at all? | Near zero | Milliseconds | Every commit, every pull request |
| Golden dataset scoring | Is it as good as it was on cases we care about? | Low to medium (inference cost) | Seconds to minutes | Every pull request that touches the AI path |
| LLM-as-judge | Is it good on qualities code cannot check? | Medium (extra model calls) | Minutes | Pull request (small sample) plus nightly (full) |
| Human review | Are we deceiving ourselves about all of the above? | High (people time) | Days | Before launch, on a periodic sample, on incidents |
Unit-style assertions are the cheapest thing you will ever build and they catch a disproportionate share of embarrassment. Does the output parse as valid JSON against the schema. Is the required order_reference field present and non-empty. Is it under the 500-character limit the UI enforces. Does it contain no email addresses, phone numbers, or anything matching your PII patterns. Does it avoid the ten phrases legal told you never to use. These are ordinary code, they run in milliseconds, and they should block a merge unconditionally. Write these first, on day one, before you build anything sophisticated.
Golden dataset scoring runs the system over a curated set of inputs and scores each output against an expected answer or programmatic criteria. This is the backbone. It answers the only question that matters during iteration: did this change help or hurt, and where. The cost is inference over N cases, which is why you keep a fast subset for pull requests and the full set for nightly runs.
LLM-as-judge covers what code cannot: is the answer grounded in the retrieved context, is the tone right for a distressed customer, did it answer the question actually asked. It scales to volumes human review never will. It is also the component most often deployed badly, which is why it gets its own section below.
Human review is the anchor for everything else. It is slow and expensive, so spend it deliberately: to calibrate your judge, to build the first golden dataset, to review a sample before a launch, and to investigate incidents. The mistake is treating human review as either the whole strategy (does not scale, so it quietly stops happening) or as unnecessary (your automated metrics drift away from reality and nobody notices).
Build it from real traffic, not imagination. Your intuition about what users send is wrong in specific, predictable ways: you imagine well-formed questions, they send fragments. Pull the last few weeks of real inputs and sample from them.
50-100 cases beats zero, and it beats waiting for 1,000. A team that has 60 cases running in CI this month learns more than a team designing the perfect 1,000-case corpus for next quarter. Volume comes later, and it comes for free from production.
Stratify across the input types you actually see. If 70% of traffic is simple lookups and 30% is multi-part reasoning, a dataset that is 95% lookups will tell you nothing useful about the 30% that generates the complaints. Count your real categories and sample each deliberately. Then deliberately over-weight the hard ones relative to their natural frequency, because that is where the score has room to move.
Include, on purpose: the known failures (every bug you have already fixed, so it stays fixed), the edge cases (empty input, enormous input, wrong language, contradictory instructions), and the adversarial inputs (prompt injection attempts, requests for things the system must refuse). See Guardrails and safety for what belongs in the adversarial slice.
For each case, write either the expected answer or the grading criteria. Expected answers work for extraction and classification. Grading criteria work for open-ended generation: "must state the refund window is 30 days", "must not promise a specific delivery date", "must direct the user to the returns portal". Criteria are more work to write and far more durable than a single golden string.
Keep a held-out slice. Nobody looks at it during development. It exists to tell you whether your improvements are real or whether you have been tuning against the answer key.
Who owns it: the team, with domain experts contributing the labels. Not a contractor, not a one-off project with an end date. It is a permanent living asset with a named owner, reviewed like code, versioned like code, and discussed in planning like code.
How it grows: every production bug becomes a new case. That is the flywheel. A user complains, you reproduce it, the input goes in the dataset with the correct behaviour written down, and the suite now protects that behaviour forever. Over a year this turns your dataset into an encoded specification of what your product actually promises, written in examples rather than prose.
Be clear-eyed about what you are building here. This dataset outlives your prompts, your model choice, and possibly your vendor. Prompts get rewritten monthly. Models get swapped when a better one appears. Vendors get renegotiated. The dataset survives all of it and is what lets you evaluate the replacement in an afternoon rather than a quarter. It is the most durable thing your team will build, and it is worth saying that out loud to the team, because labelling data feels like unglamorous work right up until the day it saves you.
An LLM judge is a model call whose job is to score another model's output against a rubric. It works because grading is easier than generating. Deciding whether an answer is supported by a passage is a far narrower task than writing the answer, and narrower tasks are where models are most reliable.
The rules that make it trustworthy:
Use a specific rubric with a small scale. Ask for 1-5 on "quality" and you get noise: the model clusters on 4, and the difference between a 3 and a 4 is not reproducible across runs. Ask a binary question with an explicit definition ("Is every factual claim in the answer supported by the provided context? Answer YES or NO") and you get signal. If you need gradation, use three points with written definitions of each point. Small, defined scales beat large, vague ones.
One criterion per judge call. Do not ask a single call to rate groundedness, tone, and completeness. The scores contaminate each other and you cannot tell which one moved. Separate calls cost more and are worth it.
Swap positions for pairwise comparison. If you ask which of two answers is better, models have a position bias. Run it both ways (A then B, B then A) and only count it as a win if the judgement is consistent. Ties tell you the difference is not real.
Calibrate against human labels before you trust it. Take 40-60 cases, label them by hand with a domain expert, run the judge over the same cases, and compare. If the judge disagrees with humans on a third of cases, fix the rubric before you build anything on top of it. Re-calibrate periodically, and always after you change the judge model or prompt.
Never judge with the same prompt that generated the answer. Self-consistency is not correctness. A model that made a reasoning error is likely to endorse the same error when asked to check it with the same framing.
Know the biases. Judges favour longer answers. They favour outputs from the same model family (self-preference). They favour confident, well-formatted prose over correct-but-plain answers. Design around this: normalise length in the rubric, use a different model family for the judge than the generator where you can, and include cases in your calibration set where the wrong answer is prettier than the right one.
Say this to your team plainly: an uncalibrated judge is a random number generator with a confident tone, and building a roadmap on its output is worse than having no metric at all, because no metric leaves you appropriately uncertain.
| Metric | What it measures | Watch out for |
|---|---|---|
| Exact match | Output equals the expected string | Only valid for extraction and classification. Punishes correct paraphrase. |
| Similarity score | Semantic closeness to a reference answer | High similarity to a wrong reference is still wrong. Insensitive to a single flipped negation. |
| Groundedness / faithfulness | Are claims supported by retrieved context | Needs a judge. Grounded and useless is possible; pair with relevance. |
| Answer relevance | Does it address the question asked | A relevant but fabricated answer scores well. Pair with groundedness. |
| Retrieval recall@k | Did the right documents make it into the context | Requires labelled relevant documents. Perfect recall with a bad generator still fails. |
| Task success rate | Did the end-to-end job get done | Needs a crisp definition of done. Ambiguous definitions make this unfalsifiable. |
| Refusal rate | How often the system declines | Falls with looser guardrails and users cheer, right up to the incident. Track both directions. |
| Latency p95 | Tail response time users feel | Averages lie. Judge and retrieval calls inflate it. Measure the pipeline, not the model call. |
| Cost per request | Spend per unit of work | Rises silently when context grows. See Cost management. |
Two rules make this table useful rather than decorative.
Measure retrieval and generation separately. If a RAG answer is wrong, there are two suspects: the right document never made it into the context, or it did and the model ignored it. These have completely different fixes (chunking, embeddings, reranking versus prompt, model, context ordering) and one blended score cannot distinguish them. Score recall@k on retrieval and groundedness on generation, then read them together. RAG explained covers the mechanics.
Pair every quality metric with cost and latency. Otherwise someone optimises quality into the ground: they add a reranker, three retrieval passes, and a self-critique loop, the score goes up two points, and the p95 goes from 1.2s to 9s at four times the cost. That is not an improvement, it is a trade you did not consent to. Put the three numbers on the same dashboard row so the trade is always visible.
This is the operational heart. Evals that run when someone remembers are not evals, they are a hobby. The suite runs automatically on every change to a prompt, model, retrieval config, or tool definition.
flowchart TD
A["Change pushed"] --> B["Fast tier: assertions plus subset"]
B --> C{"Pass?"}
C -->|No| D["Block merge"]
C -->|Yes| E["Full suite"]
E --> F["Compare to baseline"]
F --> G{"Aggregate above threshold?"}
G -->|No| H["Gate fails"]
G -->|Yes| I{"Any critical-case regression?"}
I -->|Yes| H
I -->|No| J["Deploy"]
H --> K["Human override with reason"]
K --> L["Logged and reviewed"]
L --> J
Two tiers, because you cannot have both fast and thorough. The fast tier runs on every pull request: all the deterministic assertions plus a stratified subset of the golden dataset large enough to catch obvious damage. It has to finish in a couple of minutes or people will route around it. The full tier runs nightly and on release candidates: the entire dataset, every judge criterion, per-slice breakdowns.
The gate has two conditions and both must hold. First, the aggregate score must be at or above your threshold. Second, and this is the one teams leave out, a no-regression rule on the critical subset: any regression on a case marked must-never-fail blocks the merge regardless of the average. The average will happily hide the one case that gets you in the newspaper. If your suite has 200 cases and the 3 that involve giving medical dosage information now fail, an aggregate that rose from 0.84 to 0.86 is not good news.
Results are posted to the pull request as a comment: score, delta versus baseline, which cases flipped, links to the diffs. The reviewer sees it in the place where they decide. A dashboard nobody opens is not a control.
A score drop needs an explicit human decision to override, with a written reason, logged. Sometimes the override is correct (the dataset had a wrong label, the regression is on a case you intentionally deprecated). The point is not to make override impossible, it is to make it visible and attributable. An override that costs nothing gets used every Friday afternoon.
Offline evals prove you did not break what worked. They run against a fixed dataset, they are reproducible, they gate merges. What they cannot tell you is whether the thing works for real users, because your dataset is a snapshot of a past that users have already moved on from.
Online evals prove it works in production. Three mechanisms. Sampling real traffic: score a percentage of live requests with your assertions and judge, and alert on drift. User feedback signals: thumbs, edits, copy events, escalation to a human agent, abandonment. The implicit signals are often better than the explicit ones, because almost nobody clicks thumbs-down but everybody abandons. A/B tests: the only way to attribute a business outcome to a change, and the slowest, so reserve it for changes big enough to justify the wait.
You need both, and the loop is what makes them a system rather than two disconnected activities. Online tells you what is failing now. You triage those failures, label them, and promote them into the offline golden set. Now the failure is permanently guarded, and next month's suite is a better model of reality than this month's. The teams that get compounding returns from evals are the ones that run this loop weekly and boringly. See Observability and monitoring for the instrumentation that makes sampling and tracing possible.
You have inherited a team with no evals. Do this, in this order.
- Collect 50 real inputs this week. From logs, from support tickets, from the sales demo transcripts. Real ones. Do not write them yourself.
- Hand-label the outputs with a domain expert. Sit with them for two hours. Run the current system over the 50 inputs and mark each output good or bad, and write one line on why. This is also how you discover that your team disagrees about what good means, which is the more valuable finding.
- Write three assertions and one judge criterion. The assertions come from the failures you just saw (it produced malformed JSON, it exceeded the length limit, it leaked an internal ID). The judge criterion is the single most important quality dimension, expressed as a binary question with a written definition.
- Wire it to CI. One command, runs on pull requests touching the AI path, posts the score as a comment. Do not build a platform. A script and a JSON file is fine.
- Never ship a prompt change without it. Make this a rule with no exceptions for two months. The habit is the deliverable, not the tooling.
A scrappy suite this week beats a perfect one next quarter. The scrappy suite starts catching regressions immediately and tells you what to build next. The perfect one is still in design review while your team ships three untested prompt changes. Use the eval plan template to write it down in an hour.
- What is our eval suite's score today, and what was it a month ago? A good answer is two numbers and a reason for the difference. A bad answer is a pause, or a number with no history, which means the suite is decorative.
- Which case in the suite would we never allow to regress, and does the gate enforce that? A good answer names a specific case and points at the config that blocks the merge. If they name a category but no mechanism, the protection is aspirational.
- Where did the cases in our golden dataset come from? A good answer is "sampled from production, stratified by intent, plus every bug we have fixed". A bad answer is "we wrote them", which means you are measuring your assumptions.
- How well does our judge agree with a human on the same cases? A good answer cites a calibration exercise and when it was last redone. "We use a strong model so it is fine" means nobody has checked.
- What is our worst-performing slice, and why? A good answer names the slice, has a hypothesis, and has it on the roadmap. If they only know the aggregate, they cannot see the subgroup that is failing.
This is the single strongest differentiator between a real AI EM and a keen amateur. Interviewers listen for whether evals come up before they ask. Given an open design prompt ("build an AI assistant for our support team"), the amateur designs the pipeline, picks a model, and stops. The real EM designs the pipeline and then, unprompted, says: here is how we know it works, here is the dataset, here is what blocks a deploy, here is what we watch in production. That happens in the first five minutes or it does not happen.
A shallow answer says "we would evaluate it" and reaches for a metric name. A strong answer is specific about the mechanism and the trade-offs: which checks are deterministic and which need a judge, why the judge needs calibration and what you do if it disagrees with humans, why you gate on the critical subset and not just the average, how offline and online close the loop, and what you would accept as a quality drop in exchange for halving latency. The trade-offs are what signal experience, because only someone who has run this has been forced to make them.
The other strong move is scoping. Asked how you would start with no evals, a shallow answer proposes a comprehensive framework. A strong answer says 50 cases, three assertions, one judge criterion, in CI this week, and explains why the scrappy version is the correct engineering decision rather than a compromise. Interviewers hear that as someone who has shipped rather than someone who has read.
Practise the concrete version: a change you gated, a regression the suite caught before users saw it, a time the metric was green and the users were not, and what you changed about the dataset afterwards. See top 10 questions and evaluation-driven development for how this shows up as a team practice rather than a technique.
Next: AI system design patterns
Related: Evaluation-driven development | Eval plan template | Observability and monitoring