AI system design is now a standard interview loop stage for AI/GenAI/LLM engineer roles at frontier labs, big tech, and AI startups - usually a 45-60 minute whiteboard session like "Design a customer support agent" or "Design semantic search for our catalog." It tests whether you can turn a fuzzy product idea into a shippable, measurable, affordable system with a model inside it. Senior candidates fail this round more often on missing evals and cost math than on missing boxes in the diagram.
Classic system design intuitions still apply (load balancing, sharding, caching, queues), but four things change fundamentally:
| Dimension | Classic system design | AI system design |
|---|---|---|
| Core component | Deterministic services you write | A model you don't fully control - treated as a fallible, versioned dependency |
| Correctness | Provable: tests, invariants, ACID | Probabilistic quality: you can't prove correctness, so evals replace correctness proofs - you argue with measured accuracy, faithfulness, task success rates |
| Cost model | Mostly fixed infra; marginal request cost ≈ 0 | Cost per request is significant (fractions of a cent to dollars). Token math is the new capacity planning; model choice is a first-class cost lever |
| Latency | Single number (p99 of an RPC) | Shaped latency: time-to-first-token vs tokens/sec vs total; streaming UX; agent loops take seconds-to-minutes |
| Failure modes | Crashes, timeouts, data loss | Plus semantic failures: hallucination, prompt injection, refusal, drift after a silent model update |
| Iteration | Ship features | Quality flywheel: log traces → build eval sets → improve prompts/retrieval/models → re-measure |
If your answer would be identical with a deterministic service where the model sits, you're not answering an AI system design question.
Use this skeleton for any prompt. In a 45-minute interview: ~5 min on step 1, ~10 min on steps 2-3, ~15 min on step 4, ~15 min on steps 5-8 (interviewers weight evals and ops heavily - don't run out of time before them).
Clarify users, scale (QPS, corpus size, growth), latency expectations, and what "good" means numerically - e.g. "answer accuracy ≥ 90% on a golden set, deflection rate ≥ 50%, p95 TTFT < 1.5s, cost < $0.05/query." Name the north-star metric and the guardrail metrics before drawing anything.
API vs self-host (data sensitivity, scale economics, latency control, ops burden), and tiering: route easy/high-volume traffic to a small cheap model, hard traffic to a frontier model. State fallbacks across providers and how you'll pin + upgrade model versions.
How does the model get the knowledge it needs? Default decision order: prompting → RAG → tools → fine-tuning. RAG for facts that change; fine-tuning for style/format/narrow skills, not knowledge freshness; tools for live state and actions. Define chunking, indexing, freshness pipeline, and ACLs here.
Draw the request path: client → gateway → (router) → orchestration → retrieval/tools → model → guardrails → response, plus the offline paths (ingestion, eval, training). Say what's synchronous vs queued, what's cached, where state lives.
The step that separates seniors: a golden set (100-1,000 labelled cases), automated scoring (exact-match where possible, LLM-as-judge with a rubric where not, calibrated against human labels), component-level metrics (retrieval recall@k separate from answer faithfulness), CI gating on prompt/model changes, and online A/B or interleaving.
Latency budget per stage, streaming, caching (prompt caching, semantic caching, KV cache implications), rate limits and provider quota management, graceful degradation (fallback model, "I don't know" responses, cached answers), tracing every LLM call with prompt/version/tokens/latency.
Input guards (injection detection, PII redaction, topic filters), output guards (schema validation, citation checks, moderation), action guards for agents (allowlisted tools, spend/blast-radius caps, human approval for irreversible actions), and tenant isolation.
What ships in week 1 vs quarter 1: start with the simplest system that can be measured (often a single prompt + retrieval), instrument everything, and let eval data drive complexity. Explicitly name what you would not build yet.
Most AI systems assemble the same ~10 blocks. Naming these fluently signals experience:
| Block | What it does | Design notes |
|---|---|---|
| Model gateway | Single choke point for all LLM calls: auth, quotas, retries, provider failover, usage metering | Buy or build thin; never let services call providers directly |
| Router | Sends each request to the right model tier (small/fast vs frontier) by classifier, heuristics, or task type | Biggest single cost lever; measure quality delta per tier |
| Retrieval service | Hybrid (BM25 + dense) search over chunked, permission-tagged corpus with reranking | Own service with its own evals (recall@k, nDCG) |
| Context builder | Assembles the prompt: system instructions, retrieved chunks, memory, tool schemas, under a token budget | Deterministic and testable; log the exact final prompt |
| Orchestrator | Runs multi-step flows: agent loops, tool dispatch, retries, timeouts, max-iteration caps | Durable execution (queue-backed) for long-running agents |
| Guardrails layer | Input/output/action checks: injection, PII, schema validation, moderation, policy engine for tool calls | Fail closed on actions, fail open (with logging) on style |
| Eval harness | Runs golden sets against any prompt/model/retrieval config; scores, diffs, gates CI | Version everything: prompts, datasets, judge prompts |
| Tracing & observability | Per-request trace of every model call, tool call, retrieval, with tokens/cost/latency/feedback attached | The raw material for eval sets and debugging |
| Caches | Exact-match response cache, semantic cache, provider prompt caching for shared prefixes | Prompt caching often cuts input cost 50-90% on long shared contexts |
| Feedback store | Thumbs, edits, acceptances, escalations, linked to traces | Feeds the quality flywheel and fine-tuning data |
| Human-in-the-loop queue | Review UI for low-confidence outputs and high-risk actions | Design the escalation path before the model is trusted |
- Clarifying questions first. Jumping straight to an architecture is the most common fail. Scope, scale, quality bar, and constraints are never fully specified on purpose.
- Tradeoff articulation. Every choice ("RAG over fine-tuning", "self-host the completion model") stated with the alternative, the reason, and what would change your mind.
- Eval literacy. Can you say concretely how you'd know the system works - datasets, metrics, judges, online experiments? This is the sharpest senior/junior discriminator in 2026 loops.
- Cost awareness. Back-of-envelope token math: requests/day × tokens/request × $/token. Being within 10x with clearly stated assumptions is the bar.
- Pragmatism. Simplest thing first, complexity justified by measured need. "I'd start with one prompt and a golden set" beats a 12-box agent swarm.
- Failure-mode thinking. Injection, hallucinated citations, provider outage, silent model updates, stale indexes - raised unprompted.
- Latency realism. Knowing that TTFT, streaming, and agent loop counts dominate perceived speed, and budgeting per stage.
- Designing for the model as an infallible oracle - no eval plan, no fallback, no "when it's wrong" path.
- Fine-tuning as the first answer to a knowledge-freshness problem.
- No numbers anywhere: no QPS, no token counts, no cost estimate, no latency budget.
- Agent maximalism: multi-agent swarms for a problem a single prompt + retrieval solves.
- Ignoring ACLs/tenancy in enterprise scenarios ("just embed everything into one index").
- Treating "LLM-as-judge" as free and infallible rather than a component that itself needs calibration.
- No mention of prompt injection when the design ingests untrusted content or takes actions.
Practice the 8-step skeleton on these until it's reflexive (one-liners on purpose - generating the clarifying questions is the exercise):
- Design a company-wide knowledge assistant over wikis, docs, and tickets.
- Design an AI coding assistant with inline completion, chat, and agentic edits.
- Design a customer support agent that can look up orders and issue refunds.
- Design semantic search for a 100M-item e-commerce catalog.
- Design a content moderation system for user-generated posts at 10k posts/sec.
- Design a document intelligence pipeline for invoices and contracts.
- Design a natural-language-to-SQL analytics agent for business users.
- Design a meeting assistant: transcription, summaries, action items, follow-up search.
- Design an email triage and drafting assistant for a sales team.
- Design a personalised news/feed summariser for 10M daily users.
- Design an AI feature flag: roll out a new model to 5% of traffic safely.
- Design a batch pipeline to classify and tag 500M legacy documents.
- Design a voice agent for restaurant reservations end-to-end.
- Design a code review bot for a 2,000-engineer monorepo.
- Design an LLM gateway for a company with 40 teams calling 5 model providers.
Full worked solutions following a consistent template (problem → clarifications → requirements → architecture → deep-dives → data strategy → evals → cost math → failure modes → ops → follow-ups):
- Enterprise RAG Assistant - company-wide knowledge assistant: ACLs, freshness, citations, hybrid search, eval harness.
- AI Code Assistant - IDE completion + chat + agents: latency tiers, FIM, repo context, privacy, acceptance-rate evals.
- Customer Support Agent - tool-using agent over orders/refunds: action guardrails, handoff, deflection metrics, injection defence.
- Semantic Search at Scale - e-commerce search: hybrid retrieval, reranking, ANN index ops, embedding refresh, sub-100ms budgets.
- Content Moderation - high-throughput classification with tiered models and human review.
- Document Intelligence - extraction from invoices/contracts with schema validation and HITL.
- Text-to-SQL Agent - NL analytics over a warehouse with correctness verification.
- Meeting Assistant - transcription, summarisation, action items, and search over meetings.
- Real-time Voice Agent - a phone agent under a sub-second voice-to-voice budget, with barge-in and turn-taking.
- LLM Gateway and Serving Platform - multi-tenant routing, quotas, failover, and the serving data plane behind them.
- Building Effective Agents - Anthropic - the canonical "start simple, add agency only when needed" argument.
- Patterns for Building LLM-based Systems & Products - Eugene Yan - evals, RAG, guardrails, caching as reusable patterns.
- What We Learned from a Year of Building with LLMs - O'Reilly - practitioner field notes across the whole stack.
- LLM Powered Autonomous Agents - Lilian Weng - the reference taxonomy for agent components.
- Lost in the Middle: How Language Models Use Long Contexts - why context assembly and ordering matter.
- Hidden Technical Debt in Machine Learning Systems - the classic on why the model is the small box in the diagram.
- Designing Data-Intensive Applications - Kleppmann - the classic-system-design half of the interview still rests on this.