switchyard

A governed AI platform: a model gateway, a job queue, and an observability stack, running on local Kubernetes.

It routes requests across local models by policy, falls back when one is unhealthy, meters and bills every caller, runs long jobs through a queue that survives a dead worker, traces and alerts on all of it, deploys itself from this repository through Argo CD, and has been broken on purpose three times to find out what it actually does.

Every line of code here was written by an AI (Claude Code) under a human director who wrote none of it and instead verified every phase by hand — the project is as much a study in governed AI orchestration as in platform engineering.¹ That discipline shows in the test suite: 658 tests, 0 failing, of which roughly 56% cover the product and 44% cover the governance tooling that keeps the AI honest — the claim checkers, the rehearsal harness, and the drill instruments.²

A note on the codes. Short IDs such as M3, Q-33, D-26, ENV-04 and S109 appear throughout — they are stable references into the decision ledger. See Notation for what each prefix means.

Contents

• What it does · Milestones
• See it running — architecture and live captures
• Install and run
• Notation — what the M, Q, D, ENV … codes mean
• Where the record lives
• How this project is built
• The two published numbers
• Honest limits · License

What it does

The gateway exists and works: it forwards OpenAI-format chat completions to a local Ollama, carries the undocumented reasoning field and the token counts through intact, maps every upstream failure to a documented status, and survives its upstream being killed and restarted without a restart of its own. It has one provider and binds to localhost.

M3 adds identity and metering. Callers are named by API key, with secrets held outside the tracked configuration; each key carries a short-window rate limit and a long-window token quota, refused with distinguishable 429s and an honest Retry-After. Every request writes a cost record to Redis carrying two figures that are never both simply "cost" - what was consumed and what the caller is answerable for - plus a count of attempts whose cost could not be observed at all. Until a key is declared the gateway authenticates nothing and limits nothing, and says so on every response.

M4 adds asynchronous work. Jobs are submitted to a Redis Stream and run by a separate worker process through the gateway's own provider path, so a queued completion is routed, retried, priced and recorded exactly like a direct one. Idempotency is the server's rather than this project's. A worker that dies mid-job leaves its entry pending, a recovery sweep reclaims it, and a job that has been delivered too many times is moved to a dead-letter stream with its reason attached instead of being retried forever.

The submission path fails CLOSED while completions fail OPEN, on the same Redis, in the same process - accepting a job you did not store is losing it while telling the caller you kept it. Two workers can occasionally finish the same job, and the acknowledgement decides which answer is served: the loser keeps its own cost record, marked, because the tokens were really spent. Cost records are append-only per attempt, so a redelivery cannot erase what an earlier attempt consumed.

Milestones

The platform was built in nine milestones (M0–M8), each with its own contract (docs/decisions/STEP-NN-*.md), its own plain-English concept doc (concepts/), and closed only on evidence the Director approved.

#	Milestone	What it added
M0	Scaffold	Repository, charter, tooling, Ollama and two local models
M1	Gateway core	FastAPI service serving OpenAI-format /v1/chat/completions, forwarding to Ollama behind a provider abstraction
M2	Routing + fallback	Policy-based model selection (capability, cost weight, latency), health checks, a fallback chain, and bounded retries
M3	Quotas + cost	Per-key rate limits and token quotas in Redis, per-request cost accounting, and distinguishable 429s
M4	Queue + workers	Async jobs on Redis Streams consumer groups, a reclaim sweep, idempotency, and a dead-letter stream
M5	Observability	OpenTelemetry GenAI traces, Prometheus metrics, Grafana dashboards, and alert rules
M6	Kubernetes	Dockerized services, a kind cluster, a Helm chart, health probes, and rolling update plus rollback
M7	GitOps	Argo CD syncing the Helm chart from this repo, with drift detected and reconciled
M8	SRE drills	Killing a worker, Redis, and an Ollama backend on purpose, each with a runbook and an RCA, plus a capacity note

See it running

docs/showcase/ presents the whole system wired together and proof of it running — captured live, with timestamps, on 2026-08-22. It has three parts:

• How it connects — an architecture diagram and a walk through the data path for a synchronous completion, an async job, and the GitOps reconcile loop.
• The full system running — a real completion (routed, priced, answered by a model) and an async job carried through the queue and worker, end to end.
• Observability — five completions moving switchyard_requests_total from 0 to 5 in Prometheus, with cost recorded, plus the M5 Grafana dashboards.
• GitOps self-heal — Argo CD reverting hand-planted drift on the live cluster automatically, with no human, in under a second, with the Argo CD UI captured in docs/showcase/gitops-argocd.md.

The raw, timestamped captures behind those pages are under evidence/showcase/.

Install and run

Everything is local — Python 3.14, Docker, and Ollama. No paid APIs are used at any point in this project.

switchyard is packaged as one container image with two entry points (gateway and worker), published on the GitHub Container Registry:

docker pull ghcr.io/mohdsaifhussain/switchyard:1.0.0

The fastest way to run it is the demo stack, which brings up Redis and the gateway plus a worker wired to Ollama on your host. On first run it builds the image locally from the Dockerfile at the repository root, or you can pull the published image above.

# Prerequisite: Ollama running with the two models pulled
ollama pull qwen3:4b
ollama pull llama3.2:1b

# Bring up the stack (auth disabled in the demo)
docker compose -f deploy/demo/docker-compose.yml up

Then:

curl http://localhost:8000/livez     # 200 — a heartbeat independent of Redis/Ollama
curl http://localhost:8000/health    # names every dependency and the posture toward it
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Say hello in one word."}]}'

Full prerequisites and the operator guide — verifying it, turning authentication on, and the two deliberate "refuses to start" cases — are in deploy/demo/README.md.

Notation

Throughout this repository, short IDs reference the append-only decision ledger, DECISIONS.md (browsable at docs/ledger/). An ID, once issued, always refers to the same entry — so Q-33 means the same ruling wherever it appears.

Prefix	Meaning
M0–M8	The nine milestones; each has a contract at docs/decisions/STEP-NN-*.md
Q-NN	A ruling — a decision the Director made, with its conditions
D-NN	A defect found, with how it was found and how it was resolved
DEV-NN	A deviation from a contract, with its reason
OPEN-NN	An open question the project knows it has not answered
ENV-NN	An environment trap — a property of this machine or toolchain
S-NN	A source: a URL fetched, when, and what it settled (SOURCES.md)

Where the record lives

Nine milestones, each closed on evidence in this repository that a stranger can re-read. 9 milestones closed; the charter's map is complete. Released as v1.0.0 on 2026-08-22, with the container image published to GHCR. This remains a self-operated portfolio project — no production traffic — and the honest limits below stand unchanged.

Path	What it holds
CLAUDE.md	The charter: roles, hard rules, milestone map
docs/decisions/STEP-NN-*.md	One contract per milestone: objective, deliverables, rulings, outcome
DECISIONS.md	Append-only ledger: every ruling, defect, and deviation, by ID
docs/ledger/	A generated, browsable view of the ledger, one page per namespace
SOURCES.md	Every URL fetched, when, and what it settled
docs/showcase/	The system wired together, and proof of it running
docs/HARVEST.md	Candidate rules earned here, queued for the skill at project end
RUNBOOK.md	Playbooks to investigate and runbooks to act, written from the three drills after they ran
docs/CAPACITY.md	This machine's measured ceilings, what hit each one, and what the platform did
docs/rca/	One root-cause analysis per drill, on the Google SRE postmortem fields
CHANGELOG.md	Keep a Changelog format
concepts/	One plain-English concept doc per milestone, no code
evidence/	Command transcripts: what was actually run, and its real output
tools/check_claims.py	Verifies counts stated in prose against the ledger
tools/verify_phase.py	One command for every mechanical phase-close check
tools/config_refusals.py	Shows the shipped config refusing four silent misconfigurations
tools/prepush_audit.py	What leaves the machine, and where it lands: history scan plus a visibility read
tools/check_cluster.py	Cluster state against the tracked kind config: reachable, Ready, unskewed, undrifted, no residue

How this project is built

Every milestone runs the same way, and nothing self-closes: a research pass from official documentation logged in SOURCES.md, then the build, then a green test suite, then evidence captured in evidence/, then a concept doc in concepts/ written in plain English. The Director's approval closes the milestone.

Two rules do the heavy lifting:

• Official sources only. Every technical decision is researched from the vendor's own documentation and cited by URL with the date it was fetched. Where the documentation is silent, the question is settled by a measured probe and recorded as a measurement, never as an assumption.
• Every gate proves it can fail. A check that has only ever passed is a decoration. Each gate carries a negative control that must refuse and a positive control that must accept, with a distinct reason per failure mode.

The two published numbers, and why they are checked

The suite currently has 658 tests, 0 failing. Of those, 55 are integration tests that exercise a real running Ollama or a real running Redis, and 6 read the live Kubernetes cluster; each of those skips, with its reason printed, when what it needs is absent.

The count is passed plus skipped, and that is a weaker claim than a passed-only figure — so it is not written that way. The passed-only figure read 555 with the cluster up and 551 with it down when it was measured on 2026-08-19, against the suite as it stood that day; those two readings are what ruling Q82(b) was decided on, and they are left exactly as measured rather than restated at today's size (Q-135). The count says how big the suite is and that nothing in it is failing; it does not say every test ran. Counting only passes made the published number move with whatever happened to be running on the machine that day rather than with the suite, so a stopped container turned the check red for a reason that had nothing to do with the code.

That number is not maintained by hand: tools/check_claims.py re-derives it from an actual pytest run and fails if this line disagrees with it. A real failure still turns it red — any failing or erroring test makes the count unreadable rather than merely smaller, so a skip can never mask a failure. It has now caught this line drifting thirty-two separate times.

That count is now verified rather than asserted. Its previous wording put a word between the noun and "closed", which the claim checker's pattern never matched - so the one number describing this project's progress was the one number nothing checked. The obvious rewording was worse and was tested before being rejected: the checker reads the digit nearest the noun, so a phrasing that names the total immediately before it would have had the tool verifying the wrong figure. This sentence deliberately carries no digit before that noun, because an illustration is indistinguishable from a claim to anything that reads by pattern (D-17).

Honest limits

This section is standing. It is not removed as the project matures, and it is narrowed only when something genuinely narrows.

• This platform is self-operated on one machine. A single Windows laptop: 15.6 GB RAM, an NVIDIA RTX 4060 Laptop GPU with 8188 MiB of VRAM, an Intel i7-13650HX. Every ceiling in this project is that machine's ceiling.

• No production traffic, ever. All load is self-generated. Nothing here has served a user, and no number in this repository should be read as a production measurement.

• SRE drills are self-inflicted. The failures exercised in M8 are ones this project causes on purpose. They are rehearsals, not incidents, and finding that a rehearsed failure was handled says nothing about an unrehearsed one.

• Model quality is never evaluated here. Routing decisions are made on size, latency and health. This project makes no claim about which model answers better, because it has not measured that and will not.

• Latency figures are single observations on a laptop under unknown background load. They are recorded as observations. They are not benchmarks and are not cited as benchmarks.

• One provider. Provider-level routing is not exercised. The gateway has a provider abstraction and routes between targets, but every target sits behind the same local Ollama, so what is actually proven is model routing behind that abstraction. Calling it provider routing would overstate what has been run. The standing evidence that a second provider can be added without editing the request handler is a test that registers one and resolves it; that test passes, and it is not the same thing as having run two. Recorded by ruling Q-20 and carried forward until a second provider lands.

• This gateway authenticates nothing until its first key is declared. With no [keys.*] table in switchyard.toml, every request is served unauthenticated. That is deliberate - a freshly cloned repository has to be usable before it is understood - and it is made loud rather than quiet: startup logs it, /health reports "auth": "disabled", and every response carries x-switchyard-auth: disabled (no keys configured) so a caller learns the truth from any reply. This is the one line that closes it:

  [keys.your-name]
  limits = "standard"

  then set SWITCHYARD_KEY_YOUR-NAME in the environment, or add the secret to switchyard.secrets.toml, which is gitignored. A declared key with no secret anywhere refuses startup rather than silently never matching. Recorded by ruling Q-33.

• Some costs are unknowable, and are recorded as unknown rather than as zero. When an upstream attempt fails it reports no token usage at all: a 5xx carries no usage block, and a timeout carries no body. But a timeout may mean the upstream generated and billed an entire completion nobody ever saw. So a failed attempt is counted as consuming an unknown amount, labelled wherever that count appears, and never priced at zero. Recorded by ruling Q-34.

• A caller can achieve double the configured rate across a window boundary. Rate limiting is fixed-window, and this weakness is not merely documented - it is run: three requests per minute became six in a two second span, in evidence/M3/d6-boundary-burst.md. A sliding window would prevent it at the cost of more state per caller. Recorded by ruling Q-27.

• A refused request still counts against its rate window. The counter increments before the check, per redis.io's own documented pattern. For a rate limit this is arguably correct - it is the caller ignoring the refusal that the backend needs protecting from - but 429s are not free.

• A caller can overshoot their quota by one request. Consumption cannot be predicted before the call - M2-D2 measured a fourfold spread in completion tokens for identical input, and 86x to 357x between targets - so the check admits on what is already recorded and the charge records what was used. A quota bounds what a caller may START, not what they may consume.

• A queued job that runs twice charges the quota once, and the second run's tokens are recorded but never billed. A worker can be reclaimed while it is still alive, so two workers occasionally finish the same job; only one answer is served and only that one is charged. The tokens the other burned are real, and they appear in that attempt's own cost record marked as a duplicate serve - so an operator reconciling total consumption sees them, and the caller's allowance does not drain for them. The reason is that the second run is system-caused: the recovery sweep's bet going wrong is the operator's cost, not the caller's, and a quota that drained because a reclaim threshold guessed wrong would punish a caller for the platform's uncertainty. Answer not served, cost recorded, quota not charged. Recorded by the ruling on OPEN-15.

• When Redis is unavailable, requests are served unlimited and unrecorded. Deliberate: a gateway that stops serving because its BOOKKEEPING is down has turned one outage into two. It is never silent - every response says limits were not enforced, /health reports the store unreachable, and the blind window is logged with a start and an end so reconciliation knows exactly which period is unaccounted. The per-request numbers from that window are lost permanently. Recorded by ruling Q-28.

• Counters survive a graceful Redis stop, and do not survive a kill. This was stated backwards until it was measured, and the correction is kept rather than quietly edited (D-26). Redis snapshots to disk by default; docker stop sends SIGTERM, which triggers a blocking save, and the container reloads that snapshot on start. What is lost is everything written since the last snapshot when Redis is killed rather than stopped - docker kill, a crash, or power loss - and everything, without exception, if the container is removed, because no volume is mounted and the snapshot lives in the container's own writable layer. Quotas are therefore durable enough to be misleading: durable across the stop-start an operator does deliberately, and not across the failure they do not. And that durability is a property of the default configuration, not of stopping: redis.io saves on shutdown only "if at least one save point is configured", so a Redis run as a pure cache with save "" loses everything on the same graceful stop.

• With no keys declared, nothing is limited either. Limits attach to a caller, and an unauthenticated request has none to attach to. The open-by-default state announced on every response covers this too.

• Health information is never fresher than its cache window. Routing checks a target on demand and caches the answer briefly, so the first request after a target dies still pays that failure. Where a fallback exists the caller pays it as latency rather than as an error, which softens the cost without removing it. Recorded by ruling Q-21.

License

MIT — see LICENSE. Copyright © 2026 Mohd Saif Hussain.

Footnotes

1. Governed AI orchestration is the method this project is built under: a written charter, per-milestone contracts, and verification gates so that no claim is trusted wider than its evidence. The rules are in CLAUDE.md, and the reasoning behind each decision is in the ledger, DECISIONS.md. ↩

2. Measured by test-function count across tests/: roughly 232 functions exercise the product (gateway, routing, quotas, queue, worker, telemetry, the Helm chart) and roughly 180 exercise the governance tooling (the claim checker, phase verifier, rehearsal harness, and drill instruments). ↩