feat(examples): a runnable LLM gateway showing the policies under load - #17
Merged
Merged
Conversation
"Show me the code" is the first thing anyone asks, and adaptive concurrency limiting is close to impossible to believe from prose. This is the same claim as a program that prints what it is doing. A simulated provider degrades from ~140ms to seconds at a FLAT error rate, then pushes back with 429s, then recovers. Roughly 25 requests/second across three tenants, one in five background. Per-second table: t phase limit inflight p90ms | ok 4xx shed fail 7s healthy 25 0 143 | 148 27 0 0 14s degrading 23 56 1186 | 190 34 68 2 19s degrading 21 42 3358 | 203 38 190 2 23s overloaded 5 36 7880 | 215 40 272 12 Between 7s and 19s latency rises 23x while failures stay at 2. That is the incident this library exists for, and it is the window a failure-rate breaker cannot see — it is watching errors and there are none. The limiter is watching latency, so concurrency goes 25 → 21 → 5 and the excess is shed before the timeouts start. Fifty healthy 4xx arrive across the run and none of them counts against the breaker, which is the verdict model earning its place. Covered in three files rather than one, so each use case has an obvious home: gateway.ts is policy ordering, verdicts, keys, tenants, criticality and the shared retry budget; run.ts is ctx.mark() and RejectedError.reason; upstream.ts is why concurrency rather than rate is the right lever, since latency there rises with concurrency. Type-checked and run in CI — tsconfig now includes examples/, and CI runs it for nine seconds at 4x speed, which traverses all four provider phases. An example that does not compile is worse than no example, and this is the first code anyone will read. Deliberately not a benchmark. Seeded PRNG, tuned so the transitions are visible in under a minute; the README and the docs page both say the shape is the point and the numbers are not for quoting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pnpm example:gateway— a provider that degrades from ~140ms to seconds at a flat error rate, then recovers, with a per-second table.Between 7s and 19s latency rises 23× while failures stay at 2 — the window a failure-rate breaker cannot see. The limiter walks concurrency 25 → 21 → 5 and sheds before the timeouts start.
Covers policy ordering, verdicts, isolation keys, tenant fairness, criticality, shared retry budget,
ctx.mark(),RejectedError.reason.Type-checked (tsconfig now includes
examples/) and run in CI at 4× speed.