New to client-go's workqueue? Read this first, then README.md for how to run it.
This lab exists to make one sentence visible:
A workqueue is a funnel that turns a firehose of "something changed" events into each key being processed one at a time, never lost, never duplicated — and it does it with just two sets and a slice.
Inside client-go's workqueue.Type there is no cleverness, just this:
Add(key) worker Get() worker Done(key)
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌────────────────┐ ┌────────────────┐
│ DIRTY set │ Get moves │ PROCESSING set │ Done │ PROCESSING set │
│ "needs work" │ ───────────▶ │ "being worked │ ────────▶ │ (key removed) │
│ │ dirty→proc │ on right now" │ │ │
└───────┬───────┘ └────────────────┘ └───────┬────────┘
│ key not dirty & not processing │ key dirty?
▼ ▼
┌───────────────┐ re-push to FIFO
│ FIFO slice │ EXACTLY ONCE
│ (the queue) │
└───────────────┘
A key's life as a state machine:
The states are just set memberships:
| state | dirty | processing | meaning |
|---|---|---|---|
| Absent | ✗ | ✗ | queue doesn't know the key |
| Waiting | ✓ | ✗ | sitting in the FIFO |
| Processing | ✗ | ✓ | one worker owns it |
| Processing + pending redelivery | ✓ | ✓ | owned, AND changed again meanwhile |
Two invariants do all the work:
- A key is in
processingat most once → exactly one worker can ever hold a given key. You never need your own "am I already reconciling obj-7?" lock —Get()hands out each key to one worker, period. - A key in
dirtyis never forgotten → everyAdd()either puts the key in the FIFO, or marks it soDone()re-queues it. No change is ever lost, no matter when it arrives.
And the subtle one you get for free: collapse. If a key changes 50 times while you process it, you reprocess it once — not 50, not 0. For a level-based reconciler ("make reality match the spec", not "handle every event") that's exactly right: the intermediate states are irrelevant, only the latest matters.
From the actual slow-motion run — watch default/obj-0 (worker holds it
for 300ms while the generator keeps firing):
[ENQUEUE] key=default/obj-0 (generator)
[REQUEUE-WHILE-PROCESSING] key=default/obj-0 a worker holds this key — ... redelivered exactly once when Done() runs
[ENQUEUE] key=default/obj-0 (generator)
[DEDUP] key=default/obj-0 already marked for redelivery — Add is a no-op ← 2nd+ Adds: swallowed
[ENQUEUE] key=default/obj-0 (generator)
[DEDUP] key=default/obj-0 already marked for redelivery — Add is a no-op
[WORKER-1] key=default/obj-0 Done seq=68 result=OK → Forget
[REDELIVERY] key=default/obj-0 was re-added while processing → Done() re-queued it (once)
[WORKER-1] key=default/obj-0 Get seq=69 ← exactly one redelivery
N Adds during processing → one redelivery. That log trio
(REQUEUE-WHILE-PROCESSING → DEDUP×N → REDELIVERY with seq+1) is the
whole proof.
Default run — 50,000 enqueues/sec onto 20 keys:
[STATS] enqueued=499,506 adds-total=499,705 dedup-noop=490,493 rwp-marked=5,328
processed=9,194 rate-limited=913 | adds-absorbed=99.2%
Half a million Adds in 10 seconds; 99.2% collapsed inside the queue; ~920 actual processings/sec. This is why controllers can watch high-churn resources without melting: the queue absorbs the event rate and hands workers a calm, deduplicated stream of keys.
workqueue.DefaultControllerRateLimiter() is two limiters, and the
effective delay is the max of them:
flowchart TD
F["worker fails key K"] --> W["When(K)"]
W --> A["per-key exponential<br/>ItemExponentialFailureRateLimiter<br/>base 5ms, cap 1000s"]
W --> B["global token bucket<br/>BucketRateLimiter<br/>10 qps, burst 100, ALL keys share"]
A --> M["delay = max(A, B)"]
B --> M
M --> T["time.AfterFunc(delay) → Add(K)"]
Per-key exponential — each key has its OWN failure counter:
| consecutive failures of key K | backoff |
|---|---|
| 1 | 5ms |
| 2 | 10ms |
| 3 | 20ms |
| 4 | 40ms |
| … | doubles |
| ≥ 18 | capped at 1000s (16m40s) |
Real log excerpt (obj-1 failing three times in a row, then succeeding):
[RATELIMIT] key=default/obj-1 attempt=1 backoff=5ms
[RATELIMIT] key=default/obj-1 attempt=2 backoff=10ms
[RATELIMIT] key=default/obj-1 attempt=3 backoff=20ms
[WORKER-2] key=default/obj-1 Done seq=70 result=OK → Forget (backoff reset) ← counter wiped
Forget(K) on success deletes the counter — the next failure starts back
at 5ms. The textbook worker pattern is exactly what this lab's workers do:
if err != nil { queue.AddRateLimited(key) } else { queue.Forget(key) }
queue.Done(key)The token bucket exists for thundering herds: even if 1,000 distinct keys fail at once, re-adds are smoothed to ~10/sec overall.
flowchart LR
K6["k6 / curl<br/>concurrency"] -- "POST /trigger/obj-7" --> H["HTTP handler<br/>(this program)"]
H -- "Update ConfigMap<br/>~2ms" --> API["kube-apiserver<br/>(envtest)"]
API -- "write" --> E[("etcd")]
E -- "Cacher fan-out<br/>~60µs" --> I["SharedInformer<br/>handlers"]
I -- "queue.Add(default/obj-7)" --> Q["workqueue<br/>dedup + sets + backoff"]
Q -- "Get / Done" --> W["worker pool<br/>simulated reconcile"]
Measured on this machine (203 requests, 10-way concurrent):
[TRIGGER] POST /trigger/obj-7 action=created rv=197 took=2.209ms
[ENQUEUE] key=default/obj-7 (informer: ADDED) ← 63µs after the write
[WORKER-3] key=default/obj-7 Get seq=1 ← 24µs after that
[STATS] ... | /trigger reqs=203 p50=1.94ms p95=2.53ms p99=3.10ms
The server half of this path (write → etcd → Cacher → watch stream) is dissected frame-by-frame in the sibling project informer-lab.
Look at where the milliseconds live in mode 2:
| stage | cost | why |
|---|---|---|
apiserver write (/trigger p50) |
~2ms | network + authn/authz + admission + etcd txn |
| informer → enqueue | ~60µs | Cacher fan-out + JSON decode |
| queue Add/Get/Done | ~1µs | two map writes + a slice append, in memory |
| worker reconcile (simulated) | 5ms | whatever YOUR code does |
The queue is four orders of magnitude cheaper than everything around it.
Its only failure mode is depth: if workers are slower than the
deduplicated arrival rate, queue.Len grows and memory follows. Watch
queue.Len in the [STATS] line — in a healthy run it hovers near the
number of distinct hot keys, not the event count.
Corollary worth internalizing: tuning worker count or queue internals
almost never speeds up a controller. The latency lives in the apiserver
write path and in your reconcile logic — which is exactly what mode 2's
/trigger percentiles put numbers on.
| term | meaning |
|---|---|
| key | a string naming work to do — by convention namespace/name from cache.MetaNamespaceKeyFunc. |
| dirty set | keys that need (re)processing. Add() of an already-dirty key is a no-op → dedup. |
| processing set | keys currently owned by a worker. Guarantees one-worker-per-key. |
| dedup | N Adds for a waiting key collapse into the one copy already queued. |
| requeue-while-processing | an Add for a key a worker holds: marked dirty, redelivered once at Done(). |
| AddRateLimited | schedule a re-add after a per-key backoff (When computes it and bumps the failure counter). |
| Forget | reset a key's backoff counter — call it on success. |
| NumRequeues | how many consecutive failures a key has had (drives the exponent). |
| ShutDown | stops the queue; Get() drains what's left, then returns shutdown=true. |
| level-based | reconcile philosophy the queue assumes: only the LATEST state matters, so collapsing intermediate events is safe. |