informer-lab

Watch client-go's SharedInformer talk to a real kube-apiserver watch cache (the Cacher) — no cluster, no containers, no YAML.

• envtest boots a real etcd + kube-apiserver as local processes
• two independent SharedInformerFactory instances (A and B) watch core/v1 ConfigMaps
• a mutator CREATEs / UPDATEs / DELETEs one ConfigMap every 3 seconds
• a logging http.RoundTripper prints every request and every streamed watch event
• you observe one etcd write fan out to two watch streams in real time

Everything lives in main.go — read it top to bottom, the header comment is the map.

📖 New to informers / the watch cache? Read HOW-IT-WORKS.md first — the whole mechanism as diagrams, with real log lines mapped onto them.

Prerequisites

• Go ≥ 1.23 (developed on go1.26)
• macOS or Linux (arm64/amd64)
• No docker, no kind, no cluster, no root

One-time setup: envtest binaries

envtest needs real etcd + kube-apiserver binaries. The setup-envtest tool fetches them:

# 1. install the tool (lands in $(go env GOPATH)/bin)
go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest

# if needed, put GOPATH/bin on PATH (zsh):
export PATH="$(go env GOPATH)/bin:$PATH"

# 2. download the control-plane binaries (one time, ~100 MB)
setup-envtest use 1.31.x -p path

The last command prints the store path, e.g. ~/Library/Application Support/io.kubebuilder.envtest/k8s/1.31.4-darwin-arm64.

Then either:

export KUBEBUILDER_ASSETS="$(setup-envtest use 1.31.x -p path)"

or do nothing — main.go auto-detects setup-envtest's default store.

Run

go run .
# or: go build -o informer-lab . && ./informer-lab

Ctrl-C to stop. The apiserver and etcd are killed on exit; nothing persists, nothing needs cleanup.

What to look for

1. Startup: the list-then-watch handoff (×2, once per watcher)

[HTTP][A] GET https://127.0.0.1:PORT/api/v1/configmaps?limit=500&resourceVersion=0
[LIST][A] initial LIST returned collection resourceVersion=413 — watch will resume from exactly here
[HTTP][A] GET https://127.0.0.1:PORT/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=413&timeoutSeconds=358&watch=true
[HTTP][A] watch established (200 OK) — events stream below as [WATCH-EVENT][A]

• resourceVersion=0 on the LIST means "give me current state". The returned collection rv is the high-water mark; the WATCH resumes from exactly that rv — no gaps, no duplicates.
• Bonus detail: resourceVersion=0 requests are served from the Cacher's memory, not a quorum read of etcd. The cache is on the read path too.

2. Steady state: one write → two streams — the Cacher fan-out

[MUTATOR] writing: UPDATE lab-cm-1 (data.tick=8)
[HTTP][MUT] PATCH https://127.0.0.1:PORT/api/v1/namespaces/default/configmaps/lab-cm-1
[WATCH-EVENT][A] type=MODIFIED name=lab-cm-1 rv=521     <-- 1-2 ms apart:
[WATCH-EVENT][B] type=MODIFIED name=lab-cm-1 rv=521         SAME rv, two streams
[HANDLER][A] informer handler fired: event=MODIFIED name=lab-cm-1 rv=521
[HANDLER][B] informer handler fired: event=MODIFIED name=lab-cm-1 rv=521

One etcd write → the Cacher's internal reflector ingests it → fan-out to cacheWatcher A and cacheWatcher B → two chunked HTTP streams → two client-go Reflectors → two DeltaFIFOs → two handler calls.

The identical rv on both streams is the proof: both events carry the same etcd revision because both watches are projections of the same cached store — the apiserver did not go back to etcd twice for this.

Notice both watchers get the event within milliseconds of each other — that's the Cacher fan-out.

3. The layered view, per event

For every mutation you'll see the three layers in order:

prefix	layer	what it proves
[MUTATOR] / [HTTP][MUT]	the write	one request, one etcd revision
[WATCH-EVENT][A/B]	wire (chunked HTTP)	the Cacher pushed to both watchers
[HANDLER][A/B]	client-go internals	Reflector → DeltaFIFO → local store → your code

4. Background behavior worth noticing

• type=BOOKMARK — periodic progress pings; the reflector asked for them (allowWatchBookmarks=true) so it can advance its rv cheaply.
• Watch re-establishment — the server closes each watch after timeoutSeconds (random 5–10 min); the reflector silently re-lists and re-watches. If you let it run that long you'll see the handoff repeat.
• [HTTP][MUT] POST /api/v1/namespaces — startup; the bare apiserver has no namespace controller, so the lab creates default itself.

Log prefix cheat sheet

prefix	meaning
[SETUP]	lab lifecycle (envtest, sync, shutdown)
[HTTP][tag]	every outgoing request — transport level, uninterpreted
[LIST][tag]	reflector's initial LIST + returned resourceVersion
[WATCH-EVENT][tag]	one decoded chunk from a watch stream
[MUTATOR]	the single writer goroutine
[HANDLER][tag]	informer handler fired (after DeltaFIFO + local store)

Tags: A / B = the two watchers, MUT = the mutator.

Why client-go v0.31?

From v0.32 on, client-go reflectors may do their initial sync with a "streaming list" (watch=true&sendInitialEvents=true) instead of a plain LIST. Same semantics, different wire shape. v0.31 guarantees the textbook LIST → WATCH handoff so requirement "see the handoff" is literal. Everything else about the Cacher path is identical across versions.