|
| 1 | +# E4 — two-real-pods replica-correctness topology proof (kind) |
| 2 | + |
| 3 | +> Executed: `2026-07-11` on `deproject-mac` (kind + Docker on Colima vz, 6 GiB / 4 CPU, |
| 4 | +> macOS 13.7.8 Intel). Code: `main` @ `9935bdc`. |
| 5 | +> Closes the cutover-plan Phase 3 item that was **attempted-but-blocked on 2026-07-06** |
| 6 | +> (shared-host resource crisis — kube control-plane crash-loop, build stall). This run |
| 7 | +> freed the VM first and completed the proof. |
| 8 | +
|
| 9 | +## Result |
| 10 | + |
| 11 | +`scripts/k8s_replica_correctness_verify.sh` → **exit 0, both checks PASS**: |
| 12 | + |
| 13 | +- **[Check 1] ≥2 ready pods on the postgres store** — `deployment/agentflow` 2/2 Ready, |
| 14 | + every pod carries `AGENTFLOW_CONTROLPLANE_STORE=postgres`. |
| 15 | +- **[Check 2] cross-pod registration visibility** — one webhook registered through the |
| 16 | + Service (`webhook_id=4a4709a0-0bdc-42bc-803a-2d49c1fb8f04`) was visible on **all 8 |
| 17 | + round-robin reads across the two pods**. On the embedded per-pod store a read served by |
| 18 | + the pod that did not register would miss it (the sharpest split-brain, ADR 0010 class 5); |
| 19 | + on PostgreSQL the single registration table makes every read see it. |
| 20 | + |
| 21 | +Checks 2–3 in the recipe (exactly-one delivery per (webhook,event); one alert page per |
| 22 | +incident) need an event/alert emitter + capture sink and are not part of this automated |
| 23 | +script. Their **store-level guarantee** (idempotent enqueue insert-win, single-flight |
| 24 | +`claim_alert_tick`, outbox↔dead-letter atomicity) is already live-verified 31/31 by the |
| 25 | +slice-5 standalone-PostgreSQL probe suite (`docs/perf/control-plane-pg-verify-2026-07-03.md`). |
| 26 | +Phase 3 adds only the two-real-pods topology layer on top — done here. |
| 27 | + |
| 28 | +## Topology stood up |
| 29 | + |
| 30 | +- kind cluster `agentflow-staging` (`kindest/node:v1.31.0`), `k8s/kind-config.yaml` |
| 31 | + (NodePort 30080 → host 8080). |
| 32 | +- Chart: `helm/agentflow` with `-f values-staging.yaml -f values-staging-scale.yaml.example` |
| 33 | + `--set serving.clickhouse.user=agentflow` → |
| 34 | + `replicaCount=2`, `persistence.enabled=false` (stateless request path), |
| 35 | + `serving.backend=clickhouse`, `controlPlane.store=postgres`. |
| 36 | +- In-cluster deps (the chart ships neither PG nor CH): single-replica **PostgreSQL 16** |
| 37 | + (`agentflow-postgres`, control-plane store), **ClickHouse 24.8** (`agentflow-clickhouse`, |
| 38 | + serving backend), **Redis 7.4** (`agentflow-redis`, rate limit) + the two operator secrets |
| 39 | + `agentflow-controlplane-pg` (keyword-form DSN) and `agentflow-clickhouse`. |
| 40 | +- Pods reach Ready without Kafka/Flink: `/v1/health` returns 200 with degraded components |
| 41 | + (each `HealthCollector` check is defensive), so readiness is process-up, not dependency-up. |
| 42 | + |
| 43 | +## Environment gotchas found this run (Mac stand) |
| 44 | + |
| 45 | +- The docker data disk (`/dev/vdb1`, ~59 G) is **separate from the VM root** (`/`, ~19 G): |
| 46 | + the first build died `[Errno 28] No space left on device` at the *pip install* step with |
| 47 | + the docker disk at 96 %, while `df /` still showed 18 G free. Freed it by removing the |
| 48 | + orphaned old DE-compose **named** volumes (`agentflow-docker-check_{clickhouse,kafka,minio}-data`, |
| 49 | + `…_flink-checkpoints`) — never `docker volume prune`/`system prune`, which would also |
| 50 | + delete co-tenant data (`rag_support_assistant_pgdata`, anonymous datalens/auto_bi volumes) |
| 51 | + and the stopped co-tenant containers themselves. |
| 52 | +- The rendered Service takes an auto-assigned nodePort; patch it to **30080** so kind's |
| 53 | + fixed host mapping (`30080 → 127.0.0.1:8080`) reaches it, then `BASE_URL=http://127.0.0.1:8080`. |
| 54 | +- Right after the nodePort patch, `curl` can time out for a few seconds until kube-proxy |
| 55 | + rewires the mapping — the verify script (run a moment later) succeeds. |
| 56 | +- kind + a 2-replica agentflow + PG + CH + Redis fits in 6 GiB **only with the co-tenant |
| 57 | + containers stopped** (DataLens ×7 + auto_bi). The 2026-07-06 failure was exactly this |
| 58 | + contention; free the VM first (stop co-tenants, prune orphaned DE volumes/images), and |
| 59 | + return the co-tenants on teardown. |
| 60 | + |
| 61 | +## Teardown |
| 62 | + |
| 63 | +`kind delete cluster --name agentflow-staging`; co-tenant containers restarted from the |
| 64 | +snapshot (`/tmp/agentflow_foreign_stopped.txt`). VM returned to ~5.3 GiB available. |
0 commit comments