|
| 1 | +# S10 paced produce @ 100 eps — 1-hour arm |
| 2 | + |
| 3 | +> Measured: `2026-07-16T23:30Z` → `2026-07-17T00:30Z` on `deproject-mac` |
| 4 | +> (Colima vz 6 GiB / 4 CPU). Code: `main` @ `b5d9ce0`. |
| 5 | +> Path: Kafka → Flink stream_processor (**1 TM**) → `events.validated` → |
| 6 | +> host `bridge_consumer` → ClickHouse. |
| 7 | +> JSON: `/tmp/s10_paced100_1h.json` on the stand. |
| 8 | +
|
| 9 | +## Result |
| 10 | + |
| 11 | +| Metric | Value | |
| 12 | +|--------|------:| |
| 13 | +| Produce | **100.0 eps** (360 000 events in 3600.0 s, paced) | |
| 14 | +| Flink hop | **99.5 eps** (360 000 validated) | |
| 15 | +| Bridge apply | **99.5 eps** (+360 000 applied over 3617 s wall incl. catch-up) | |
| 16 | +| Failures / duplicates | **0 / 0** | |
| 17 | +| Lag start → end | 0 → **0** | |
| 18 | +| Lag peak | **1679** (recovered) | |
| 19 | + |
| 20 | +**Gate (this arm):** produce ≥ 95, apply ≥ 90, flink ≥ 90, failures = 0, |
| 21 | +duplicates < 5 000, lag_end ≤ 100 → **PASS**. |
| 22 | + |
| 23 | +## Honest claim |
| 24 | + |
| 25 | +| Claim | Status | |
| 26 | +|-------|--------| |
| 27 | +| Drain-window ≥100 | Met (107.3 @ 2000) — separate doc | |
| 28 | +| Paced ~10 min @ 100 produce | Met (~96.5 apply) — 10 min doc | |
| 29 | +| **Paced 1 hour @ 100 produce with apply/hop ≈100, lag → 0** | **Met (this doc)** | |
| 30 | +| Multi-hour (e.g. 4 h+) sustained ≥100 | **Still open** | |
| 31 | + |
| 32 | +Do **not** call this “multi-hour soak.” It is a **single continuous hour** of |
| 33 | +paced ingress at the product target rate on the Mac compose stand. |
| 34 | + |
| 35 | +## Stand hygiene (required) |
| 36 | + |
| 37 | +Earlier 1h attempts failed for ops reasons, not apply-path capacity: |
| 38 | + |
| 39 | +1. **Stale ClickHouse journal + Flink earliest replay** → tens of thousands of |
| 40 | + journal-guard **duplicates**, `applied≈0` for the benchmark window. |
| 41 | + **Fix:** `docker compose … down -v` for *this* project only before the arm. |
| 42 | +2. **`docker compose up` stuck** waiting on a one-shot `kafka-init` that stayed |
| 43 | + Running after topics were created — unstick by removing the init container |
| 44 | + and bringing CH/redis separately. |
| 45 | + |
| 46 | +## Reproduce |
| 47 | + |
| 48 | +```bash |
| 49 | +export PATH=$HOME/bin:/usr/local/bin:$PATH |
| 50 | +# colima start --cpu 4 --memory 6 --vm-type vz |
| 51 | +cd ~/agentflow-docker-check |
| 52 | +docker compose -f docker-compose.yml -f docker-compose.flink.yml down -v |
| 53 | +docker compose -f docker-compose.yml -f docker-compose.flink.yml \ |
| 54 | + up -d --build --scale flink-taskmanager=1 flink-job-runner |
| 55 | +docker compose -f docker-compose.yml -f docker-compose.flink.yml up -d clickhouse redis |
| 56 | +# wait job RUNNING + CH ping; start bridge_consumer on :9108 |
| 57 | +.venv/bin/python scripts/benchmark_throughput_realpath.py \ |
| 58 | + --bootstrap 127.0.0.1:19092 \ |
| 59 | + --count 360000 --pace-eps 100 \ |
| 60 | + --bridge-metrics http://127.0.0.1:9108/metrics \ |
| 61 | + --latency-samples 0 --catchup-timeout-seconds 1200 --no-md |
| 62 | +``` |
| 63 | + |
| 64 | +## Status |
| 65 | + |
| 66 | +**1 h paced @ 100 eps: PASS** (produce 100.0 / flink 99.5 / apply 99.5, |
| 67 | +zero failures/duplicates, lag ends at 0). Multi-hour remains open. |
0 commit comments