Skip to content

Commit ea66fbb

Browse files
Merge pull request #174 from brownjuly2003-code/s6-bridge-consumer
S6: bridge events.validated into the serving store
2 parents 8fab113 + 087da21 commit ea66fbb

33 files changed

Lines changed: 2519 additions & 117 deletions

‎.github/workflows/ci.yml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,10 +89,20 @@ jobs:
8989
- name: Run rate limiter coverage gate
9090
run: |
9191
# Security-critical sliding-window rate limiter (Redis + in-memory
92-
# fail-open fallback); local module coverage is 98% (only the optional
92+
# fail-closed fallback); local module coverage is 98% (only the optional
9393
# redis auto-construct line is env-gated), so the 90% gate keeps a
9494
# cushion on a mutmut target.
95-
python -m pytest tests/unit/test_rate_limiter.py -v --tb=short --cov=src.serving.api.rate_limiter --cov-report=term-missing --cov-fail-under=90
95+
#
96+
# Uses `coverage run` + `coverage report --include` (auth/outbox pattern).
97+
# Scope is pure RateLimiter tests only (`-k "not auth_middleware"`): the
98+
# three middleware+TestClient cases still run in the main unit suite,
99+
# but under coverage they tear down redis.asyncio + Starlette and have
100+
# SIGSEGV'd (exit 139) on ubuntu-latest after a green 12/12 report
101+
# (PR #174). Pure module coverage stays ≥90% without those three.
102+
# Also disable the pytest-cov plugin so it cannot double-instrument.
103+
python -m coverage run -m pytest tests/unit/test_rate_limiter.py \
104+
-k "not auth_middleware" -v --tb=short -p no:schemathesis -p no:cov
105+
python -m coverage report --include="*/serving/api/rate_limiter.py" --show-missing --fail-under=90
96106
- name: Run auth manager coverage gate
97107
run: |
98108
# Security-critical auth manager (key match/verify, tenant isolation,

‎docker-compose.yml‎

Lines changed: 38 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -36,13 +36,24 @@ services:
3636
entrypoint: ["/bin/bash", "-c"]
3737
command:
3838
- |
39+
set -e
3940
echo "Creating Kafka topics..."
4041
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic orders.raw --partitions 6 --replication-factor 1
4142
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic payments.raw --partitions 6 --replication-factor 1
4243
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic clicks.raw --partitions 6 --replication-factor 1
4344
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic products.cdc --partitions 3 --replication-factor 1
4445
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic events.validated --partitions 6 --replication-factor 1
4546
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic events.deadletter --partitions 3 --replication-factor 1
47+
# The stream processor's KafkaSource subscribes to the CDC topics too
48+
# (src/processing/flink_jobs/stream_processor.py). Kafka refuses to hand
49+
# out metadata for a topic that does not exist, so the job dies in a
50+
# restart loop -- "Failed to get metadata for topics [...]" -- before it
51+
# ever reads orders.raw. Debezium creates these in a full deployment;
52+
# locally they stay empty, and empty is fine. Absent is not.
53+
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic cdc.postgres.public.orders_v2 --partitions 1 --replication-factor 1
54+
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic cdc.postgres.public.users_enriched --partitions 1 --replication-factor 1
55+
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic cdc.mysql.agentflow_demo.products_current --partitions 1 --replication-factor 1
56+
kafka-topics --bootstrap-server kafka:9092 --create --if-not-exists --topic cdc.mysql.agentflow_demo.sessions_aggregated --partitions 1 --replication-factor 1
4657
echo "Topics created."
4758
4859
# ── Flink ─────────────────────────────────────────────────────
@@ -117,10 +128,24 @@ services:
117128
entrypoint: ["/bin/bash", "-c"]
118129
command:
119130
- |
120-
mc alias set local http://minio:9000 minio minio123
121-
mc mb local/agentflow-lake --ignore-existing
122-
mc mb local/agentflow-lake/warehouse --ignore-existing
123-
mc mb local/agentflow-lake/checkpoints --ignore-existing
131+
set -e
132+
# `local` is one of mc's BUILT-IN aliases (http://localhost:9000). When
133+
# `alias set local` failed, every `mc mb local/...` below silently went
134+
# to the mc container's own loopback, failed too, and the script still
135+
# printed success -- leaving Flink to block forever on a checkpoint
136+
# bucket that did not exist. Use a name mc does not ship, and let a
137+
# failure fail the container.
138+
for attempt in $$(seq 1 30); do
139+
mc alias set lake http://minio:9000 minio minio123 && break
140+
echo "waiting for minio (attempt $$attempt)"
141+
sleep 2
142+
done
143+
# `-p` IS `--ignore-existing` in mc (one flag, two spellings); passing
144+
# both is a hard error.
145+
mc mb -p lake/agentflow-lake
146+
mc mb -p lake/agentflow-lake/warehouse
147+
mc mb -p lake/agentflow-lake/checkpoints
148+
mc ls lake/agentflow-lake
124149
echo "MinIO buckets created."
125150
126151
redis:
@@ -138,7 +163,15 @@ services:
138163
image: clickhouse/clickhouse-server:24.8
139164
ports:
140165
- "127.0.0.1:8123:8123"
141-
- "127.0.0.1:9000:9000"
166+
# ClickHouse's native protocol, published on 9010 rather than its default
167+
# 9000: MinIO already owns host 9000 in this file, and config/iceberg.yaml
168+
# points the S3 endpoint at localhost:9000. Publishing both collided, so
169+
# the Flink/MinIO lake path and the ClickHouse serving backend could not
170+
# be brought up together — which is exactly the topology the serving
171+
# bridge needs (Flink -> events.validated -> bridge -> ClickHouse).
172+
# Nothing in the project speaks the native protocol; every ClickHouse
173+
# client here uses HTTP on 8123 (config/serving.yaml, CLICKHOUSE_PORT).
174+
- "127.0.0.1:9010:9000"
142175
environment:
143176
CLICKHOUSE_DB: agentflow
144177
CLICKHOUSE_USER: ${CLICKHOUSE_USER:-default}

‎docs/architecture.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,8 @@ Consumers are anything that needs the current number at the moment of decision:
4444
2. **Processing**: Flink validates (schema + semantic), enriches, deduplicates, and routes events
4545
3. **Storage**: Valid events land in Iceberg tables; production uses AWS Glue as the catalog over object storage
4646
4. **Quality**: Pre-storage gates check schema + semantic rules. Failures → dead letter topic
47-
5. **Serving**: Agent API reads from the configured serving backend; the checked-in paths are DuckDB by default and optional ClickHouse for higher read concurrency, while Iceberg remains the lakehouse storage layer
47+
5. **Bridge**: A consumer of `events.validated` applies each event to the serving store, idempotently by `event_id` — this is what makes the serving store move when a real Kafka event happens. After a successful apply the bridge also **pushes** metric-cache invalidation (Redis channel + in-process callback); a journal-scan fallback covers writers that do not push. See [Serving Bridge](serving-bridge.md#cache-invalidation-s7).
48+
6. **Serving**: Agent API reads from the configured serving backend; the checked-in paths are DuckDB by default and optional ClickHouse for higher read concurrency, while Iceberg remains the lakehouse storage layer
4849

4950
For CDC sources, Debezium/Kafka Connect handles source capture while a shared normalizer converts Postgres/MySQL envelopes into one canonical AgentFlow CDC contract before validation. See [ADR 0005](decisions/0005-cdc-ingestion-strategy.md).
5051

‎docs/freshness-benchmark.md‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,15 @@
55
66
Generated: `2026-06-06T10:10:41+03:00`
77

8+
> **S7 note (2026-07-09, not yet re-measured):** cache invalidation is no
9+
> longer hostage to the webhook dispatcher poll. Production wiring is push
10+
> from the serving bridge (Redis channel `agentflow:cache:metrics_invalidate`)
11+
> plus an independent journal scan in `MetricCacheController` — see
12+
> [`serving-bridge.md`](serving-bridge.md#cache-invalidation-s7). The numbers
13+
> below still describe the pre-S7 poll path. **S8** will re-run this benchmark
14+
> end-to-end on the real Kafka→Flink→bridge path (and must use the N2-aware
15+
> ClickHouse timestamp convention).
16+
817
## What is measured
918

1019
The end-to-end delay between an event entering the pipeline (schema
@@ -17,9 +26,8 @@ Metric reads are authenticated (tenant-scoped cache path — unauthenticated
1726
requests bypass the metric cache when multi-tenancy is configured) and
1827
served through the production `QueryCache` (Redis semantics via in-process
1928
`fakeredis`; localhost RTT to a real Redis is sub-millisecond and
20-
excluded). Cache invalidation is driven by the webhook dispatcher poll
21-
loop seeing new pipeline events — the same wiring as
22-
`src/serving/api/main.py` production defaults.
29+
excluded). At measurement time, cache invalidation was driven by the webhook
30+
dispatcher poll loop seeing new pipeline events.
2331

2432
## System Under Test
2533

@@ -39,7 +47,7 @@ loop seeing new pipeline events — the same wiring as
3947

4048
## Reading the numbers
4149

42-
- `event_driven` is the production configuration: staleness is bounded by the 2 s dispatcher poll, not the 30 s cache TTL.
50+
- `event_driven` was the production configuration at measurement time: staleness was bounded by the 2 s dispatcher poll, not the 30 s cache TTL. Post-S7 the production path is push-driven (see note above); these numbers are the pre-S7 baseline.
4351
- A TTL-only cache at the production TTL (30 s) would sit at ~U(0, TTL) staleness — p50 ≈ 15 s (linear scaling of the measured ttl_only arm: 2.75 s at TTL 5 s ⇒ ≈ 16.51 s at TTL 30 s). Event-driven invalidation measured 1.06 s p50 / 1.99 s p95 on the same pipeline.
4452
- `fast_poll` shows the same wiring with the dispatcher poll tuned to 0.25 s — freshness is a configuration knob, not an architectural ceiling.
4553
- `no_cache` is the floor: with the cache off every read hits the store directly, so the measured delay collapses to this script's 25 ms read granularity plus one query.

‎docs/perf/freshness-realpath-2026-06-30.md‎

Lines changed: 25 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@
66
77
Measured: `2026-06-30`
88

9+
> **Update 2026-07-09.** The missing link this document reports — "nothing
10+
> bridges `events.validated` into the serving store" — has since been built:
11+
> see [`../serving-bridge.md`](../serving-bridge.md). The numbers below are the
12+
> streaming-hop measurement as taken on 2026-06-30 and are left untouched; an
13+
> end-to-end *event → serving metric* measurement on the real path is a separate
14+
> run and is not reported here.
15+
916
## Why this exists
1017

1118
`docs/freshness-benchmark.md` measures freshness through the **in-process
@@ -69,12 +76,14 @@ tighten the tail percentiles; the p50 is stable.
6976
- The two columns measure **complementary segments, not the same thing.** The
7077
DuckDB shortcut measures *event → serving metric* (and is dominated by the 2 s
7178
cache-invalidation poll); this benchmark measures *Kafka produce → validated
72-
event on Kafka* through the real Flink operators. The serving/DuckDB metric
73-
store is **not wired to the real streaming path** (Flink sinks to
74-
`events.validated`; nothing bridges that topic into the serving DuckDB — that
75-
is a deliberate demo shortcut, production target is a ClickHouse sink). So
76-
there is no honest single "event → metric on the real path" number on this
77-
stand; this is the real **streaming-hop** number.
79+
event on Kafka* through the real Flink operators. **At the time of this
80+
measurement** the serving store was not wired to the real streaming path
81+
(Flink sank to `events.validated`; nothing bridged that topic into serving).
82+
So there was no honest single "event → metric on the real path" number on this
83+
stand; this is the real **streaming-hop** number. The bridge that closes the
84+
remaining segment landed on 2026-07-09
85+
([`../serving-bridge.md`](../serving-bridge.md)); measuring the full path
86+
end-to-end is a separate run.
7887
- The real path's ~2.5 s p50 is the same order of magnitude as the headline
7988
freshness claim. The extra ~1.4 s over the shortcut is the real cost of the
8089
streaming hop: Kafka produce + source poll + the **Beam Python portability
@@ -98,13 +107,16 @@ surfaced three latent pyflink-2.x / Docker issues:
98107
→ `AttributeError` crashed the Python worker on every event. Fix:
99108
`Time.minutes(10)` (same class of bug as the already-fixed watermark
100109
`timedelta`→`Duration`).
101-
3. **`ctx.output` side output (found, not yet fixed).** `ValidateAndEnrich`
102-
routes invalid events with `ctx.output(DEAD_LETTER_TAG, …)` — the Java side-
103-
output API; in pyflink the `ProcessFunction` context has no `.output()`
104-
(`AttributeError: 'InternalProcessFunctionContext' object has no attribute
105-
'output'`). **Valid** events are unaffected (this benchmark measures valid
106-
events), but the dead-letter path is currently broken. Left as a follow-up so
107-
the streaming-freshness measurement isn't blocked on it.
110+
3. **`ctx.output` side output (fixed since; re-verified 2026-07-09).**
111+
`ValidateAndEnrich` used to route invalid events with
112+
`ctx.output(DEAD_LETTER_TAG, …)` — the Java side-output API; in pyflink the
113+
`ProcessFunction` context has no `.output()`. It now `yield`s
114+
`(DEAD_LETTER_TAG, payload)` pairs, which is the pyflink idiom, and the
115+
dead-letter path works on a live cluster: a schema-invalid `order.created`
116+
produced to `orders.raw` lands on `events.deadletter` (payload
117+
`{event_id, error, stage: "schema_validation"}`) and never reaches
118+
`events.validated`. Verified on the Mac stand while gating the serving
119+
bridge.
108120

109121
## Reproduce
110122

0 commit comments

Comments
 (0)