Merge pull request #188 from brownjuly2003-code/flink-configurable-backend

feat(flink): configurable state.backend (default rocksdb) + changelog for #184-186

Author: brownjuly2003-code

CHANGELOG.md

@@ -4,6 +4,33 @@ All notable changes to AgentFlow are documented in this file.
 
 ## [Unreleased]
 
+### Added — Flink state backend is configurable (unblocks stand throughput runs)
+
+- The Flink `state.backend.type` in `docker-compose.yml` is now
+  `${FLINK_STATE_BACKEND:-rocksdb}` for both JobManager and TaskManager.
+  Production and CI keep the RocksDB default (incremental checkpoints, off-heap
+  state); a stand can export `FLINK_STATE_BACKEND=hashmap` to run the small
+  dedup/TTL state in heap. Offered as a candidate workaround for a
+  RocksDB-async-checkpoint native TaskManager crash observed on a constrained
+  macOS/Lima Intel VM during a sustained-throughput window (root cause
+  unconfirmed — may be the VM kernel, so this is a lever to try on the stand,
+  not a proven fix). The default is unchanged, so CI `flink-smoke` keeps
+  exercising RocksDB.
+
+### Fixed — webhook dedup and cache-scan window (audit #184-186)
+
+- **Webhook dispatcher** dropped a dead bare-`event_id` membership test — the
+  seen-set only ever holds `tenant:event_id`, so dedup is strictly per-tenant
+  (regression test added).
+- **Startup scan cursor** now seeds from the newest journal row whose
+  `processed_at` parses, not blindly the newest row: a malformed newest
+  timestamp left the cursor `None`, which made the first scan fetch from the
+  oldest row and re-deliver the seeded batch.
+- **Metric-cache scan window** raised 200 → 2000 rows
+  (`DEFAULT_SCAN_WINDOW_ROWS`). At the ≥100 eps target the pre-merge journal
+  emits ~2 rows/event, so 200 left no margin for a non-pushing writer; push
+  feeds still cover writers that publish.
+
 ### Fixed — SSE per-connection dedup cache is bounded (issue #183 follow-up)
 
 - **`/v1/stream/events` kept a bare per-connection seen-set** that grew one

docker-compose.yml

@@ -67,7 +67,7 @@ services:
         jobmanager.rpc.address: flink-jobmanager
         jobmanager.bind-host: 0.0.0.0
         rest.bind-host: 0.0.0.0
-        state.backend.type: rocksdb
+        state.backend.type: ${FLINK_STATE_BACKEND:-rocksdb}
         execution.checkpointing.dir: s3://agentflow-lake/checkpoints
         s3.endpoint: http://minio:9000
         s3.access-key: minio
@@ -94,7 +94,7 @@ services:
         jobmanager.rpc.address: flink-jobmanager
         taskmanager.bind-host: 0.0.0.0
         taskmanager.numberOfTaskSlots: 4
-        state.backend.type: rocksdb
+        state.backend.type: ${FLINK_STATE_BACKEND:-rocksdb}
         s3.endpoint: http://minio:9000
         s3.access-key: minio
         s3.secret-key: minio123