|
| 1 | +# ADR 0011: Operational serving split — transactional reads on the control-plane store, analytical reads on the serving backend |
| 2 | + |
| 3 | +## Status |
| 4 | + |
| 5 | +Accepted - 2026-07-03 |
| 6 | + |
| 7 | +Design decision for the operational layer (the three ops surfaces of |
| 8 | +[`domain.md`](../domain.md) §4: **Order 360 timeline**, **stuck-orders |
| 9 | +worklist**, **exception inbox**). The companion specification that pins |
| 10 | +endpoint contracts, the SLA stage model, and per-slice scope is |
| 11 | +[`docs/ops-surfaces-spec.md`](../ops-surfaces-spec.md); implementation is |
| 12 | +staged as slices D2–D4 (see § Rollout). |
| 13 | + |
| 14 | +## Context |
| 15 | + |
| 16 | +The operations team of the modeled business juggles five tools to answer one |
| 17 | +question about one order (`domain.md` §4). The serving layer exists to replace |
| 18 | +that with one API surface, and the three planned ops surfaces are its most |
| 19 | +read-shape-diverse consumers yet: |
| 20 | + |
| 21 | +- **Order 360** is a *point-read composition*: one order row, its event |
| 22 | + history, its customer, its failures — "everything about ORD-20260404-1001, |
| 23 | + now, in one place". |
| 24 | +- **Stuck-orders** is an *analytical scan*: every open order, joined to its |
| 25 | + stage-entry time, filtered against per-stage SLA budgets, ranked by |
| 26 | + overshoot. |
| 27 | +- **Exception inbox** is a *transactional worklist*: failed events and |
| 28 | + consistency findings with a triage lifecycle (acknowledge / resolve / |
| 29 | + replay / dismiss) and a counter of absorbed manual work. |
| 30 | + |
| 31 | +The platform has exactly two data planes, with different semantics, both |
| 32 | +already behind ports: |
| 33 | + |
| 34 | +| Plane | Port | Engines | Semantics | |
| 35 | +|---|---|---|---| |
| 36 | +| Serving / analytics | `ServingBackend` via `QueryEngine` (ADR 0006) | DuckDB (demo) / ClickHouse (scale) | Analytical reads: entity point-reads, metric aggregates, the `pipeline_events` journal scan (`QueryEngine.fetch_pipeline_events` — the freshness axis) | |
| 37 | +| Control plane / transactional | `ControlPlaneStore` (ADR 0010) | embedded DuckDB+YAML (demo) / PostgreSQL (scale) | Mutable row state with claim semantics: webhook queue, alert state, outbox, **dead-letter lifecycle** (`failed → replay_pending → replayed` / `dismissed`), usage | |
| 38 | + |
| 39 | +### Verified inventory — where each surface's data lives today (`main=c2749dd`) |
| 40 | + |
| 41 | +| Data | Home | Access path | Notes | |
| 42 | +|---|---|---|---| |
| 43 | +| Order row, open-order set | `orders_v2` on the serving backend | `QueryEngine.get_entity` / catalog SQL | CH table is `ReplacingMergeTree ORDER BY order_id` — key point-reads are cheap on both engines | |
| 44 | +| Per-order event history | `pipeline_events` journal | `QueryEngine.fetch_pipeline_events(entity_id=…)` | The journal has an `entity_id` axis, **but only the demo seed populates it** — the live pipeline writes journal rows without `entity_id` (`local_pipeline.py` insert sites, `clickhouse_sink.record_pipeline_event` has no such parameter). On CH the journal is `ORDER BY (tenant_id, topic, processed_at, event_id)`, so an `entity_id` lookup is scan-shaped — acceptable for one bounded order trail, wrong as a triage backbone | |
| 45 | +| Customer golden record | `bv_customer_mdm` in the DV2 vault | engine-side governed views (ADR 0006 Phase 2) | PII-full, behind per-jurisdiction roles designed for warehouse principals. The serving store carries the PII-free projection `users_enriched` (the serving tier holds no PII by construction — 2026-07-01 decision) | |
| 46 | +| Failed events + replay lifecycle | `dead_letter_events` + `outbox` | `ControlPlaneStore` | Tenant-scoped reads, `replay`/`dismiss` actions already live on `/v1/deadletter`; invariant 8 (outbox↔dead-letter transactional flip) preserved by the port | |
| 47 | +| Webhook deliveries parked `dead` | webhook queue | `ControlPlaneStore` | Visible today only via per-webhook logs; no unified triage view | |
| 48 | +| SLA budgets | nowhere yet | — | `contracts/entities/order.yaml` documents the status vocabulary but carries no stage budgets | |
| 49 | +| OLTP hot tier (warehouse contour) | `ops_<branch>` PostgreSQL schemas (`pg_ops__*`) | DV2 promotion + LISTEN/NOTIFY | The warehouse-side archetype of the same split: "Postgres is a buffer, not a system of record; cross-branch joins live in CH, never in Postgres". Not wired to the serving API | |
| 50 | + |
| 51 | +### Constraints |
| 52 | + |
| 53 | +1. **The demo profile stays zero-dependency.** Both ports have embedded |
| 54 | + adapters; the ops surfaces must run on them unchanged. |
| 55 | +2. **The serving tier stays PII-free by construction.** Whatever the timeline |
| 56 | + shows about a customer must come from the PII-free projection, not the |
| 57 | + vault. |
| 58 | +3. **Freshness is the point.** These surfaces are the business reading of the |
| 59 | + event→metric axis (`domain.md` §4.1, oversell): a TTL-cached or |
| 60 | + batch-materialized ops read would re-create the very lag the five programs |
| 61 | + are being killed for. |
| 62 | +4. **No third data path.** The 2026-07-02 architecture audit (§3.1, §3.6) |
| 63 | + spent an entire phase (E) evicting direct `query_engine._conn` reads from |
| 64 | + the control plane; the ops layer must not grow the seam back. |
| 65 | + |
| 66 | +## Options considered |
| 67 | + |
| 68 | +### 1. Serve everything from the analytics backend (ClickHouse at scale) |
| 69 | + |
| 70 | +Rejected. Triage is mutable row state with lifecycle transitions and actions — |
| 71 | +this re-litigates ADR 0010 option 1 (asynchronous mutations, no row locks, |
| 72 | +no transactional flip). Worse, the dead-letter lifecycle *already* lives |
| 73 | +behind the control-plane port; reading a CH copy of it would fork the state |
| 74 | +the replay/dismiss actions mutate. |
| 75 | + |
| 76 | +### 2. Serve everything from the transactional store (PostgreSQL at scale) |
| 77 | + |
| 78 | +Rejected. The worklist and aging aggregates are analytical scans over serving |
| 79 | +data; serving them from PG means copying `orders_v2` and the journal into the |
| 80 | +control-plane store — a second serving engine, against ADR 0006. The |
| 81 | +warehouse's own hot tier is explicitly a rolling 30-day buffer, not history. |
| 82 | + |
| 83 | +### 3. Read the DV2 vault directly for the customer block |
| 84 | + |
| 85 | +Rejected for v1. Vault PII sits behind per-jurisdiction engine-side roles |
| 86 | +(officers, analysts) that were deliberately designed for warehouse |
| 87 | +principals, not for an always-on API service account; and the timeline's |
| 88 | +customer block needs no PII — `users_enriched` (orders, spend, recency, |
| 89 | +preferred category) is exactly the PII-free projection of the golden record. |
| 90 | +A dedicated non-PII vault role for a richer MDM block is recorded as roadmap, |
| 91 | +not v1. |
| 92 | + |
| 93 | +### 4. Precompute ops marts (materialized stuck-list / inbox tables) |
| 94 | + |
| 95 | +Rejected for v1. The open-order set is small and bounded; computing on read |
| 96 | +keeps the surfaces at serving-store freshness (constraint 3) and adds no new |
| 97 | +moving parts. Materialization remains a later optimization that must not |
| 98 | +change the API contract. |
| 99 | + |
| 100 | +### 5. Two-port composition rule — chosen |
| 101 | + |
| 102 | +Formalize the split the platform already converged on (ADR 0010: "ClickHouse |
| 103 | += analytics/serving, PostgreSQL = transactional state, Redis = cache/limits") |
| 104 | +as a *rule for the ops layer*, executable today on the embedded profiles. |
| 105 | + |
| 106 | +## Decision |
| 107 | + |
| 108 | +1. **Every ops surface composes exactly the two existing ports.** Analytical |
| 109 | + reads — entity point-reads, journal scans and history, open-order |
| 110 | + aggregates — go through `QueryEngine` / `ServingBackend`. Transactional |
| 111 | + state — triage lifecycles, actions, the manual-work counter — goes through |
| 112 | + `ControlPlaneStore`. No ops code path touches `query_engine._conn`, a raw |
| 113 | + engine connection, or a vault DSN (spec invariant I1, pinned by a |
| 114 | + structural test like the slice-1 ratchet). |
| 115 | + |
| 116 | +2. **Per-surface mapping:** |
| 117 | + |
| 118 | + | Surface | Serving backend (analytics) | Control-plane store (transactional) | |
| 119 | + |---|---|---| |
| 120 | + | Order 360 timeline | order row · stage history + pipeline trail from the journal · `users_enriched` customer block | dead-letter detail for the order's failed events (journal rows are the index, the store is the truth) | |
| 121 | + | Stuck-orders worklist | the whole computation (open orders × latest stage-entry row, budgets from the contract) | — | |
| 122 | + | Exception inbox | journal only as the entity index for findings | item truth: dead-letter lifecycle, dead webhook deliveries, reconciliation findings + triage overlay, stats/counter | |
| 123 | + |
| 124 | +3. **The journal's entity axis becomes real.** The serving-store projection |
| 125 | + writes stage-entry rows (topic `orders.status`, event_type |
| 126 | + `order.status.<status>`, `entity_id` = order id) on every order status |
| 127 | + transition, and the existing validated/dead-letter journal writes carry |
| 128 | + `entity_id` where it is derivable from the event payload |
| 129 | + (`clickhouse_sink.record_pipeline_event` gains the parameter). Without |
| 130 | + this, the timeline and the stage clock exist only for seeded demo rows — |
| 131 | + the current live-write sites leave `entity_id` NULL. |
| 132 | + |
| 133 | +4. **SLA budgets are catalog data, not code.** A `stages:` block in |
| 134 | + `contracts/entities/order.yaml` (ordered ladder, per-stage budget, |
| 135 | + terminal markers) is parsed into the catalog's `EntityDefinition` and is |
| 136 | + the only source of budgets for the worklist and the timeline's breach |
| 137 | + flag. Agents and SDKs see the SLA model through the same catalog surface |
| 138 | + as everything else. |
| 139 | + |
| 140 | +5. **The exception-triage overlay is the seventh control-plane state class.** |
| 141 | + ADR 0010's inventory grows by one table (triage status for items that have |
| 142 | + no native lifecycle: dead webhook deliveries, reconciliation findings), |
| 143 | + implemented in both adapters. Items with a native lifecycle — dead-letter |
| 144 | + events — are *mapped* into the inbox read model and are never duplicated |
| 145 | + into the overlay; their actions stay on `/v1/deadletter`. |
| 146 | + |
| 147 | +6. **Consistency semantics, stated:** ops surfaces read at serving-store |
| 148 | + freshness and are not metric-cached; the stage clock falls back to |
| 149 | + `created_at` (and says so) for orders with no stage rows; reconciliation |
| 150 | + checks run on inbox read with idempotent, dedupe-keyed finding upserts — |
| 151 | + safe at `replicaCount > 1` under ADR 0010's semantics (reads are |
| 152 | + stateless, writes are single-row transactions). |
| 153 | + |
| 154 | +7. **Scale-profile evolution stays a swap, not a rewrite.** Rebinding the |
| 155 | + order point-read to a true OLTP hot tier (the `pg_ops__` pattern the |
| 156 | + warehouse contour already demonstrates) is a recorded F-phase option |
| 157 | + behind the same port boundary; the surface contracts in the spec must not |
| 158 | + change if that rebinding happens. |
| 159 | + |
| 160 | +## Consequences |
| 161 | + |
| 162 | +### Positive |
| 163 | + |
| 164 | +- All three surfaces land on the zero-dependency demo profile — both ports |
| 165 | + already have embedded adapters, so D2–D4 add no services. |
| 166 | +- The audit's "no third path" concern becomes an enforced rule (structural |
| 167 | + ratchet), not a review habit. |
| 168 | +- Triage state inherits the PG adapter's transactional/claim semantics at |
| 169 | + scale for free; the pinned control-plane regression suites transfer. |
| 170 | +- The SLA model is data (contract), so budget changes are a config edit with |
| 171 | + contract-versioning discipline, and the catalog/search surfaces expose it. |
| 172 | + |
| 173 | +### Negative |
| 174 | + |
| 175 | +- The stage clock depends on journal hygiene: orders written before the |
| 176 | + stage-row writer shipped (or by writers that bypass it) degrade to the |
| 177 | + `created_at` fallback — visible in the response (`clock: "fallback"`) |
| 178 | + rather than hidden. |
| 179 | +- Seeding stage rows moves the demo `error_rate` denominator (it counts all |
| 180 | + journal rows in the window); the affected pinned tests are re-pinned by |
| 181 | + arithmetic, per the B3 house rule — recorded in the spec, not discovered in |
| 182 | + CI. |
| 183 | +- The worklist is computed on read: O(open orders × journal window) per call. |
| 184 | + Bounded and cheap at demo scale; the materialization option is recorded for |
| 185 | + when it is not. |
| 186 | +- Inbox reads trigger reconciliation checks against both stores — bounded |
| 187 | + read amplification, accepted for v1 (no scheduler dependency). |
| 188 | + |
| 189 | +## Rollout |
| 190 | + |
| 191 | +- **D2** — Order 360: timeline endpoint, journal entity axis (§ Decision 3), |
| 192 | + stage-entry seed trails, `error_rate` re-pins. |
| 193 | +- **D3** — stuck-orders: `stages:` contract block + catalog parsing, |
| 194 | + `/v1/ops/stuck-orders`, aging/summary. |
| 195 | +- **D4** — exception inbox: triage overlay (state class 7), sources |
| 196 | + (dead-letter mapping, dead webhook deliveries, reconciliation checks R1/R2), |
| 197 | + `/v1/ops/exceptions` + stats + manual-resolutions counter. |
| 198 | + |
| 199 | +Each slice is one PR with the full suite green; endpoint contracts, schemas, |
| 200 | +demo-story pins, and per-slice test obligations are in |
| 201 | +[`docs/ops-surfaces-spec.md`](../ops-surfaces-spec.md). |
| 202 | + |
| 203 | +## Follow-up |
| 204 | + |
| 205 | +- F-phase option: order point-read rebinding to an OLTP hot tier behind the |
| 206 | + same port (Decision 7). |
| 207 | +- Per-channel SLA budgets need a channel column on serving orders — recorded |
| 208 | + as out of v1 scope (`ops-surfaces-spec.md` § Non-goals). |
| 209 | +- A non-PII vault role for a richer MDM customer block — roadmap (option 3). |
| 210 | +- Container-ETA and marking-code exception sources (`domain.md` §4.2, §3) — |
| 211 | + named roadmap sources for the inbox, blocked on `excel__`/WMS feeds landing |
| 212 | + in the serving contour. |
0 commit comments