Skip to content

Commit d0b6353

Browse files
committed
refactor(feeds)!: region-first feed and format ids
Replace ad-hoc feed ids (country-suffix, inconsistent) and serialization-baked format ids with a region-first scheme. Feed id is now derived in the shared schema from country/subdivision/operator/stream ([region]-[operator]-[stream]); format ids name the interchange standard or a provider slug without a -json/-xml serialization suffix. Credentials (auth env vars) are unchanged. Fixes a latent bug where scheduler.ts gated Fintraffic native-baseline derivation on the old format string. BREAKING CHANGE: feed ids and source formats are renamed; the single existing deployment needs a one-off DB source/sensor_key migration (handled by hand).
1 parent 2056efc commit d0b6353

115 files changed

Lines changed: 769 additions & 506 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/datex-conformance.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ description, and that's what the code says.
110110

111111
Separately from schema conformance, actually _publishing_ to a National
112112
Access Point is a per-member-state administrative process, not a technical
113-
one. Per the commons ADR (§10.1, verified 2026-07-07):
113+
one. Per OpenConditions' NAP-publication findings (verified 2026-07-07):
114114

115115
- **France — `transport.data.gouv.fr`.** Fully self-service, no entity-type
116116
gate, already carries real-time DATEX II (Bison Futé). This is the

docs/federation-onboarding.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,14 @@ Field rules (enforced by `parseRegistryEntry` in
4343
`packages/federation/src/registry.ts`, which fails closed on anything
4444
malformed):
4545

46-
| Field | Meaning |
47-
| ----------- | ------------------------------------------------------------------------------------------------------ |
48-
| `id` | Lowercase slug (`[a-z0-9-]`). Doubles as the TUF target file name, so no dots, slashes, or case games. |
49-
| `actor` | The instance's Actor document URL (`/.well-known/openconditions/actor.json`). |
50-
| `operator` | Legal operator: `name`, `contact`, `jurisdiction` (all required). |
51-
| `coverage` | `iso3166` country codes and/or a `[minLon, minLat, maxLon, maxLat]` bbox. |
52-
| `trustTier` | 0 (observer), 1 (standard), or 2 (anchor) — see the peering tiers in the federation design. |
53-
| `keys` | The instance's **authorized Ed25519 keys** as `publicKeyMultibase` (`z6Mk…`) values. |
46+
| Field | Meaning |
47+
| ----------- | ----------------------------------------------------------------------------------------------------------- |
48+
| `id` | Lowercase slug (`[a-z0-9-]`). Doubles as the TUF target file name, so no dots, slashes, or case games. |
49+
| `actor` | The instance's Actor document URL (`/.well-known/openconditions/actor.json`). |
50+
| `operator` | Legal operator: `name`, `contact`, `jurisdiction` (all required). |
51+
| `coverage` | `iso3166` country codes and/or a `[minLon, minLat, maxLon, maxLat]` bbox. |
52+
| `trustTier` | 0 (observer), 1 (standard), or 2 (anchor) — governs peering scope; admission is the governance review (§5). |
53+
| `keys` | The instance's **authorized Ed25519 keys** as `publicKeyMultibase` (`z6Mk…`) values. |
5454

5555
### Ed25519 key authorization
5656

docs/probe-p1-feasibility-spike.md

Lines changed: 27 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,17 @@
55
> thresholds, and the DAP + differential-privacy design agenda for the independent reviewer. It exists so the
66
> decision to build (or defer) the probe layer can be made **on evidence, before any Phase P0–P6 code is written**.
77
>
8-
> Binding sources (this doc must not contradict them): the commons ADR §3 (D4/D5/D6/D7) and Plan 4
9-
> (`4-privacy-probe-data.md`) in the OpenMapX plan set; the shared Privacy-Pass token layer from Plan 1
10-
> (`services/contributions-api` Attester/Issuer); the speed-congestion fusion seam (`segment_observation`).
8+
> Binding sources (this doc must not contradict them): OpenConditions' commons and probe-layer design
9+
> decisions; the shared Privacy-Pass token layer already built in `services/contributions-api`
10+
> (Attester/Issuer); the speed-congestion fusion seam (`segment_observation`).
1111
1212
## 1. Why this gate exists
1313

1414
The entire probe privacy model rests on one load-bearing assumption: **raw trajectories never leave the device —
1515
only `(segment, window, clamped speed)` tuples do.** That is only true if continuous on-device map-matching is
1616
feasible on a real, mid-range phone at acceptable battery, storage and accuracy. If it is not, the model collapses
1717
to "send GPS to a server," which is exactly what this layer exists to prevent. So feasibility is decided **before**
18-
committing to build, not discovered mid-build (ADR §8; Plan 4 Phase P-1).
18+
committing to build, not discovered mid-build.
1919

2020
P-1 has six sub-gates. All six must pass to commit to P0–P6. They split into work that is verifiable here (in CI,
2121
without a device or an external party) and work that is **structurally blocked** on a physical device, an
@@ -31,8 +31,8 @@ independent reviewer, or a maintained implementation that does not currently exi
3131
| 6 | DAP + DP design review (privacy unit, adjacency, ε/δ accountant, mechanism) | design + independent review | **BLOCKED — acceptance is reviewer-gated (P0)** |
3232

3333
**Nothing in this spike marks any sub-gate "passed."** Simulator or desktop numbers do not satisfy 1–4; only real
34-
mid-range-phone evidence does. Item 6 acceptance requires the P0 independent privacy reviewer, by the plan's own
35-
text ("P0's independent reviewer must approve it").
34+
mid-range-phone evidence does. Item 6 acceptance requires the P0 independent privacy reviewer, by the probe-layer
35+
design's own rule ("P0's independent reviewer must approve it").
3636

3737
## 2. Reconnaissance summary (what exists today)
3838

@@ -43,7 +43,7 @@ text ("P0's independent reviewer must approve it").
4343
`services/contributions-api/src/attester/policy.ts`). This is the intended greenfield starting point.
4444
- **The Privacy-Pass token layer a probe would reuse already exists and works**, server-side, in
4545
`services/contributions-api/src/issuer/` + `.../attester/` on `@cloudflare/privacypass-ts@0.9.0` (RFC 9578
46-
Blind-RSA). Reusable surface, verbatim from Plan 1:
46+
Blind-RSA). Reusable surface, verbatim from the contributions-api crowd-report layer:
4747
- `PublicContext { purpose; taskId?; epoch }`, `publicContextString``` `${purpose}:${taskId ?? "-"}:${epoch}` ``,
4848
`redemptionContext(ctx)`**server-authoritative** `SHA-256(purpose:taskId:epoch)` — the probe context is
4949
literally `purpose: "probe"` (the context hashing is already proven purpose-agnostic in tests).
@@ -55,8 +55,8 @@ text ("P0's independent reviewer must approve it").
5555
- **The landing target exists**: `segment_observation` (`packages/core/src/db/schema.ts`) with
5656
`(segmentId, source)` PK and `sourceTier`, and the tier-priority reducer `fuseSegmentSpeed`
5757
(`services/ingest/src/pipeline/segment-speed.ts`) — `SELECT DISTINCT ON (segment_id) … ORDER BY tier, observed_at DESC`,
58-
where `crowd` is the lowest tier. The privacy-accounting columns Plan 4 P5 needs are **not yet present** (that is
59-
the P5 additive migration — not built here, it is gated behind P-1 passing).
58+
where `crowd` is the lowest tier. The privacy-accounting columns the probe-landing step (P5) needs are **not yet
59+
present** (that is the P5 additive migration — not built here, it is gated behind P-1 passing).
6060

6161
## 3. The decisive tooling finding (item 5 interoperability spike result)
6262

@@ -79,7 +79,7 @@ hand-rolling DAP. The interop finding is decisive and blocks a _conformant_ buil
7979
by autonomously committing non-conformant code. This is a legitimate spike _finding_, and it is a gate that
8080
cannot go green here.
8181
- **Differential-privacy primitive.** No maintained **in-process** JS DP bounded-sum/mean primitive exists. The
82-
non-hand-rolled options are all **out-of-process**, which the plan anticipates ("Google `differential-privacy`
82+
non-hand-rolled options are all **out-of-process**, which the probe-layer design anticipates ("Google `differential-privacy`
8383
run as a Go binary beside the BFF"): Google's `differential-privacy` Go module (`dpagg.BoundedSumFloat64`,
8484
v4.1.0, 2025-02) invoked as a subprocess wrapper, or OpenDP/PyDP via a Python service. Adopting either introduces
8585
a **new toolchain (Go or Python) into this TypeScript monorepo** — an operator/architecture decision, not a
@@ -93,8 +93,8 @@ hand-rolling DAP. The interop finding is decisive and blocks a _conformant_ buil
9393

9494
This is the artifact needed to **obtain** the P-1 evidence. It must be run by the operator on a real mid-range phone
9595
(not a flagship, not a simulator). The proposed thresholds below are **starting values to be confirmed by the
96-
operator + the independent reviewer** — the plan requires the threshold to be "set explicitly," and the review
97-
signs off on the numbers.
96+
operator + the independent reviewer** — the probe-layer design requires the threshold to be "set explicitly," and
97+
the review signs off on the numbers.
9898

9999
### 4.1 Battery / thermal (item 1)
100100

@@ -147,19 +147,19 @@ signs off on the numbers.
147147

148148
## 5. DAP + DP design agenda for the independent reviewer (items 5–6)
149149

150-
This is the **agenda a P0 independent reviewer signs off**, not a settled design. It states the ADR-committed
150+
This is the **agenda a P0 independent reviewer signs off**, not a settled design. It states the design-committed
151151
constraints (fixed) and the open decisions (reviewer + operator). Writing settled ε/δ values, a chosen VDAF profile,
152-
or a chosen DP mechanism into code here would pre-empt the review the plan reserves — and would risk manufacturing
152+
or a chosen DP mechanism into code here would pre-empt the review the design reserves — and would risk manufacturing
153153
exactly the plausible-but-wrong privacy artifact the whole gated design exists to prevent.
154154

155-
### 5.1 Committed constraints (fixed by the ADR/Plan 4 — not open)
155+
### 5.1 Committed constraints (fixed by the commons design — not open)
156156

157157
- **On-device only.** Map-match to `way_id:dir`; trim first/last ~200 m of every trip _before_ bucketing; reduce
158158
candidates to `(segment_id, ~5-min UTC window, speed clamped [0,200])` with at most one candidate per
159159
`(segment, window)`, then **sample at most one tuple per epoch-bound entitlement** — a multi-segment trip exports
160160
**at most one measurement per epoch**, not one per segment crossed (this per-epoch bound is what makes the DP
161161
sensitivity accounting sound). Raw trajectories never leave the device.
162-
- **Submission.** Reuse Plan 1's single-use RFC 9578 Privacy-Pass token (self-hosted Attester + Issuer) as the
162+
- **Submission.** Reuse the contributions-api's single-use RFC 9578 Privacy-Pass token (self-hosted Attester + Issuer) as the
163163
anonymous admission credential; the Origin sees only "a valid token," never identity or which token. **No probe
164164
pseudonym** enters either input share. No-log ingest edge (drop IP before persistence; if abuse control is
165165
demonstrably needed, a **short-epoch keyed HMAC behind a rotating key**, documented as personal data, deleted on
@@ -226,7 +226,7 @@ exactly the plausible-but-wrong privacy artifact the whole gated design exists t
226226
non-viable at our thousands-scale. Rejected.
227227
- **A plaintext / single-aggregator / raw-tuple / server-GPS bootstrap** — breaks the model's whole premise.
228228
**Never build it.** If DAP+DP is not viable, the probe layer is **deferred**, and the sensor + official-feed base
229-
(speed-congestion) remains the coverage mechanism; the crowd-report layer (Plan 1) is unaffected.
229+
(speed-congestion) remains the coverage mechanism; the crowd-report layer is unaffected.
230230

231231
## 6. The STOP boundary — what only the operator / an outside party can do
232232

@@ -239,21 +239,22 @@ P-1 cannot be completed in this environment. It is blocked on three independent
239239
2. **The DAP/VDAF tooling decision (item 5, §3 + §5.2.1).** No maintained JS client exists at the draft-18-DAP /
240240
current-VDAF level; hand-rolling DAP is forbidden. The operator + reviewer must choose the profile + implementation path (and accept any new
241241
Rust/Go/Python toolchain) before conformant DAP code is written.
242-
3. **The P0 institutional gate (item 6 acceptance + launch).** Per Plan 4 Phase P0, before any probe code ships:
242+
3. **The P0 institutional gate (item 6 acceptance + launch).** Before any probe code ships:
243243
a **public DPIA** + a **scoped public threat-model doc**; a **commissioned independent privacy review** (academic
244244
/ civil-society) of the DPIA + threat model + the k/ε parameters, **passed and published**; a **DSB appointed**
245-
and **Cyber-Versicherung bound**; the **crowd-report layer (Plan 1) proven durable in production** (D7: probe
245+
and **Cyber-Versicherung bound**; the **crowd-report layer proven durable in production** (probe
246246
ships _after_ crowd reports prove durability); a **contracted independent DAP Helper operator** with a reviewed
247247
MoU/DPA (task config, key rotation, retention/deletion, incident response, termination, emergency migration) and
248248
an **out-of-band key exchange** — a cloud account or second service controlled by us does **not** pass this gate;
249249
and an **independently operated OHTTP relay** (§5.1 metadata boundary) run by a _different_ operator than the
250250
aggregator.
251251

252-
Only when all three clear does P-1 pass. Building P0→P6 then _additionally_ requires the Plan-4 prerequisite gate —
253-
the substrate 0b commons fields on `segment_observation` (the P5 migration), the Plan-1 Attester/Issuer, and the
254-
live speed-congestion `segment_observation` seam — so a P-1 pass is necessary, not sufficient, for build.
252+
Only when all three clear does P-1 pass. Building P0→P6 then _additionally_ requires the probe-layer prerequisite
253+
gate — the substrate commons fields on `segment_observation` (the P5 additive migration), the contributions-api
254+
Attester/Issuer, and the live speed-congestion `segment_observation` seam — so a P-1 pass is necessary, not
255+
sufficient, for build.
255256

256-
## 7. Honesty channel (ADR D6 — non-negotiable)
257+
## 7. Honesty channel (non-negotiable)
257258

258259
Even a fully-built stack is honest about its limits, and so is this spike:
259260

@@ -277,12 +278,12 @@ Even a fully-built stack is honest about its limits, and so is this spike:
277278

278279
## 8. What was intentionally not built here, and why
279280

280-
- **No P0–P6 production code** (client pipeline, Leader/Helper/Collector, landing migration). ADR §8: the spike must
281-
pass before any other Plan-4 work is committed.
281+
- **No P0–P6 production code** (client pipeline, Leader/Helper/Collector, landing migration). The spike must
282+
pass before any other probe-layer work is committed.
282283
- **No DAP/VDAF client on the stale draft-09 lib, and no new Go/Python DP toolchain** committed to `main`. Both are
283284
operator + reviewer decisions (§3, §5.2); committing a chosen profile/mechanism would pre-empt the P0 review and
284285
risk a non-conformant or unsupportable privacy artifact.
285286
- **No claim that any sub-gate passed.** Simulator/desktop numbers do not count; item 6 acceptance is reviewer-gated.
286287

287-
The probe layer stays deferred until P-1's three gates clear. The sensor + official-feed base and the Plan 1
288+
The probe layer stays deferred until P-1's three gates clear. The sensor + official-feed base and the
288289
crowd-report layer are the coverage mechanisms in the meantime, and neither is affected by this deferral.

0 commit comments

Comments
 (0)