Skip to content

Commit 90a50fb

Browse files
ryan-williamsclaude
andcommitted
Add EventBridge drift detection; repin pyrmts for aws verify
Repins pyrmts `fea864a` → `90896ec` (Python only — no JS changed in the range) for `pyrmts-ops aws verify`, the `verify`-shaped counterpart to `d1 verify`, and the batch prefix fix (`bootstrap` no longer shadows the 16-vCPU default; `-p/--prefix` defaults unchanged, so ctbk's submit is untouched — confirmed by dry-run). `infra-drift.yml` runs `aws verify` daily over the three avail cascade rules. Its load-bearing check is invoke-permission: a rule that exists, is ENABLED, and resolves its target still fails silently at every tick if the function doesn't grant it permission — how the avail-v6 tick was lost (2026-08-06), and both live ticks still share one function, the shape that trips a constant StatementId. Verified locally: `schedules up to date` against prod, and exit 1 on a wrong schedule expr or a re-enabled v3 rule (which must stay DISABLED per `gc-legacy-pyramids.md`). Its own workflow, not a `gbfs.yml` job: unlike the D1 schema (verified on the push that re-pins), the schedules have no pinned artifact to hang off of — they're deployed imperatively, so drift is periodic. Daily cron + `workflow_dispatch`, read-only, non-gating. Specs: `iac-finish-the-stack.md` frames this as increment 0 of finishing the stalled `infra/` Pulumi stack (companion to pyrmts' `iac-boundary`), and `avail-outage-aggregations.md` is refreshed for v6 — no-bikes and no-ebikes are both cheap marginal histogram reads; only group co-emptiness ("all 4 dry at once") needs a fine scan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GgLAzeRJeNeyizG4XHWRBK
1 parent e9178ce commit 90a50fb

5 files changed

Lines changed: 264 additions & 84 deletions

File tree

.github/workflows/infra-drift.yml

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
name: Infra drift
2+
3+
# Periodic, read-only drift detection for infrastructure this repo deploys
4+
# imperatively (so there's no pinned artifact to verify on push):
5+
#
6+
# - EventBridge schedules + their Lambda targets, via `pyrmts-ops aws verify`.
7+
# The load-bearing check is invoke-permission: a rule that exists, is
8+
# ENABLED, and resolves its target still fails silently at every tick if the
9+
# function doesn't grant it permission — how the avail-v6 tick was lost
10+
# (2026-08-06). Both live ticks share one function, which is exactly the
11+
# shape that trips a constant StatementId.
12+
#
13+
# Not a gate on anything — a red run is the signal. Also runnable on demand.
14+
# (The D1 schema is verified separately in `gbfs.yml`, on the push that re-pins
15+
# pyrmts; this file is the piece with no push to hang off of.)
16+
on:
17+
schedule:
18+
- cron: '17 13 * * *' # 13:17 UTC daily — off the top of the hour
19+
workflow_dispatch:
20+
21+
jobs:
22+
aws-schedules:
23+
runs-on: ubuntu-latest
24+
env:
25+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
26+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
27+
AWS_DEFAULT_REGION: us-east-1
28+
steps:
29+
- uses: actions/checkout@v5
30+
- uses: astral-sh/setup-uv@v6
31+
- name: Install pyrmts-ops at the pinned rev
32+
run: |
33+
# Same single-source-of-truth trick as gbfs.yml's d1-schema job:
34+
# install whatever `pyproject.toml` pins, don't paste a second SHA.
35+
rev=$(sed -nE 's/^pyrmts +=.*rev = "([0-9a-f]+)".*/\1/p' pyproject.toml)
36+
[ -n "$rev" ] || { echo "no pyrmts rev found in pyproject.toml" >&2; exit 1; }
37+
echo "pyrmts @ $rev"
38+
base="git+https://github.com/runsascoded/pyrmts.git@$rev"
39+
uv venv
40+
uv pip install "boto3" \
41+
"pyrmts @ $base#subdirectory=python/pyrmts" \
42+
"pyrmts-engine @ $base#subdirectory=python/pyrmts_engine" \
43+
"pyrmts-ops @ $base#subdirectory=python/pyrmts_ops"
44+
- name: Verify avail EventBridge schedules
45+
run: |
46+
# The three avail cascade rules. Both the v5 and v6 ticks target the
47+
# v5 function (v6 rides the same config-driven image); the v3 rule is
48+
# retired and MUST stay DISABLED (`specs/gc-legacy-pyramids.md`).
49+
.venv/bin/pyrmts-ops aws verify \
50+
'ctbk-avail-cascade-v5-tick=ctbk-avail-cascade-v5@cron(3/5 * * * ? *)' \
51+
'ctbk-avail-cascade-v6-tick=ctbk-avail-cascade-v5@cron(4/5 * * * ? *)' \
52+
'ctbk-avail-cascade-hourly=ctbk-avail-cascade@cron(1/5 * * * ? *)' \
53+
-d ctbk-avail-cascade-hourly

pyproject.toml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,9 @@ dependencies = [
4040
# dvx: pinned past v0.5.0 (PyPI's latest) for `cache comm` — the dir-manifest
4141
# audit tool — plus `gc --safe` and the `.dvc`-rewrite/gitignore fixes.
4242
dvx = { git = "https://github.com/runsascoded/dvx.git", rev = "9c22fc08c" }
43-
pyrmts = { git = "https://github.com/runsascoded/pyrmts.git", rev = "fea864a", subdirectory = "python/pyrmts" }
44-
pyrmts-engine = { git = "https://github.com/runsascoded/pyrmts.git", rev = "fea864a", subdirectory = "python/pyrmts_engine" }
45-
pyrmts-ops = { git = "https://github.com/runsascoded/pyrmts.git", rev = "fea864a", subdirectory = "python/pyrmts_ops" }
43+
pyrmts = { git = "https://github.com/runsascoded/pyrmts.git", rev = "90896ec", subdirectory = "python/pyrmts" }
44+
pyrmts-engine = { git = "https://github.com/runsascoded/pyrmts.git", rev = "90896ec", subdirectory = "python/pyrmts_engine" }
45+
pyrmts-ops = { git = "https://github.com/runsascoded/pyrmts.git", rev = "90896ec", subdirectory = "python/pyrmts_ops" }
4646
# Local (dev) — swap all three together (the Python `pds l`): from ctbk, the
4747
# `pyrmts` entry here is what resolves `pyrmts-engine`'s own `pyrmts` dep (its
4848
# workspace source only applies inside the pyrmts repo). Don't commit this

specs/avail-outage-aggregations.md

Lines changed: 108 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -1,79 +1,112 @@
1-
# avail: outage aggregations + plot issue-bands
1+
# avail: outage / reliability aggregations (v6 refresh)
22

3-
Status: proposed (captured 2026-07-13; separate direction from the
4-
rides+pyrmts refactor).
3+
Status: proposed. Original captured 2026-07-13 (pre-v6, pre-drop-LUC); this
4+
rewrite brings it to the avail-v6 serving surface and adds the concrete
5+
questions that motivate it (single-station reliability, group co-emptiness,
6+
peak-hour slicing).
57

68
## Goal
79

8-
Fast aggregate answers to "how often / how long is availability bad?"
9-
over any station set, plus visual annotation of problem windows on
10-
avail plots:
11-
12-
1. **Fraction-of-time stats** — % of minutes at 0 bikes / 0 ebikes /
13-
0 docks, or < N total, over a window and station set.
14-
2. **Span stats** — number and lengths of contiguous outage spans
15-
(e.g. "0 bikes for ≥ 15 min"), incl. distributions.
16-
3. **Group emptiness** — for a station set, time at k stations empty
17-
(k = 0, 1, 2, …) — "how often is the whole neighborhood dry?"
18-
4. **Plot issue-bands** — semi-transparent red vertical background
19-
spans on avail plots over x-ranges where outages (or data issues)
20-
occurred. Prior art: the homicides plot in `$hccs/crashes`
21-
(`www/src/njsp/HomicidesComparisonPlot.tsx` + `src/annotations/*`
22-
`useAnnotations``toPlotLayers` → plotly shapes/layers), which
23-
layers annotation ranges behind the traces with hover/click detail.
24-
25-
## What the pyramid already answers (cheap)
26-
27-
(1) falls out of the histogram monoid today — proven during the OGI
28-
work: ONE `reducer=hist` query over 30d at a coarse tier returns
29-
per-bin `{value: minute_count}` histograms; merging gives
30-
`pct_zero = h[0]/Σh` and mean in O(bins) client-side. Works for any
31-
`cells=` station set, any window, no schema change. The og card's
32-
"% of the time no bikes, last 30d" stat is exactly this.
33-
34-
`< N total` similarly: `Σ_{v<N} h[v] / Σh`.
35-
36-
Caveat: multi-station sets' histograms mix stations (a bin's histogram
37-
counts (station, minute) observations) — (1) then reads as "fraction
38-
of station-minutes", not "fraction of minutes where ANY/ALL empty".
39-
Per-station loops (one hist query per LUC cell) recover per-station
40-
stats; (3) needs more (below).
41-
42-
## What it doesn't (spans, group-k)
43-
44-
(2) and (3) are **not monoidal** — span boundaries don't survive
45-
histogram aggregation. Options:
46-
47-
- **Query-time scan** of `/1m` (or `/5m`) bins over the window for the
48-
set's cells: runs/spans computed in the worker (or FE) from the fine
49-
series. Fine for ≤ ~a month of 1m bins × small sets; wide windows
50-
want coarser bins with a "min" reducer (a 1h bin with min=0 ⇒ some
51-
zero minute inside — conservative span detection at 1h resolution).
52-
- **Materialized runs**: a Lambda-side pass emitting per-station
53-
outage-span records (start, end, metric, threshold) to a small
54-
parquet/D1 table, maintained incrementally like the pyramid. Exact
55-
spans, O(1) query; new moving part. Prefer starting with query-time
56-
scan; materialize only if usage warrants.
57-
58-
(3) group-k: needs per-station series aligned per bin → k(t) =
59-
#stations with value 0 at t. Query per-LUC-cell series (one request,
60-
`cells=` with per-cell grouping — the /cells route already returns
61-
per-cell rows) and fold client-side.
62-
63-
## Issue-bands (4)
64-
65-
- Data source: outage spans from (2) (query-time at first).
66-
- Rendering: uPlot (StationDetail avail chart) supports background
67-
band plugins; mirror the crashes annotations shape — bands behind
68-
series, tooltip/hover shows span detail (metric, duration).
69-
- Also usable for *data* issues (feed gaps — the poller's missed
70-
minutes are visible as absent observations; /health already knows
71-
scrape gaps).
72-
73-
## Sketch of increments
74-
75-
1. Worker: `/api/avail-v3/stats?cells=…&from=…&to=…&thresholds=…`
76-
hist-based (1) + scan-based (2) over ≤ 31d windows.
77-
2. FE StationDetail: issue-bands from (2) + a small "reliability"
78-
stat block (reuse og card's 30d numbers).
79-
3. Group page (post station-sets work): (3) fold + set-level bands.
10+
Fast answers to "how often / how long is availability bad?" over any station
11+
set, plus visual annotation of problem windows on avail plots. Driving
12+
questions (user, 2026-08-29):
13+
14+
- How often is a station **out of bikes** (0 total) or **out of ebikes**
15+
(0 ebikes)? — the two primary metrics of interest.
16+
- How often are my 4 closest stations in that state **all at once**?
17+
- …restricted to **peak commute hours** (e.g. 7–10am weekdays).
18+
19+
(Not a priority: "classic-only" = `ebikes==0 AND bikes>0`. It's the one
20+
*joint-across-metrics* case and the only reason to touch two metrics together;
21+
demoted to the footnote below since the questions above don't need it.)
22+
23+
## What already exists (don't rebuild)
24+
25+
- **Station sets.** `/stations` has a multi-select set (`?sel=` URL codec,
26+
`www/src/pages/Stations.tsx`) and named sets via `neighborhoods.json`
27+
(`ctbk neighborhoods`). "My 4 closest" is a `?sel=` set or a neighborhood.
28+
- **Per-metric histograms.** avail-v6 stores five histogram-monoid metrics —
29+
`bikes`, `ebikes`, `docks`, `disabled`, `pending` — per (`s2_cell`, `dt`)
30+
(`configs/pyramids/avail-v6.yaml`). GBFS `num_bikes_available` is TOTAL, so
31+
classic = `bikes − ebikes`.
32+
- **Serving.** `/api/avail-v3[/cells]?from=&to=&cells=|bbox=&reducer=` (serves
33+
the default pyramid = v6; the `-v3` in the path is legacy naming). `reducer`
34+
`mean|min|max|p05|p25|p50|p75|p95|hist`; `hist` returns full per-metric
35+
histograms. `/cells` returns one row per station; the rollup route collapses
36+
`dims` and sums across the set (`gbfs/api/src/avail_geo.ts`).
37+
- **FE chart.** `StationAvailabilityChart.tsx` (uPlot) already renders a
38+
station's series.
39+
40+
## Monoidal (cheap, any tier/window) vs. joint (needs a fine scan)
41+
42+
The split is the whole design. A histogram monoid answers questions about ONE
43+
metric's MARGINAL distribution; anything joint (across metrics, or across
44+
stations, at a specific instant) is not recoverable from merged histograms.
45+
46+
**Monoidal — cheap, any tier/window — and it covers both primary questions:**
47+
48+
- `% time 0 bikes` for one station over a window = `h_bikes[0] / Σ h_bikes`.
49+
- `% time 0 ebikes` = `h_ebikes[0] / Σ h_ebikes`. **Also cheap** — it's the
50+
`ebikes` metric's own bin 0, a separate marginal from `bikes`, so no joint is
51+
involved. (`0 docks` likewise.) Proven during the OGI work (the og card's
52+
"% of the time no bikes, last 30d" is exactly this shape).
53+
- `% time < N` = `Σ_{v<N} h[v] / Σh`.
54+
- Per-station stats over a set: one `/cells` request, fold each row.
55+
56+
Caveat (unchanged): on the ROLLUP route the histogram mixes stations —
57+
`h[0]/Σh` reads as "fraction of station-minutes empty", NOT "fraction of
58+
minutes where ANY/ALL empty". Per-station needs `/cells`.
59+
60+
So "how often is station X out of bikes / out of ebikes, last 30d" is a
61+
histogram read at any tier — the cheap path, shippable first.
62+
63+
**Joint — NOT monoidal, needs a scan of fine bins:**
64+
65+
1. **Group co-emptiness** (k of N stations at 0 at the same t) — the "are all 4
66+
dry at once?" question, and a joint ACROSS stations. Needs per-station series
67+
aligned per bin, then `k(t) = #{stations : bikes(t)=0}` folded
68+
client/worker-side. Simultaneity requires bins fine enough that a bin ≈ an
69+
instant: at a coarse tier a station's bin is a distribution, and even
70+
`reducer=min=0` only says "some minute in this bin was empty" without telling
71+
you WHICH minute — so cross-station alignment is lost above the fine tiers.
72+
The per-station "% time empty" fractions above are cheap; only their
73+
*simultaneity* costs a scan.
74+
2. **Spans / runs** ("0 bikes for ≥15 min", count + length distribution) —
75+
boundaries don't survive histogram aggregation. Same fine-scan path.
76+
77+
Footnote — **classic-only** (`ebikes==0 AND bikes>0`), not a current priority:
78+
a joint across two metrics. Marginals give `P(ebikes=0)` and `P(bikes=0)`
79+
separately, never the joint, so it'd need the same fine scan. Left out of the
80+
increments below.
81+
82+
**Peak-hour slicing** is orthogonal to both: it's a periodic (hour-of-day,
83+
day-of-week) filter, not a window. Neither the monoid nor a plain window gives
84+
it — scan fine bins and bucket by `hour_of_day(dt)`. Cheap for bounded windows
85+
(a month of 1m bins × a handful of stations); wide windows want a coarser bin
86+
with a conservative reducer, accepting the simultaneity caveat above.
87+
88+
## Increments
89+
90+
1. **Worker `/api/avail-v3/stats?cells=&from=&to=&thresholds=&hours=`**
91+
monoidal fractions (1) from `hist` + fine-scan spans (3) over ≤ ~31d
92+
windows; optional `hours=7-10` weekday-peak filter. Returns per-station
93+
`{pct_zero_bikes, pct_zero_ebikes, pct_zero_docks, spans:[…]}` — all three
94+
cheap from `hist` — and, with `group=1`, the co-emptiness histogram
95+
`k → minutes` (1), the one part that scans.
96+
2. **FE reliability block** on StationDetail: reuse the og card's 30d numbers
97+
(% time 0 bikes / 0 ebikes) and, for a `?sel=` set, the group-k summary.
98+
3. **Issue-bands** on the avail chart: semi-transparent red x-spans over
99+
outage windows from (3). uPlot supports background-band plugins; mirror the
100+
crashes annotations shape (`$hccs/crashes` `HomicidesComparisonPlot.tsx` +
101+
`src/annotations/*``useAnnotations``toPlotLayers`), bands behind the
102+
series with hover detail (metric, duration). Doubles for feed-gap bands
103+
(`/health` already knows scrape gaps).
104+
105+
## Materialization (only if usage warrants)
106+
107+
Spans and co-outage records could be precomputed by a Lambda/engine pass into a
108+
small parquet/D1 table (start, end, metric, threshold, k), maintained
109+
incrementally like the pyramid — exact spans, O(1) query, new moving part.
110+
Start query-time; materialize only if the fine-scan windows people actually
111+
ask for get too wide. Group-k especially is a candidate: a "co-outage" metric
112+
keyed by a station SET is the one thing no per-station monoid can ever give.

specs/iac-finish-the-stack.md

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Finish the ctbk infra stack (don't start a new one)
2+
3+
Status: proposed (2026-08-29). Companion to the pyrmts session's
4+
`specs/pyrmts-iac-answer.md` / pyrmts `specs/iac-boundary.md`, which argued
5+
the *shape*; this is the ctbk-side plan and the first increment is already
6+
landed.
7+
8+
## The finding that frames everything
9+
10+
ctbk **already has Pulumi**: `infra/` is a live `pulumi-cloudflare` project
11+
(`infra/__main__.py`, `specs/pulumi-cf-infra.md` behind it) declaring four
12+
resources — the imported R2 bucket (`protect=True`), the `ctbk-gbfs` D1
13+
database, the `gbfs-status-events` queue, and the R2→queue event notification.
14+
Last touched **2026-04-12**.
15+
16+
Since then ctbk grew, imperatively, around that stalled stack: 3 avail Lambdas
17+
(`ctbk-avail-cascade{,-v5}`, `-rebuild`), 3 EventBridge rules, 2 ECR repos, a
18+
Batch job definition + compute environments, 6 Workers (its `WORKERS` dict
19+
lists 3), and 2 custom domains. `pulumi up` needed a human to remember it from
20+
a laptop; the imperative scripts run in CI on every push. The scripts won.
21+
22+
So the task is **not** "adopt IaC" — it's finish the stack that exists, and
23+
close the gap that let it stall (nothing ran it). A half-adopted IaC layer is
24+
worse than scripts because it implies coverage it doesn't have; `infra/` is
25+
that sentence already true in the tree (its `WORKERS` dict is 3 of 6).
26+
27+
## Increment 0 — DONE (2026-08-29): drift detection, no state migration
28+
29+
The cheapest, highest-value slice, and it needs no Pulumi at all: *detect*
30+
drift on the imperative resources so the stalled stack's blind spots become
31+
visible.
32+
33+
- **D1 schema**`gbfs.yml`'s `d1-schema` job runs `pyrmts-ops d1 verify` on
34+
the push that re-pins pyrmts (verifies the newly-pinned DDL matches prod).
35+
Read-only, scoped token (`ctbk-gha-d1-read`).
36+
- **EventBridge schedules**`infra-drift.yml`, daily cron, runs
37+
`pyrmts-ops aws verify` over the three avail rules incl. the retired-and-must-
38+
stay-DISABLED v3 rule. Its invoke-permission check is exactly what would have
39+
caught the avail-v6 tick outage; verified locally that it flags a wrong
40+
schedule expr and a re-enabled v3 rule (exit 1).
41+
42+
These are `verify`-shaped, so they're safe to run against prod forever and
43+
graduate cleanly: whatever a later `pulumi up` would manage, the same `verify`
44+
keeps checking.
45+
46+
## Increment 1 — import the AWS resources into Pulumi (no-op first diff)
47+
48+
Leverage pyrmts' new Pulumi component library (`32cb08e`, `95c17eb` — a
49+
pyramid's cloud footprint as components, validated against real providers).
50+
Per `iac-boundary.md`, the useful primitive is a pure `config → desired
51+
resource set` function: read a pyramid YAML, emit {function, schedule, env,
52+
bucket, database} — pyrmts owns the description, ctbk's program declares it.
53+
54+
Order (OA's stated sequence — import first, so the first `pulumi up` is a no-op
55+
diff that just gives you drift detection):
56+
57+
1. Add an AWS provider to `infra/` alongside the CF one.
58+
2. Import the live Lambdas, EventBridge rules (with the `enabled=False` v3 rule
59+
as a *declared* fact), ECR repos, and the Batch job definition. Match the
60+
imperative scripts' current output so the diff is empty.
61+
3. Reconcile the two imperative deployers against the imported resources:
62+
`deploy-image.py` becomes the thing Pulumi manages, or is retired in favor
63+
of it. (`deploy.py` is already gone, `fb0dbd6a`.)
64+
65+
Only after the import is a clean no-op do the imperative scripts get retired —
66+
never a flag-day rewrite.
67+
68+
## Increment 2 — make it run without being remembered
69+
70+
The four-months-unrun problem is a CI problem, and it recurs with any tool a
71+
human must invoke by hand.
72+
73+
1. Move state off the committed local-file backend (the thing that made
74+
concurrent/CI runs awkward). Reuse OA's Pulumi GHA workflow (`pulumi-v1`)
75+
for PR diffs — `pulumi-cf-infra.md` already intended this; the `infra.yml`
76+
it drafts was never created.
77+
2. Wire `infra.yml` to run `pulumi preview` on PRs and `pulumi up` on merge, so
78+
the stack converges on push like the worker deploys already do.
79+
3. Fold the drift `verify`s (increment 0) into the same story: preview *is* a
80+
superset of verify, but keep the standalone verifies as the cheap, always-on
81+
floor until `pulumi up` actually owns each resource.
82+
83+
## Explicit non-goals
84+
85+
- A pyrmts-side IaC framework. pyrmts ships the `config → resources`
86+
description and the `verify` commands; the stack lives in the consumer
87+
(`iac-boundary.md`, OA's `cf-iac.md`). D1 *migrations* stay with the app
88+
(schema is not IaC'd).
89+
- CF-side expansion beyond the four existing resources until the AWS import is
90+
green — one incomplete stack at a time.
91+
- Re-bootstrapping Batch to 16 vCPU is a separate, one-line op (`pyrmts-engine
92+
batch bootstrap` via the CLI, now that the shadowed default is fixed
93+
upstream); it mutates the shared job definition and wants explicit sign-off,
94+
independent of this IaC work.

0 commit comments

Comments
 (0)