|
1 | | -# avail: outage aggregations + plot issue-bands |
| 1 | +# avail: outage / reliability aggregations (v6 refresh) |
2 | 2 |
|
3 | | -Status: proposed (captured 2026-07-13; separate direction from the |
4 | | -rides+pyrmts refactor). |
| 3 | +Status: proposed. Original captured 2026-07-13 (pre-v6, pre-drop-LUC); this |
| 4 | +rewrite brings it to the avail-v6 serving surface and adds the concrete |
| 5 | +questions that motivate it (single-station reliability, group co-emptiness, |
| 6 | +peak-hour slicing). |
5 | 7 |
|
6 | 8 | ## Goal |
7 | 9 |
|
8 | | -Fast aggregate answers to "how often / how long is availability bad?" |
9 | | -over any station set, plus visual annotation of problem windows on |
10 | | -avail plots: |
11 | | - |
12 | | -1. **Fraction-of-time stats** — % of minutes at 0 bikes / 0 ebikes / |
13 | | - 0 docks, or < N total, over a window and station set. |
14 | | -2. **Span stats** — number and lengths of contiguous outage spans |
15 | | - (e.g. "0 bikes for ≥ 15 min"), incl. distributions. |
16 | | -3. **Group emptiness** — for a station set, time at k stations empty |
17 | | - (k = 0, 1, 2, …) — "how often is the whole neighborhood dry?" |
18 | | -4. **Plot issue-bands** — semi-transparent red vertical background |
19 | | - spans on avail plots over x-ranges where outages (or data issues) |
20 | | - occurred. Prior art: the homicides plot in `$hccs/crashes` |
21 | | - (`www/src/njsp/HomicidesComparisonPlot.tsx` + `src/annotations/*` — |
22 | | - `useAnnotations` → `toPlotLayers` → plotly shapes/layers), which |
23 | | - layers annotation ranges behind the traces with hover/click detail. |
24 | | - |
25 | | -## What the pyramid already answers (cheap) |
26 | | - |
27 | | -(1) falls out of the histogram monoid today — proven during the OGI |
28 | | -work: ONE `reducer=hist` query over 30d at a coarse tier returns |
29 | | -per-bin `{value: minute_count}` histograms; merging gives |
30 | | -`pct_zero = h[0]/Σh` and mean in O(bins) client-side. Works for any |
31 | | -`cells=` station set, any window, no schema change. The og card's |
32 | | -"% of the time no bikes, last 30d" stat is exactly this. |
33 | | - |
34 | | -`< N total` similarly: `Σ_{v<N} h[v] / Σh`. |
35 | | - |
36 | | -Caveat: multi-station sets' histograms mix stations (a bin's histogram |
37 | | -counts (station, minute) observations) — (1) then reads as "fraction |
38 | | -of station-minutes", not "fraction of minutes where ANY/ALL empty". |
39 | | -Per-station loops (one hist query per LUC cell) recover per-station |
40 | | -stats; (3) needs more (below). |
41 | | - |
42 | | -## What it doesn't (spans, group-k) |
43 | | - |
44 | | -(2) and (3) are **not monoidal** — span boundaries don't survive |
45 | | -histogram aggregation. Options: |
46 | | - |
47 | | -- **Query-time scan** of `/1m` (or `/5m`) bins over the window for the |
48 | | - set's cells: runs/spans computed in the worker (or FE) from the fine |
49 | | - series. Fine for ≤ ~a month of 1m bins × small sets; wide windows |
50 | | - want coarser bins with a "min" reducer (a 1h bin with min=0 ⇒ some |
51 | | - zero minute inside — conservative span detection at 1h resolution). |
52 | | -- **Materialized runs**: a Lambda-side pass emitting per-station |
53 | | - outage-span records (start, end, metric, threshold) to a small |
54 | | - parquet/D1 table, maintained incrementally like the pyramid. Exact |
55 | | - spans, O(1) query; new moving part. Prefer starting with query-time |
56 | | - scan; materialize only if usage warrants. |
57 | | - |
58 | | -(3) group-k: needs per-station series aligned per bin → k(t) = |
59 | | -#stations with value 0 at t. Query per-LUC-cell series (one request, |
60 | | -`cells=` with per-cell grouping — the /cells route already returns |
61 | | -per-cell rows) and fold client-side. |
62 | | - |
63 | | -## Issue-bands (4) |
64 | | - |
65 | | -- Data source: outage spans from (2) (query-time at first). |
66 | | -- Rendering: uPlot (StationDetail avail chart) supports background |
67 | | - band plugins; mirror the crashes annotations shape — bands behind |
68 | | - series, tooltip/hover shows span detail (metric, duration). |
69 | | -- Also usable for *data* issues (feed gaps — the poller's missed |
70 | | - minutes are visible as absent observations; /health already knows |
71 | | - scrape gaps). |
72 | | - |
73 | | -## Sketch of increments |
74 | | - |
75 | | -1. Worker: `/api/avail-v3/stats?cells=…&from=…&to=…&thresholds=…` — |
76 | | - hist-based (1) + scan-based (2) over ≤ 31d windows. |
77 | | -2. FE StationDetail: issue-bands from (2) + a small "reliability" |
78 | | - stat block (reuse og card's 30d numbers). |
79 | | -3. Group page (post station-sets work): (3) fold + set-level bands. |
| 10 | +Fast answers to "how often / how long is availability bad?" over any station |
| 11 | +set, plus visual annotation of problem windows on avail plots. Driving |
| 12 | +questions (user, 2026-08-29): |
| 13 | + |
| 14 | +- How often is a station **out of bikes** (0 total) or **out of ebikes** |
| 15 | + (0 ebikes)? — the two primary metrics of interest. |
| 16 | +- How often are my 4 closest stations in that state **all at once**? |
| 17 | +- …restricted to **peak commute hours** (e.g. 7–10am weekdays). |
| 18 | + |
| 19 | +(Not a priority: "classic-only" = `ebikes==0 AND bikes>0`. It's the one |
| 20 | +*joint-across-metrics* case and the only reason to touch two metrics together; |
| 21 | +demoted to the footnote below since the questions above don't need it.) |
| 22 | + |
| 23 | +## What already exists (don't rebuild) |
| 24 | + |
| 25 | +- **Station sets.** `/stations` has a multi-select set (`?sel=` URL codec, |
| 26 | + `www/src/pages/Stations.tsx`) and named sets via `neighborhoods.json` |
| 27 | + (`ctbk neighborhoods`). "My 4 closest" is a `?sel=` set or a neighborhood. |
| 28 | +- **Per-metric histograms.** avail-v6 stores five histogram-monoid metrics — |
| 29 | + `bikes`, `ebikes`, `docks`, `disabled`, `pending` — per (`s2_cell`, `dt`) |
| 30 | + (`configs/pyramids/avail-v6.yaml`). GBFS `num_bikes_available` is TOTAL, so |
| 31 | + classic = `bikes − ebikes`. |
| 32 | +- **Serving.** `/api/avail-v3[/cells]?from=&to=&cells=|bbox=&reducer=` (serves |
| 33 | + the default pyramid = v6; the `-v3` in the path is legacy naming). `reducer` |
| 34 | + ∈ `mean|min|max|p05|p25|p50|p75|p95|hist`; `hist` returns full per-metric |
| 35 | + histograms. `/cells` returns one row per station; the rollup route collapses |
| 36 | + `dims` and sums across the set (`gbfs/api/src/avail_geo.ts`). |
| 37 | +- **FE chart.** `StationAvailabilityChart.tsx` (uPlot) already renders a |
| 38 | + station's series. |
| 39 | + |
| 40 | +## Monoidal (cheap, any tier/window) vs. joint (needs a fine scan) |
| 41 | + |
| 42 | +The split is the whole design. A histogram monoid answers questions about ONE |
| 43 | +metric's MARGINAL distribution; anything joint (across metrics, or across |
| 44 | +stations, at a specific instant) is not recoverable from merged histograms. |
| 45 | + |
| 46 | +**Monoidal — cheap, any tier/window — and it covers both primary questions:** |
| 47 | + |
| 48 | +- `% time 0 bikes` for one station over a window = `h_bikes[0] / Σ h_bikes`. |
| 49 | +- `% time 0 ebikes` = `h_ebikes[0] / Σ h_ebikes`. **Also cheap** — it's the |
| 50 | + `ebikes` metric's own bin 0, a separate marginal from `bikes`, so no joint is |
| 51 | + involved. (`0 docks` likewise.) Proven during the OGI work (the og card's |
| 52 | + "% of the time no bikes, last 30d" is exactly this shape). |
| 53 | +- `% time < N` = `Σ_{v<N} h[v] / Σh`. |
| 54 | +- Per-station stats over a set: one `/cells` request, fold each row. |
| 55 | + |
| 56 | + Caveat (unchanged): on the ROLLUP route the histogram mixes stations — |
| 57 | + `h[0]/Σh` reads as "fraction of station-minutes empty", NOT "fraction of |
| 58 | + minutes where ANY/ALL empty". Per-station needs `/cells`. |
| 59 | + |
| 60 | +So "how often is station X out of bikes / out of ebikes, last 30d" is a |
| 61 | +histogram read at any tier — the cheap path, shippable first. |
| 62 | + |
| 63 | +**Joint — NOT monoidal, needs a scan of fine bins:** |
| 64 | + |
| 65 | +1. **Group co-emptiness** (k of N stations at 0 at the same t) — the "are all 4 |
| 66 | + dry at once?" question, and a joint ACROSS stations. Needs per-station series |
| 67 | + aligned per bin, then `k(t) = #{stations : bikes(t)=0}` folded |
| 68 | + client/worker-side. Simultaneity requires bins fine enough that a bin ≈ an |
| 69 | + instant: at a coarse tier a station's bin is a distribution, and even |
| 70 | + `reducer=min=0` only says "some minute in this bin was empty" without telling |
| 71 | + you WHICH minute — so cross-station alignment is lost above the fine tiers. |
| 72 | + The per-station "% time empty" fractions above are cheap; only their |
| 73 | + *simultaneity* costs a scan. |
| 74 | +2. **Spans / runs** ("0 bikes for ≥15 min", count + length distribution) — |
| 75 | + boundaries don't survive histogram aggregation. Same fine-scan path. |
| 76 | + |
| 77 | +Footnote — **classic-only** (`ebikes==0 AND bikes>0`), not a current priority: |
| 78 | +a joint across two metrics. Marginals give `P(ebikes=0)` and `P(bikes=0)` |
| 79 | +separately, never the joint, so it'd need the same fine scan. Left out of the |
| 80 | +increments below. |
| 81 | + |
| 82 | +**Peak-hour slicing** is orthogonal to both: it's a periodic (hour-of-day, |
| 83 | +day-of-week) filter, not a window. Neither the monoid nor a plain window gives |
| 84 | +it — scan fine bins and bucket by `hour_of_day(dt)`. Cheap for bounded windows |
| 85 | +(a month of 1m bins × a handful of stations); wide windows want a coarser bin |
| 86 | +with a conservative reducer, accepting the simultaneity caveat above. |
| 87 | + |
| 88 | +## Increments |
| 89 | + |
| 90 | +1. **Worker `/api/avail-v3/stats?cells=&from=&to=&thresholds=&hours=`** — |
| 91 | + monoidal fractions (1) from `hist` + fine-scan spans (3) over ≤ ~31d |
| 92 | + windows; optional `hours=7-10` weekday-peak filter. Returns per-station |
| 93 | + `{pct_zero_bikes, pct_zero_ebikes, pct_zero_docks, spans:[…]}` — all three |
| 94 | + cheap from `hist` — and, with `group=1`, the co-emptiness histogram |
| 95 | + `k → minutes` (1), the one part that scans. |
| 96 | +2. **FE reliability block** on StationDetail: reuse the og card's 30d numbers |
| 97 | + (% time 0 bikes / 0 ebikes) and, for a `?sel=` set, the group-k summary. |
| 98 | +3. **Issue-bands** on the avail chart: semi-transparent red x-spans over |
| 99 | + outage windows from (3). uPlot supports background-band plugins; mirror the |
| 100 | + crashes annotations shape (`$hccs/crashes` `HomicidesComparisonPlot.tsx` + |
| 101 | + `src/annotations/*` → `useAnnotations` → `toPlotLayers`), bands behind the |
| 102 | + series with hover detail (metric, duration). Doubles for feed-gap bands |
| 103 | + (`/health` already knows scrape gaps). |
| 104 | + |
| 105 | +## Materialization (only if usage warrants) |
| 106 | + |
| 107 | +Spans and co-outage records could be precomputed by a Lambda/engine pass into a |
| 108 | +small parquet/D1 table (start, end, metric, threshold, k), maintained |
| 109 | +incrementally like the pyramid — exact spans, O(1) query, new moving part. |
| 110 | +Start query-time; materialize only if the fine-scan windows people actually |
| 111 | +ask for get too wide. Group-k especially is a candidate: a "co-outage" metric |
| 112 | +keyed by a station SET is the one thing no per-station monoid can ever give. |
0 commit comments