Repository navigation
feat: run analysis on DFO's engine and serve cards and shares from dfo-apis - #170
Merged
Merged
Conversation
The nine analysis verbs, op specs, id minting, execution identity, units, rolling periods and the pure half of the recipe normal form now live in packages/dfo-analysis-engine (module dfo_analysis_engine), copied from dfo-ai's analysis-mcp at eefa15be with their tests, id vectors, goldens and 433 differential cases. Only import lines change, apart from the store half of recipe.py, a Trace protocol in steps.py and pyarrow-stubs suppressions; scripts/diff-moved-engine.sh prints every other changed line against the source commit. Every pinned ops_, ocv_ and op_spec_hash vector matches, and ENGINE_VERSION stays 0.9.0. The 206 recorded refusal cases are skipped until the engine maps its exceptions to codes.
…trees refusal.py holds one table from each verb's engine exceptions to the refusal code, retry mode, message and fix the MCP tools return today, so the 433 differential cases all replay again, refusals included. calls.py is the pure half of each verb tool, and anchor.py resolves the latest period over admitted frames. evaluate(recipe, input_frames, policy=...) runs a recipe tree with one latest-period resolution per tree, refuses rolling branches over different populations (rolling_populations_differ) even when their periods agree, and returns the frame, resolved periods, exclusions, execution identity, the computed tree and a JSON summary of each call's facts.
sum_of and compare take an optional `on` mapping from data's columns to other's, so each side may hold many series and members pair the way quotient_of pairs them. A call without `on` hashes and computes exactly as before, so every existing op spec, vector, golden and differential case keeps its ids and rows, and ENGINE_VERSION stays 0.9.0. compare stays right minus left in the engine; a call names `data` as the right side, so it returns data minus other, and each summary names which side is which. Unmatched members and periods are counted per side, and no matched key refuses compare_no_key_matched or sum_of_no_key_matched.
dfo_platform.caller.resolve_caller names each request's caller without an API key. A request vouched for by the website proxy (x-dfo-caller-principal) is named by its x-dfo-caller digest; any other is named by the X-Forwarded-For entry DFO_TRUSTED_PROXY_HOPS from the right, hashed with the website's preimage (IPv6 folded to its /64) so both routes share one namespace and no raw address is stored. datasets.caller_allowance (migration 0039) counts each caller's pins and shares per UTC day; a spent allowance answers 429 allowance_exhausted with Retry-After at the next midnight. The pin door now admits any caller: grant holders keep the global budget alone, and other callers spend their allowance inside an open-caller ceiling of 200 copies per 24 hours. spec-kit moves to 2765bc3, which registers the new endpoints and codes.
POST /api/datasets/analysis takes a recipe tree whose roots are Specs, exact versions, derived frames or refs, with verb arguments in the caller's words (roles such as origin, destination and place; `on` pairing data with other). The runner reads the roots with their renderings, settles the rolling period once, evaluates the tree with dfo_analysis_engine, digests the output and stores the derived frame and its recipe, answering a format 2 ref. A rerun stores nothing new, and `latest` and exact runs share one path. A root still building answers 202 with Retry-After. Runs have their own pool of 2, hold a materialise permit, and are bounded by nodes, root rows and join rows. POST /api/datasets/analysis/anchor returns the latest period for given roots and stores nothing.
GET /api/datasets/derived/{ops}/{ocv} answers a derived frame's
manifest: retention (pinned or held_until), the roots' merged citation,
the recipe, resolved periods, and left_out, every series dropped for
missing data at any step of the tree, naming the step, input side,
reason and rows. .../file serves its rows as csv, xlsx or parquet,
alone or bundled. The engine now reports each step's drops, and the
runner stores them in the recipe.
POST /api/datasets/analysis also takes a bare derived root, re-running
its stored recipe, so fetching a derived ref with latest=True keeps the
same ops_ and ocv_ when no root moved.
datasets-maintenance diff-recipes re-runs every stored recipe on its
own root versions without writing anything and compares each output
frame's cv_ and output digest with the stored frame's, reporting equal,
differs, roots unreadable and frame unreadable. record-recipe-golden
records recipes and their frames for a test that the runner reproduces
each one; three synthetic cases stand in until dev recipes are
recorded.
`just local-stack up|seed|smoke|down|status` starts Postgres 18 and fake-gcs-server in docker, applies both migrators, and serves the Datasets and Presentation surfaces on 127.0.0.1:58083 with fixed local principals and caller settings. Upstream sources are fixture doubles, so nothing reaches GCP, Gemini or Data Commons. The seed mints four readings through the real mint door (German ODA to Kenya, Ghana and Nigeria; ODA to Ethiopia by donor; Kenya population; Kenya GNI), idempotently, and the smoke script checks each one's versions, manifest, observations and file bytes over HTTP.
artifact_store.signing.sign_get_url returns the storage emulator's media link when STORAGE_EMULATOR_HOST is set, with no credential work, and signs exactly as before otherwise. dfo_apis.app exposes build_presentation_surface, which create_app and the local stack both use, so the local app patches nothing and imports no private name.
…endering Card revisions store the rendering of the frame they draw (component_revision.frame_rendering, migration 0011), read from the version's records on every load, and a render judges its target against that recorded rendering. Card reads and renders carry the frame's retention: pinned, or held_until counted from when DFO stored the frame. An expired frame's refusal names dfo.fetch(..., latest=True). Creating, revising and live-rendering a card pin nothing. presentation_store.pin_card_frames pins a revision's frame when a share first includes it, and pinning a derived frame also pins the root versions its recipe names.
POST /api/presentation/shares publishes cards as a share with an 8-character id and answers its url, embed_url, one data URL per slot and a shw_ write token. latest is required, the caller's daily share allowance is spent, and the cards' frames are pinned when a share first includes them. PUT replaces a live share's cards with the token, keeping each slot's URL by position; a snapshot cannot be replaced (share_not_replaceable). DELETE takes the share down with the token. GET reads a share publicly and never shows the token. The token is checked before the mode, and a share with no token refuses every token. Presentation migration 0012 adds share, share_revision, share_slot and served_frame.
The analysis runner named flow datasets' enrichment columns by role word (origin, destination) while stored identities key them by column (financingProvider), so every analysis over a flow dataset refused column_unknown. It now matches both. The analysis route answers every Datasets code at its datasets.analysis status, with no fallback, now that spec-kit 4bcb6dc maps them. The definition and refresh helpers the runner reads roots through are public (served_spec, serve_spec, answer_frame_refresh). The local stack's smoke now runs a quotient_of over seeded readings, reads the derived manifest and Parquet rows, re-runs the derived root with latest, draws a card on the result, and creates, replaces and deletes a live share with its write token.
GET /api/presentation/shares/{id}/slots/{n}/data serves a slot's rows
as JSON, csv or parquet, publicly and across origins. A snapshot slot
serves its card's stored frame; a live slot draws the card on the
newest data, through the refresh path for datasets and the runner for
derived frames, checking at most once every 15 minutes per slot and
stating current, stale or pending in the body and x-dfo-freshness. A
strong ETag answers If-None-Match with 304.
Each frame a slot serves is recorded, and GET .../history lists them
newest first with their retention and whether each is still fetchable.
POST .../snapshot with the write token turns a live share into a
snapshot in place, pinning what each slot serves. Create takes a kind
(card, canvas, search, conversation) and a website-owned payload.
datasets-maintenance import-shares reads the website's share export
idempotently, keeps each id, and writes a fresh write token for every
signed-in owner's share to a 0600 file the website reads.
A card on an ops_ frame cited "Unattributed source": the frame store cited a derived frame from its rows' provenance refs, which a quotient of two sources leaves null, and a null ref mapped straight to the unattributed source. The frame store now cites a derived frame from its recipe's roots through datasets_api.derived.recipe_citation, the function the derived manifest uses, and a row with no ref stands for every source the manifest cites. A frame cached before its recipe was written is cited again once the recipe exists.
… rules on replace A live slot whose newest frame is still building is checked again after the retry time the refresh path or runner gave, clamped to 1 to 900 seconds, so a finished rebuild shows within that time; current and stale slots keep the 15-minute window. The slot check stores its next check time (migration 0014). Replacing a share applies the per-kind card rule create uses, so a live search share takes no cards. A snapshot share, conversations included, refuses share_not_replaceable before its cards are read.
The derived file route serves a bare Parquet request as the stored canonical frame's bytes, so the body hashes to the ref's artifact digest and clients can check and cache it; csv, xlsx and bundles still render. Runner roots take the groupby a fetched frame was minted with (a StatVar sum or a breakdown) beside without_party_totals and columns. The groupby is part of the root's rendering, so it enters the recipe's normal form, the output digest and the manifest's roots, and the root reads through the same head rendering a definition read uses. recorded_dataset_root is the one reader from a mint record to a root, shared by the runner, recipe replay and the golden.
…nown A card on a derived frame published without steps was refused schema_invalid: a row with no provenance ref took the id of the first cited source, while the card description listed the computed version's inputs, whose ids are null for such a publish. with_card_description now gives a null row a description only when the record knows an input id, and then the row and the description share it; otherwise the card carries no description and its provenance lists the roots' sources. The analysis door builds every refusal through classify_analysis_exception, and the code sweep lists its two engine code passthroughs. The local-stack smoke checks that a card on a derived frame cites the derived manifest's sources in order.
…tryable
GET /api/presentation/shares/{id}/slots/{n} answers the slot's card in
the components read shape, drawn on the frame the slot serves, with the
same freshness, ETag and CORS as the slot data route; both routes take
their frame from serve_on_slot_frame, so they never disagree. A site
can draw a live share without the render principal.
Share create takes an optional client-minted shw_ write token: a retry
with the same token and body answers the first share, and the same
token with another body is refused (migration 0015). Card create
replies carry retention as reads do, and a frame with no version record
states null retention. binding_invalid and constraint_violation answer
retryable: false, since their fix is a corrected request. A share
payload may be up to 10 MiB, on create and on import.
…es its own A derived frame reached clients as the canonical frame, whose unit is a tagged struct, while a dataset version's file carries unit as text. A published derived frame now also stores its bare frame-layout Parquet file through render_and_store, the pipeline dataset files use, and the ref's artifact digest and the publish response name that file. The derived file route serves it, rendering it again from the canonical frame if its bytes are gone, so both kinds of frame read the same way. The canonical frame stays stored for cards, replay and pins.
…hment and site The derived frame file rendered with an optional enrichment and a fixed placeholder site and time. The analysis stores now carry the request's enrichment and site, the publish path passes them and a real prepared time to render_and_store, and render_and_store requires its enrichment again. A bare frame file reads none of them, so its digest is the same whenever the frame is published. A missing-field schema_invalid takes its retryability from retryable_for, like every other refusal, and answers false.
tests/e2e/test_dashboard_journey.py runs one client's path through both surfaces sharing one Datasets cell: a compare over a summed root and a second reading answers a format 2 ref whose file hashes to its digest; a card cites the roots' sources; a live share serves the derived frame with ETags, follows new data after the freshness window, and lists both frames in its history; replace, a wrong token, snapshot conversion and its pins, a retried create, unshare, and an open caller's pins up to the allowance each answer as a client expects. A strict xfail reproduces a derived frame over a summed root that stops following new data. tests/live_refresh/composed_app.py mounts the share routes and the shared cell as the app does, and conformance clause C12 states the pin rule and names the test.
A derived frame over a root read with a groupby sum stopped following new data when, at analysis time, the sum equalled the plain frame: the latest re-run refreshed the root by version alone, and a version that links both renderings is ambiguous once new data separates them, so it refused rendering_not_recorded and a live share served its old frame as stale. LiveRootSource now serves a root under latest through the same path as a fresh Spec root, with the rendering the recipe recorded, so a root is always read by version and rendering.
…wnloads spec-kit 5a1959de maps the share routes' refusals: an oversize share payload answers 413 payload_too_large before any allowance is spent, an I/O failure on a share route answers retryable 503 service_unavailable, and pin_not_permitted is retired with the PinNotPermitted refusal. The presentation surface classifies each route by its own door. A card's select lists are a set: canonical_select sorts them and drops repeats before an intent is stored, so the same intent draws the same table whatever order a client sends. A derived frame's card takes its default title from the frame's recipe, naming each composed variable from its roots' labels. The local stack serves signed download links on the API origin and answers the graph node reads card citations make, so clients need no link rewriting and card creates no longer wait on a refused read.
… over a population The analysis route published derived frames without steps, so their records named no inputs and their cards were titled by ops_ id. Each verb call now builds its step with steps.build_step (call_steps.py), and the runner publishes the steps with a retrieve per root, so a derived card reads like one the MCP published. A derived frame's party and type columns stay out of card series labels when their role column is present. A request may carry anchor_roots, the roots /analysis/anchor was asked about, so a rolling call over one side of a paired population resolves its latest period over the whole population. They enter the recipe's normal form and replay key. A token holder's repeat DELETE of a share answers 204.
GET /api/presentation/shares/{id}/slots/{n}/data takes the formats and
bundle option the components data route takes, through the same
export_component_data call on the frame the slot serves, so a live
card's bundle and spreadsheet downloads follow new data. A bundle of
json is refused.
An analysis step list over the byte bound sheds its coverage members,
largest first, until it fits (call_steps.shed_coverage), and the frame
is published with those steps.
Analysis sums the roots' recorded rows and refuses past the root-row ceiling before reading a frame, and refuses an output past the publish ceiling before storing it. Columns are put in one order in mint_options, so equal requests give one normal form; a stored pair whose recipe or bytes differ answers 500 internal_error and logs both identities. The latest period comes from the engine's latest_period_of. A request whose caller-principal header does not match DFO_CALLER_PRINCIPAL is refused 503, where it fell back to the address and pooled every website user under one allowance. The pin budgets and the caller's allowance are spent together in one transaction per UTC day (migration 0040), so concurrent pins cannot pass a ceiling, and Retry-After counts to the next UTC midnight.
A replace refuses under the row lock when the share became a snapshot or its revision moved. A slot check takes one claim, the first check included, and the slot keeps its last frame, freshness and check time while the claim runs; a dead claimer's claim lapses after twice the components timeout (migration 0016). Create, replace, snapshot and a replayed create record each slot's frame, so history lists a new share at once. import-shares pins every imported card's and slot's frame as create and snapshot do, and writes and fsyncs each token line before the share commits. A row with no provenance takes a derived frame's cited source on ops_ frames only; a dataset version keeps "Unattributed source". The shares door's public GET routes answer a cross-origin preflight with If-None-Match and keep their ETag and freshness headers through the platform CORS layer. Unsigned emulator links need DFO_LOCAL_STACK=1, which the local stack sets.
POST /api/datasets/analysis keys each request before reading a frame (the working tree, the resolved root records, latest and the execution identity) and stores its 200 body in datasets.analysis_answer (migration 0041). A request whose answer is stored, with its frame file still served, answers at once; one whose key is running answers 202 with Retry-After 5, so a client retrying after a 504 joins the one run and then reads the stored answer. The shares door waits up to 5 s for a free thread before refusing 503, so a share page with more slots than threads loads.
A share's history records an entry whenever what a slot shows changes, including a return to a frame it showed before, and an empty entry (no card, revision or frame, never fetchable) when a write leaves the slot without a card. A client asking what a slot showed after a replace emptied it reads that it showed nothing. Migration 0017 keys served_frame by an identity column and lets the card and frame columns be null together.
Creating or revising a card spends one unit of the caller's daily card allowance (DATASETS_CARD_DAILY_ALLOWANCE, default 500; migration 0042); a principal holding the publish grant spends none. A share create that fails after spending, at the pin or the store, gives its unit back. A live share holds at most 20 cards that draw a frame, checked on create and replace. A slot read takes the history lock only when what the slot shows changed, and a slot that becomes a note records that it shows no frame.
…30968 datasets-maintenance normalise-recipes sorts each stored recipe root's columns, as the analysis route now writes them, recomputes its normal-form hash and replay key, and moves the presentation revisions that carry the old hash, then the recipe row; --dry-run counts, a second run changes nothing, and a recipe it cannot reproduce or whose new key another row holds is reported and left. Run it after import-shares. A card create replayed with its write token spends no card unit. spec-kit is pinned to 3e30968, which maps allowance_exhausted on presentation writes.
jm-rivera
changed the title
… engine Analysis counts nodes as it walks, bounds anchor reads by root rows, refuses malformed ref members and rolling shapes, cuts the left-out list to the recipe's byte bound, and refuses a failed rebuild in place of waiting on it; GET on the analysis paths answers 405, and diff-recipes keys stored roots by record. Live share checks run analyses through the stored-answer path and share one running set with the analysis route, and refresh under the card's recorded rendering. Sharing a derived card checks its roots first, pins its frame, and pins each root still held; a frame with no recipe yet refuses retryable recipe_pending. Stated retention follows the frame's own version and revocation, and card reads and renders answer null when it cannot be read. A card records its rendering at create. A share create answers once its write commits, history records what each slot serves and pages by keyset (migration 0018), every GET on a public path carries Access-Control-Allow-Origin, a NaN payload is refused before any spend, and a conditional slot read answers 304 from its inputs. settle leaves caller keys alone and locks out concurrent pins, a pin racing to an existing copy refunds its units, pin budgets name their reset, and a derived frame's rendered file is linked to its version. call_verb refuses op_spec_invalid unless its inputs fill exactly the slots the op spec names, an `on` list holding anything but column names refuses op_spec_invalid, and a non-object rolling predicate refuses op_spec_invalid. A paired side with several rows per member and period refuses role_ambiguous naming that side and its entity binding. Any `on` makes rolling pairing explicit, so an anchor never reads flat populations the call does not use. A paired summary names the fields the single-series summary does, and a step that cannot be built is logged and left out of a computed result. Compare's data side lives in one table, and the engine tests run in every shard of the suite.
3 tasks
The local app builds one Datasets link and passes its cell and running analyses to both surfaces, as dfo_apis.app does, so the stack starts again and live share checks join analysis runs.
This was referenced Oct 29, 2026
Closed
minted_variable_names names a minted variable mv_ plus the first 32 hex
of SHA-256 over the RFC 8785 JSON of {verb, slot: sorted input names}.
A name document holds its inputs' names, so a name is 35 characters at
any depth; the JSON-text form doubled with each minting step.
_stored_file_digest reads through read_servable_version, so a cached
answer or a run that reuses a stored pair refuses snapshot_revoked (410)
when that output is revoked.
refund_together locks the allowance row before choosing between delete
and decrement, so a spend that commits between the two no longer leaves
the failed operation charged.
create_share, replace_share and snapshot_share take each card's rendering and write every slot's served_frame entry after the revision insert, under the share row lock. The revision's created_at is read from clock_timestamp() while that lock is held, and each entry takes it as first_served_at, so history lists writes in commit order and as_of between two replaces selects the first. A write whose history insert fails commits nothing and answers 503. The door's post-commit _record_shown and _record_slots, and the store's record_emptied, are removed. record_served takes the same lock and records only while the latest revision still shows that card in that slot, so a read that loaded an older revision adds nothing after a replace.
frame_ref_for puts "without_party_totals": true in the cv_ preimage when the cut is set and sorts groupby.by, and mint_options sorts by, so a request with the cut and one without, over a dataset the cut leaves unchanged, name two pairs and both answer 200; a replay of the cut recipe keeps the cut. data-mcp in dfo-ai computes the same pairs, and both repos pin cv_bb722d06259f357e6961a081f1c1f141 for one input. When a run's own pair holds no frame, _store answers the pair read_replay names for its recipe, so a frame stored under the earlier naming is served instead of refusing ReplayKeyHeld. The ref, the top tree node and any divergence report name the pair the frame is stored under.
copy_to_pinned (datasets_api.retention_admission) is the one path into pinned storage: the pin door's artifact, and each frame and derived root a share create, replace or snapshot conversion pins. It spends the global pin budget, the open-caller budget and the caller's pin allowance for each new physical copy, skips copies that already exist, and refunds only a copy that turned out deduplicated or whose source is gone. A copy that stands keeps its units when the share write fails. Share routes answer PinBudgetExhausted and OpenPinBudgetExhausted as 503 with Retry-After, and allowance_exhausted as 429. LiveFrameStore takes the ledger as a required argument; share_import passes None. DATASETS_PIN_DAILY_ALLOWANCE defaults to 200. datasets-maintenance take-down-share <share_id> takes a share down as the owner's delete does, for the shares the website records in its share_cleanup table; an id with no share up exits 0. normalise-recipes also sorts groupby.by. An imported share records each slot's history entry, with its card revision's rendering, in the import transaction.
b7a40ad names minted variables mv_ plus a fixed-size digest and admits a derived ref root that carries the groupby DFO read it with.
When an analysis run answers the pair its replay key names, every field of the reply (ref, tree, summary and both frame_content_version values) names that pair and describes the stored frame. If that pair's file is no longer servable, supersede_replay marks the old recipe superseded and the run publishes under its own pair; migration 0043 adds frame_recipe.superseded_at and keeps replay keys unique among recipes not superseded. record_served records only while the revision the read served is still the latest, so a read in flight during a replace or snapshot conversion adds nothing. The shares door passes its allowance ledger to the frame store it builds, and live.py builds none of its own. Pin defaults: 100 new copies per caller a day, 2,000 for open callers together and 2,500 in all, so one caller can spend at most 5% of the open budget. Docs and refusal text say share routes spend them too.
0cb55e3 adds a nested minted-name vector with name documents and records the grouped-root ref vectors from a dfo-apis run, with by sorted.
A share replace pins the frames it adds against its caller's pin allowance, so the smoke PUTs send X-Forwarded-For as the create does.
# Conflicts: # src/datasets_api/README.md # src/datasets_api/config.py
# Conflicts: # packages/dfo-data-tools/pyproject.toml # pyproject.toml # uv.lock
jm-rivera
changed the title
jm-rivera
added a commit
that referenced
this pull request
Oct 30, 2026
… frames by spec_id, and roll a pairing over both sides (#194) Adds a lasting download link for any frame, fetched or derived, and fixes how rolling analysis counts the two sides of a pairing. ## Exports - `POST /api/datasets/exports` takes `{frame: {spec_id, content_version}, format: csv|xlsx|parquet, bundle}`. It renders the file through the existing file writers, pins it, and records it under `export_id` = `exp_` + the first 32 hex of the SHA-256 of the RFC 8785 form of the request. A repeat request returns the stored record, so it renders nothing and spends nothing. Callers are admitted as the pin door admits them. A new pinned copy spends the caller's `pin` allowance; a file already pinned spends nothing, even when two prepares race for the last unit. Each export holds an `artifact_reference` row of its own kind, so the reconciliation sweep counts it. A rendering linked incomplete refuses retryably, with nothing pinned or recorded. - `GET /api/datasets/exports/{export_id}` answers 302 to a five-minute signed URL on the pinned tier, with `Cache-Control: no-store`. It never renders. One guard runs on a prepare, a repeat prepare and the link: a version revoked under any bound id, a quarantined identity, or a revoked root of a derived frame refuses. The response to a prepare never holds a signed URL. - Both routes check `licence_restrictions.json`, a deny-list keyed by source ref. It ships empty: no source's terms are recorded yet, and today's file routes already serve every dataset. - csv and xlsx stop at 250,000 rows (`over_ceiling`). An `ops_` frame with no stored recipe, or with a root that cannot be cited, refuses `provenance_unavailable`. - Migration `0044_export.sql` adds the exports table. The website's datasets proxy already forwards `/api/datasets/*`, so the public link is `https://developmentfinance.org/api/datasets/exports/{export_id}`. - spec-kit is pinned to DevelopmentFinanceObservatory/spec-kit `2dd2400` (the `datasets.exports` and `datasets.export_link` codes, with the rebuild refusals on `datasets.exports` and `identity_quarantined` on the link); merge spec-kit#37 with a merge commit so the pin stays on its dev history, or re-pin to the squash commit. ## Frames by spec_id - `GET /api/presentation/frames/{spec_id}` resolves the newest served version as the analysis runner resolves a Spec root (`LiveRootSource.resolve(SpecRoot(spec_id))`), honouring a revocation under the id sent, and answers the pair route's body over that version's canonical frame, plus the `spec_id` and `content_version` it read. A version still building answers 503 `frame_unavailable`; a rebuild that failed for good answers 500. A rebuild larger than the route's wait goes to an operation, so a retry finds it. The route runs on its own executor (`PRESENTATION_FRAMES_BARE_MAX_CONCURRENCY`, default 2), so a slow read cannot take the pair route's permits. - `POST /api/datasets/exports` accepts `frame: {spec_id}` alone and resolves it through `analysis_roots.newest_spec_version`, the same resolve the bare frames route uses. The export record keeps the id sent, so a later takedown of that id reaches the link. A rebuild refusal registered on `datasets.exports` passes through unchanged. - `frame_unavailable` carries its reason as `cause.code`, so a client can tell a frame it may rebuild from a revoked or quarantined one. - `POST /api/presentation/components` accepts an intent whose `frame_ref` holds only a `spec_id` and stores the resolved pair. A create retried with its write token replays its first reply. ## Default enrichment - `datasets_api.default_enrichment` holds one rule for a frame's default type and party columns: the `_type` column of each of `observationAbout`, `financingProvider` and `financingRecipient` among the version's coverage dimensions, plus `party_kind`, `party_class` and `party_whole` when a party role is present. It reads the roles the version holds, so a flow identity that binds no role gets its party columns where the identity-based rule asked for `observationAbout_type` and refused `column_unknown`. Frames that resolved before keep their columns and names. - A mint takes `options.enrichment: "default"`, resolved against the served version before the rendering is keyed, so the answer equals a mint that names those columns. Unset options dump and key as before. Spec roots send it and serve once. dfo-ai's public `get_data` sends it. - A fetched-frame export renders with the default enrichment, so the file carries each role's `_type` column beside its `_label` and `_kind` cells. ## Rolling over a whole tree - `dfo_analysis_engine.rolling._sides` found a node's frame slots from its `*_frame_ref` parameter names. The analysis runner removes those from its working tree, so a compare or join took its rolling populations and latest period from the left side alone. It now reads the slots from the node's `inputs` too, and `ENGINE_VERSION` moves to 0.9.1, so an answer stored before this fix is computed again. A narrowing above a two-sided verb crosses the verb's key map to the other side. A `join` with no `join_keys` counts each side flat, as `compare` and `sum_of` with no `on` do, because no tree holds the frames the engine detects keys from. A stored recipe names its slots in its parameters, so a recipe the runner stores with this fix replays on `latest` to the same frame (`test_a_rolling_step_over_a_two_sided_verb_stores_its_slots_and_replays_on_latest`). A two-sided rolling recipe the runner stored before this fix, since #170 deployed, can hold a left-only population list; its `latest` replay refuses `rolling_anchor_unresolved` on dev today and still does after this change. - `POST /api/datasets/analysis` takes `rolling: "tree"`. With it, each node expanded from a derived root drops its overlay population, its `anchor_roots` and the literals its overlay governs, so the whole tree takes one population list and one latest period. dfo-ai's public analysis tools send it. Without it a request is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #175
Datasets runs analysis on DFO's own engine, and Presentation serves cards and shares that follow new data.
POST /api/datasets/analysistakes a recipe tree whose roots are Specs, exact versions, derived frames or refs, with arguments in role words. It evaluates the tree withdfo_analysis_engine, stores the derived frame and its recipe, and answers a format 2 ref. Requests are keyed before any read and their answers stored (datasets.analysis_answer), so a retry joins the running request. Runs are bounded by nodes as the tree is walked and by root rows before any read.GET /api/datasets/derived/{ops}/{ocv}answers a derived frame's manifest (retention, citation, recipe, periods, andleft_out, every series dropped for missing data), and.../fileserves its rows as the stored Parquet file or as csv, xlsx or a bundle.The analysis engine moves into dfo-apis as
packages/dfo-analysis-engine(moduledfo_analysis_engine). The nine verbs, op specs, id minting, execution identity, units, rolling periods and the pure half of the recipe normal form are copied from dfo-ai's analysis-mcp with their tests, id vectors, goldens and 433 differential cases;scripts/diff-moved-engine.shprints every line that changed beyond imports against the source commit. Every pinnedops_,ocv_andop_spec_hashvector matches, andENGINE_VERSIONstays 0.9.0.refusal.pymaps each verb's engine exceptions to the refusal the MCP tools return, andevaluate(recipe, input_frames, policy=...)runs a recipe tree with one latest-period resolution per tree.sum_ofandcomparetake an optionalonmapping, so each side may hold many series; a call withoutonhashes and computes as before.The output digest (
out2) names each row'sstat_var. A variable that compare, sum_of, share_of or quotient_of writes is namedmv_plus a digest of its verb and its inputs' names, fixed size at any depth (output_digest.minted_variable_names), so equal computations over equal variables share a digest across recipes, engine releases and rebuilds. A frame pair'scv_preimage includes the party-totals cut and sortsgroupby.by, as dfo-ai's data-mcp computes it, so a request with the cut and one without name two pairs. A run whose pair holds no frame answers the pair its replay key names, and supersedes that recipe when its file has aged out (frame_recipe.superseded_at). A cached answer or reused pair whose output is revoked refuses 410snapshot_revoked.dfo_platform.caller.resolve_callernames each caller without an API key, by the website's vouchedx-dfo-callerdigest or the trustedX-Forwarded-Forentry.datasets.caller_allowancecounts each caller's pins, shares and cards per UTC day. A spent allowance answers 429allowance_exhausted, not retryable, with Retry-After naming the reset, and a refund locks the counter row first. Every new copy into pinned storage (pin, share create, replace and snapshot conversion) goes throughretention_admission.copy_to_pinned. It spends the caller's pin allowance (default 100 a day), the open callers' budget (2,000) and the global budget (2,500), skips copies that exist, and refunds only a deduplicated copy or a missing source.Card revisions record the rendering of the frame they draw, and creating or rendering a card pins nothing. A share (
/api/presentation/shares) publishes cards with ashw_write token that replaces, snapshots or takes it down, and pins each frame on first inclusion. A derived card pins its frame and each root still held. Public slot routes serve each slot's card and rows with freshness, ETags and CORS, and a live slot checks for new data at most every 15 minutes through the analysis route's stored-answer path. History records what each slot serves, written in the transaction that writes each revision and dated in commit order, and pages newest first (limit,before,next).datasets-maintenance take-down-share <id>takes down, without its token, the shares the website records inshare_cleanup.just local-stack up|seed|smoke|downruns both surfaces on Postgres 18 and fake-gcs-server with fixture upstreams, for the SDK and website journeys.Deploying: DevelopmentFinanceObservatory/infra#128 mounts
DFO_CALLER_HMAC_KEYandDFO_CALLER_PRINCIPALand must apply first. The deploy workflow applies Datasets migrations 0039 to 0043 and presentation migrations 0011 to 0018 before traffic. After it, rundatasets-maintenance import-shareswith the website's export, thendatasets-maintenance normalise-recipes. The website (DevelopmentFinanceObservatory/website#77), the SDK and dfo-ai's runner go out after this deploys. The spec-kit pin is 6b606ec on spec-kit dev (DevelopmentFinanceObservatory/spec-kit#30).Left for follow-ups: a row count in the derived manifest (#176), several source refs per card series (#177), and reclaiming pinned copies nothing references (#183). Nothing deletes pinned storage today, so the daily budgets bound how fast it grows.
Testing: the CI jobs, with
DFO_SPEC_KIT_REQUIRED=1overtestsand the engine package after merging dev (43,844 and 1,625 passed), including the engine's id vectors, goldens and differential cases and every output-digest and operation-outputs vector in the client contract. 43 Datasets and 18 presentation migrations applied on Postgres 18, a re-run was a no-op, and the live store tests passed, including the refund race and history ordering under row locks. On the local stack the smoke passed 22 checks, the SDK journeys passed (Python 13, TypeScript 7, a grouped-root share in both, and the crossing test), the share journey through the website proxy passed 3 of 3, and the website's share migration round trip passed./code-review xhighran over every commit, and an outside review's findings are fixed here or listed above.