Repository navigation
Conversation
dfo_platform.caller.resolve_caller names each request's caller without an API key. A request vouched for by the website proxy (x-dfo-caller-principal) is named by its x-dfo-caller digest; any other is named by the X-Forwarded-For entry DFO_TRUSTED_PROXY_HOPS from the right, hashed with the website's preimage (IPv6 folded to its /64) so both routes share one namespace and no raw address is stored. datasets.caller_allowance (migration 0039) counts each caller's pins and shares per UTC day; a spent allowance answers 429 allowance_exhausted with Retry-After at the next midnight. The pin door now admits any caller: grant holders keep the global budget alone, and other callers spend their allowance inside an open-caller ceiling of 200 copies per 24 hours. spec-kit moves to 2765bc3, which registers the new endpoints and codes.
POST /api/datasets/analysis takes a recipe tree whose roots are Specs, exact versions, derived frames or refs, with verb arguments in the caller's words (roles such as origin, destination and place; `on` pairing data with other). The runner reads the roots with their renderings, settles the rolling period once, evaluates the tree with dfo_analysis_engine, digests the output and stores the derived frame and its recipe, answering a format 2 ref. A rerun stores nothing new, and `latest` and exact runs share one path. A root still building answers 202 with Retry-After. Runs have their own pool of 2, hold a materialise permit, and are bounded by nodes, root rows and join rows. POST /api/datasets/analysis/anchor returns the latest period for given roots and stores nothing.
GET /api/datasets/derived/{ops}/{ocv} answers a derived frame's
manifest: retention (pinned or held_until), the roots' merged citation,
the recipe, resolved periods, and left_out, every series dropped for
missing data at any step of the tree, naming the step, input side,
reason and rows. .../file serves its rows as csv, xlsx or parquet,
alone or bundled. The engine now reports each step's drops, and the
runner stores them in the recipe.
POST /api/datasets/analysis also takes a bare derived root, re-running
its stored recipe, so fetching a derived ref with latest=True keeps the
same ops_ and ocv_ when no root moved.
datasets-maintenance diff-recipes re-runs every stored recipe on its
own root versions without writing anything and compares each output
frame's cv_ and output digest with the stored frame's, reporting equal,
differs, roots unreadable and frame unreadable. record-recipe-golden
records recipes and their frames for a test that the runner reproduces
each one; three synthetic cases stand in until dev recipes are
recorded.
`just local-stack up|seed|smoke|down|status` starts Postgres 18 and fake-gcs-server in docker, applies both migrators, and serves the Datasets and Presentation surfaces on 127.0.0.1:58083 with fixed local principals and caller settings. Upstream sources are fixture doubles, so nothing reaches GCP, Gemini or Data Commons. The seed mints four readings through the real mint door (German ODA to Kenya, Ghana and Nigeria; ODA to Ethiopia by donor; Kenya population; Kenya GNI), idempotently, and the smoke script checks each one's versions, manifest, observations and file bytes over HTTP.
artifact_store.signing.sign_get_url returns the storage emulator's media link when STORAGE_EMULATOR_HOST is set, with no credential work, and signs exactly as before otherwise. dfo_apis.app exposes build_presentation_surface, which create_app and the local stack both use, so the local app patches nothing and imports no private name.
jm-rivera
changed the title
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Datasets gains the analysis route, derived-frame reads and per-caller allowances.
POST /api/datasets/analysistakes a recipe tree whose roots are Specs, exact versions, derived frames or refs, with verb arguments in the caller's words (roles such as origin and destination,onpairing data with other). The runner reads the roots with their renderings, settles the rolling period once, evaluates the tree withdfo_analysis_engine, digests the output and stores the derived frame and its recipe, answering a format 2 ref. A rerun stores nothing new, a root still building answers 202 with Retry-After, and runs have their own pool of 2 bounded by nodes, root rows and join rows.POST /api/datasets/analysis/anchorreturns the latest period for given roots.GET /api/datasets/derived/{ops}/{ocv}answers a derived frame's manifest: retention, the roots' merged citation, the recipe, resolved periods andleft_out, every series dropped for missing data at any step..../fileserves its rows as csv, xlsx or parquet.datasets-maintenance diff-recipesre-runs every stored recipe on its own root versions and compares outputs without writing anything.dfo_platform.caller.resolve_callernames each request's caller without an API key: by the website'sx-dfo-callerdigest whenx-dfo-caller-principalvouches for it, otherwise by theX-Forwarded-ForentryDFO_TRUSTED_PROXY_HOPSfrom the right, hashed with the website's preimage.datasets.caller_allowance(migration 0039) counts each caller's pins and shares per UTC day; a spent allowance answers 429allowance_exhausted. The pin door admits any caller.just local-stack up|seed|smoke|downruns Postgres 18 and fake-gcs-server with both surfaces on 127.0.0.1:58083 and fixture upstreams, seeds four readings through the real mint door, and checks them over HTTP.artifact_store.signing.sign_get_urlreturns the emulator's media link whenSTORAGE_EMULATOR_HOSTis set.Second of four stacked PRs, on top of the engine move.
Testing: covered by the full suite on the top layer; this layer's own tests are in
tests/datasets_api,tests/platformandtests/fixtures/analysis_golden.