feat(contract): derived refs, output digests and the dashboard routes' codes

python/client-contract/CHANGELOG.md

@@ -11,9 +11,9 @@ All notable changes to the `dfo-spec-kit-client-contract` Python package will be
 - `strict-fields.json` and its schema: 29 members that stay required and non-nullable, each one a client reads, and the answer kinds, part statuses and operation statuses the clients branch on.
 - `actions.json` and its schema: the caller's next step for each of the 104 active refusal codes on `problem_details` or `no_data_reason`, with the gap kind of each no_data reason.
 - The status fallback for a refusal whose code has no row: 400 and 422 to `InvalidRequestError`, 401 to `AuthenticationError`, 403 to `PermissionDeniedError`, 404 and 410 to `NotFoundError`, 408 to `DfoTimeoutError`, 429 to `RateLimitError`, and any other 4xx or 5xx to `InvalidRequestError` or `ServerError`.
-- `scenario.schema.json`, the scenario record, and 125 scenarios: 34 recorded against the deployed dfo-apis dev app at `d80df9f5f`, and 91 synthetic, each derived from a recorded one by a one-line edit, with 21 binary bodies totalling 1,134,603 bytes and records and bodies together 13,693,847 bytes. `search-place-ambiguous-twice` pins two entity_ambiguous groups of one text, each option and slot token naming its group by `entity_choice`, and `choose-refused-label-in-two-groups` pins that a label both groups offer matches two options, so a choice there names its option by index. The scenarios cover the several-part choice entries and the assumptions each identity entry echoes, a picked reading whose assumption Search states in its answer, the place choice through a no_data part's options, the choice refusals, a part without variable_text, an unknown status without a restatement, a shared label, a one-part option whose data_query holds two parts, the separate and operations echo, an unsupported part's placeholder, the timed_out, unsupported and unknown gaps, an unknown answer kind, the `SearchWarning` rule, the stacked dataset's `part` column, the frame layout, a strict fetch's `DataWarning`, a spec fetch that rebinds one of two entity roles and keeps the other, one that drops a role bound to null, a spec whose destination is a named group fetched with a window or with its origin rebound, whose sentence lookup posts the groups of the roles the build left bound, a twice cut-off build, a cut-off build retried after its Retry-After, a build refused 504 then 503 and one refused 503 then 504, a refused build, a build refused 501 `capability_unavailable`, a digest mismatch, a signed URL refused on both rounds, a relative files-route Location, an artifact url on a foreign origin, on the storage host and absolute on the API's origin, a manifest definition without identity or schema_version, a Search spec without identity or resolved, nested specs no rule reads, an empty parts list, a Search body that is not an object, a codeless 503 problem document that is not retryable, a 401, an identity resubmit answering another spec, a part whose status is not a string, a Search spec whose definition names another spec_id, a Search response with no JSON body, an option with no data_query, empty entities and facets, a Sunset header issued once, `x-dfo-warning` messages and a Deprecation and Sunset pair, an unreadable versions page, a mint and a spec fetch whose build answers 202 and is polled to done, a resumed operation, a rate-limited poll, 503 polls retried within the budget an answered poll restores, a 202 that reads running, polls whose `retry_after_ms` changes from one to the next, a 404 poll that raises, a failed operation posted again, a failed operation with no error, and a ref whose stored file expired, read through the file selector, rebuilt at the same version, at a newer one, or refused with the newest version read from the versions route: the first version it lists with no `base_content_version`, the newest plain build, so a derived version listed before it is skipped.
+- `scenario.schema.json`, the scenario record, and 126 scenarios: 34 recorded against the deployed dfo-apis dev app at `d80df9f5f`, and 92 synthetic, each derived from a recorded one by a one-line edit, with 21 binary bodies totalling 1,134,603 bytes and records and bodies together 13,695,441 bytes. `search-place-ambiguous-twice` pins two entity_ambiguous groups of one text, each option and slot token naming its group by `entity_choice`, and `choose-refused-label-in-two-groups` pins that a label both groups offer matches two options, so a choice there names its option by index. The scenarios cover the several-part choice entries and the assumptions each identity entry echoes, a picked reading whose assumption Search states in its answer, the place choice through a no_data part's options, the choice refusals, a part without variable_text, an unknown status without a restatement, a shared label, a one-part option whose data_query holds two parts, the separate and operations echo, an unsupported part's placeholder, the timed_out, unsupported and unknown gaps, an unknown answer kind, the `SearchWarning` rule, the stacked dataset's `part` column, the frame layout, a strict fetch's `DataWarning`, a spec fetch that rebinds one of two entity roles and keeps the other, one that drops a role bound to null, a spec whose destination is a named group fetched with a window or with its origin rebound, whose sentence lookup posts the groups of the roles the build left bound, a twice cut-off build, a cut-off build retried after its Retry-After, a retryable refusal whose Retry-After passes the 60-second cap, a build refused 504 then 503 and one refused 503 then 504, a refused build, a build refused 501 `capability_unavailable`, a digest mismatch, a signed URL refused on both rounds, a relative files-route Location, an artifact url on a foreign origin, on the storage host and absolute on the API's origin, a manifest definition without identity or schema_version, a Search spec without identity or resolved, nested specs no rule reads, an empty parts list, a Search body that is not an object, a codeless 503 problem document that is not retryable, a 401, an identity resubmit answering another spec, a part whose status is not a string, a Search spec whose definition names another spec_id, a Search response with no JSON body, an option with no data_query, empty entities and facets, a Sunset header issued once, `x-dfo-warning` messages and a Deprecation and Sunset pair, an unreadable versions page, a mint and a spec fetch whose build answers 202 and is polled to done, a resumed operation, a rate-limited poll, 503 polls retried within the budget an answered poll restores, a 202 that reads running, polls whose `retry_after_ms` changes from one to the next, a 404 poll that raises, a failed operation posted again, a failed operation with no error, and a ref whose stored file expired, read through the file selector, rebuilt at the same version, at a newer one, or refused with the newest version read from the versions route: the first version it lists with no `base_content_version`, the newest plain build, so a derived version listed before it is skipped.
 - `client-codes.json` and its schema: the 23 codes a client sets on the errors it raises itself, each with its error class, retryable flag and condition, disjoint from `actions.json` and `registries/codes.json`, and the `client_codes()` and `client_codes_schema()` accessors. Every scenario error the client raises itself carries its code, and `refusal_without_code` names a refusal whose body carries no code.
-- `proxy-codes.json` and its schema, the website proxy's `rate_limited` and `transient_unavailable` codes with their statuses, which the proxy sends in problem documents of the shape every refusal has, both retryable, with `Retry-After` on the 429, so a client reads one refusal shape, and `warning-codes.json` and its schema, the 6 codes a client sets on the warnings it issues itself, with the `proxy_codes()` and `warning_codes()` accessors.
+- `proxy-codes.json` and its schema, the website proxy's `rate_limited` and `transient_unavailable` codes with their statuses, which the proxy sends in problem documents of the shape every refusal has, both retryable, with `Retry-After` on the 429, so a client reads one refusal shape, and `warning-codes.json` and its schema, the 7 codes a client sets on the warnings it issues itself, with the `proxy_codes()` and `warning_codes()` accessors.
 - `ref_digest_changed` in `warning-codes.json`, a `DfoWarning` for a ref member whose `content_version` serves a file with another digest than the ref's, because the server rendered the same data again. The client returns the data, with no error under strict.
 - `timed_out` in `actions.json` and `registries/codes.json`, the no_data reason Search gives a part it left unresolved at the request's time limit, with action `ask_data_question` and gap kind `timed_out`, which `actions.schema.json` now admits. The client-side `timeout` code, a transport timeout, is unchanged.
 - A no_data part's gap kind comes from its own reason, whatever the answer's `diagnostics.timed_out` says. `answer-timed-out` gives its part the reason `timed_out`, and `answer-timed-out-keeps-reasons` pins a part that keeps `variable_not_resolved` as a `not_understood` gap in a timed-out answer.
@@ -31,5 +31,13 @@ All notable changes to the `dfo-spec-kit-client-contract` Python package will be
 - The `base-url` rule states the whole validity rule, and seven new vectors pin it: no query, fragment, `.` or `..` segment, character outside visible ASCII, empty userinfo or non-decimal IPv4 host; trailing slashes dropped; a relative URL valid only in a TypeScript client inside a browser page.
 - The `base-url` rule also refuses a `.` or `..` segment written with `%2e`, a backslash, and a port outside 1 to 65535 or written other than in decimal digits, an empty one included, so a client on Python's `urlsplit` and one on the WHATWG URL parser accept the same values. Ten vectors pin them.
 - The `file-selector` rule and its vectors: the file selector's query holds the `format`, `bundle` and `layout` of a ref member's rendering, the keys `query_keys` lists, in that order, and leaves out `without_party_totals`, which the selector refuses with 400 `request_invalid` because the version records its own cut. `fetch-ref-expired-file-selector` and `fetch-ref-artifact-unlisted` ask that query, and the validator and the capture build it from `query_keys`.
+- Ref format 2 in `dataset-ref.schema.json`: `format_version: 2` and one or more derived members, each `{parts, sentence, spec: {spec_id, recipe: {normal_form_hash, replay_key}}, content_version, output_digest, roots, execution_identity, resolved_periods, artifact}`, with an `ops_` id, an `ocv_` version, `rnf_` and `rpk_` recipe hashes, an `out_` output digest, the root dataset versions with the renderings they were read with, and each period stated relative to the latest with the year it resolved to. Format 1 is unchanged and holds dataset members only, and format 2 holds no dataset member. A derived member's `parts` is empty. A scenario's fetch-by-ref step admits a format 1 ref only, and a format 2 ref fails there on its `format_version`. Nine new ref examples, two accepted and seven rejected, each derived identity computed by the dfo-apis analysis runner. The validator checks a derived member's roots against the scenario it names, each root's rendering as the posted one in the frame layout, and that the derived members of the ref examples and the `output-comparison` vectors agree wherever one identity fixes another.
+- A derived member's root records its rendering in `root_rendering`, the artifact selector's members and `groupby`, the sum DFO read the root with as `options.groupby` states it: `by`, one or more distinct column names, `op` `sum`, and `levels`, a depth from 0 to 99 for each kept hierarchical axis, absent when none is set. A breakdown or a sum over StatVars reads a root that way. The artifact's `rendering` admits no `groupby`. `ref-derived-grouped-root` is accepted, its `by` sorted as DFO records it and its identities computed by the dfo-apis analysis runner over the scenario's rows summed to their purpose's sector code, and `ref-derived-root-groupby-op-rejected` and `ref-artifact-groupby-rejected` are rejected. The validator checks a summed root's rendering against its scenario's build without its `groupby`, which no recorded build posts.
+- The analysis and publication rules DFO applies, each with vectors and a reference implementation in `tests/test_analysis_rules.py`: `output-digest`, the `out2` digest of a derived frame's values with the `stat_var`, members, period and unit each belongs to, without `measurementMethod`, `provenance`, `status` or a unit's `recipe`, reading the variable `compare`, `sum_of`, `share_of` or `quotient_of` writes under an `mv_` name hashed from its verb and its inputs' variables, 35 characters at any depth, so an engine release or upstream rebuild that moves no number keeps the digest, with 22 canonicalisation vectors, each minted name given with its name document and one naming a `quotient_of` over a `sum_of` by the inner step's name; `output-comparison`, how `==` and `same_data` compare two refs, with 12 vectors; `operation-outputs`, `share_of` in per cent under `Percent`, `join` in long format by `stat_var`, `compare` as data minus other, `sum_of` as data plus other, and `growth` as a fraction in `value` with no unit, leaving out a member with no value at an end of its window, a relative change over a zero or sign-crossing base and a change that is not finite, and refusing two rows of one member in one period with `role_ambiguous`, with 10 vectors; `rolling-composition`, one latest period per recipe tree, refusing rolling steps over different populations with `rolling_populations_differ` even when they settle on one year, with 6 vectors; and `publication`, a share's numbered slots that keep their URLs across a live replace, a snapshot that refuses `share_not_replaceable`, and the write token checks, with 8 vectors.
+- `actions.json` rows for `allowance_exhausted` (`wait_for_reset`), `rolling_populations_differ`, `several_members`, `share_not_replaceable` and `write_token_required` (`fix_request`), and `write_token_invalid` (`stop`).
+- The `stop` action, class `PermissionDeniedError`, for a refusal no permission or credential the caller can obtain clears: `write_token_invalid`, since a share takes only the write token issued when it was created and a share created without one takes none.
+- The retry rules: a refusal whose code's action is `wait_for_reset` is never retried and raises `QuotaExceededError` at once, whatever its document and status say, and a refusal whose `Retry-After` is longer than 60 seconds (`RETRY_AFTER_MAX_S`) raises at once with that wait, a refused poll included. `search-retry-after-past-cap` pins the cap.
+- `series_left_out` in `warning-codes.json`, class `SeriesLeftOutWarning`, and the `series-left-out` rule with 8 vectors: a fetch of derived members issues it once when any member's derived manifest lists left-out series in `left_out` or says `left_out_complete` false, a null `left_out` issues nothing, and a strict fetch raises it in place of its data. `tools/validate_client_contract.py` takes the code from the rule.
+- `actions.json` rows (`fix_request`) for the 34 refusal codes the analysis engine serves on `datasets.analysis`, including `compare_no_key_matched` and `sum_of_no_key_matched`.
 
 [unreleased]: https://github.com/DevelopmentFinanceObservatory/spec-kit/commits/main/python