CHANGELOG.md

Changelog

All notable changes to the dfo-spec-kit-client-contract Python package will be documented in this file. The format is based on Keep a Changelog, and this package adheres to Semantic Versioning.

Unreleased

Added

• Package scaffolding (uv, ruff, ty, pytest).
• served/openapi.json, the composed dfo-apis app's OpenAPI document, and served/SOURCE.json, naming dfo-apis commit d80df9f5fd598b7345255470932d202ee5d49942. The document admits timed_out as a no_data reason and the optional entity_refs on each answer part, a list of EntityRefs with one entry per restatement entity row, the graph nodes that row binds, and a stored manifest warning states its counts and the refs it names (CountedRefsModel, CountedEntityYearsModel, EntityYearsModel). NoDataOption and SentenceToken carry entity_choice, the index into NoData.ambiguous_entities of the group an entity_ambiguous option settles or a slot token asks, so a client groups a place menu by it when two wordings share one text.
• strict-fields.json and its schema: 29 members that stay required and non-nullable, each one a client reads, and the answer kinds, part statuses and operation statuses the clients branch on.
• actions.json and its schema: the caller's next step for each of the 106 active refusal codes on problem_details or no_data_reason, with the gap kind of each no_data reason.
• The status fallback for a refusal whose code has no row: 400 and 422 to InvalidRequestError, 401 to AuthenticationError, 403 to PermissionDeniedError, 404 and 410 to NotFoundError, 408 to DfoTimeoutError, 429 to RateLimitError, and any other 4xx or 5xx to InvalidRequestError or ServerError.
• scenario.schema.json, the scenario record, and 126 scenarios: 34 recorded against the deployed dfo-apis dev app at d80df9f5f, and 92 synthetic, each derived from a recorded one by a one-line edit, with 21 binary bodies totalling 1,134,603 bytes and records and bodies together 13,695,441 bytes. search-place-ambiguous-twice pins two entity_ambiguous groups of one text, each option and slot token naming its group by entity_choice, and choose-refused-label-in-two-groups pins that a label both groups offer matches two options, so a choice there names its option by index. The scenarios cover the several-part choice entries and the assumptions each identity entry echoes, a picked reading whose assumption Search states in its answer, the place choice through a no_data part's options, the choice refusals, a part without variable_text, an unknown status without a restatement, a shared label, a one-part option whose data_query holds two parts, the separate and operations echo, an unsupported part's placeholder, the timed_out, unsupported and unknown gaps, an unknown answer kind, the SearchWarning rule, the stacked dataset's part column, the frame layout, a strict fetch's DataWarning, a spec fetch that rebinds one of two entity roles and keeps the other, one that drops a role bound to null, a spec whose destination is a named group fetched with a window or with its origin rebound, whose sentence lookup posts the groups of the roles the build left bound, a twice cut-off build, a cut-off build retried after its Retry-After, a retryable refusal whose Retry-After passes the 60-second cap, a build refused 504 then 503 and one refused 503 then 504, a refused build, a build refused 501 capability_unavailable, a digest mismatch, a signed URL refused on both rounds, a relative files-route Location, an artifact url on a foreign origin, on the storage host and absolute on the API's origin, a manifest definition without identity or schema_version, a Search spec without identity or resolved, nested specs no rule reads, an empty parts list, a Search body that is not an object, a codeless 503 problem document that is not retryable, a 401, an identity resubmit answering another spec, a part whose status is not a string, a Search spec whose definition names another spec_id, a Search response with no JSON body, an option with no data_query, empty entities and facets, a Sunset header issued once, x-dfo-warning messages and a Deprecation and Sunset pair, an unreadable versions page, a mint and a spec fetch whose build answers 202 and is polled to done, a resumed operation, a rate-limited poll, 503 polls retried within the budget an answered poll restores, a 202 that reads running, polls whose retry_after_ms changes from one to the next, a 404 poll that raises, a failed operation posted again, a failed operation with no error, and a ref whose stored file expired, read through the file selector, rebuilt at the same version, at a newer one, or refused with the newest version read from the versions route: the first version it lists with no base_content_version, the newest plain build, so a derived version listed before it is skipped.
• client-codes.json and its schema: the 23 codes a client sets on the errors it raises itself, each with its error class, retryable flag and condition, disjoint from actions.json and registries/codes.json, and the client_codes() and client_codes_schema() accessors. Every scenario error the client raises itself carries its code, and refusal_without_code names a refusal whose body carries no code.
• proxy-codes.json and its schema, the website proxy's rate_limited and transient_unavailable codes with their statuses, which the proxy sends in problem documents of the shape every refusal has, both retryable, with Retry-After on the 429, so a client reads one refusal shape, and warning-codes.json and its schema, the 7 codes a client sets on the warnings it issues itself, with the proxy_codes() and warning_codes() accessors.
• ref_digest_changed in warning-codes.json, a DfoWarning for a ref member whose content_version serves a file with another digest than the ref's, because the server rendered the same data again. The client returns the data, with no error under strict.
• timed_out in actions.json and registries/codes.json, the no_data reason Search gives a part it left unresolved at the request's time limit, with action ask_data_question and gap kind timed_out, which actions.schema.json now admits. The client-side timeout code, a transport timeout, is unchanged.
• A no_data part's gap kind comes from its own reason, whatever the answer's diagnostics.timed_out says. answer-timed-out gives its part the reason timed_out, and answer-timed-out-keeps-reasons pins a part that keeps variable_not_resolved as a not_understood gap in a timed-out answer.
• Error outcomes give the error's trace_id, the body's, else the x-dfo-trace-id header's, else null. Scenario outcomes give retry_waits, the waits before each retry, and warnings, the warning codes a step's responses issued. Scenario steps take no timeout.
• dataset-ref.schema.json, the frozen DatasetRef format, with accepted and rejected examples.
• without_party_totals in the ref's artifact rendering, true for a member fetched with party totals left out and absent for a plain rendering, with ref-without-party-totals, a default fetch's cut rendering, ref-one-member, a fetch with party totals and its plain rendering, and ref-without-party-totals-false-rejected as its examples. Each ref example's rendering is the one the build that answered its file posted, and the validator checks it.
• tools/capture_client_contract.py, which captures served/openapi.json from a dfo-apis checkout in process and records the scenarios against a base URL, dev by default, or with record --offline derives their outcomes again from the records with no request. A local app's signed URLs are written as the storage URLs a deployment signs. record reads the rule tables from --package-dir before any request, and looks for each long build's spec from the year after the one its record built, stopping at the first start year whose estimate is within the sync budget.
• tools/validate_client_contract.py, which checks the contract against its schemas, the served document and registries/codes.json, checks client-codes.json against them and against the reasons the key-lookup rule raises, and checks each scenario's expect against the outcomes its exchanges, actions.json and client-codes.json give, each Search request against the input its steps derive, a sentence lookup's entity_groups included, each build's input.scope against the narrowing its step names over the spec's bound roles, each build's options against its layout, each Search request's whole body, each download against its artifact's digest, each recorded scenario's commit against served/SOURCE.json, each problem document's code against registries/codes.json or, for the website proxy's, proxy-codes.json, each proxy problem document's content type, members and retryable: true, each 202 against an operation that reads done, and each absolute URL against the storage host, each ref example's members against the dataset members and downloaded files of the scenario it names, and the scenarios directory against a 16,000,000-byte budget for records and bodies together. A record that ends a run of retries while a retry remains does not fit. The outcomes include each long build's operation id, poll waits and resubmissions, and a VersionUnavailableError's newest version.
• The contract README states that a POST for a build identical to one pending or running returns that operation, so a client that stops waiting posts the same build to join it, and that fetch waits for a build with no whole-call deadline by default, with a download that delivers no byte for 30 s a retryable network_error.
• A fetch of an answer or a spec leaves out party totals by default: its build posts without_party_totals: true in options, so the server drops the group totals of the identity's open party roles, and a role the caller binds keeps what it binds. Fetch steps take party_totals: true for the client option, party_totals in Python and partyTotals in TypeScript, false by default, which posts no without_party_totals. The wire flag and the ref format are unchanged, and a ref records without_party_totals: true for a member fetched with the default. fetch-spec-party-totals-left-out and fetch-spec-party-totals-kept pin both on DAC2a ODA to Kenya in 2022 with the origin open. A mint leaves them out by default too, and its step takes party_totals, so its result answers the table a fetch of the spec answers; mint-async-build keeps them. A resumed operation's result takes the rendering its build posted from the done manifest's artifact.
• The reading-order rule and its vectors, which carry rows: a client presents a dataset's columns with the reading columns first, the time period, provider, recipient, subject, purpose, value, unit, flow type and provenance, each slot filled by the long layout's label or the frame layout's code, the flow type by CRS crsTypeOfFlow or else DAC2a dac2aAidType, then the columns whose values differ between rows, then those that hold one value, each group in the server's order, with stat_var last. It sorts each member's rows by the filled slots without value and the time period, then the varying columns, then the time period, ascending with nulls last and ties in served order, so each series' years read together. Two cells hold the same value when they are equal as JSON values: numbers by value, so 1 and 1.0 are one value, booleans apart from numbers, NaN the same as NaN, lists item by item in order and structs field by field in any order. A column of lists or structs is no sort key. A vector's rows hold one cell per column and each column one JSON type, and integer-and-float-are-one-value and nested-cells-compare-by-value pin the comparison. Scenario outcomes give columns in that order, and the validator checks it against the served files, list and struct columns included, and reports a malformed vector.
• The client-identity rule and its vectors: every request a client sends to the API carries X-DFO-Client: <client>/<version> (<runtime>), such as dfo-python/0.1.0 (CPython 3.14) or dfo-ts/0.1.0 (Node 22.18), browser as the runtime in a browser page or browser worker, and other in any other runtime, such as Cloudflare Workers, with pattern matching every well-formed value. A request to another origin, a signed storage URL included, carries none. A caller's headers or header provider naming X-DFO-Client in any letter case raises InvalidRequestError header_invalid, whose condition in client-codes.json names that case, and User-Agent stays as each client sends it. A scenario replay checks the header on each path and origin_path request and its absence on each url request.
• The role-keys rule and its vectors: a caller narrows a spec with entities= keyed by place, origin and destination, and a client posts place as the wire role observationAbout. Spec.roles, specRoles(spec) and the clients' errors name the caller keys, and a flow spec takes both directions. A key outside the spec's roles raises InvalidRequestError invalid_argument naming the key and the spec's roles, so observationAbout on GDP is refused naming place and financingRecipient on a flow spec naming origin and destination. A spec that names no role posts place as observationAbout and other keys as typed, and refuses observationAbout naming place.
• The client rules and their vectors: the credentials path and file format, the key lookup order, with an empty api_key refused as key_empty and an empty profile key as profile_without_key, the key format check, the base URL order, base_url_invalid for an invalid value, and the cache directory and layout.
• The base-url rule states the whole validity rule, and seven new vectors pin it: no query, fragment, . or .. segment, character outside visible ASCII, empty userinfo or non-decimal IPv4 host; trailing slashes dropped; a relative URL valid only in a TypeScript client inside a browser page.
• The base-url rule also refuses a . or .. segment written with %2e, a backslash, and a port outside 1 to 65535 or written other than in decimal digits, an empty one included, so a client on Python's urlsplit and one on the WHATWG URL parser accept the same values. Ten vectors pin them.
• The file-selector rule and its vectors: the file selector's query holds the format, bundle and layout of a ref member's rendering, the keys query_keys lists, in that order, and leaves out without_party_totals, which the selector refuses with 400 request_invalid because the version records its own cut. fetch-ref-expired-file-selector and fetch-ref-artifact-unlisted ask that query, and the validator and the capture build it from query_keys.
• Ref format 2 in dataset-ref.schema.json: format_version: 2 and one or more derived members, each {parts, sentence, spec: {spec_id, recipe: {normal_form_hash, replay_key}}, content_version, output_digest, roots, execution_identity, resolved_periods, artifact}, with an ops_ id, an ocv_ version, rnf_ and rpk_ recipe hashes, an out_ output digest, the root dataset versions with the renderings they were read with, and each period stated relative to the latest with the year it resolved to. Format 1 is unchanged and holds dataset members only, and format 2 holds no dataset member. A derived member's parts is empty. A scenario's fetch-by-ref step admits a format 1 ref only, and a format 2 ref fails there on its format_version. Nine new ref examples, two accepted and seven rejected, each derived identity computed by the dfo-apis analysis runner. The validator checks a derived member's roots against the scenario it names, each root's rendering as the posted one in the frame layout, and that the derived members of the ref examples and the output-comparison vectors agree wherever one identity fixes another.
• A derived member's root records its rendering in root_rendering, the artifact selector's members and groupby, the sum DFO read the root with as options.groupby states it: by, one or more distinct column names, op sum, and levels, a depth from 0 to 99 for each kept hierarchical axis, absent when none is set. A breakdown or a sum over StatVars reads a root that way. The artifact's rendering admits no groupby. ref-derived-grouped-root is accepted, its by sorted as DFO records it and its identities computed by the dfo-apis analysis runner over the scenario's rows summed to their purpose's sector code, and ref-derived-root-groupby-op-rejected and ref-artifact-groupby-rejected are rejected. The validator checks a summed root's rendering against its scenario's build without its groupby, which no recorded build posts.
• The analysis and publication rules DFO applies, each with vectors and a reference implementation in tests/test_analysis_rules.py: output-digest, the out2 digest of a derived frame's values with the stat_var, members, period and unit each belongs to, without measurementMethod, provenance, status or a unit's recipe, reading the variable compare, sum_of, share_of or quotient_of writes under an mv_ name hashed from its verb and its inputs' variables, 35 characters at any depth, so an engine release or upstream rebuild that moves no number keeps the digest, with 22 canonicalisation vectors, each minted name given with its name document and one naming a quotient_of over a sum_of by the inner step's name; output-comparison, how == and same_data compare two refs, with 12 vectors; operation-outputs, share_of in per cent under Percent, join in long format by stat_var, compare as data minus other, sum_of as data plus other, and growth as a fraction in value with no unit, leaving out a member with no value at an end of its window, a relative change over a zero or sign-crossing base and a change that is not finite, and refusing two rows of one member in one period with role_ambiguous, with 10 vectors; rolling-composition, one latest period per recipe tree, refusing rolling steps over different populations with rolling_populations_differ even when they settle on one year, with 6 vectors; and publication, a share's numbered slots that keep their URLs across a live replace, a snapshot that refuses share_not_replaceable, and the write token checks, with 8 vectors.
• actions.json rows for allowance_exhausted (wait_for_reset), rolling_populations_differ, several_members, share_not_replaceable and write_token_required (fix_request), and write_token_invalid (stop).
• The stop action, class PermissionDeniedError, for a refusal no permission or credential the caller can obtain clears: write_token_invalid, since a share takes only the write token issued when it was created and a share created without one takes none, licence_not_redistributable, since a source's licence forbids the export for every caller, and provenance_unavailable, since a recipe root that cannot be cited survives a rebuild.
• The retry rules: a refusal whose code's action is wait_for_reset is never retried and raises QuotaExceededError at once, whatever its document and status say, and a refusal whose Retry-After is longer than 60 seconds (RETRY_AFTER_MAX_S) raises at once with that wait, a refused poll included. search-retry-after-past-cap pins the cap.
• series_left_out in warning-codes.json, class SeriesLeftOutWarning, and the series-left-out rule with 8 vectors: a fetch of derived members issues it once when any member's derived manifest lists left-out series in left_out or says left_out_complete false, a null left_out issues nothing, and a strict fetch raises it in place of its data. tools/validate_client_contract.py takes the code from the rule.
• actions.json rows (fix_request) for the 34 refusal codes the analysis engine serves on datasets.analysis, including compare_no_key_matched and sum_of_no_key_matched.
• actions.json rows (stop) for the two refusals of the exports door, provenance_unavailable and licence_not_redistributable.
• tools/validate_client_contract.py lets a proxy code share a registry code whose every mapping is mcp_refusal, as rate_limited and transient_unavailable do.