diff --git a/python/client-contract/CHANGELOG.md b/python/client-contract/CHANGELOG.md index aa12883..1a91094 100644 --- a/python/client-contract/CHANGELOG.md +++ b/python/client-contract/CHANGELOG.md @@ -9,7 +9,7 @@ All notable changes to the `dfo-spec-kit-client-contract` Python package will be - Package scaffolding (uv, ruff, ty, pytest). - `served/openapi.json`, the composed dfo-apis app's OpenAPI document, and `served/SOURCE.json`, naming dfo-apis commit `d80df9f5fd598b7345255470932d202ee5d49942`. The document admits `timed_out` as a no_data reason and the optional `entity_refs` on each answer part, a list of `EntityRefs` with one entry per restatement entity row, the graph nodes that row binds, and a stored manifest warning states its counts and the refs it names (`CountedRefsModel`, `CountedEntityYearsModel`, `EntityYearsModel`). `NoDataOption` and `SentenceToken` carry `entity_choice`, the index into `NoData.ambiguous_entities` of the group an entity_ambiguous option settles or a slot token asks, so a client groups a place menu by it when two wordings share one text. - `strict-fields.json` and its schema: 29 members that stay required and non-nullable, each one a client reads, and the answer kinds, part statuses and operation statuses the clients branch on. -- `actions.json` and its schema: the caller's next step for each of the 106 active refusal codes on `problem_details` or `no_data_reason`, with the gap kind of each no_data reason. +- `actions.json` and its schema: the caller's next step for each of the 148 active refusal codes on `problem_details` or `no_data_reason`, with the gap kind of each no_data reason. - The status fallback for a refusal whose code has no row: 400 and 422 to `InvalidRequestError`, 401 to `AuthenticationError`, 403 to `PermissionDeniedError`, 404 and 410 to `NotFoundError`, 408 to `DfoTimeoutError`, 429 to `RateLimitError`, and any other 4xx or 5xx to `InvalidRequestError` or `ServerError`. - `scenario.schema.json`, the scenario record, and 126 scenarios: 34 recorded against the deployed dfo-apis dev app at `d80df9f5f`, and 92 synthetic, each derived from a recorded one by a one-line edit, with 21 binary bodies totalling 1,134,603 bytes and records and bodies together 13,695,441 bytes. `search-place-ambiguous-twice` pins two entity_ambiguous groups of one text, each option and slot token naming its group by `entity_choice`, and `choose-refused-label-in-two-groups` pins that a label both groups offer matches two options, so a choice there names its option by index. The scenarios cover the several-part choice entries and the assumptions each identity entry echoes, a picked reading whose assumption Search states in its answer, the place choice through a no_data part's options, the choice refusals, a part without variable_text, an unknown status without a restatement, a shared label, a one-part option whose data_query holds two parts, the separate and operations echo, an unsupported part's placeholder, the timed_out, unsupported and unknown gaps, an unknown answer kind, the `SearchWarning` rule, the stacked dataset's `part` column, the frame layout, a strict fetch's `DataWarning`, a spec fetch that rebinds one of two entity roles and keeps the other, one that drops a role bound to null, a spec whose destination is a named group fetched with a window or with its origin rebound, whose sentence lookup posts the groups of the roles the build left bound, a twice cut-off build, a cut-off build retried after its Retry-After, a retryable refusal whose Retry-After passes the 60-second cap, a build refused 504 then 503 and one refused 503 then 504, a refused build, a build refused 501 `capability_unavailable`, a digest mismatch, a signed URL refused on both rounds, a relative files-route Location, an artifact url on a foreign origin, on the storage host and absolute on the API's origin, a manifest definition without identity or schema_version, a Search spec without identity or resolved, nested specs no rule reads, an empty parts list, a Search body that is not an object, a codeless 503 problem document that is not retryable, a 401, an identity resubmit answering another spec, a part whose status is not a string, a Search spec whose definition names another spec_id, a Search response with no JSON body, an option with no data_query, empty entities and facets, a Sunset header issued once, `x-dfo-warning` messages and a Deprecation and Sunset pair, an unreadable versions page, a mint and a spec fetch whose build answers 202 and is polled to done, a resumed operation, a rate-limited poll, 503 polls retried within the budget an answered poll restores, a 202 that reads running, polls whose `retry_after_ms` changes from one to the next, a 404 poll that raises, a failed operation posted again, a failed operation with no error, and a ref whose stored file expired, read through the file selector, rebuilt at the same version, at a newer one, or refused with the newest version read from the versions route: the first version it lists with no `base_content_version`, the newest plain build, so a derived version listed before it is skipped. - `client-codes.json` and its schema: the 23 codes a client sets on the errors it raises itself, each with its error class, retryable flag and condition, disjoint from `actions.json` and `registries/codes.json`, and the `client_codes()` and `client_codes_schema()` accessors. Every scenario error the client raises itself carries its code, and `refusal_without_code` names a refusal whose body carries no code. diff --git a/python/client-contract/README.md b/python/client-contract/README.md index 9f475e8..6407aa4 100644 --- a/python/client-contract/README.md +++ b/python/client-contract/README.md @@ -180,14 +180,14 @@ The records also fix these client rules: - A fetch of an answer issues one `SearchWarning` when a definite part has an operation Search did not run, a departure, or a warning at severity `warn` or `degraded` with a `message`, or the answer's `diagnostics.warnings` holds such a warning. An `info` warning, a warning with no `message`, and an operation that does not read under Strict fields issue nothing. Each way is one line a client shows: `Not done: ()` coded by the verb, the phrase quoted as a JSON string that keeps non-ASCII characters as they are, the departure's `text` coded by its code, or `Note: ` coded by the warning's code, in part order with the diagnostics warnings last, and ways that show the same line count once, under the first one's code. A strict fetch raises the `SearchWarning` before any build, with the code of its one line, or a null code when there are several lines. A strict fetch raises it before the `frame` layout's several-member check. - A ref's version is found in this order: the snapshot route and the files route at the stored digest; the file selector, `GET /datasets/{id}/{content_version}/file` with the query the file-selector rule in `rules/README.md` builds from the ref's rendering, which leaves out `without_party_totals`; and a build of the ref's spec in the ref's rendering, which must answer the ref's `spec_id` and the same `content_version`. Both recovery steps keep the ref's identity, so a recovered member carries the ref's `spec_id` and `content_version`: a snapshot manifest whose `definition.spec_id` or `content_version` is not the ref's, or a rebuild manifest whose `definition.spec_id` is not the ref's, raises `ContractError` `contract_violation` before any download, which `fetch-ref-expired-rebuild-other-spec` pins for the rebuild. A snapshot refusal other than `artifact_expired` ends the lookup, so `snapshot_not_found` raises `NotFoundError` with no further request. A files route or selector refusal of `artifact_expired`, `artifact_not_found` or `not_found` takes the next step, and a selector file of another digest is not the stored file. A snapshot manifest whose artifacts list no artifact of the ref member's digest sends the client to the selector for the ref member's own rendering, and a file that hashes to that digest gives the dataset, which `fetch-ref-artifact-unlisted` pins. A rebuild that answers another version, or that raises `InvalidRequestError`, `NotFoundError` or `VersionUnavailableError`, ends in `VersionUnavailableError` with the first refusal's code, or `version_unavailable` when nothing refused. Its `newest_version` is the version the rebuild answered, or else the newest stored version, as a `newest_version` step finds it: the first version `GET /datasets/{id}/versions?limit=100` lists with a null `base_content_version`, the newest plain build, read page by page through `&cursor=` with each page's `next_cursor` until one is listed. The route lists versions newest first and needs no key. A version with a `base_content_version` is derived from a plain build, a groupby collapse or a party-totals cut, and is skipped, which `fetch-ref-newest-version` pins. The newest version is null when the list ends without a plain version or the route refuses with `NotFoundError`. It names the newest plain build, which a fetch with the default party totals rule answers only when the cut leaves nothing out; when the cut leaves rows out, the fetch answers a version whose `base_content_version` is that build. So a client does not present `newest_version` as comparable to the `content_version` of a cut ref or a cut fetch: two different versions there need not mean newer data. Any other refusal of the versions route raises, and a page whose `versions` is not a list of versions each with a `content_version` and a `base_content_version` that is a string or null, or whose `next_cursor` is neither a string nor null, raises `ContractError` `contract_violation`, which `newest-version-unreadable-page` pins. - A build answers 200 with the manifest, or 202 with an operation the client polls at `/datasets/operations/{operation_id}`, the id of the operation it holds. The 202 reads `pending` or `running`, because a submission starts a build or joins one under way, and a `done` operation left by an earlier submission of a rejected head is built again. A deterministic failure answers a 202 that reads `failed`. `fetch-spec-async-build` pins `pending` and `fetch-spec-accepted-running` pins `running`. A POST for a build identical to one that is pending or running returns that same operation, because the server derives the operation id from the request: dfo-apis `src/datasets_api/operation_store.py` `compute_operation_id` hashes the request identity, the origin catalogue build id and the rendering. A client that stops waiting for a build can post the same build again to join it. A mint's build answers 202. A record whose mint answers 200 fits no mint step. Before each poll it waits the `Retry-After` of the refusal that answered the last poll, whatever its status, as every retry of a refused request waits; else the last operation resource's `retry_after_ms`, else the `Retry-After` of the last answered poll, the response that carried that resource, else a backoff of 1, 2, 4 and then 8 seconds that advances one step each time it is used, a poll after a refusal included. `fetch-spec-poll-pacing-changes` pins a `retry_after_ms` that changes from one poll to the next, so a client waits each poll's own value. -- A retryable 429 poll spends no retry, and a 429 whose body says it is not retryable raises at once, as a refusal's `retryable` outranks its status, which `fetch-spec-poll-rate-limited-not-retryable` pins. Any other retryable refused poll spends one of the two retries, which a poll that answers restores, and a refusal that is not retryable raises. `fetch-spec-poll-refused` pins two 503s polled again after an answered poll, `fetch-spec-poll-refused-backoff` a 503 with no `Retry-After` among polls no response paces, after which the backoff advances, `fetch-spec-poll-refused-retry-after` a 503 whose `Retry-After` paces the next poll, and `fetch-spec-poll-not-found` a 404 that raises at once. A poll that answers a failed operation whose error is retryable posts the same build once more and polls the operation it starts; a second failure, an error that is not retryable, and a resumed operation, whose build request the client does not hold, raise the operation's error. A refused poll whose code is `operation_failed` is a failed operation with that error. A failed operation whose resource carries no `error` raises `ServerError` with `operation_failed`, retryable by status 500 as a refusal without a document is, which `operation-failed-without-error` pins. A 202 or resumed operation that already reads `failed` raises its error before any poll. +- A retryable 429 poll spends no retry, and a 429 whose body says it is not retryable raises at once, as a refusal's `retryable` outranks its status, which `fetch-spec-poll-rate-limited-not-retryable` pins. Any other retryable refused poll spends one of the two retries, which a poll that answers restores, and a refusal that is not retryable raises. `fetch-spec-poll-refused` pins two 503s polled again after an answered poll, `fetch-spec-poll-refused-backoff` a 503 with no `Retry-After` among polls no response paces, after which the backoff advances, `fetch-spec-poll-refused-retry-after` a 503 whose `Retry-After` paces the next poll, and `fetch-spec-poll-not-found` a 404 that raises at once. A poll that answers a failed operation whose error is retryable posts the same build once more and polls the operation it starts; a second failure, an error that is not retryable, and a resumed operation, whose build request the client does not hold, raise the operation's error. A refused poll whose code is `operation_failed` is a failed operation with that error, and one whose `Retry-After` passes 60 seconds raises at once, as every refused poll does. Under the cap, the client posts the build again at once, and that `Retry-After` paces nothing. A failed operation whose resource carries no `error` raises `ServerError` with `operation_failed`, retryable by status 500 as a refusal without a document is, which `operation-failed-without-error` pins. A 202 or resumed operation that already reads `failed` raises its error before any poll. - Any other refusal is retried when it is retryable: never when its code's action is `wait_for_reset`, whatever its document and status say, else as its problem document's `retryable` says, else when its status is 408, 429, 500, 502, 503 or 504. A client makes two retries by default, and the scenarios assume that default. A build's POST shares those two retries with every other refusal of it, and once a 504 cuts the build off the client posts it once more only, whatever that post answers: a build refused 504 then 503 raises the 503, a build refused 503 then 504 is posted once more after the 504, and a second 504 raises. `fetch-spec-504-then-503`, `fetch-spec-503-then-504` and `fetch-spec-504-twice` pin the three. A refusal that is not retryable, or one past the retries, raises its error. - Each retry waits the refusal's `Retry-After` when it carries one, and otherwise the client's backoff for that attempt: 0.5 seconds before the first retry, doubling to at most 8. The build's 504 cut-off retry follows the same rule. A client waits a `Retry-After` of at most 60 seconds (`RETRY_AFTER_MAX_S`). A refusal whose `Retry-After` is longer raises at once with that wait on the error, a refused poll included, so no call sleeps for minutes unasked. `retry_waits` pins these waits, `fetch-spec-504-retry-after` pins a cut-off that carries `Retry-After`, and `search-retry-after-past-cap` a retryable 503 whose `Retry-After` of 61 raises with no retry. - A 401 resolves the key again by the key-lookup rule. When the key changed, the client sends the request once more at once, with no wait and spending no retry. When it did not, or the second answer is a 401 too, the 401 raises `AuthenticationError`. A replay's key never changes, so `search-unauthorized` pins the second case and no scenario the first. - A `request(method, path)` whose `path` is not a path on the base URL, because it starts with `//` or names a scheme, raises `InvalidRequestError` `invalid_argument` before any request. No scenario step makes such a request. - A 2xx response of a JSON route with no JSON body raises `ContractError` `unexpected_response`. `search-no-json-body` pins it for Search. - A `Deprecation` or `Sunset` header on an API response, either alone, issues `route_deprecated` once per process per distinct pair of their values, and an `x-dfo-warning` header issues `server_warning` once per process per message. A signed fetch's headers issue nothing. `search-sunset-warning` pins a Sunset header alone, issued once over two responses that carry the same value, and `search-server-warning` pins a repeated `x-dfo-warning` message issued once, a new message issued again, and a Deprecation beside the same Sunset issued as a new pair. -- A fetch of derived members issues `series_left_out`, a `SeriesLeftOutWarning`, once when the recipe anchor of any member lists left-out series (`Recipe.anchor.left_out` non-empty in the served document): series the rolling anchor dropped because they hold no row at the anchor period. The client returns the frame. A strict fetch raises the warning in place of its data. A recipe with no rolling period has a null anchor and issues nothing. No scenario serves a derived member yet, so the `series-left-out` rule's vectors in `rules/` pin it. +- A fetch of derived members issues `series_left_out`, a `SeriesLeftOutWarning`, once when any member's manifest, from `GET /datasets/derived/{ops_id}/{ocv_version}`, lists in `left_out` a series that a step of its recipe dropped for a data gap, or says `left_out_complete` false. The client returns the frame. A strict fetch raises the warning in place of its data. A null `left_out` issues nothing. No scenario serves a derived member yet, so the `series-left-out` rule's vectors in `rules/` pin it: `growth-left-series-out` warns for a growth step's dropped series, `list-incomplete` and `incomplete-with-none-listed` for `left_out_complete` false. - Python synchronous platform name resolution can exceed the following timeout and deadline limits. Asynchronous name resolution is cancellable. Timeouts follow the clients' design and are not replayed, since the replay advances the clock only by the waits the client asks for: the client's `timeout`, 30 s by default, bounds one request attempt; `search(timeout=60)` bounds the whole search, each attempt reading until that deadline, and None means no limit; `fetch` waits for a build with no whole-call deadline by default, and `fetch(timeout=60)` bounds the whole fetch, each download getting the time left on the fetch's deadline; a download that delivers no byte for 30 s raises the retryable `network_error`; a synchronous build POST, and mint's POST, read for up to 150 s within any outer deadline; `result(timeout=1800)` bounds the wait, and None means no limit. A poll attempt that hangs past the per-attempt timeout spends a retry, and a deadline that passes during a body read raises `DfoTimeoutError` `deadline_exceeded`. In TypeScript an infinite remaining time arms no timer, and a finite one above 2^31-1 ms is clamped. No scenario step takes a timeout. ## Recording diff --git a/tools/tests/test_validate_client_contract.py b/tools/tests/test_validate_client_contract.py index f17cd0e..d70ab02 100644 --- a/tools/tests/test_validate_client_contract.py +++ b/tools/tests/test_validate_client_contract.py @@ -2932,7 +2932,9 @@ def _write_long_build(package_dir: Path, kind: str) -> Path: 202 and two polls finish it, the first after the 202's 1.5 s pacing and the second after a 1 s backoff. `rate_limited` adds a 429 with Retry-After 3 before the first poll; `resubmitted` has the first poll - answer a retryable failed operation and the build posted again.""" + answer a retryable failed operation and the build posted again, and + `refused_resubmitted` has it refused 500 with that operation's error and + Retry-After 3.""" scenarios = package_dir / "scenarios" parent = json.loads( (scenarios / "fetch-spec-known-sentence.json").read_text(encoding="utf-8") @@ -2971,7 +2973,7 @@ def polled(status: str, **members: Any) -> dict[str, Any]: }, ) waits = [1.5, 3.0, 1.0] - elif kind == "resubmitted": + elif kind in ("resubmitted", "refused_resubmitted"): error = { "type": "/codes/operation_failed", "title": "Internal Server Error", @@ -2982,7 +2984,11 @@ def polled(status: str, **members: Any) -> dict[str, Any]: "capability_version": "synthetic", "retryable": True, } - polls[:0] = [polled("failed", error=error), started] + failed = polled("failed", error=error) + if kind == "refused_resubmitted": + headers = {"content-type": "application/problem+json", "retry-after": "3"} + failed["response"] = {"status": 500, "headers": headers, "json": error} + polls[:0] = [failed, started] waits = [1.5, 1.5, 1.0] resubmissions = 1 outcome = copy.deepcopy(parent["expect"][0]) @@ -3015,7 +3021,9 @@ def _long_builds(change: Callable[[Any], None]) -> Callable[[Any], None]: return _set_outcome(0, lambda o: change(o["dataset"]["long_builds"][0])) -@pytest.mark.parametrize("kind", ["plain", "rate_limited", "resubmitted"]) +@pytest.mark.parametrize( + "kind", ["plain", "rate_limited", "resubmitted", "refused_resubmitted"] +) def test_a_long_build_record_is_valid(package_dir: Path, kind: str) -> None: _write_long_build(package_dir, kind) assert validator.validate() == [] @@ -3054,6 +3062,13 @@ def test_a_long_build_record_is_valid(package_dir: Path, kind: str) -> None: _set_body(1, lambda body: body["error"].update(retryable=False)), "the steps never make exchanges 2 to 6", ), + ( + "refused_resubmitted", + lambda doc: doc["exchanges"][1]["response"]["headers"].update( + {"retry-after": "300"} + ), + "the steps never make exchanges 2 to 6", + ), ], ids=[ "poll-waits-without-backoff", @@ -3062,6 +3077,7 @@ def test_a_long_build_record_is_valid(package_dir: Path, kind: str) -> None: "rate-limited-poll-without-its-wait", "resubmission-uncounted", "failed-operation-not-retryable", + "refused-failed-operation-past-retry-after-cap", ], ) def test_a_broken_long_build_is_reported( diff --git a/tools/validate_client_contract.py b/tools/validate_client_contract.py index 2f03997..7fbd422 100644 --- a/tools/validate_client_contract.py +++ b/tools/validate_client_contract.py @@ -2458,9 +2458,10 @@ class LongBuild: """A build the server runs on, followed the way a client polls it. Each wait before a poll is `PollPacing`'s. A retryable 429 poll spends no retry, any other retryable refused poll one of MAX_RETRIES, and a refused - poll that is not retryable raises, a 429 included. The poll after a + poll `tries_again` declines raises, a 429 included. The poll after a refusal waits as `PollPacing` gives. A poll answering a failed - operation, retryable by its error, posts the build again once when the + operation, retryable by its error, or refused with `operation_failed` + and allowed by `tries_again`, posts the build again once when the client holds its request. Otherwise, and for an operation that failed before the client polled it, the operation's error raises. `entry` is the build's `long_builds` entry, and `result` its manifest once it is @@ -2517,9 +2518,9 @@ def polled(self, status: int, headers: Mapping[str, str], body: Any) -> bool: error, again = failed_operation(body, self.actions, headers) else: error = refusal_error(status, body, self.actions, headers) - again = retryable(status, body, self.actions) + again = tries_again(status, body, headers, self.actions) if error.code != OPERATION_FAILED: - if not tries_again(status, body, headers, self.actions): + if not again: raise error if status != 429: if self.retries >= MAX_RETRIES: