fix(client-contract): ask the file selector only for the keys it reads

CONTRIBUTING.md

@@ -41,7 +41,7 @@ uv run tools/validate_unit_vectors.py
 uv run tools/validate_provenance_docs_vectors.py
 uv run tools/validate_descriptions_vectors.py
 uv run tools/validate_client_contract.py
-uv run --with jsonschema --with pytest pytest tools/tests -q
+uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q
 ```
 
 The last command tests the validators themselves. [tools/README.md](tools/README.md) says what each validator checks. Each package's tests drive its implementation against the same vectors or fixtures, and `uv run pytest` runs them.

python/client-contract/src/dfo_client_contract/contract.py

@@ -19,6 +19,7 @@
     "reading-order",
     "client-identity",
     "role-keys",
+    "file-selector",
 )
 
 

python/client-contract/src/dfo_client_contract/rules/README.md

@@ -1,6 +1,6 @@
 # Client rules
 
-Clients implement these rules for the capabilities available in their runtime. Python implements every rule. TypeScript implements the key, credentials and base-URL rules in Node, and the reading-order, client-identity and role-keys rules in every runtime; browser runtimes use the key sources available there. TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Clients implementing the same capability use the same rules and formats. Each rule's vectors are in `<rule>.json`, validated by `rules.schema.json`. In a vector, `env` lists the environment variables set, and an empty value counts as unset everywhere below.
+Clients implement these rules for the capabilities available in their runtime. Python implements every rule. TypeScript implements the key, credentials and base-URL rules in Node, and the reading-order, client-identity, role-keys and file-selector rules in every runtime; browser runtimes use the key sources available there. TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Clients implementing the same capability use the same rules and formats. Each rule's vectors are in `<rule>.json`, validated by `rules.schema.json`. In a vector, `env` lists the environment variables set, and an empty value counts as unset everywhere below.
 
 ## Credentials path (`credentials-path`)
 
@@ -42,7 +42,7 @@ A client resolves its base URL when it is built. The first source found wins:
 1. `DFO_BASE_URL`;
 1. the public default, `https://developmentfinance.org/api/v1`.
 
-Only the chosen source is checked. A base URL is valid when it is an absolute `http` or `https` URL with a host and none of these: userinfo, an empty one as in `https://@host/api` included; a query; a fragment; a `.` or `..` path segment; a character outside visible ASCII, a control character, a space or a non-ASCII character; or a host written as an IPv4 address in any form but four decimal numbers, such as `127.1`, `0x7f.1` or `2130706433`. An invalid one raises `base_url_invalid`, so an empty `base_url` argument raises it where an empty `DFO_BASE_URL` counts as unset. A relative URL is invalid, except in a TypeScript client inside a browser page, which resolves a relative base URL against the page's address. A valid value is used as given, less its trailing slashes, so the client's base URL reads `https://developmentfinance.org/api/v1` for `https://developmentfinance.org/api/v1/`; the rule fixes the source and the validity, and leaves the rest of the URL's form to the client. A key sent over plain `http` to a host that is not loopback is refused by `insecure_transport`, which this rule does not decide.
+Only the chosen source is checked. A base URL is valid when it is an absolute `http` or `https` URL with a host and none of these: userinfo, an empty one as in `https://@host/api` included; a query; a fragment; a `.` or `..` path segment, with any of its dots written `%2e` or `%2E` included; a backslash; a port that is not 1 to 65535 in decimal digits, an empty port after a `:` included; a character outside visible ASCII, a control character, a space or a non-ASCII character; or a host written as an IPv4 address in any form but four decimal numbers, such as `127.1`, `0x7f.1` or `2130706433`. An invalid one raises `base_url_invalid`, so an empty `base_url` argument raises it where an empty `DFO_BASE_URL` counts as unset. A relative URL is invalid, except in a TypeScript client inside a browser page, which resolves a relative base URL against the page's address. A valid value is used as given, less its trailing slashes, so the client's base URL reads `https://developmentfinance.org/api/v1` for `https://developmentfinance.org/api/v1/`; the rule fixes the source and the validity, and leaves the rest of the URL's form to the client. A key sent over plain `http` to a host that is not loopback is refused by `insecure_transport`, which this rule does not decide.
 
 ## Key format (`key-format`)
 
@@ -93,15 +93,15 @@ The columns come in four groups:
 1. the other columns that hold one value in every row, null included, in the order the server wrote them;
 1. `stat_var`, last whether or not it varies.
 
-A table with no row or one row has no column that varies. A dataset of several members puts `part` first and judges the groups over the stacked table, where a member's rows hold null in a column that member lacks. So a CRS series of Germany's basic health ODA to Nigeria, split by co-operation modality, reads `time_period`, `financingProvider_label`, `financingRecipient_label`, `crsPurpose_label`, `value`, `unit_label`, `crsTypeOfFlow_label` and `provenance_label`, then `crsCooperationModality` and `crsCooperationModality_label`, which tell its rows apart, then its 15 constant columns, `stat_var` last.
+A table with no row or one row has no column that varies. Two cells hold the same value when they are equal as JSON values: numbers by value, so `1` and `1.0` are one value, booleans apart from numbers, so `true` and `1` differ, and null the same as null. NaN, which a Parquet double column can hold, is the same as NaN. A list cell is the same as another when their items are the same, in order, and a struct cell when it has the same fields with the same values, in any field order. A dataset of several members puts `part` first and judges the groups over the stacked table, where a member's rows hold null in a column that member lacks. So a CRS series of Germany's basic health ODA to Nigeria, split by co-operation modality, reads `time_period`, `financingProvider_label`, `financingRecipient_label`, `crsPurpose_label`, `value`, `unit_label`, `crsTypeOfFlow_label` and `provenance_label`, then `crsCooperationModality` and `crsCooperationModality_label`, which tell its rows apart, then its 15 constant columns, `stat_var` last.
 
-The rows of each member are sorted by the filled slots in slot order, leaving out `value`, a measure, and the time period, then by the varying columns in presented order, `stat_var` after them when it varies, then by the time period, so each series' years read together. The keys of that series are the provider, recipient, purpose, unit, flow type and provenance labels, then `crsCooperationModality` and `crsCooperationModality_label`, then the time period, so it reads every A02 year, then every C01 year. Each key sorts ascending, numbers by value and strings by Unicode code point, with nulls last. Rows whose keys tie keep the order the server wrote them. The members follow one another in member order.
+The rows of each member are sorted by the filled slots in slot order, leaving out `value`, a measure, and the time period, then by the varying columns in presented order, `stat_var` after them when it varies, then by the time period, so each series' years read together. A column that holds lists or structs is no sort key, since the rule gives them no order. The keys of that series are the provider, recipient, purpose, unit, flow type and provenance labels, then `crsCooperationModality` and `crsCooperationModality_label`, then the time period, so it reads every A02 year, then every C01 year. Each key sorts ascending, numbers by value and strings by Unicode code point, with nulls last. Rows whose keys tie keep the order the server wrote them. The members follow one another in member order.
 
-A vector gives the dataset's `members`, each with its `part` when there are several, its `columns` and its `rows` as the server wrote them, and the `columns` and `rows` a client presents. The order moves columns and rows and changes no value. A client hashes a downloaded file before it orders anything, so the digest and the ref keep the server's file.
+A vector gives the dataset's `members`, each with its `part` when there are several, its `columns` and its `rows` as the server wrote them, and the `columns` and `rows` a client presents. Each row holds one cell per column, and the cells of a column, nulls aside, are all strings, all numbers, all booleans, all lists or all structs, since the rule orders no value against one of another type. The order moves columns and rows and changes no value. A client hashes a downloaded file before it orders anything, so the digest and the ref keep the server's file.
 
 ## Client identity (`client-identity`)
 
-Every request a client sends to the API carries `X-DFO-Client: <client>/<version> (<runtime>)`. `<client>` is `dfo-python` or `dfo-ts`, and `<version>` is the package version as the package states it. `<runtime>` names the runtime and its major and minor version. In Python it is the implementation as `platform.python_implementation()` names it, then the interpreter's major and minor version, such as `CPython 3.14`. In TypeScript it is `Node`, `Deno` or `Bun` with the major and minor of that runtime's version, such as `Node 22.18`, and `browser` in any other runtime, a page or a worker. `pattern` in `client-identity.json` matches every well-formed value. A vector gives the `client`, the `version`, the `runtime` as its `name` and full `version`, or `browser` with no version, and the `value` the client sends.
+Every request a client sends to the API carries `X-DFO-Client: <client>/<version> (<runtime>)`. `<client>` is `dfo-python` or `dfo-ts`, and `<version>` is the package version as the package states it. `<runtime>` names the runtime and its major and minor version. In Python it is the implementation as `platform.python_implementation()` names it, then the interpreter's major and minor version, such as `CPython 3.14`. In TypeScript it is `Node`, `Deno` or `Bun` with the major and minor of that runtime's version, such as `Node 22.18`, `browser` in a browser page or browser worker, and `other` in any other runtime, such as Cloudflare Workers. `pattern` in `client-identity.json` matches every well-formed value. A vector gives the `client`, the `version`, the `runtime` as its `name` and full `version`, or `browser` or `other` with no version, and the `value` the client sends.
 
 An API request is one to the base URL's origin, a `path` or `origin_path` request in a scenario. A request to any other origin, a signed storage URL included, carries no `X-DFO-Client`. In a browser, the files-route request the TypeScript client sends with `redirect: "follow"` carries the API key alone, because the browser forwards every other header to the storage host it is redirected to.
 
@@ -116,3 +116,7 @@ The roles a spec takes are those its `definition.identity` binds in `entities` o
 A client checks each key the caller types before any request. A key outside the spec's roles raises `InvalidRequestError` `invalid_argument`, and the message names the key and the spec's roles. So GDP for Kenya refuses `origin` and `observationAbout` naming `place`, and a flow spec refuses `place` and a column name such as `financingRecipient` naming `origin` and `destination`. A spec that names no role leaves the check to the server, since the client cannot tell which roles it takes. Such a spec posts `place` as `observationAbout` and any other key as typed, and still refuses `observationAbout` naming `place`.
 
 The server replaces a spec's whole binding with the posted one, so a client posts the spec's own `identity.entities` with each role the caller names replaced by its dcids and each role the caller binds to null dropped. A vector gives the spec's `identity` and its `roles`. A vector that narrows adds the caller's `entities`, then either `posted`, the wire entities the build posts, or the `error` with its `reason` and the keys its message `names`, the typed key first.
+
+## File selector (`file-selector`)
+
+The file selector, `GET /datasets/{spec_id}/{content_version}/file`, redirects to a stored version's file in a rendering. A client asks it for a ref member's version when the files route at the stored digest refuses, or when the snapshot manifest lists no artifact of that digest. Its query holds the keys `query_keys` in `file-selector.json` lists, `format`, `bundle` and `layout`, in that order, each one the member's rendering holds, as `key=value` pairs joined by `&`, with booleans written `true` and `false`. The query leaves out `without_party_totals`: the version records the cut it was minted with, and the server lays that over the query. The server refuses any other key with 400 `request_invalid`, so a selector query that carried the cut would end the lookup. A vector gives the `rendering` and the `query` a client sends, without the leading `?`.

python/client-contract/src/dfo_client_contract/rules/base-url.json

@@ -172,6 +172,84 @@
       "env": {},
       "source": "argument",
       "url": "https://developmentfinance.org/api/v1"
+    },
+    {
+      "name": "encoded-dot-segment-raises",
+      "base_url": "https://developmentfinance.org/api/%2e%2e/v1",
+      "env": {},
+      "error": {
+        "reason": "base_url_invalid"
+      }
+    },
+    {
+      "name": "mixed-encoded-dot-segment-raises",
+      "base_url": "https://developmentfinance.org/api/.%2E/v1",
+      "env": {},
+      "error": {
+        "reason": "base_url_invalid"
+      }
+    },
+    {
+      "name": "encoded-single-dot-segment-raises",
+      "base_url": "https://developmentfinance.org/api/%2E/v1",
+      "env": {},
+      "error": {
+        "reason": "base_url_invalid"
+      }
+    },
+    {
+      "name": "backslash-raises",
+      "base_url": "https://developmentfinance.org\\api\\v1",
+      "env": {},
+      "error": {
+        "reason": "base_url_invalid"
+      }
+    },
+    {
+      "name": "port-out-of-range-raises",
+      "base_url": "https://developmentfinance.org:99999/api/v1",
+      "env": {},
+      "error": {
+        "reason": "base_url_invalid"
+      }
+    },
+    {
+      "name": "port-zero-raises",
+      "base_url": "https://developmentfinance.org:0/api/v1",
+      "env": {},
+      "error": {
+        "reason": "base_url_invalid"
+      }
+    },
+    {
+      "name": "port-not-decimal-raises",
+      "base_url": "https://developmentfinance.org:abc/api/v1",
+      "env": {},
+      "error": {
+        "reason": "base_url_invalid"
+      }
+    },
+    {
+      "name": "empty-port-raises",
+      "base_url": "https://developmentfinance.org:/api/v1",
+      "env": {},
+      "error": {
+        "reason": "base_url_invalid"
+      }
+    },
+    {
+      "name": "highest-port-is-valid",
+      "base_url": "https://developmentfinance.org:65535/api/v1",
+      "env": {},
+      "source": "argument",
+      "url": "https://developmentfinance.org:65535/api/v1"
+    },
+    {
+      "name": "dot-inside-a-segment-is-valid",
+      "base_url": "https://developmentfinance.org/api/v1.%2e",
+      "env": {},
+      "source": "argument",
+      "url": "https://developmentfinance.org/api/v1.%2e"
     }
   ]
 }

python/client-contract/src/dfo_client_contract/rules/client-identity.json

@@ -2,7 +2,7 @@
   "format_version": 1,
   "rule": "client-identity",
   "header": "X-DFO-Client",
-  "pattern": "^(dfo-python|dfo-ts)/[0-9A-Za-z.+-]+ \\(([A-Za-z]+ [0-9]+\\.[0-9]+|browser)\\)$",
+  "pattern": "^(dfo-python|dfo-ts)/[0-9A-Za-z.+-]+ \\(([A-Za-z]+ [0-9]+\\.[0-9]+|browser|other)\\)$",
   "vectors": [
     {
       "name": "python-cpython",
@@ -93,6 +93,15 @@
       },
       "value": "dfo-ts/0.1.0 (browser)"
     },
+    {
+      "name": "ts-other-runtime",
+      "client": "dfo-ts",
+      "version": "0.1.0",
+      "runtime": {
+        "name": "other"
+      },
+      "value": "dfo-ts/0.1.0 (other)"
+    },
     {
       "name": "ts-pre-release-package",
       "client": "dfo-ts",

python/client-contract/src/dfo_client_contract/rules/file-selector.json

@@ -0,0 +1,47 @@
+{
+  "format_version": 1,
+  "rule": "file-selector",
+  "query_keys": [
+    "format",
+    "bundle",
+    "layout"
+  ],
+  "vectors": [
+    {
+      "name": "long-cut",
+      "rendering": {
+        "format": "parquet",
+        "bundle": false,
+        "without_party_totals": true
+      },
+      "query": "format=parquet&bundle=false"
+    },
+    {
+      "name": "long-plain",
+      "rendering": {
+        "format": "parquet",
+        "bundle": false
+      },
+      "query": "format=parquet&bundle=false"
+    },
+    {
+      "name": "frame-cut",
+      "rendering": {
+        "format": "parquet",
+        "bundle": false,
+        "layout": "frame",
+        "without_party_totals": true
+      },
+      "query": "format=parquet&bundle=false&layout=frame"
+    },
+    {
+      "name": "keys-in-listed-order",
+      "rendering": {
+        "layout": "frame",
+        "bundle": true,
+        "format": "csv"
+      },
+      "query": "format=csv&bundle=true&layout=frame"
+    }
+  ]
+}

python/client-contract/src/dfo_client_contract/rules/reading-order.json

@@ -1394,6 +1394,134 @@
           "dc/sv/gdp"
         ]
       ]
+    },
+    {
+      "name": "integer-and-float-are-one-value",
+      "members": [
+        {
+          "columns": [
+            "time_period",
+            "value",
+            "currency",
+            "base_year"
+          ],
+          "rows": [
+            [
+              2021,
+              5,
+              "USD",
+              2020
+            ],
+            [
+              2020,
+              4,
+              "USD",
+              2020.0
+            ]
+          ]
+        }
+      ],
+      "columns": [
+        "time_period",
+        "value",
+        "currency",
+        "base_year"
+      ],
+      "rows": [
+        [
+          2020,
+          4,
+          "USD",
+          2020.0
+        ],
+        [
+          2021,
+          5,
+          "USD",
+          2020
+        ]
+      ]
+    },
+    {
+      "name": "nested-cells-compare-by-value",
+      "members": [
+        {
+          "columns": [
+            "time_period",
+            "value",
+            "meta",
+            "tags",
+            "flags"
+          ],
+          "rows": [
+            [
+              2021,
+              5,
+              {
+                "a": 1,
+                "b": 2
+              },
+              [
+                1
+              ],
+              [
+                true
+              ]
+            ],
+            [
+              2020,
+              4,
+              {
+                "b": 2,
+                "a": 1
+              },
+              [
+                1.0
+              ],
+              [
+                1
+              ]
+            ]
+          ]
+        }
+      ],
+      "columns": [
+        "time_period",
+        "value",
+        "flags",
+        "meta",
+        "tags"
+      ],
+      "rows": [
+        [
+          2020,
+          4,
+          [
+            1
+          ],
+          {
+            "b": 2,
+            "a": 1
+          },
+          [
+            1.0
+          ]
+        ],
+        [
+          2021,
+          5,
+          [
+            true
+          ],
+          {
+            "a": 1,
+            "b": 2
+          },
+          [
+            1
+          ]
+        ]
+      ]
     }
   ]
 }