feat(registry): map every Presentation route and state where a frame refusal lands

python/frame-grammar/CHANGELOG.md

@@ -153,6 +153,15 @@ and this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   refuses a `composed` member whose Arrow type is neither a string type nor
   null with `schema_violation`. Canonical bytes, `frame_sha256` and
   `content_version` are unchanged for every frame.
+- `unit_cells` and `Frame.rows` read each stored `unit` struct value as the
+  Arrow build reads it, and raise `FrameError` for a struct with no member or
+  more than one or an empty ref (`schema_violation`) and for a member outside
+  its form: a request outside its closed shape, or `composed` text that is not
+  JSON, repeats a member name, or holds no composed unit (`invalid_unit`).
+  `unit_cells` reads each distinct value once. `unit_column_cells` gives a
+  `unit` column's per-row cells, plain or in JSON form, through it, and
+  `Frame.rows` and `canonical_frame_bytes` read the column that way, so
+  neither validates per row.
 - `unit_cells` returns `ComposedUnit` cells, and `collation_keys` ranks a
   composed unit 3, after a unit request at 2.
 - `canonical_frame_bytes` and `frame_sha256` stream through

python/frame-grammar/README.md

@@ -59,7 +59,7 @@ A consumer that keeps frames in Arrow uses the package root exports.
 
 - `Frame.from_table(columns=..., table=...)` wraps a table this package built, such as `frame.table` written to Parquet and read back. It checks `columns` and the schema against `canonical_frame_schema(columns)`, accepts the `unit` column in every layout a release stored, and trusts the cells and row order. Untrusted tables go through `build_frame_from_arrow`.
 - `canonical_frame_bytes` and `frame_sha256` serialise through `stream_canonical_frame_bytes`, 8,192 rows at a time. Hashing a 250,000-row frame raises peak RSS by about 4 MiB, against about 197 MiB for one whole-table document (`benchmarks/arrow_frame_rss.py`). A table whose rows are out of canonical order refuses with `FrameError` `row_key_not_increasing`; only `Frame.from_table` or the `Frame` constructor can produce one.
-- `unit_codes(unit)` gives each row an integer code, equal exactly where the `unit` values are equal, read from the struct's leaf dictionary codes. PyArrow cannot `dictionary_encode`, `unique` or `group_by` the `unit` struct itself. `UnitCodes.count` is the number of distinct units, `unit_cells` their values, and `map_units` applies a function once per distinct unit.
+- `unit_codes(unit)` gives each row an integer code, equal exactly where the `unit` values are equal, read from the struct's leaf dictionary codes. PyArrow cannot `dictionary_encode`, `unique` or `group_by` the `unit` struct itself. `UnitCodes.count` is the number of distinct units, `unit_cells` their values, validated as the Arrow build validates a `unit` column and raising `FrameError` for one outside the grammar's forms, and `map_units` applies a function once per distinct unit.
 - A non-null `unit` cell is a `UnitCell`: a ref string, a `UnitRequest` (`currency`, `prices`, `base_year` and an optional `recipe`), or a `ComposedUnit` whose `numerator` and `denominator` are each a ref string, a `UnitRequest`, or a `StatVarPart`. `unit_cell_from_json(value)` validates the JSON form and returns the cell, `unit_cell_to_json(cell)` gives the JSON form back, and `unit_cell_array(cell, length)` builds `length` rows of the cell in the canonical `unit` column type. Both take and return `None` for a null unit. `unit_cell_array` refuses a hand-built cell that validation would not return with `FrameError` `invalid_unit`. A reader on frame grammar `1.0.0` refuses a cell that carries `recipe`, so every reader takes `1.1.0` before any producer writes one.
 - The `unit` column is `struct<ref, request, composed>`. Its `request` struct gains a `recipe` child only in a frame where some cell carries a recipe, so a frame without one stores as frame grammar `1.0.0` stored it and a `1.0.0` reader reads it. `canonical_frame_schema(columns)` is that layout and `canonical_frame_schema(columns, recipe=True)` the recipe layout. `composed` is a string holding the composed unit's JCS text, so `unit_codes` and `collation_keys` read it as one leaf, and a column with no composed unit spends about 4 bytes a row on it (`benchmarks/unit_column_bytes.py`).
 - `widen_frame_table(table, recipe=False)` takes a canonical table whose `unit` column is in any layout a release stored, `struct<ref, request>` or `struct<ref, request, composed>` with or without `recipe`, and returns it in the recipe layout when `recipe` is true or some cell carries a recipe, and in the layout without `recipe` otherwise. `Frame.from_table` applies it, so a reader never calls it first. A table already in the result's layout comes back unchanged, and any other schema, a table with no `unit` column included, refuses with `FrameError` `schema_violation`.

python/frame-grammar/benchmarks/arrow_frame_rss.py

@@ -19,7 +19,7 @@
 from dfo_frame_grammar import grammar
 from dfo_frame_grammar.arrow import build_frame_from_arrow
 from dfo_frame_grammar.content_version import frame_sha256
-from dfo_frame_grammar.frame import Frame, _row_cells
+from dfo_frame_grammar.frame import Frame, _python_rows
 
 ROW_COUNT = 250_000
 CHUNK_COUNT = 44
@@ -62,7 +62,13 @@ def _rss_mib() -> float:
 
 
 def _whole_table_sha256(frame: Frame) -> str:
-    rows = [_row_cells(frame, row) for row in frame.table.to_pylist()]
+    rows = [
+        [
+            grammar.unit_cell_to_json(cell) if column == grammar.UNIT_COLUMN else cell
+            for column, cell in zip(frame.columns, row, strict=True)
+        ]
+        for row in _python_rows(frame.columns, frame.table)
+    ]
     document = {
         "frame_version": grammar.FRAME_VERSION,
         "columns": list(frame.columns),

python/frame-grammar/src/dfo_frame_grammar/__init__.py

@@ -37,6 +37,7 @@
     scalar_kernel,
     unit_cells,
     unit_codes,
+    unit_column_cells,
 )
 from dfo_frame_grammar.content_version import (
     ContentVersionError,
@@ -138,5 +139,6 @@
     "unit_cell_to_json",
     "unit_cells",
     "unit_codes",
+    "unit_column_cells",
     "widen_frame_table",
 ]

python/frame-grammar/src/dfo_frame_grammar/_unit_codec.py

@@ -9,12 +9,13 @@
 
 import json
 from collections.abc import Mapping
-from typing import cast
+from typing import Any, cast
 
 import pyarrow as pa
 from dfo_spec_kit.identity import canonicalize
 
 from dfo_frame_grammar import grammar
+from dfo_frame_grammar.grammar import FrameError
 
 
 def _unit_request_type(*, recipe: bool) -> pa.StructType:
@@ -130,33 +131,83 @@ def _unit_to_arrow(cell: grammar.UnitCell | None) -> dict[str, object] | None:
     return {"composed": canonicalize(cell.as_json()).decode("utf-8")}
 
 
-def _arrow_part_to_cell(value: object) -> grammar.UnitPart:
-    """A ref string, a `request` struct, or one side of a parsed composed
-    unit as its cell, trusted as canonical."""
-    if isinstance(value, str):
-        return value
-    part = cast("Mapping[str, object]", value)
-    if grammar.STAT_VAR_COLUMN in part:
-        return grammar.StatVarPart(stat_var=cast(str, part[grammar.STAT_VAR_COLUMN]))
-    return grammar.UnitRequest(
-        currency=cast(str, part["currency"]),
-        prices=cast(str, part["prices"]),
-        base_year=cast("int | None", part.get("base_year")),
-        recipe=cast("str | None", part.get("recipe")),
+def _without_nulls(struct: Mapping[Any, object]) -> dict[str, object]:
+    """The non-null members of a struct value as `to_pylist` reads it."""
+    return {key: value for key, value in struct.items() if value is not None}
+
+
+def _wrong_unit_type(value: object) -> FrameError:
+    return FrameError(
+        "schema_violation",
+        f"unit must be a string, mapping, or null, got {type(value)!r}",
     )
 
 
-def _arrow_unit_to_cell(value: object) -> grammar.UnitCell | None:
-    if value is None:
+def _unit_request_from_mapping(raw: object) -> grammar.UnitRequest | None:
+    if raw is None:
         return None
-    assert isinstance(value, Mapping)
-    tagged = cast("Mapping[str, object]", value)
-    composed = tagged["composed"]
-    if composed is None:
-        member = tagged["ref"] if tagged["ref"] is not None else tagged["request"]
-        return cast("str | grammar.UnitRequest", _arrow_part_to_cell(member))
-    sides = json.loads(cast(str, composed))
-    return grammar.ComposedUnit(
-        numerator=_arrow_part_to_cell(sides["numerator"]),
-        denominator=_arrow_part_to_cell(sides["denominator"]),
-    )
+    if not isinstance(raw, Mapping):
+        raise _wrong_unit_type(raw)
+    return grammar.unit_request_from_json(_without_nulls(raw))
+
+
+def _unique_members(pairs: list[tuple[str, object]]) -> dict[str, object]:
+    members = dict(pairs)
+    if len(members) != len(pairs):
+        raise FrameError("invalid_unit", "composed unit text repeats a member name")
+    return members
+
+
+def _composed_unit_from_text(text: str) -> grammar.ComposedUnit:
+    """Parse and validate the JSON text of a composed unit, refusing text
+    that is not JSON, repeats a member name, or holds anything but a
+    composed unit."""
+    try:
+        parsed = json.loads(text, object_pairs_hook=_unique_members)
+    # `ValueError` covers `JSONDecodeError` and an integer past Python's digit
+    # limit; `RecursionError` covers nesting past the interpreter's depth.
+    except (ValueError, RecursionError) as exc:
+        raise FrameError(
+            "invalid_unit", f"composed unit text is not JSON: {exc}"
+        ) from exc
+    if not isinstance(parsed, dict):
+        raise FrameError(
+            "invalid_unit",
+            f"composed unit text must hold a JSON object, got {type(parsed)!r}",
+        )
+    return grammar._composed_unit_from_json(parsed)
+
+
+def _canonical_unit_from_mapping(raw: object) -> grammar.UnitCell | None:
+    """The validated cell of one tagged struct value, read from its single
+    non-null member: a `ref` string, a `request` struct with its null
+    members dropped, or the JSON text of a `composed` unit. A null struct
+    is a null unit. `unit_cells`, `Frame.rows` and the Arrow build read a
+    `unit` column through it, so each refuses a stored cell outside these
+    forms.
+
+    Raises:
+        FrameError: `schema_violation` for a struct with no member or more
+            than one, a member of the wrong type, or an empty ref;
+            `invalid_unit` for a request or composed member outside its
+            form.
+    """
+    if raw is None:
+        return None
+    if not isinstance(raw, Mapping):
+        raise _wrong_unit_type(raw)
+    variants = _without_nulls(raw)
+    if len(variants) != 1:
+        raise FrameError(
+            "schema_violation", "unit tagged struct must carry one variant"
+        )
+    ((member, value),) = variants.items()
+    if member == "request":
+        return _unit_request_from_mapping(value)
+    if member == "composed":
+        if not isinstance(value, str):
+            raise _wrong_unit_type(value)
+        return _composed_unit_from_text(value)
+    if not isinstance(value, str):
+        raise _wrong_unit_type(value)
+    return grammar.unit_cell_from_json(value)

python/frame-grammar/src/dfo_frame_grammar/arrow.py

@@ -3,17 +3,17 @@
 from __future__ import annotations
 
 import io
-import json
 import unicodedata
-from collections.abc import Callable, Iterable, Mapping
-from typing import Any, cast
+from collections.abc import Callable, Iterable
 
 import pyarrow as pa
 import pyarrow.csv as pacsv
 
 from dfo_frame_grammar import grammar
 from dfo_frame_grammar._unit_codec import (
+    _canonical_unit_from_mapping,
     _conform_unit,
+    _unit_request_from_mapping,
     _unit_to_arrow,
     _unit_type,
 )
@@ -158,79 +158,6 @@ def _normalise_value(array: pa.ChunkedArray) -> pa.Array:
     )
 
 
-def _without_nulls(struct: Mapping[Any, object]) -> dict[str, object]:
-    """The non-null members of a struct value as `to_pylist` reads it."""
-    return {key: value for key, value in struct.items() if value is not None}
-
-
-def _wrong_unit_type(value: object) -> FrameError:
-    return FrameError(
-        "schema_violation",
-        f"unit must be a string, mapping, or null, got {type(value)!r}",
-    )
-
-
-def _unit_request_from_mapping(raw: object) -> grammar.UnitRequest | None:
-    if raw is None:
-        return None
-    if not isinstance(raw, Mapping):
-        raise _wrong_unit_type(raw)
-    return grammar.unit_request_from_json(_without_nulls(raw))
-
-
-def _unique_members(pairs: list[tuple[str, object]]) -> dict[str, object]:
-    members = dict(pairs)
-    if len(members) != len(pairs):
-        raise FrameError("invalid_unit", "composed unit text repeats a member name")
-    return members
-
-
-def _composed_unit_from_text(text: str) -> grammar.ComposedUnit:
-    """Parse and validate the JSON text of a composed unit, refusing text
-    that is not JSON, repeats a member name, or holds anything but a
-    composed unit."""
-    try:
-        parsed = json.loads(text, object_pairs_hook=_unique_members)
-    # `ValueError` covers `JSONDecodeError` and an integer past Python's digit
-    # limit; `RecursionError` covers nesting past the interpreter's depth.
-    except (ValueError, RecursionError) as exc:
-        raise FrameError(
-            "invalid_unit", f"composed unit text is not JSON: {exc}"
-        ) from exc
-    if not isinstance(parsed, dict):
-        raise FrameError(
-            "invalid_unit",
-            f"composed unit text must hold a JSON object, got {type(parsed)!r}",
-        )
-    return grammar._composed_unit_from_json(parsed)
-
-
-def _canonical_unit_from_mapping(raw: object) -> grammar.UnitCell | None:
-    """The validated cell of one tagged struct value, read from its single
-    non-null member: a `ref` string, a `request` struct with its null
-    members dropped, or the JSON text of a `composed` unit. A null struct
-    is a null unit. `_refuse_untyped_composed` has already refused a
-    `composed` member that is not text.
-    """
-    if raw is None:
-        return None
-    if not isinstance(raw, Mapping):
-        raise _wrong_unit_type(raw)
-    variants = _without_nulls(raw)
-    if len(variants) != 1:
-        raise FrameError(
-            "schema_violation", "unit tagged struct must carry one variant"
-        )
-    ((member, value),) = variants.items()
-    if member == "request":
-        return _unit_request_from_mapping(value)
-    if member == "composed":
-        return _composed_unit_from_text(cast(str, value))
-    if not isinstance(value, str):
-        raise _wrong_unit_type(value)
-    return grammar.unit_cell_from_json(value)
-
-
 # The tagged spellings of the `unit` struct: the canonical three-member
 # layout and the two-member layout, which holds no composed unit. A struct
 # with any other member names is a bare unit request.

python/frame-grammar/src/dfo_frame_grammar/columnar.py

@@ -8,7 +8,8 @@
 
 from __future__ import annotations
 
-from collections.abc import Callable, Sequence
+import json
+from collections.abc import Callable, Mapping, Sequence
 from dataclasses import dataclass
 from typing import cast
 
@@ -18,9 +19,9 @@
 
 from dfo_frame_grammar import grammar
 from dfo_frame_grammar._unit_codec import (
-    _arrow_part_to_cell,
-    _arrow_unit_to_cell,
+    _canonical_unit_from_mapping,
     _unit_type,
+    _without_nulls,
 )
 from dfo_frame_grammar.grammar import FrameError
 
@@ -34,6 +35,7 @@
     "scalar_kernel",
     "unit_cells",
     "unit_codes",
+    "unit_column_cells",
 ]
 
 _UNIT_TYPES = (_unit_type(recipe=False), _unit_type(recipe=True))
@@ -226,14 +228,48 @@ def unit_cells(
     `grammar.UnitRequest`, a `grammar.ComposedUnit`, or `None`.
     """
     distinct = array_kernel("take", _combined(unit), codes.first_rows).to_pylist()
-    return [_arrow_unit_to_cell(value) for value in distinct]
+    return [_canonical_unit_from_mapping(value) for value in distinct]
+
+
+def unit_column_cells(
+    unit: pa.Array | pa.ChunkedArray, *, json_form: bool = False
+) -> list[object]:
+    """Each row's cell of a canonical `unit` column, read through
+    `unit_cells`, so each distinct struct value is validated once and the
+    rows that share it share one object.
+
+    A cell has the form `Frame.rows` gives it, or with `json_form` its JSON
+    form as `json.loads` parses its JCS text: a ref string, a mapping with
+    sorted keys, or `None`.
+
+    Raises:
+        FrameError: For a stored value outside the `unit` cell's forms
+            (`unit_cells`).
+    """
+    codes = unit_codes(unit)
+    distinct: list[object] = list(unit_cells(unit, codes))
+    if json_form:
+        distinct = [
+            _parsed_json_form(cast("grammar.UnitCell | None", cell))
+            for cell in distinct
+        ]
+    return [distinct[code] for code in codes.codes.to_pylist()]
+
+
+def _parsed_json_form(cell: grammar.UnitCell | None) -> object:
+    """`cell`'s JSON form as `json.loads` parses its JCS text."""
+    if cell is None or isinstance(cell, str):
+        return cell
+    return json.loads(canonicalize(cell.as_json()))
 
 
 def _request_payload_of(value: object) -> bytes | None:
     if value is None:
         return None
-    request = cast("grammar.UnitRequest", _arrow_part_to_cell(value))
-    return canonicalize(request.as_json())
+    request = cast("Mapping[str, object]", value)
+    return canonicalize(
+        grammar.unit_request_from_json(_without_nulls(request)).as_json()
+    )
 
 
 def _unit_payload(unit: pa.Array) -> pa.Array: