[d-v-b]

🤖 AI text below 🤖

Reading a metadata document is 2.5 to 4 times faster: a document is refined to JSON once, the model holds what was refined, and a model's key -- what == and hash compare -- is computed the first time the model is compared rather than when it is built. A definition's canonical is therefore first asked when a model is first compared. Every problem is reported as before, with the same loc, kind, message and input.

Timings

Per read, best of ten timeit repeats, Python 3.12, PYTHONHASHSEED=0, Apple M5, on documents as zarr==3.4.1 writes them (zarr.create_array(MemoryStore(), shape=(100, 100), chunks=(10, 10), dtype="f8") as v3 and v2, and a v3 group of 200 int32 arrays consolidated with zarr.consolidate_metadata), each round-tripped through json.dumps/json.loads:

case	before (c414f94)	after	speedup
ZarrV3ArrayMetadata.from_json	194.1 µs	72.5 µs	2.7x
ZarrV2ArrayMetadata.from_json	75.0 µs	18.2 µs	4.1x
ZarrV3GroupMetadata.from_json, 200 consolidated arrays	37.4 ms	14.7 ms	2.5x

For scale, zarr-python 3.4.1's own ArrayV3Metadata.from_dict reads the v3 array document in about 23 µs on the same machine. The benchmark is committed as tests/benchmarks/read_documents.py; run it from the package directory with PYTHONHASHSEED=0 uv run --group test python -m tests.benchmarks.read_documents (--profile prints a cProfile of the v3 array read). A smoke test reads each case once so the script keeps working.

Where the time went, and what changed

Profiling 1000 v3 array reads on the base branch put 55% of the time in resolve for the five extension points, 23% in _refine (58 calls per document: each field refined inside resolve, then the whole document refined again to build the model's document), 13% in the model's _adopt (key and claims computed at construction), 6% in json.dumps for the eager JSON-text keys, and 5% in Scope.format. The compiled type checker was about 4% and is untouched.

• One refine walk. read_array_v3, read_array_v2 and the group reader refine each member where they read it, hand the refined value to the new internal resolve_refined (what resolve is after it has refined), and assemble the document the model holds from what was refined, in the document's own order. ArrayMembersV3, ArrayMembersV2 and GroupMembersV3 carry that document; consolidated children's documents are taken from their members, so a group of 200 arrays is no longer refined a second time as a whole. refined_object is no longer called on the read paths.
• Lazy keys and claims. Keyed._model_key computes _key_of() the first time ==, hash, or a holding model asks, and keeps it; claims likewise. A model that is only read pays nothing for them. (This surfaced a latent sharing bug in ZarrV2GroupMetadata.__init__, which kept the caller's nested attribute containers; it now holds the refined document's.)
• Caches of what never changes. Context._format is computed once per scope in __post_init__; as_kind remembers its answer per way a kind is named; kind_of walks the MRO once per definition class; well_named is an lru_cache over names; json_text reuses one JSONEncoder; the get_args(...) calls in storage_of, the float fill-value rules and ValidationProblem.__post_init__ are module constants.
• Cheaper walks. _refine and frozen tell the exact types json.loads builds apart by type() before falling back to the ABC isinstance checks (subclasses, mapping proxies and other sequences take the same path as before); frozen keeps scalar leaves without a call per leaf; _put_back walks nothing when a configuration holds no nested field; one_of checks a frozenset of (type, value); within and _located return at once for no problems.

Verification

• just test (2543 passed, 1 skipped), just lint, just typecheck (pyright 1.1.x on 3.11, 0 errors) and just docs-check all pass.
• A differential run of the base branch's sources against this branch over 25,780 documents -- the three benchmark documents, every JSON fixture under tests/, and systematic mutations of each member and nested member (deleted, replaced by 90 kinds of junk, extra keys, must_understand: false, bad names, NaN, big ints, non-string keys, model entries in consolidated metadata, ...) -- produced byte-identical records of every validator's problems (loc, kind, message, input, ctx), every reading's fields, each model's to_json/to_key_value, equality and hash round trips, claims and canonical spellings.
• New tests: resolve_refined agrees with resolve on eight kinds of field, and its two error paths (a field that is not JSON; a scope of the other format); the model key is computed on first comparison (a counting canonical); mapping proxies, UserDict and dict/list subclasses refine as dict/list do; as_kind refuses an unhashable value as a non-kind; _document_of refuses members with no document; each benchmark case reads.

Stack bookkeeping

Base is feat/zarr-metadata-document-context-stack (upstream zarr-developers#4490, head c414f94). This PR is one commit on top of it plus the changelog fragment; rebase onto whatever zarr-developers#4490 becomes before it lands, and renumber the fragment if it is folded into an upstream PR.

🤖 Generated with Claude Code

[d-v-b]

🤖 AI text below 🤖

The agent folded this branch into feat/zarr-metadata-document-context-stack (upstream zarr-developers#4490) by fast-forward, with the merge of upstream main and the fixes from the review round, so this PR has nothing left to merge.