Skip to content

Commit a994a4f

Browse files
d-v-bd-v-b
authored
feat: add the zarr-indexing package (TensorStore-style index transforms, ndsel wire format) (#4196)
* fix: byte-order handling for structured dtypes in the bytes codec (#220) * fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes #4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5 * feat: add the zarr-indexing package (TensorStore-style index transforms) Standalone workspace package extracted from the lazy-indexing branch (#3906): composable, lazy coordinate transforms (IndexTransform / IndexDomain / output maps), dependency-aware chunk resolution against a DimensionGridLike protocol, and an ndsel-conformant JSON wire format validated against the vendored conformance corpus. zarr itself does not depend on zarr-indexing yet — the runtime wiring lands separately once 0.1.0 is published. The package is numpy-only; its tests exercise chunk resolution against zarr's concrete ChunkGrid, so they run from the workspace root (uv sync --all-packages). Assisted-by: ClaudeCode:claude-fable-5 * style: conventional submodule import in the chunk-resolution tests Assisted-by: ClaudeCode:claude-fable-5 * perf(zarr-indexing): joint chunk enumeration for correlated vindex maps Candidate-chunk enumeration took the cartesian product of each correlated ArrayMap's per-dimension distinct chunk ids and relied on intersect() to filter untouched combinations. For a diagonal selection of P scattered points that is P**2 intersect calls — quadratic in the number of selected points, the same workload shape as #4174 (400 points: ~2.6s; 10k points: ~30min). Group correlated maps jointly instead: broadcast their per-point chunk ids, take the distinct rows (np.unique(axis=0), O(P log P)), and enumerate exactly the touched combinations. Candidate slots now carry chunk-coordinate tuples covering one or more output dimensions; orthogonal/constant/slice dimensions keep their existing per-dimension candidates. 400-point diagonal resolution drops from 2628ms to 14ms and scales linearly. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): standalone documentation site; add package justfile Mirror the treatment zarr-metadata received in #4208/#4210 onto zarr-indexing: a self-contained mkdocs site under the package (own mkdocs.yml, landing page, ndsel wire-format guide, mkdocstrings page per module, and .readthedocs.yaml for a dedicated RTD project), so the package presents as a separate project with docs versioned by its own zarr_indexing-v* release tags rather than zarr-python's. The zarr-python site's API Reference nav links out to it, and each RTD project now skips PR builds that do not touch its half of the repo. The package gains a pinned docs dependency group, a docs build job in its CI workflow, and a justfile with package-scoped dev recipes. Two recipes deviate from the zarr-metadata original by design: - `test` runs against the workspace-root environment (`uv run --project ../.. --all-packages --group test`), because the chunk-resolution tests exercise this package against zarr's chunk grids and `zarr` is deliberately not a dependency of this package. - `typecheck` uses plain `pyright`, unpinned and on the default interpreter, mirroring this package's own CI invocation. The zarr-metadata pin exists for a PEP 661 sentinel regression that zarr-indexing's sources do not hit. composition.py gains the module docstring the other modules already have, since mkdocstrings renders it as the page introduction. Assisted-by: ClaudeCode:claude-fable-5 * chore: drop the already-released 4141 changelog fragment The bytes-codec byte-order fix this fragment describes shipped upstream and its entry is already in docs/release-notes.md; the fragment survived on this branch only as a rebase remnant, and would emit a duplicate entry in the next release. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): canonicalize ndsel references to zarr-developers/ndsel Also aligns the zarr-indexing workflow's setup-uv pin (v8.3.2) with the rest of the repo. The vendored-corpus sha is present upstream; the historical d-v-b/ndsel#1 PR reference stays as provenance. Assisted-by: ClaudeCode:claude-fable-5 * chore: drop the root uv-workspace wiring for zarr-indexing Per review: the root pyproject.toml should not change in this PR. The package now operates fully standalone (like zarr-metadata); the test invocations layer the package into the repo-root environment as an editable overlay instead (python -m pytest, since a base-env console script would not see the overlay). Assisted-by: ClaudeCode:claude-fable-5
1 parent 53e6dc6 commit a994a4f

57 files changed

Lines changed: 6955 additions & 8 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/check_changelogs.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,6 @@ jobs:
2929

3030
- name: Check zarr-metadata changelog entries
3131
run: uv run --no-sync python ci/check_changelog_entries.py packages/zarr-metadata/changes
32+
33+
- name: Check zarr-indexing changelog entries
34+
run: uv run --no-sync python ci/check_changelog_entries.py packages/zarr-indexing/changes

‎.github/workflows/zarr-indexing-release.yml‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
name: zarr-indexing release
2+
3+
on:
4+
workflow_dispatch:
5+
push:
6+
tags:
7+
- 'zarr_indexing-v*'
8+
9+
permissions:
10+
contents: read
11+
12+
concurrency:
13+
group: ${{ github.workflow }}-${{ github.ref }}
14+
cancel-in-progress: false
15+
16+
jobs:
17+
build:
18+
name: Build wheel and sdist
19+
runs-on: ubuntu-latest
20+
defaults:
21+
run:
22+
shell: bash
23+
working-directory: packages/zarr-indexing
24+
steps:
25+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
26+
with:
27+
persist-credentials: false
28+
fetch-depth: 0 # hatch-vcs needs full history + tags
29+
30+
- name: Install Hatch
31+
uses: pypa/hatch@257e27e51a6a5616ed08a39a408a21c35c9931bc
32+
with:
33+
version: '1.16.5'
34+
35+
- name: Build
36+
run: hatch build
37+
38+
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
39+
with:
40+
name: zarr-indexing-dist
41+
path: packages/zarr-indexing/dist
42+
43+
test_artifacts:
44+
name: Test built artifacts
45+
needs: [build]
46+
runs-on: ubuntu-latest
47+
steps:
48+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
49+
with:
50+
name: zarr-indexing-dist
51+
path: dist
52+
53+
- name: Install uv
54+
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
55+
with:
56+
enable-cache: false
57+
58+
- name: Set up Python
59+
run: uv python install 3.12
60+
61+
- name: Install built wheel and run import smoke test
62+
run: |
63+
wheel=$(ls dist/*.whl)
64+
uv run --with "${wheel}" --python 3.12 --no-project \
65+
python -c "import zarr_indexing; print('zarr_indexing', zarr_indexing.__version__)"
66+
67+
upload_pypi:
68+
name: Upload to PyPI
69+
needs: [build, test_artifacts]
70+
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/zarr_indexing-v')
71+
runs-on: ubuntu-latest
72+
environment:
73+
name: zarr-indexing-releases
74+
url: https://pypi.org/p/zarr-indexing
75+
permissions:
76+
id-token: write # required for OIDC trusted publishing
77+
attestations: write # required for artifact attestations
78+
steps:
79+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
80+
with:
81+
name: zarr-indexing-dist
82+
path: dist
83+
84+
- name: Generate artifact attestation
85+
uses: actions/attest@f7c74d28b9d84cb8768d0b8ca14a4bac6ef463e6 # v4.2.0
86+
with:
87+
subject-path: dist/*
88+
89+
- name: Publish package to PyPI
90+
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
91+
92+
upload_testpypi:
93+
name: Upload to TestPyPI
94+
needs: [build, test_artifacts]
95+
if: github.event_name == 'workflow_dispatch'
96+
runs-on: ubuntu-latest
97+
environment:
98+
name: zarr-indexing-releases-test
99+
url: https://test.pypi.org/p/zarr-indexing
100+
permissions:
101+
id-token: write
102+
attestations: write
103+
steps:
104+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
105+
with:
106+
name: zarr-indexing-dist
107+
path: dist
108+
109+
- name: Generate artifact attestation
110+
uses: actions/attest@f7c74d28b9d84cb8768d0b8ca14a4bac6ef463e6 # v4.2.0
111+
with:
112+
subject-path: dist/*
113+
114+
- name: Publish package to TestPyPI
115+
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
116+
with:
117+
repository-url: https://test.pypi.org/legacy/

‎.github/workflows/zarr-indexing.yml‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
name: zarr-indexing
2+
3+
on:
4+
push:
5+
branches: [main]
6+
paths:
7+
- 'packages/zarr-indexing/**'
8+
- '.github/workflows/zarr-indexing.yml'
9+
pull_request:
10+
paths:
11+
- 'packages/zarr-indexing/**'
12+
- '.github/workflows/zarr-indexing.yml'
13+
workflow_dispatch:
14+
15+
permissions:
16+
contents: read
17+
18+
concurrency:
19+
group: ${{ github.workflow }}-${{ github.ref }}
20+
cancel-in-progress: true
21+
22+
jobs:
23+
test:
24+
name: pytest py=${{ matrix.python-version }}
25+
runs-on: ubuntu-latest
26+
defaults:
27+
run:
28+
shell: bash
29+
strategy:
30+
fail-fast: false
31+
matrix:
32+
python-version: ['3.12', '3.13', '3.14']
33+
steps:
34+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
35+
with:
36+
persist-credentials: false
37+
- name: Install uv
38+
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
39+
with:
40+
enable-cache: true
41+
- name: Set up Python ${{ matrix.python-version }}
42+
run: uv python install ${{ matrix.python-version }}
43+
# The transform tests exercise chunk resolution against zarr's ChunkGrid,
44+
# so they run from the repo root against the root environment (which
45+
# provides `zarr`) with this package as an editable overlay rather than in
46+
# package isolation.
47+
- name: Sync test dependency group
48+
run: uv sync --group test --python ${{ matrix.python-version }}
49+
- name: Run pytest
50+
run: uv run --no-sync --group test --with-editable ./packages/zarr-indexing python -m pytest packages/zarr-indexing/tests
51+
52+
ruff:
53+
name: ruff
54+
runs-on: ubuntu-latest
55+
defaults:
56+
run:
57+
shell: bash
58+
working-directory: packages/zarr-indexing
59+
steps:
60+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
61+
with:
62+
persist-credentials: false
63+
- name: Install uv
64+
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
65+
- name: Run ruff
66+
run: uvx ruff check .
67+
68+
pyright:
69+
name: pyright
70+
runs-on: ubuntu-latest
71+
defaults:
72+
run:
73+
shell: bash
74+
working-directory: packages/zarr-indexing
75+
steps:
76+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
77+
with:
78+
persist-credentials: false
79+
- name: Install uv
80+
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
81+
with:
82+
enable-cache: true
83+
- name: Set up Python
84+
run: uv python install 3.12
85+
- name: Sync test dependency group
86+
run: uv sync --group test --python 3.12
87+
- name: Run pyright
88+
run: uv run --group test --with pyright pyright src
89+
90+
docs:
91+
name: docs
92+
runs-on: ubuntu-latest
93+
defaults:
94+
run:
95+
shell: bash
96+
working-directory: packages/zarr-indexing
97+
steps:
98+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
99+
with:
100+
persist-credentials: false
101+
- name: Install uv
102+
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
103+
with:
104+
enable-cache: true
105+
- name: Install just
106+
uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4
107+
- name: Build docs
108+
# The strict mkdocs build lives in packages/zarr-indexing/justfile.
109+
run: just docs-check
110+
111+
zarr-indexing-complete:
112+
name: zarr-indexing complete
113+
needs: [test, ruff, pyright, docs]
114+
if: always()
115+
runs-on: ubuntu-latest
116+
steps:
117+
- name: Check failure
118+
if: |
119+
contains(needs.*.result, 'failure') ||
120+
contains(needs.*.result, 'cancelled')
121+
run: exit 1
122+
- name: Success
123+
run: echo Success!

‎.readthedocs.yaml‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,14 @@ build:
66
python: "3.12"
77
jobs:
88
post_checkout:
9-
# Cancel pull request builds whose changes are confined to the
10-
# zarr-metadata package, which has its own Read the Docs project. Exit
11-
# code 183 cancels the build and reports success to the Git provider.
12-
# Scoped to PR builds ("external" versions) because origin/main is only
13-
# a meaningful diff base there. Read the Docs strips shell quoting from
14-
# commands, so the exclude pathspec must use the quote-free :! form,
15-
# not ':(exclude)'.
9+
# Cancel pull request builds whose changes are confined to the packages
10+
# that have their own Read the Docs projects. Exit code 183 cancels the
11+
# build and reports success to the Git provider. Scoped to PR builds
12+
# ("external" versions) because origin/main is only a meaningful diff
13+
# base there. Read the Docs strips shell quoting from commands, so the
14+
# exclude pathspecs must use the quote-free :! form, not ':(exclude)'.
1615
- |
17-
if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/main -- :!packages/zarr-metadata;
16+
if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/main -- :!packages/zarr-metadata :!packages/zarr-indexing;
1817
then
1918
exit 183;
2019
fi

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,7 @@ nav:
9595
- '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.zeros</code>': api/zarr/functions/zeros.md
9696
- '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.zeros_like</code>': api/zarr/functions/zeros_like.md
9797
- 'zarr-metadata ↪': https://zarr-metadata.readthedocs.io/
98+
- 'zarr-indexing ↪': https://zarr-indexing.readthedocs.io/
9899
- release-notes.md
99100
- contributing.md
100101
- Blog:

‎packages/zarr-indexing/.readthedocs.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Read the Docs configuration for the zarr-indexing docs site, separate from
2+
# the zarr-python site configured by the repo-root .readthedocs.yaml. The RTD
3+
# project for zarr-indexing must set its configuration-file path to
4+
# packages/zarr-indexing/.readthedocs.yaml.
5+
version: 2
6+
7+
build:
8+
os: ubuntu-22.04
9+
tools:
10+
python: "3.12"
11+
jobs:
12+
post_checkout:
13+
# Cancel pull request builds that do not touch this package. Exit code
14+
# 183 cancels the build and reports success to the Git provider. Scoped
15+
# to PR builds ("external" versions) because origin/main is only a
16+
# meaningful diff base there.
17+
- |
18+
if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/main -- packages/zarr-indexing;
19+
then
20+
exit 183;
21+
fi
22+
install:
23+
- pip install --upgrade pip
24+
- pip install ./packages/zarr-indexing --group packages/zarr-indexing/pyproject.toml:docs
25+
build:
26+
html:
27+
- mkdocs build --strict -f packages/zarr-indexing/mkdocs.yml --site-dir $READTHEDOCS_OUTPUT/html
28+
29+
mkdocs:
30+
configuration: packages/zarr-indexing/mkdocs.yml

‎packages/zarr-indexing/CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Release notes
2+
3+
<!-- towncrier release notes start -->

‎packages/zarr-indexing/LICENSE.txt‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
The MIT License (MIT)
2+
3+
Copyright (c) 2015-2025 Zarr Developers <https://github.com/zarr-developers>
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎packages/zarr-indexing/README.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# zarr-indexing
2+
3+
Composable, lazy coordinate transforms for Zarr array indexing.
4+
5+
Documentation: <https://zarr-indexing.readthedocs.io/>
6+
7+
This package implements TensorStore-inspired index transforms. The core idea:
8+
every indexing operation (slicing, fancy indexing, etc.) produces a coordinate
9+
mapping from user space to storage space. These mappings compose lazily — no
10+
I/O until you explicitly read or write.
11+
12+
Key types:
13+
14+
- `IndexDomain` — a rectangular region of integer coordinates
15+
- `IndexTransform` — maps input coordinates to storage coordinates
16+
- `ConstantMap`, `DimensionMap`, `ArrayMap` — the three ways a single output
17+
dimension can depend on the input
18+
- `compose` — chain two transforms into one
19+
20+
The package depends only on NumPy and the standard library; it does not import
21+
`zarr`. It is developed in the [zarr-python](https://github.com/zarr-developers/zarr-python)
22+
repository and consumed by `zarr` to resolve array indexing operations.
23+
24+
## Installation
25+
26+
```bash
27+
pip install zarr-indexing
28+
```
29+
30+
## Developing
31+
32+
Package-scoped development commands live in the [`justfile`](./justfile)
33+
(requires [just](https://github.com/casey/just)):
34+
35+
```
36+
just test # run the test suite (extra args go to pytest)
37+
just lint # ruff, same invocation as CI
38+
just typecheck # pyright, same invocation as CI
39+
just docs-check # strict build of the docs site
40+
just check # all of the above
41+
just docs-serve # serve the docs site locally
42+
```
43+
44+
Run them from this directory, or from anywhere in the repository as
45+
`just packages/zarr-indexing/<recipe>`.
46+
47+
The test recipe runs against the workspace-root environment, because the
48+
chunk-resolution tests exercise this package against `zarr`'s chunk grids and
49+
`zarr` is deliberately not a dependency of this package.
50+
51+
## License
52+
53+
MIT

‎packages/zarr-indexing/changes/3906.feature.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Reworked the JSON layer to conform to the [ndsel](https://github.com/zarr-developers/ndsel) draft wire format, which adapts TensorStore's `IndexTransform`. A new `zarr_indexing.messages` module (`parse_ndsel`, `normalize_ndsel`, `NdselError`) is a pure JSON-to-JSON layer that accepts all five message kinds (`point`/`box`/`slice`/`points`/`transform`) and normalizes them to the canonical transform body, enforcing the full ndsel error taxonomy. The package is checked against the vendored, language-agnostic ndsel conformance corpus. `index_transform_to_json`/`index_transform_from_json` (and the domain variants) now produce and consume the canonical body. On serialization, orthogonal (`oindex`) `index_array` maps no longer emit `input_dimension` alongside `index_array` (a combination both ndsel and TensorStore reject), and degenerate all-singleton index arrays collapse to constant maps; the in-memory `input_dimension` is reconstructed from the array's dependency axes on load.

0 commit comments

Comments
 (0)