Skip to content

Commit 87bd5bc

Browse files
d-v-bd-v-b
authored
Merge branch 'main' into fix/validate-before-overwrite
2 parents c664fc6 + f9c1438 commit 87bd5bc

129 files changed

Lines changed: 12198 additions & 734 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/big-endian.yml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Run the byte-order-sensitive tests on a big-endian host: s390x, emulated with QEMU.
2+
# Emulation makes the full suite take about 40 minutes, so this job runs only the tests
3+
# that exercise data types, codecs and metadata, and only when that code changes (plus
4+
# a weekly run to catch anything the path filter misses).
5+
6+
name: Big-endian
7+
8+
on:
9+
pull_request:
10+
branches: [ main ]
11+
paths:
12+
- "src/zarr/codecs/**"
13+
- "src/zarr/core/buffer/**"
14+
- "src/zarr/core/dtype/**"
15+
- "src/zarr/core/metadata/**"
16+
- "src/zarr/metadata/**"
17+
- "tests/test_codecs/**"
18+
- "tests/test_dtype/**"
19+
- "tests/test_metadata/**"
20+
- "ci/big-endian/**"
21+
- ".github/workflows/big-endian.yml"
22+
schedule:
23+
- cron: "0 3 * * 1"
24+
workflow_dispatch:
25+
26+
permissions:
27+
contents: read
28+
29+
concurrency:
30+
group: ${{ github.workflow }}-${{ github.ref }}
31+
cancel-in-progress: true
32+
33+
jobs:
34+
s390x:
35+
name: s390x (QEMU)
36+
runs-on: ubuntu-latest
37+
timeout-minutes: 90
38+
steps:
39+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
40+
with:
41+
fetch-depth: 0 # hatch-vcs needs the tags to resolve a version
42+
persist-credentials: false
43+
- uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
44+
with:
45+
platforms: s390x
46+
- uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
47+
- name: Build test image
48+
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
49+
with:
50+
context: ci/big-endian
51+
platforms: linux/s390x
52+
tags: zarr-big-endian
53+
load: true
54+
cache-from: type=gha,scope=big-endian
55+
cache-to: type=gha,scope=big-endian,mode=max
56+
- name: Run tests
57+
run: |
58+
docker run --rm --platform linux/s390x -v "$PWD":/src:ro zarr-big-endian bash -euc '
59+
cp -r /src /work && cd /work
60+
pip install -q --no-deps --no-build-isolation .
61+
python -c "import sys; assert sys.byteorder == \"big\", sys.byteorder"
62+
python -m pytest -p no:cacheprovider --hypothesis-profile ci -n auto \
63+
tests/test_dtype tests/test_dtype_registry.py \
64+
tests/test_codecs tests/test_metadata \
65+
tests/test_array.py tests/test_v2.py tests/test_info.py \
66+
tests/test_pipeline_parity.py tests/test_cli/test_migrate_v3.py
67+
'

‎changes/4356.bugfix.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
Coordinate selections (including `vindex` masks) and orthogonal integer-array
2+
selections no longer allocate counts proportional to the total number of chunks.
3+
Previously, selecting a few points on an array with billions of chunks could raise
4+
`MemoryError`. The indexers now store occupied chunk IDs and run-end offsets
5+
(`chunk_run_ends`); unsorted selections can still require sorting. The dense
6+
`chunk_nitems` / `chunk_nitems_cumsum` attributes of `CoordinateIndexer` and
7+
`IntArrayDimIndexer` remain available without deprecation warnings. Each array is
8+
materialized on first access and cached, preserving its identity and in-place
9+
mutations on subsequent reads. These attributes remain dataclass fields, so
10+
`dataclasses.asdict()` materializes them; `repr` omits them to avoid dense allocation
11+
during inspection. Accessing a dense attribute still allocates memory proportional
12+
to the full chunk grid.
13+
14+
The new `zarr.core.common.ceildiv_int` helper uses exact integer arithmetic for
15+
chunk counts and slice calculations. For example, `ceildiv_int(2**62 - 1, 1)`
16+
returns `2**62 - 1`. The existing `ceildiv` helper retains its floating-point behavior.

‎changes/4414.bugfix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
An array `config` with a key that is not a runtime configuration option (for example a misspelled `write_empty_chunk`) now raises a `TypeError` naming the unknown and valid keys. Previously `create_array`, `create`, `open_array` and other functions silently ignored such keys.

‎changes/4435.bugfix.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Fix byte-order bugs found by running the test suite on a big-endian (s390x) host.
2+
3+
- Zarr V3 data type metadata has no byte order, so V3 data types without an explicit `endianness` (such as `Float64()`, or a data type parsed from `"float64"`) now use the host byte order instead of always little-endian. On big-endian hosts, arrays opened from V3 metadata now return native arrays, and `dtype="float64"` and `dtype=np.float64` produce the same data type. Nothing changes on little-endian hosts. Stored chunks are still little-endian by default.
4+
- `ShardingCodec` now defaults its inner and index `bytes` codecs to little-endian, so sharded arrays written with default settings are byte-identical on every host.
5+
- The `scale_offset` codec no longer fails on arrays whose byte order is not the host's, for example a `">f8"` or `">u8"` array on a little-endian host.
6+
- Base64-encoded fill values of structured data types in Zarr V3 metadata are now read and written as little-endian bytes, whatever the byte order of the fields in memory.
7+
- Tests no longer assume that the host is little-endian.

‎ci/big-endian/Dockerfile‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Test image for a big-endian host (s390x, run under QEMU by .github/workflows/big-endian.yml).
2+
# Debian packages supply numpy and numcodecs, so nothing heavy compiles under emulation;
3+
# this is also the environment Debian tests zarr in. Debian does not package Google's
4+
# crc32c library, so it is built here for the google-crc32c C extension.
5+
FROM debian:sid
6+
7+
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
8+
build-essential cmake git python3 python3-dev python3-pip python3-venv \
9+
python3-crc32c python3-donfig python3-fsspec python3-hatch-vcs python3-hatchling \
10+
python3-hypothesis python3-numcodecs python3-numpy python3-numpydoc python3-packaging \
11+
python3-pytest python3-pytest-asyncio python3-pytest-benchmark python3-pytest-xdist \
12+
python3-tomlkit python3-typer python3-typing-extensions \
13+
&& rm -rf /var/lib/apt/lists/*
14+
15+
RUN git clone -q --depth 1 --branch 1.1.2 https://github.com/google/crc32c /tmp/crc32c \
16+
&& cmake -S /tmp/crc32c -B /tmp/crc32c/build -DBUILD_SHARED_LIBS=1 \
17+
-DCRC32C_BUILD_TESTS=0 -DCRC32C_BUILD_BENCHMARKS=0 -DCRC32C_USE_GLOG=0 \
18+
-DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
19+
&& cmake --build /tmp/crc32c/build -j"$(nproc)" \
20+
&& cmake --install /tmp/crc32c/build \
21+
&& ldconfig \
22+
&& rm -rf /tmp/crc32c
23+
24+
RUN python3 -m venv --system-site-packages /venv
25+
ENV PATH=/venv/bin:$PATH
26+
RUN pip install -q --no-binary google-crc32c google-crc32c msgspec pytest-accept

‎packages/zarr-metadata/README.md‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Two layers and an optional integration:
1414
specifications, plus types for [`zarr-extensions`](https://github.com/zarr-developers/zarr-extensions/)
1515
and a few widely-used-but-unspecified entities (e.g. consolidated metadata).
1616
- **Document models** (`zarr_metadata.model`): canonical frozen-dataclass
17-
models of whole metadata documents, with structural validators, loc-aware
17+
models of whole metadata documents, with validators, loc-aware
1818
parsers, and store-key (de)serialization. A document produced by `to_json`
1919
shares no mutable state with the model that produced it.
2020
- **Optional Pydantic integration** (`zarr_metadata.pydantic`, requires
@@ -56,11 +56,20 @@ members that the strict model parser rejects.
5656

5757
The model validators enforce the declared document structure and a small set
5858
of context-free consistency rules, including fixed format literals, finite
59-
JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
60-
`dimension_names` entry per array dimension. They do not interpret extension
61-
names or configurations, resolve codec pipelines, or decide whether a data
62-
type, chunk grid, codec, or storage transformer is supported. Those decisions
63-
belong to consumer implementations.
59+
JSON numbers, non-negative dimensions, and one `dimension_names` entry per
60+
array dimension. In a v3 document they also read
61+
each extension point -- the data type, chunk grid, chunk key encoding, each
62+
codec and each storage transformer -- through the definition that claims its
63+
name in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: a
64+
configuration its definition refuses is refused, and a key it does not
65+
declare is reported as `unknown_key`. A name nothing in the scope claims is
66+
left unjudged, and whether to support it is the consumer's decision. A v3
67+
fill value is judged against the data type it names, by that data type's
68+
definition, the chunk grid against the shape, by the grid's definition,
69+
and the codecs as a pipeline: in order, each judged by its definition
70+
against the chunk it is handed, a shard's inner and index codecs too.
71+
The validators do no arithmetic on values: whether a fill value survives
72+
a `cast_value` round trip is not judged.
6473

6574
The Pydantic integration's generated JSON Schemas express independently
6675
checkable document structure and field constraints, but they are not a
@@ -75,7 +84,7 @@ versus `shape`. Consumers should run the model parser after schema validation.
7584
At minimum, this library supports what Zarr-Python needs: the complete
7685
Zarr v2 and v3 specs, consolidated metadata, and a subset of the metadata
7786
defined in `zarr-extensions`. We are generally open to contributions that
78-
add types, models, or structural validation for Zarr metadata with a
87+
add types, models, or validation for Zarr metadata with a
7988
published spec.
8089

8190
Runtime array behavior is out of scope: nothing here encodes or decodes

‎packages/zarr-metadata/changes/4420.bugfix.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
Attributes are user data, and a non-finite number in them is read,
2+
validated and written back. zarr-python writes attributes with the
3+
defaults of Python's `json` module, so an xarray `_FillValue` or a CF
4+
`missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks;
5+
the model refused such a document and could not read the store at all.
6+
A node's `attributes`, a v2 `.zattrs`, the `.zattrs` entries of a v2
7+
`.zmetadata`, and the attributes of the nodes a v3 group's inline
8+
`consolidated_metadata` holds may now hold `NaN`, `Infinity` and
9+
`-Infinity`, in every document `validate_*`, `parse_*` and `is_*`, and in
10+
`from_key_value` and `to_key_value`. Wherever the spec interprets a value
11+
it spells those numbers as strings, so anywhere else a non-finite number
12+
is still refused, and the store reader now says where one is instead of
13+
failing to decode the document. `is_json`, `validate_json` and
14+
`parse_json` judge by RFC 8259 alone, so a document the node guards
15+
accept can hold a value `is_json` refuses.
16+
17+
`to_key_value` validates the document it writes as `from_key_value`
18+
validates what it reads, and raises `MetadataValidationError` with every
19+
problem instead of writing a document the reader refuses. A model built
20+
by hand is not validated, so the writer is where these are caught:
21+
22+
- a non-finite number outside attributes, which was refused before too;
23+
- a non-string attribute key, which was written as a string;
24+
- a value that is not JSON, which raised `TypeError`;
25+
- anything else the reader refuses, such as `dimension_names` longer than
26+
`shape`, which was written and then refused on read.
27+
28+
Any sequence is written as an array. A number too large for a float, such
29+
as `1e400`, is read as infinity, so in attributes it is now accepted and
30+
written back as `Infinity`.
31+
32+
Through `zarr_metadata.pydantic`, pydantic writes a non-finite attribute
33+
as `null` unless the enclosing model sets `ser_json_inf_nan="constants"`.

‎packages/zarr-metadata/changes/4421.bugfix.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
`from_key_value` takes a mapping keyed by `str` or by any narrower key
2+
type, so reading back what `to_key_value` wrote type-checks. `to_key_value`
3+
keys its result by the store-key literals, and since a mapping's key type is
4+
invariant, a `Mapping[str, bytes]` parameter refused it.

‎packages/zarr-metadata/changes/4422.bugfix.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
`MetadataValidationError` pickles and copies with its message. A copy was
2+
built from the error's message, and its message came out as the
3+
original's characters, one per line.

‎packages/zarr-metadata/changes/4422.feature.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
**Breaking:** `ValidationProblem` and `MetadataValidationError` check at run
2+
time what their annotations declare, and raise `TypeError` otherwise. A
3+
problem's `loc` is a tuple of strings and integers, so a list, which was
4+
accepted before, is now refused, as is a `bool`; its `message` is a string
5+
and its `kind` one of the kinds; an error holds problems. A caller that is
6+
not type-checked fails where it builds the value, not later, where a
7+
location is read.

0 commit comments

Comments
 (0)