fix(metadata)!: require int chunk sizes of at least 1 in metadata constructors and chunk specifications

changes/4375.bugfix.md

@@ -0,0 +1,3 @@
+Arrays whose stored `regular` chunk grid mixes chunk sizes with lists of chunk edge lengths, such as `"chunk_shape": [2, [5, 10, 5]]` (written by zarr 3.2.0 and 3.2.1 for `chunks=(2, (5, 10, 5))`, and as `[2, [5.0, 10.0, 5.0]]` for float edges), can be read again, without enabling `array.rectilinear_chunks`. The grid is read as the rectilinear chunk grid it describes, with a `ZarrUserWarning`; re-saving the array's metadata (or writing chunks to it) stores that rectilinear grid, which requires the flag. A group's consolidated metadata keeps such an array's metadata as it was stored.
+
+The `array.rectilinear_chunks` flag now gates reading and storing array metadata that declares a rectilinear chunk grid, including in a group's consolidated metadata, instead of constructing `RectilinearChunkGridMetadata`. An operation that would store such metadata with the flag off (creating an array, `Array.resize`, deleting a member of a group with consolidated metadata, `create_hierarchy`) raises before it deletes or writes anything, and the error names the array, also when it is a member of a group's consolidated metadata.

changes/4376.bugfix.md

@@ -0,0 +1 @@
+Chunk specifications are judged only by the chunk normalizer: the separate duck-typed check for rectilinear input is gone, so the Zarr format 2 and sharding restrictions apply to what the normalized grid says was declared. A chunk size is an integer by Python's integer protocol: numpy integer scalars and 0-d integer arrays are accepted wherever a chunk or shard shape is. The legacy `zarr.create(..., zarr_format=2)` no longer tests the truth value of a numpy array given as `chunks`, so numpy arrays work there. A non-integer scalar chunk specification raises a `TypeError` naming the problem. An explicit list of chunk edges declares a rectilinear dimension everywhere, including a stored rectilinear chunk grid passed as `chunks=` whose edges happen to be uniform, so the Zarr format 2 and sharding restrictions treat it the same whether it is given as lists or as metadata, and `zarr.create` stores it as a rectilinear chunk grid, as `create_array` does. A `RectilinearChunkGridMetadata` made only of bare integers declares no explicit edge list, so it is read as the regular grid it describes: it is accepted for Zarr format 2 and as the chunk shape of a sharded array.

changes/4377.misc.md

@@ -0,0 +1 @@
+The hypothesis strategies in `zarr.testing.strategies` now sample the full space of rectilinear chunk grid declarations: `arrays()` passes a rectilinear grid to `create_array` either as metadata or as a `chunks=` specification mixing bare-int steps and explicit edge lists in any arrangement, and the new `rectilinear_chunk_shape_declarations` and `rectilinear_chunk_grids` cover stored metadata, including run-length encoded edge lists in canonical and arbitrary groupings and edges overhanging the array extent. `rectilinear_chunks` and `chunk_grids` also draw zero-length dimensions.

changes/4431.removal.md

@@ -0,0 +1,7 @@
+Metadata built in code is strict about chunk edge lengths: `ArrayV2Metadata(chunks=...)`, `RegularChunkGridMetadata`, `RectilinearChunkGridMetadata` and `ShardingCodec(chunk_shape=...)` take them as Python `int`s of at least 1. A chunk size of 0 now raises a `ValueError`, as in `ArrayV2Metadata(chunks=(0,))` for an empty array. Code that passes an array's shape as its chunk shape, such as `ArrayV2Metadata(chunks=arr.shape)` or `RegularChunkGridMetadata(chunk_shape=arr.shape)`, fails for an array with a zero-length axis: use `tuple(max(1, s) for s in arr.shape)`. A `bool` or a NumPy integer, including a scalar `chunks=np.int64(5)`, raises a `TypeError`, as a float already did, and so does a chunk shape that is not a list or tuple. This reverses what 3.4.1 said of these classes: `ArrayV2Metadata(chunks=(0,))` is no longer accepted, and a `bool` chunk edge length is no longer read as the `int` it equals. (This entry assumes 3.4.1 is released before this release.) The array creation functions still accept NumPy integers in `chunks=` and `shards=`.
+
+This applies to stored metadata too wherever the stored-document repairs do not read it: a stored chunk shape that is a bare integer instead of a list, as in a Zarr format 2 `"chunks": 4` or a sharding codec's `"chunk_shape": 2`, which earlier releases read as a one-dimensional chunk shape, now raises a `TypeError` when the array is opened. A group whose consolidated metadata includes such an array does not open from its consolidated metadata at all, and the error names that member: open the group with `use_consolidated=False` to reach its other members. A stored `true` as a rectilinear chunk grid's bare step (`"chunk_shapes": [true]`) or as the count of a run-length pair (`[[4, true]]`), which no known writer stored, raises a `TypeError` too. The chunk sizes the repairs read open as before: 0 or `false` as a regular chunk size (in Zarr format 2 `chunks` or a `regular` grid's `chunk_shape`); `true` as a regular chunk size, as an edge in a rectilinear edge list, as the size of a run-length pair, or as a sharding codec's inner chunk size; and integral float edges in a rectilinear chunk grid (`[[4.0, 2]]`).
+
+Other than `chunks=False` or `shards=False` as the whole specification (one chunk spanning every axis), a boolean anywhere in a chunk specification given to the array creation functions (`chunks=True`, `np.True_`, `np.False_`, a boolean array, `chunks=(True, 5)`, `chunks=(False, 5)`, or a boolean edge in a list) raises one `TypeError`, instead of being read as a chunk size of 1 or 0 or raising a different error for each spelling. `chunks=True` and `create_array(chunks=None)` raised a `ValueError` before; both errors say how to ask for automatic chunking. The legacy `zarr.create(..., zarr_format=2)` no longer reads a falsy `chunks` as not given: `chunks=0`, `np.int64(0)` and `chunks=[]` raise a `ValueError`, and `chunks=False` means one chunk spanning the array, as it does for Zarr format 3, instead of automatic chunking; `chunks=None` still chunks automatically.
+
+`zarr.testing.strategies.chunks_param_from_rectilinear` is removed; nothing in zarr used it after `arrays()` began drawing its `chunks=` argument from `rectilinear_chunks`. `rectilinear_chunks` now returns `list[int | list[int]]` instead of `list[list[int]]`: a dimension may be a bare int step, which can exceed the extent, so code that treats every dimension as a list of edges (`len(dim)`, `sum(dim)`) must handle ints. It also requires `shape` to have at least one dimension.

src/zarr/codecs/sharding.py

@@ -45,9 +45,9 @@
     merge_and_encode_chunk,
 )
 from zarr.core.common import (
-    ShapeLike,
+    ChunkShape,
+    parse_chunk_shape,
     parse_named_configuration,
-    parse_shapelike,
     product,
 )
 from zarr.core.config import config as zarr_config
@@ -459,7 +459,7 @@ class ShardingCodec(
     def __init__(
         self,
         *,
-        chunk_shape: ShapeLike,
+        chunk_shape: ChunkShape,
         codecs: Iterable[Codec | dict[str, JSON]] = (BytesCodec(endian="little"),),
         index_codecs: Iterable[Codec | dict[str, JSON]] = (
             BytesCodec(endian="little"),
@@ -468,7 +468,7 @@ def __init__(
         index_location: ShardingCodecIndexLocation | IndexLocation = "end",
         subchunk_write_order: SubchunkWriteOrder = "morton",
     ) -> None:
-        chunk_shape_parsed = parse_shapelike(chunk_shape)
+        chunk_shape_parsed = parse_chunk_shape(chunk_shape)
         codecs_parsed = parse_codecs(codecs)
         index_codecs_parsed = parse_codecs(index_codecs)
         _check_index_codecs_fixed_size(index_codecs_parsed)
@@ -507,7 +507,7 @@ def __getstate__(self) -> dict[str, Any]:
 
     def __setstate__(self, state: dict[str, Any]) -> None:
         config = state["configuration"]
-        object.__setattr__(self, "chunk_shape", parse_shapelike(config["chunk_shape"]))
+        object.__setattr__(self, "chunk_shape", parse_chunk_shape(config["chunk_shape"]))
         object.__setattr__(self, "codecs", parse_codecs(config["codecs"]))
         object.__setattr__(self, "index_codecs", parse_codecs(config["index_codecs"]))
         object.__setattr__(self, "index_location", _parse_index_location(config["index_location"]))

src/zarr/core/array.py

@@ -45,7 +45,6 @@
     ChunkGrid,
     _is_auto,
     _is_keep,
-    _is_rectilinear_chunks,
     guess_chunks,
     normalize_chunks_nd,
     resolve_outer_and_inner_chunks,
@@ -120,10 +119,12 @@
 )
 from zarr.core.metadata.io import (
     ARRAY_DOCUMENTS,
+    encode_documents,
     parse_stored_array,
     read_documents,
     save_metadata,
     save_new_metadata,
+    store_documents,
     upsert_metadata,
 )
 from zarr.core.metadata.v2 import (
@@ -210,14 +211,8 @@ def _chunk_sizes_from_shape(
 
 def parse_array_metadata(data: Any, path: str | None = None) -> ArrayMetadata:
     """Array metadata from a metadata object or a metadata document, naming the array at
-    `path` in warnings about how an invalid document was read.
-
-    `ArrayV2Metadata` accepts a chunk size of 0, as it always has, though only an
-    invalid document holds one: such metadata is read as the documents it would store
-    are (see `zarr.core.metadata.repair`), so an array can be built from it. No data
-    was read or written under that chunk size, so the reading is silent."""
-    if isinstance(data, ArrayV2Metadata) and 0 in data.chunks:
-        return parse_stored_array(data.to_buffer_dict(default_buffer_prototype()), 2)
+    `path` in warnings about how an invalid document was read. A metadata object is
+    valid as built: the metadata constructors are strict."""
     if isinstance(data, ArrayMetadata):
         return data
     if isinstance(data, dict):
@@ -463,6 +458,11 @@ async def _create(
 
         # Unify the v2 (chunks) and v3 (chunk_shape) parameter names
         _raw_chunks = chunks if chunks is not None else chunk_shape
+        if _raw_chunks is None:
+            item_size = dtype_parsed.item_size if isinstance(dtype_parsed, HasItemSize) else 1
+            outer_chunks = guess_chunks(shape, item_size)
+        else:
+            outer_chunks = normalize_chunks_nd(_raw_chunks, shape)
 
         config_parsed = parse_array_config(config)
 
@@ -484,13 +484,6 @@ async def _create(
             if order is not None:
                 _warn_order_kwarg()
 
-            item_size = 1
-            if isinstance(dtype_parsed, HasItemSize):
-                item_size = dtype_parsed.item_size
-            if _raw_chunks is None:
-                outer_chunks = guess_chunks(shape, item_size)
-            else:
-                outer_chunks = normalize_chunks_nd(_raw_chunks, shape)
             chunk_grid = create_chunk_grid_metadata(outer_chunks)
             result = await cls._create_v3(
                 store_path,
@@ -516,17 +509,8 @@ async def _create(
                 )
             if dimension_names is not None:
                 raise ValueError("dimension_names cannot be used for arrays with zarr_format 2.")
-            if _is_rectilinear_chunks(_raw_chunks):
+            if not outer_chunks.is_regular:
                 raise ValueError("Zarr format 2 does not support rectilinear chunk grids.")
-
-            item_size = 1
-            if isinstance(dtype_parsed, HasItemSize):
-                item_size = dtype_parsed.item_size
-            _raw = chunks or chunk_shape
-            if _raw is None:
-                outer_chunks = guess_chunks(shape, item_size)
-            else:
-                outer_chunks = normalize_chunks_nd(_raw, shape)
             _chunks = outer_chunks.chunk_shape
 
             if order is None:
@@ -646,7 +630,6 @@ async def _create_v3(
             dimension_names=dimension_names,
             attributes=attributes,
         )
-
         array = cls(metadata=metadata, store_path=store_path, config=config)
         await save_new_metadata(store_path, metadata, overwrite=overwrite, ensure_parents=True)
         return array
@@ -727,7 +710,6 @@ async def _create_v2(
             compressor=compressor_parsed,
             attributes=attributes,
         )
-
         array = cls(metadata=metadata, store_path=store_path, config=config)
         await save_new_metadata(store_path, metadata, overwrite=overwrite, ensure_parents=True)
         return array
@@ -1607,7 +1589,9 @@ async def get_coordinate_selection(
 
     async def _save_metadata(self, metadata: ArrayMetadata, ensure_parents: bool = False) -> None:
         """Store `metadata` as this array's own documents, then clear the
-        `_stored_document` mark (see `_stored_document_replaced`)."""
+        `_stored_document` mark (see `_stored_document_replaced`). `_resize` stores the
+        documents it encoded before deleting chunks directly, then clears the mark the
+        same way."""
         await save_metadata(self.store_path, metadata, ensure_parents=ensure_parents)
         self._stored_document_replaced()
 
@@ -1642,14 +1626,14 @@ async def _store_repaired_document(self) -> None:
         zarr_format = self.metadata.zarr_format
         documents = await read_documents(self.store_path, ARRAY_DOCUMENTS[zarr_format])
         try:
-            current = parse_stored_array(documents, zarr_format)
+            current = parse_stored_array(documents, zarr_format, str(self.store_path))
         except ArrayNotFoundError:
             pass
         else:
             if _chunk_layout(current) != _chunk_layout(self.metadata):
                 raise ValueError(
-                    f"The metadata stored for the array at {str(self.store_path)!r} has "
-                    "changed since this array was opened: reopen the array to write to it."
+                    f"Array {str(self.store_path)!r}: the metadata stored has changed since "
+                    "this array was opened; reopen the array to write to it. Nothing was stored."
                 )
             if current._stored_document is not None:
                 await upsert_metadata(self.store_path, current, documents)
@@ -4568,8 +4552,17 @@ async def init_array(
         chunk_key_encoding, zarr_format=zarr_format
     )
 
-    # Validate rectilinear chunks constraints
-    if _is_rectilinear_chunks(chunks):
+    # Normalize the user's chunks into a canonical ChunkGrid
+
+    if _is_auto(chunks):
+        max_bytes = None if shards is None else SHARDED_INNER_CHUNK_MAX_BYTES
+        chunks_normalized = guess_chunks(shape_parsed, item_size, max_bytes=max_bytes)
+    else:
+        chunks_normalized = normalize_chunks_nd(chunks, shape_parsed)
+
+    # Validate rectilinear chunks constraints. The normalized grid is the one
+    # judge of what the user declared.
+    if not chunks_normalized.is_regular:
         if zarr_format == 2:
             raise ValueError("Zarr format 2 does not support rectilinear chunk grids.")
         if shards is not None:
@@ -4579,14 +4572,6 @@ async def init_array(
                 "chunks=(inner_size, ...), shards=[[shard_sizes], ...]"
             )
 
-    # Normalize the user's chunks into a canonical ChunkGrid
-
-    if _is_auto(chunks):
-        max_bytes = None if shards is None else SHARDED_INNER_CHUNK_MAX_BYTES
-        chunks_normalized = guess_chunks(shape_parsed, item_size, max_bytes=max_bytes)
-    else:
-        chunks_normalized = normalize_chunks_nd(chunks, shape_parsed)
-
     # Resolve chunks + shards into outer_chunks (grid metadata) and
     # inner (sub-chunk structure for ShardingCodec, None if no sharding)
     outer_chunks, inner = resolve_outer_and_inner_chunks(
@@ -4908,10 +4893,9 @@ def _stored_rectilinear_grid_or_none(
     """The *stored* rectilinear chunk grid, or None if the stored grid is regular
     (in which case `.chunks` and `.shards` are defined).
 
-    Dispatches on the stored metadata, not the runtime ``ChunkGrid``: the
-    runtime grid collapses a rectilinear dimension whose edges happen to be
-    uniform to a ``FixedDimension`` as an optimization, so it can report regular
-    for an array whose stored metadata — and therefore `.chunks` — is
+    Dispatches on the stored metadata, not the runtime `ChunkGrid`: a
+    rectilinear grid whose dimensions are all bare-int step sizes is regular at
+    runtime, while its stored metadata — and therefore `.chunks` — is
     rectilinear. Zarr format 2 grids are always regular.
     """
     if isinstance(metadata, ArrayV3Metadata) and isinstance(
@@ -5904,6 +5888,10 @@ async def _resize(
     # ensure deletion is only run if array is shrinking as the delete_outside_chunks path is unbounded in memory
     only_growing = all(new >= old for new, old in zip(new_shape, array.metadata.shape, strict=True))
 
+    # Encode the new metadata before deleting any chunk: metadata that cannot be stored
+    # then fails with the store untouched.
+    documents = encode_documents(array.store_path, new_metadata)
+
     if delete_outside_chunks and not only_growing:
         # Remove all chunks outside of the new shape
         old_chunk_coords = set(array._chunk_grid.all_chunk_coords())
@@ -5922,7 +5910,8 @@ async def _delete_key(key: str) -> None:
         )
 
     # Write new metadata
-    await array._save_metadata(new_metadata)
+    await store_documents(array.store_path, documents)
+    array._stored_document_replaced()
 
     # Update metadata and chunk_grid (in place)
     object.__setattr__(array, "metadata", new_metadata)