Skip to content

Commit 87c1af4

Browse files
d-v-bclaude
andcommitted
Merge the upgrade-to-repair rename from #4334
Conflicts resolved by keeping this branch's lines and applying the same rename to them; this branch's own text now says repair as well. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2 parents d542eb9 + 0992e94 commit 87c1af4

11 files changed

Lines changed: 145 additions & 145 deletions

File tree

‎changes/4334.bugfix.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,4 @@ A chunk edge length is now always at least 1, while an array extent may be 0. Ev
44

55
Stored metadata with a regular chunk size of 0 or JSON `false` (Zarr format 2 `chunks`, Zarr format 3 `regular` `chunk_shape`) is now read as chunk size 1 (the inner chunk size for sharded arrays, which previously failed to open), however long the axis is: that is the chunk size `chunks=-1` gives on a zero-length axis, and it does not depend on how far the axis has grown since. zarr-python wrote such sizes for arrays created with a zero-length axis until 3.4, and, from 2.18.7 to 3.2.1, for an explicit chunk size of 0 or `False` on an axis of any length, as in `chunks=(0,)` (those arrays could store no data). On a zero-length axis this opens silently, as before, and appending to it stores chunks of size 1. On an axis of positive length it opens with a `ZarrUserWarning`, which says that the array holds only its fill value, so that recreating it with the wanted chunk shape loses nothing (`zarr.from_array(array.store, name=array.path, data=array, chunks=..., overwrite=True, write_data=False)` keeps its data type, fill value, attributes and codecs), and how to keep it by storing valid metadata instead: `array.update_attributes({})`, then `zarr.consolidate_metadata` if the metadata is consolidated. JSON `true`, as zarr-python 3.0 and 3.2 wrote for a chunk size of `True`, is read as 1, silently, in the chunk grid (regular, and the explicit edges and run-length encoded sizes of a rectilinear one) and in the inner chunk shape of every sharding codec, nested or not. A stored rectilinear chunk grid whose edge lengths are integral JSON floats (`[[4.0, 2]]`), as zarr-python 3.2 wrote for float edges, is read with those edges as integers, silently; a float anywhere no release wrote one (a regular chunk shape, the inner chunk shape of a sharding codec, a run-length repeat count, an edge below 1) is rejected.
66

7-
Writing data to an array whose stored metadata holds a regular chunk size of 0 or `false` (other than an empty selection) first stores the upgrade of the metadata the store then holds, unless another writer has stored valid metadata since, so that other readers find the chunks written (a `true` chunk size or an integral float edge is read as the value it equals, so a write stores no metadata for it); in a `ZipStore` this adds a second entry for the metadata document, as every metadata update does. If the metadata the store then holds lays out chunks differently (another writer stored a different chunk size since, say), the write raises a `ValueError` asking to reopen the array, and stores nothing. Apart from the array's own metadata writes (`update_attributes`, `resize`), which store the upgrade as before, this is the only operation that stores it: a group's consolidated metadata keeps such an array's metadata as it was stored (`zarr.consolidate_metadata` copies it as the array's own document holds it) until the array stores its upgrade through the group's handle, and changing a group reads and writes no metadata of its members.
7+
Writing data to an array whose stored metadata holds a regular chunk size of 0 or `false` (other than an empty selection) first stores the repair of the metadata the store then holds, unless another writer has stored valid metadata since, so that other readers find the chunks written (a `true` chunk size or an integral float edge is read as the value it equals, so a write stores no metadata for it); in a `ZipStore` this adds a second entry for the metadata document, as every metadata update does. If the metadata the store then holds lays out chunks differently (another writer stored a different chunk size since, say), the write raises a `ValueError` asking to reopen the array, and stores nothing. Apart from the array's own metadata writes (`update_attributes`, `resize`), which store the repair as before, this is the only operation that stores it: a group's consolidated metadata keeps such an array's metadata as it was stored (`zarr.consolidate_metadata` copies it as the array's own document holds it) until the array stores its repair through the group's handle, and changing a group reads and writes no metadata of its members.

‎src/zarr/core/array.py‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -216,7 +216,7 @@ def parse_array_metadata(data: Any, path: str | None = None) -> ArrayMetadata:
216216
217217
`ArrayV2Metadata` accepts a chunk size of 0, as it always has, though only an
218218
invalid document holds one: such metadata is read as the documents it would store
219-
are (see `zarr.core.metadata.upgrades`), so an array can be built from it. No data
219+
are (see `zarr.core.metadata.repair`), so an array can be built from it. No data
220220
was read or written under that chunk size, so the reading is silent."""
221221
if isinstance(data, ArrayV2Metadata) and 0 in data.chunks:
222222
return parse_stored_array(data.to_buffer_dict(default_buffer_prototype()), 2)
@@ -1615,28 +1615,28 @@ async def _save_metadata(self, metadata: ArrayMetadata, ensure_parents: bool = F
16151615

16161616
def _stored_document_replaced(self) -> None:
16171617
"""Record that the store no longer holds a document of this array that needs an
1618-
upgrade: it holds the upgrade, a valid document, or none. The metadata this handle
1618+
repair: it holds the repair, a valid document, or none. The metadata this handle
16191619
holds, which a consolidated group handle may share, then stops standing for the
1620-
document it was read from (see `mark_upgraded`), so no later write through either
1620+
document it was read from (see `mark_repaired`), so no later write through either
16211621
handle stores that document again."""
16221622
object.__setattr__(self.metadata, "_stored_document", None)
16231623

1624-
async def _store_upgraded_document(self) -> None:
1625-
"""Store the upgrade of this array's current stored document, if it needs one,
1624+
async def _store_repaired_document(self) -> None:
1625+
"""Store the repair of this array's current stored document, if it needs one,
16261626
before chunks are written under this handle's metadata.
16271627
1628-
Only for metadata read from a document whose upgrade moves chunks (see
1629-
`zarr.core.metadata.upgrades`). The document is read again, because the store may
1628+
Only for metadata read from a document whose repair moves chunks (see
1629+
`zarr.core.metadata.repair`). The document is read again, because the store may
16301630
hold a newer one than this handle's metadata. If that one lays out chunks
16311631
differently (another writer stored a different chunk size since), this handle
16321632
would write chunks no reader finds, so it raises and stores nothing. If it needs
1633-
no upgrade (the array was re-saved since, possibly by another implementation), it
1634-
is left as written; if there is none, there is nothing to upgrade.
1633+
no repair (the array was re-saved since, possibly by another implementation), it
1634+
is left as written; if there is none, there is nothing to repair.
16351635
1636-
Storing the same upgrade twice is harmless, so handles that write chunks
1636+
Storing the same repair twice is harmless, so handles that write chunks
16371637
concurrently need no coordination. The read and the store are not one atomic
16381638
step, though: a metadata write by another handle between them (a resize, an
1639-
attribute update) is replaced by the upgrade of the document read before it, as
1639+
attribute update) is replaced by the repair of the document read before it, as
16401640
with any two metadata writes that race.
16411641
"""
16421642
if self.metadata._stored_document is None:
@@ -1666,9 +1666,9 @@ async def _set_selection(
16661666
fields: Fields | None = None,
16671667
) -> None:
16681668
if product(indexer.shape) > 0:
1669-
# Chunks are about to be stored under the upgraded metadata, so store it
1669+
# Chunks are about to be stored under the repaired metadata, so store it
16701670
# first: every reader of the store then agrees with them.
1671-
await self._store_upgraded_document()
1671+
await self._store_repaired_document()
16721672
return await _set_selection(
16731673
self.store_path,
16741674
self.metadata,

‎src/zarr/core/common.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -290,7 +290,7 @@ def _subject(name: str, axis: int | None) -> str:
290290
def _parse_positive_int(value: object, name: str, axis: int | None) -> int:
291291
"""`value` as an `int` of at least 1. A `bool` is read as the `int` it equals; any
292292
other type, a NumPy integer or a float (even an integral one: stored documents with
293-
integral floats are read by `zarr.core.metadata.upgrades`), is rejected."""
293+
integral floats are read by `zarr.core.metadata.repair`), is rejected."""
294294
subject = _subject(name, axis)
295295
if not isinstance(value, int):
296296
raise TypeError(f"{subject} must be an int, got {value!r}")

‎src/zarr/core/group.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -155,8 +155,8 @@ class ConsolidatedMetadata:
155155

156156
def to_dict(self) -> dict[str, JSON]:
157157
"""The consolidated metadata document. An array read from a stored document that
158-
had to be upgraded is written as that document was stored, so every reader of
159-
the consolidated metadata reads it as upgraded again (see `mark_upgraded`)."""
158+
had to be repaired is written as that document was stored, so every reader of
159+
the consolidated metadata reads it as repaired again (see `mark_repaired`)."""
160160
return {
161161
"kind": self.kind,
162162
"must_understand": self.must_understand,
@@ -387,7 +387,7 @@ class GroupMetadata(Metadata):
387387
def to_buffer_dict(self, prototype: BufferPrototype) -> dict[str, Buffer]:
388388
if self.consolidated_metadata is not None:
389389
for path, member in self.consolidated_metadata.flattened_metadata.items():
390-
# A member read from a document that had to be upgraded is stored as it
390+
# A member read from a document that had to be repaired is stored as it
391391
# was stored (see `ConsolidatedMetadata.to_dict`).
392392
if isinstance(member, ArrayV3Metadata) and member._stored_document is None:
393393
try:

‎src/zarr/core/metadata/io.py‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
from zarr.core.buffer.core import default_buffer_prototype
1111
from zarr.core.buffer.cpu import buffer_prototype as cpu_buffer_prototype
1212
from zarr.core.common import ZARR_JSON, ZARRAY_JSON, ZATTRS_JSON
13-
from zarr.core.metadata.upgrades import mark_upgraded, upgrade_array_document
13+
from zarr.core.metadata.repair import mark_repaired, repair_array_document
1414
from zarr.errors import ArrayNotFoundError, ContainsArrayError
1515
from zarr.storage._common import StorePath, ensure_no_existing_node
1616

@@ -138,9 +138,9 @@ def parse_stored_array(
138138
documents: Mapping[str, Buffer], zarr_format: ZarrFormat, path: str | None = None
139139
) -> ArrayMetadata:
140140
"""The metadata of an array from its documents (by store key, see `ARRAY_DOCUMENTS`),
141-
read with the upgrades but without their warnings (whoever asks has warned, or reads
142-
metadata built in code), and marked (see `mark_upgraded`) if they had to be
143-
upgraded. Raises `ArrayNotFoundError` if there is no array document among them.
141+
read with the repairs but without their warnings (whoever asks has warned, or reads
142+
metadata built in code), and marked (see `mark_repaired`) if they had to be
143+
repaired. Raises `ArrayNotFoundError` if there is no array document among them.
144144
145145
Only operations that store metadata read documents this way, so documents read as a
146146
rectilinear chunk grid require the rectilinear chunks flag, as storing it does; the
@@ -157,9 +157,9 @@ def parse_stored_array(
157157
stored = _array_metadata_dict_v3(documents[ZARR_JSON])
158158
else:
159159
raise ArrayNotFoundError(f"No Zarr format {zarr_format} array metadata document.")
160-
upgraded, readings = upgrade_array_document(stored, zarr_format)
161-
metadata = parse_array_metadata(dict(upgraded), path)
162-
return mark_upgraded(metadata, stored, readings, None, warn=False)
160+
repaired, readings = repair_array_document(stored, zarr_format)
161+
metadata = parse_array_metadata(dict(repaired), path)
162+
return mark_repaired(metadata, stored, readings, None, warn=False)
163163

164164

165165
def _build_parents(store_path: StorePath, zarr_format: ZarrFormat) -> dict[str, GroupMetadata]:

0 commit comments

Comments
 (0)