Skip to content

fix: tolerate unencodable default_scalar in ShardingCodec.validate - #4526

Open
inchang-ing wants to merge 3 commits into
zarr-developers:mainfrom
inchang-ing:fix/sharding-validate-default-scalar
Open

inchang-ing wants to merge 3 commits into
zarr-developers:mainfrom
inchang-ing:fix/sharding-validate-default-scalar

Conversation

@inchang-ing

Copy link
Copy Markdown

Fixes #4511

Root cause

When creating an array, codec chains are validated by threading an ArraySpec through each codec's resolve_metadata:

  • ArrayV3Metadata.__init__ threads the array's fill value (which defaults to dtype.default_scalar()) through the codec chain,
  • ShardingCodec._validate_inner_codecs threads the inner codec chain with dtype.default_scalar() as the probe fill value.

Value-transforming codecs (ScaleOffset, CastValue) encode the fill value as part of that threading. When the chain maps only a subrange of the dtype — e.g. float32 values known to lie in [100, 200], scaled to [0, 65535] and cast to uint16 — the probe value (0) scales to -65535, which CastValue rejects:

ValueError: Value -65535 out of range [0, 65535]. Set out_of_range='clamp' or out_of_range='wrap' to handle this.

Array creation failed even though no real value ever needs that encoding: the failure is a property of the probe fill value, not of the codec configuration.

Fix

Make the fill-value threading best-effort in ScaleOffset.resolve_metadata and CastValue.resolve_metadata: if encoding the fill value raises a value-domain error (ValueError/OverflowError), carry the fill value forward unchanged instead of failing. Shape/dtype threading and per-codec validate() are untouched, so:

  • structurally invalid configurations still raise at creation time (wrong dtype for a codec, scale=0, out_of_range='wrap' with a float target, ...) — verified below;
  • genuine value-domain errors for actual chunk data still surface in _encode_sync at write time (writing a value outside [100, 200] with this chain still errors, as before).

The same tolerance also fixes the runtime pipeline-evolution path, which threaded the same unencodable fill through resolve_metadata.

Changes

  • src/zarr/codecs/scale_offset.py: tolerate value-domain failures when encoding the fill value in resolve_metadata
  • src/zarr/codecs/cast_value.py: same
  • tests/test_codecs/test_scale_offset.py: creation succeeds with an unencodable default fill value (uint16 + ScaleOffset(offset=100)); invalid config still raises
  • tests/test_codecs/test_cast_value.py: the issue's exact scenario (float32 [100, 200] → uint16 via ScaleOffset+CastValue) creates and round-trips; invalid config still raises
  • tests/test_codecs/test_sharding.py: sharded array creation succeeds when the inner codec chain cannot encode default_scalar

Verification

  • Reproduced the issue on main (v3.4.2.dev14): the issue's repro script raised ValueError: Value -65535 out of range [0, 65535].
  • With the fix: the repro creates successfully; a full-chunk write and round-trip works; a sharded array with the same inner chain creates and round-trips.
  • Regression discipline: reverting the fix makes the 3 new creation tests fail; re-applying makes them pass.
  • pytest tests/test_codecs/ tests/test_metadata/test_v3.py: 757 passed, 1 skipped, 2 xfailed.
  • pytest tests/test_array.py tests/test_codec_pipeline.py tests/test_unified_chunk_grid.py: 2737 passed, 756 skipped.
  • Structurally invalid configs still rejected: ScaleOffset on a string dtype, out_of_range='wrap' with a float target, and scale=0 all raise ValueError at creation.
  • A towncrier changelog fragment will be added with the PR number once known.

ShardingCodec._validate_inner_codecs (and ArrayV3Metadata codec
threading) probe the codec chain with dtype.default_scalar() as the
fill value. When the chain maps only a subrange of the dtype — e.g.
ScaleOffset + CastValue scaling float32 values in [100, 200] to uint16
— the probe value itself is unencodable, and array creation failed even
though no real value ever needs that encoding.

Make the fill-value threading in ScaleOffset.resolve_metadata and
CastValue.resolve_metadata best-effort: if encoding the fill value
raises a value-domain error (ValueError/OverflowError), carry the fill
value forward unchanged instead of failing. Shape/dtype threading and
per-codec validate() are untouched, so structurally invalid
configurations (wrong dtype for a codec, scale=0, out_of_range='wrap'
with a float target, ...) still raise at creation time. Genuine
value-domain errors for actual chunk data still surface in
_encode_sync.

Fixes zarr-developers#4511
@github-actions github-actions Bot added the needs release notes Automatically applied to PRs which haven't added release notes label Oct 18, 2026
@github-actions github-actions Bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Oct 18, 2026
@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.71%. Comparing base (ec2c359) to head (d6a2e71).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4526   +/-   ##
=======================================
  Coverage   94.71%   94.71%           
=======================================
  Files          94       94           
  Lines       13663    13669    +6     
=======================================
+ Hits        12941    12947    +6     
  Misses        722      722           
Files with missing lines Coverage Δ
src/zarr/codecs/cast_value.py 98.75% <100.00%> (+0.02%) ⬆️
src/zarr/codecs/scale_offset.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ShardingCodec.validate incorrectly assumes that Zdtype.default_scalar is always a valid value

1 participant