Skip to content

fix: normalize dict-form scalar_map at codec construction - #20

Merged
d-v-b merged 2 commits into
mainfrom
fix/scalar-map-dict-form
Aug 12, 2026
Merged

fix: normalize dict-form scalar_map at codec construction#20
d-v-b merged 2 commits into
mainfrom
fix/scalar-map-dict-form

Conversation

@d-v-b

@d-v-b d-v-b commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Fixes zarr-developers/cast-value.rs#24 (filed on the cast-value.rs tracker, but the code lives here).

The bug

CastValueNumpyV1 and CastValueRustV1 accepted a dict for the encode/decode entries in scalar_map without error at construction, then crashed on first use:

cv = CastValueRustV1(
    data_type="int16",
    rounding="nearest-even",
    scalar_map={"encode": {"NaN": -32768}, "decode": {-32768: "NaN"}},
)
# construction succeeds; using the codec raises
# ValueError: too many values to unpack (expected 2)

extract_raw_map iterates entries as for src, tgt in pairs — iterating a dict yields keys only, so "NaN" gets unpacked as its three characters. Verified affecting both codec classes (they share _parsing.py).

The fix

The issue offered two options: reject dicts at construction, or accept them. Accepting is the consistent choice — the sibling implementations already do: cast-value.rs's binding downcasts PyMapping and calls .items(), and zarr-python's parse_scalar_map normalizes dicts too. This package was the only one that accepted the dict and then failed.

A new parse_scalar_map normalizes scalar_map at construction: each direction accepts either a mapping of source → target or an iterable of (source, target) pairs, both converted to the spec's list-of-pairs form. Normalizing at construction rather than in extract_raw_map matters for two reasons:

  • to_dict serializes scalar_map verbatim, so patching only the read path would have written dict-form (non-spec) metadata to disk — and reading it back would crash again on the decode side.
  • Malformed maps (wrong entry arity, unknown direction keys, non-mapping input) now fail fast at construction with a clear message instead of at encode/decode time.

Tests

  • test_parse_scalar_map — parametrized over all accepted forms (dict, tuple pairs, list pairs, mixed, empty, None)
  • One test per error case: non-mapping input, unknown direction key, wrong entry arity, non-pair entry
  • test_init_normalizes_dict_scalar_map — regression test for the issue: dict-form construction, spec-form to_dict output, and the encode path that used to crash (NaN → −32768 through a float32→int16 cast)

Kill test: with only the constructor normalization reverted, the regression test fails and the rest pass. Full run: 207 passed; ruff check/ruff format, ty, and pylint all clean. Also verified the exact issue repro end-to-end through zarr.create_array for both codec classes — NaN round-trips correctly.

@codspeed-hq

codspeed-hq Bot commented Aug 12, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 48 untouched benchmarks


Comparing fix/scalar-map-dict-form (f90b942) with main (c1b1270)

Open in CodSpeed

d-v-b added 2 commits August 12, 2026 19:04
CastValueNumpyV1 and CastValueRustV1 accepted a dict for the
encode/decode entries in scalar_map without error at construction, but
crashed with "ValueError: too many values to unpack (expected 2)" when
the codec was first used: extract_raw_map iterates entries as pairs,
and iterating a dict yields keys only.

Normalize scalar_map at construction with a new parse_scalar_map
function: each direction accepts either a mapping of source -> target
or an iterable of (source, target) pairs, and both are converted to
the spec's list-of-pairs form. This is also what the sibling
implementations do -- cast-value-rs's binding and zarr-python's
parse_scalar_map both take dict form -- so cast-value.py was the only
one that accepted the dict and then failed.

Normalizing at construction (rather than accepting dicts in
extract_raw_map) keeps to_dict emitting spec-form metadata and makes
malformed maps -- wrong arity, unknown direction keys, non-mapping
input -- fail fast with a clear error instead of at encode/decode
time.

Fixes zarr-developers/cast-value.rs#24

Assisted-by: ClaudeCode:claude-opus-4.8
Narrow the JSON union before indexing the configuration dict, and
restore the ty ignore that NDBuffer.from_ndarray_like needs (ruff
format had collapsed the call onto one line without it; hoist the
array like the neighboring tests do).

Assisted-by: ClaudeCode:claude-opus-4.8
@d-v-b
d-v-b force-pushed the fix/scalar-map-dict-form branch from 41f67c7 to f90b942 Compare August 12, 2026 17:04
@d-v-b
d-v-b merged commit b8f1709 into main Aug 12, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

scalar_map accepts dict but fails at encode/decode time

1 participant