Skip to content

Commit b804e6c

Browse files
committed
fix(map): D-155 — HAMT full-hash collision bucket (clj HashCollisionNode parity)
Smell-audited: 1: the 2026-05-30 "essentially never hit" premise was wrong for pointer-hashed keys (GLL workloads collide non-deterministically) — recorded in the D-155 discharge; the tag-55 reservation was honored on merit (layout reuse keeps every walker unchanged), not as a contract. Two distinct keys sharing one full 32-bit valueHash crashed assoc with a raw error.HashCollision (instaparse's GLL tramp hit it: fn/regex hash terms are pointer bits, so per-run addresses made it non-deterministic). The bucket reuses the HamtMapNode struct — pairs front-loaded with data_map=(1<<n)-1 low-bits mask, node_map=1 + slots[63] chaining past 31 pairs — so keys/vals/seq/foldHash/subset/forEachEntry/trace walk buckets with zero collision-specific code; get/contains/assoc/dissoc branch on the child slot's tag; copyHamtNode preserves the source header tag; createTwoNode at shift>30 creates the bucket. error.HashCollision is deleted from the error set (3 analyzer catch arms removed). Sets ride the same fix (map-backed). Verified: unit — brute-forced hashLong-colliding i48 pairs through the public API (coexist/replace/dissoc/walkers) + a fabricated-hash 40-key chain-overflow test; corpus hash_collision.txt (3 goldens — clj's Murmur3.hashLong is the same formula, so the SAME pairs exercise clj's HashCollisionNode); instaparse 4-grammar probe 5-run byte-identical + large-input stress. D-430 discharged: instaparse works end-to-end.
1 parent efc2d68 commit b804e6c

4 files changed

Lines changed: 299 additions & 40 deletions

File tree

‎.dev/debt.yaml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1314,11 +1314,6 @@ active:
13141314
category: "concurrency"
13151315
barrier: "blocked-by: a reliable reproducer. 2026-06-13 attempts: 200 iterations in-process = 0 mismatches; 3000 iterations under 6x `yes` CPU load = 0 mismatches. So the race needs the gate's SPECIFIC contention (≈16 parallel cljw processes + the e2e harness preempting the agent worker vs main thread vs a cross-thread GC collect), not mere in-process concurrency or coarse CPU load. LEAD for the next attempt: agent.zig docstring flags the `send` FABRICATION WINDOW (the `[f & args]` action vector is a Zig local, not yet GC-rooted, before the queue lock — GC-ROOT note ~:35) — a cross-thread collect during that window is the prime suspect for how `await`'s completion Promise (a heap value) displaces the first action's arg. NOTE: the leaked arg `1` is an IMMEDIATE (not heap), so the corruption is at the action-VECTOR level (the vector's slot/memory), not the int — re-read fabrication-window rooting + the await barrier-action injection order. Fix only with a reproducer in hand (a rare race fixed blind cannot be verified)."
13161316
last_reviewed: "2026-06-13"
1317-
- id: "D-430"
1318-
status: "NARROWED AGAIN 2026-07-06: the var-alias gap is FIXED (analyzeVar consults current_ns.aliases first, ADR-0035 D3 precedence; e2e phase14_var_special var_alias/var_alias_reader/var_alias_deref/var_fqn — corpus line infeasible: `(var alias/x)` is analysis-time so a `(do …)`-wrapped corpus expr fails in BOTH runtimes identically; cljw top-level-do sequential eval verified matching clj). instaparse.core now LOADS, `(insta/parser \"S = 'a'\")` builds, and string-literal grammars PARSE correctly (`[:S \"a\"]`, nested parens, == clj) — the ORIGINAL mis-tokenization divergence is GONE (was the GC-rooting family, ADR-0169). NEW FRONTIER (deterministic, reproduced 2026-07-06): grammars exercising regexp-full-parse (e.g. `A = 'a'+`) fail at gll.cljc:774 `(count text)` — text is instaparse's Segment deftype (declares CharSequence, defines length); cljw countFn's typed_instance arm lacks clj RT.countFrom's `instanceof CharSequence -> .length()` fallback -> protocol_no_satisfies. Fix: after the isPersistentCollection walk check, dispatch `-cs-length` (protocol CharSequence, host_interface.zig:175 remap) when the descriptor declares CharSequence. AFTER count: probe re-matcher/re-find with a Segment arg (clj Matcher accepts any CharSequence; cljw regex surface expects string — likely needs a toString coercion at the regex boundary). PRIOR HISTORY: 3-layer GC-rooting family fixed via ADR-0169 (AnalysisFrame; formToValue accumulators; TypeDescriptor trace); load-blocking var-alias fixed 2026-07-06."
1319-
category: "interop"
1320-
barrier: "none — repro in hand: cljw -cp ~/.gitlibs/libs/instaparse/instaparse/da886e71a4afa80f8b83d1d67f058b2f02cdc0e3/src, grammar `S = AB* ; AB = A B ; A = 'a'+ ; B = 'b'+`, input aaabbab. Next unit = count-on-CharSequence-deftype (S-sized, clj RT.countFrom parity), then the re-matcher CharSequence boundary probe."
1321-
last_reviewed: "2026-07-06"
13221317
- id: "D-435"
13231318
status: "open (2026-06-14, user-flagged via D-434). The dual-backend differential oracle (ADR-0005; src/lang/diff_test.zig's Fixture) is the project's correctness oracle for VM↔TreeWalk parity, BUT the Fixture is a MINIMAL runtime: it does `primitive.registerAll` + macro registration only, NOT the core.clj bootstrap. So `*out*`, `print`, `with-out-str`, and the whole clojure.core .clj surface are absent — any FULL-RUNTIME form cannot be dual-backend-diff-tested there. Today such behavior is verified single-backend (clj_corpus via `cljw -e`, which runs ONE backend) + structural arguments ('both backends call the same shared helper'). That is a real oracle-coverage HOLE: full-runtime dual-backend parity is UNVERIFIED except by hand-argument. Surfaced when D-434 (*out* writer-interop) could not get a diff case — the workaround (remove the diff test, lean on clj_corpus) is the bad smell the user caught. FINISHED FORM: a diff Fixture that bootstraps core.clj once (shared/cached so per-test cost stays low) OR a separate full-runtime dual-backend harness (run a form through both backends with the real runtime + compare), so ANY form — not just primitives — is parity-checked. Part of the D-436 大整理 epic."
13241319
category: "testing"
@@ -3565,9 +3560,9 @@ discharged:
35653560
resolution: |-
35663561
print | **Chars print as `\uXXXX` for ALL codepoints, not JVM's readable forms**: cljw's char printer emits `A` for `(char 65)` where JVM Clojure prints `\A`. JVM uses readable literals — `\A`/`\a` for printable ASCII, named forms `\space`/`\newline`/`\tab`/`\return`/`\backspace`/`\formfeed`, and `\uXXXX` only for non-printable/non-named. cljw always uses `\uXXXX`. Construction + value are CORRECT (codepoint round-trips: `(int (char 65))` → 65); only the printed surface diverges. Not a semantics drop. Discharge: the char arm of `runtime/print.zig` should emit `\<ch>` for printable ASCII, the named forms for the standard whitespace chars, and `\uXXXX` only otherwise (mirror JVM `clojure.lang.Util`/`RT.charToString`). Surfaced 2026-05-30 adding the `char` primitive (D-134). | Opportunistic — when char print fidelity (REPL / pr-str of chars) matters | 2026-05-30
35673562
- id: "D-155"
3568-
discharged_at: "Opportunistic — astronomically rare (full 32-bit Murmur3 collision)"
3563+
discharged_at: "DISCHARGED 2026-07-06 — collision bucket landed (D-430 instaparse arc); the 'essentially never' premise was WRONG for pointer-hashed keys"
35693564
resolution: |-
3570-
runtime | **HAMT depth-exhaustion raises `error.HashCollision` instead of bucketing colliding keys (cycle C)**: when two distinct map/set keys share a full 32-bit `equal.valueHash`, `map.zig createTwoNode` (at shift > 30) raises `error.HashCollision` — a legal transient stub (explicit error, no silent drop). The analyzer converts it to a `feature_not_supported` message for map/set literals; primitives propagate it to the eval loop (bare render). Discharge: implement `HashCollisionMapNode` (tag 62, declared `heap_tag.zig:123`) — a flat linear-scan bucket for keys sharing a 32-bit hash, with assoc/get/dissoc arms + (if its layout diverges from the `[64]Value` slot region) a dedicated `traceHashCollisionMapNode`. Birthday bound ~77k keys for 50% collision at 32 bits, so essentially never hit in practice. Surfaced 2026-05-30 landing D-045 cycle A. | Opportunistic — when a real workload hits a 32-bit hash collision (essentially never) or adversarial-key robustness is wanted | 2026-05-30
3565+
DISCHARGED 2026-07-06: `.hash_collision_map_node` (tag 55) collision bucket landed in map.zig (cycle C). The 2026-05-30 'essentially never hit' premise was WRONG: instaparse's GLL creates thousands of keys whose hash folds pointer-bits terms (fn/regex values via valueHash's else arm), and per-run allocation addresses vary — a NON-DETERMINISTIC full-32-bit birthday collision fired on a real workload (~1-in-few-runs on the 4-grammar probe). Design: the bucket REUSES the HamtMapNode struct — pairs front-loaded with data_map = (1<<n)-1 low-bits mask, node_map=1 + slots[63] chaining past 31 pairs — so keys/vals/seq/foldHash/subset/forEachEntry/trace walk buckets UNCHANGED; only get/contains/assoc/dissoc branch on the child slot's tag; copyHamtNode preserves the source header tag. createTwoNode at shift>30 now creates the bucket (error.HashCollision deleted from the error set; the 3 analyzer catch arms removed). Verified: unit (real brute-forced hashLong-colliding i48 pairs 99387577075583/133060659732565 + -27661886147831/123535269098939 via public assoc/get/dissoc/count/seq; fabricated-hash 40-key chain-overflow test through the internal fns), corpus test/diff/clj_corpus/hash_collision.txt (3 goldens, clj-byte-matched — clj's Murmur3.hashLong is the same formula so the SAME pairs collide there and exercise clj's HashCollisionNode), instaparse 5-run byte-identical + large-input stress. Sets ride the same fix (map-backed). ORIGINAL: HAMT depth-exhaustion raised error.HashCollision (transient stub, D-045 cycle A); tag reserved at heap_tag; trace pre-registered (same slot-walk — the reuse-the-layout design honored that reservation).
35713566
- id: "D-156"
35723567
discharged_at: "Opportunistic — micro-optimization, no observable effect"
35733568
resolution: |-
@@ -6570,6 +6565,11 @@ discharged:
65706565
category: "clj-parity"
65716566
barrier: "none for the SEQUENTIAL surface (a .clj ns over reduce/transduce). The parallel r/fold speedup is gated on a fork/join pool cljw lacks (ADR-0093 no ForkJoinPool) — sequential fallback is clj-correct. Low-value (transducers superseded reducers); take up on a real consumer."
65726567
last_reviewed: "2026-07-02"
6568+
- id: "D-430"
6569+
status: "DISCHARGED 2026-07-06 — instaparse works END-TO-END. The final arc (this session, 3 fixes): (1) var-alias — analyzeVar resolves NS aliases (Env.findNsOrAlias, ADR-0035 D3 precedence; 5-site consolidation; e2e phase14_var_special) -> instaparse.core LOADS; (2) count-on-CharSequence — countFn's typed_instance arm gained clj RT.countFrom's CharSequence->-cs-length fallback (instaparse Segment deftype; e2e phase14_instaparse_substrate charsequence_count) -> regexp grammars run; (3) HashCollision — the NON-DETERMINISTIC crash on the 4-grammar probe was the D-155 HAMT full-32-bit-collision stub firing on GLL's pointer-hash-folded keys; the collision bucket landed (see D-155's discharge). VERIFIED: 4 grammar classes (string literals / `'a'+` repeats / regex terminals `#'[0-9]+'` / left recursion `expr = add | term` / nested parens) parse == clj; failure? true on a non-match; 5-run byte-identical; 59-term + 160-char large-input stress 3-run stable. The re-matcher-CharSequence boundary suspected earlier never fires (regex grammars pass Segments through fine — no gap found). PRIOR HISTORY: the per-run-varying garbage-bytes NameError was the 3-layer GC-rooting family (ADR-0169: AnalysisFrame; formToValue accumulators; TypeDescriptor trace); the ORIGINAL mis-tokenization divergence is GONE (parse trees byte-match clj). Any future instaparse gap is a NEW discovery, not this row."
6570+
category: "interop"
6571+
barrier: "DISCHARGED (was: instaparse GLL frontier)."
6572+
last_reviewed: "2026-07-06"
65736573

65746574
conventions: |-
65756575
- Structure: two top-level lists — `active:` and `discharged:`. Active entries carry id / status / category / barrier (+ optional quality_floor / last_reviewed); discharged entries carry id / discharged_at / resolution. Edit as YAML (no Markdown table to align).

‎src/eval/analyzer/analyzer.zig‎

Lines changed: 7 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -425,11 +425,11 @@ fn buildCompilerSpecials(rt: *Runtime) AnalyzeError!Value {
425425
var m = map_collection.empty();
426426
for (SPECIAL_FORMS.keys()) |name| {
427427
const sym = try symbol_mod.intern(rt, null, name);
428-
// `assoc`'s `HashCollision` / `AssocOnNonMap` cannot fire here (interned
429-
// symbol keys, `m` is always a map) — collapse them into the AnalyzeError
430-
// set as InternalError so the resolver's error set stays unchanged.
428+
// `assoc`'s `AssocOnNonMap` cannot fire here (`m` is always a map) —
429+
// collapse it into the AnalyzeError set as InternalError so the
430+
// resolver's error set stays unchanged.
431431
m = map_collection.assoc(rt, m, sym, .nil_val) catch |e| switch (e) {
432-
error.HashCollision, error.AssocOnNonMap => return error.InternalError,
432+
error.AssocOnNonMap => return error.InternalError,
433433
else => |other| return other,
434434
};
435435
}
@@ -1301,12 +1301,8 @@ fn mapFormToValue(rt: *Runtime, env: *Env, entries: []const Form, loc: SourceLoc
13011301
const val = try formToValue(rt, env, entries[i + 1]);
13021302
roots[2] = val;
13031303
out = map_collection.assoc(rt, out, k, val) catch |err| switch (err) {
1304-
// Astronomically rare: two literal keys share a full 32-bit
1305-
// hash (collision-bucket handling is unimplemented). Convert
1306-
// the internal error so the AnalyzeError envelope holds.
1307-
error.HashCollision => return error_catalog.raise(.feature_not_supported, loc, .{
1308-
.name = "a map with hash-colliding keys",
1309-
}),
1304+
// Hash-colliding literal keys land in the D-155 collision bucket
1305+
// inside assoc — no analyzer-level error remains.
13101306
error.AssocOnNonMap => unreachable, // out always starts at .array_map
13111307
else => |e| return e,
13121308
};
@@ -1329,9 +1325,7 @@ fn setFormToValue(rt: *Runtime, env: *Env, items: []const Form) AnalyzeError!Val
13291325
const v = try formToValue(rt, env, item);
13301326
roots[1] = v;
13311327
out = set_collection.conj(rt, out, v) catch |err| switch (err) {
1332-
error.HashCollision => return error_catalog.raise(.feature_not_supported, .{}, .{
1333-
.name = "a set with hash-colliding elements",
1334-
}),
1328+
// Hash-colliding elements land in the D-155 collision bucket.
13351329
error.AssocOnNonMap => unreachable, // out always starts at .hash_set
13361330
else => |e| return e,
13371331
};

0 commit comments

Comments
 (0)