Skip to content

Commit 826cb6d

Browse files
committed
fix(vm): D-555 — restore loc fidelity on op_push_binding_frame/op_require raises; sweep CLEAN + nightly cadence
Smell-audited: 1: the e2e expectations were written against the vm's DEGRADED fallback (message once, no caret) — updated to print.zig's intended form (message + caret = 2 occurrences) now that both backends render it; the cycle case was the same class and updated with the same reason comment. The vm raised binding_target_not_dynamic and called loadOrFindNs with a bare `.{}` loc, so these errors rendered header+message only while tree_walk showed the full source window + caret. Each instruction already carries line/col — wire the existing instr_loc through both sites. check_vm_parity sweep is now CLEAN (0 failing: corpus 3903/3903 + every e2e on -Dbackend=tree_walk; was 6 findings at first run). Cadence mechanized: ci_gate.sh's FULL tier (nightly schedule + dispatch) runs the sweep as step 3/3, so an oracle-only regression surfaces within a day with no human in the loop. D-555 + D-556 discharged.
1 parent faea9ae commit 826cb6d

5 files changed

Lines changed: 36 additions & 18 deletions

File tree

‎.dev/debt.yaml‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,6 @@
2424
# md-table-align no longer pads it to ~800 KB of whitespace.
2525

2626
active:
27-
- id: "D-556"
28-
status: "ROOT CAUSE FOUND + FIXED 2026-07-07 (the deftest-BFS integer-overflow panic + the D-555 GC family). The swept-string probe showed the FIRST post-bootstrap collect sweeping foundation strings (\"map\", \"clojure.core\", \"*in*\") on the tree_walk backend: fn-body Node literals live in the heap-lifetime analyzer arena, but the ADR-0169 AnalysisFrame bracket DROPPED its roots at form end — correct only for Nodes that die with their form; a (defn f …) body Node survives via Function.body and its literal pool was unrooted from the first collect on (the vm masks the EXECUTING chunk via the A1 pool root; a not-yet-executed fn had the same latent hole). FIX: root_set.endAnalysisPersist — production seams (driver x2, loader, repl, nrepl, evaluator oracle) persist their bracket roots into gc.persisted_analysis_roots (a GcHeap field, walked after permanent_roots; same lifetime the Nodes already have). ALSO landed in the same arc: the conservative C-stack scan (mark_sweep.conservativeStackScan: stack words from the ADR-0157 anchor + a setjmp register spill (Boehm GC_push_regs idiom), validated against a per-collect live-set hash, pinned — sound under non-moving mark-sweep) as the STRUCTURAL backstop for tree_walk's C-stack intermediates, and the 7 explicit A6/A6b brackets. VERIFIED: the original repro (rush-hour solver-test, pre-edit sources preserved in the scratchpad rhmin copy) passes 3/3 at BOTH default and 1MB thresholds on tree_walk; sym-meta GC e2e green; vm untouched. REMAINING out of scope: registered-WORKER-thread stacks are not conservatively scanned (parked at alloc-prologue; agent+tree_walk is the known D-418-adjacent area)."
29-
category: "runtime"
30-
barrier: "FIXED for the reported class — keep open until the D-555 re-sweep (check_vm_parity.sh) confirms which of the 6 findings cleared, then discharge with the sweep numbers."
31-
last_reviewed: "2026-07-07"
32-
- id: "D-555"
33-
status: "SWEEP RE-RUN 2026-07-07 (post-D-556 root-cause fix): 6 findings -> 2. CLEARED by the persist-analysis-roots + conservative-scan arc: (1) watch_recursion segfault [the ADR-0157 tree_walk guard], (2) phase16_gc_torture, (3) symbol_metadata sym_meta_gc_survives, (4) seq_helpers2 interleave_large — all were the D-556 GC family. REMAINING 2 (both the SAME shape, not memory-unsafe): phase15_dynamic_var `guard` + phase15_require_fs `missing` expect `grep -c <message>` == 1 but tree_walk prints 2 — DIAGNOSED: print.zig's INTENDED error form is header+message THEN a source window with `^--- <message>` caret (message appears twice; print.zig's own unit test pins that form); the vm loses the SourceLocation for these errors and falls back to header+message only (1 occurrence), and the e2e expectations were written against the vm's DEGRADED form. Fix = restore loc fidelity on the vm path for these error classes (or consciously pin the fallback) + update the two e2e expectations; then the sweep is CLEAN and D-555 discharges. Cadence follow-up: wire check_vm_parity.sh into the nightly CI leg + Phase-boundary list. PRIOR: six findings from the first sweep (see history in this row)."
34-
category: "runtime"
35-
barrier: "none — deterministic repros in hand. Start with the phase15_watch_recursion SEGFAULT (memory safety) and the two GC-related fails (torture + symbol-meta) — likely a tree-walk-only rooting gap; the exit-code pair last (may be harness-shaped)."
36-
last_reviewed: "2026-07-06"
3727
- id: "D-554"
3828
status: "open (2026-07-06, split from D-239's discharge). The `(ns name {:attr-map} …)` ATTR-MAP and `(ns ^{:m 1} name)` name-symbol meta are still SKIPPED at analysis (special_forms.zig analyzeNs) — only the docstring lands as {:doc} ns meta. clj merges name-sym meta + attr-map + docstring into the ns meta. Rare vs the docstring (libs mostly use :author/:added attr-maps); needs a Form->Value map conversion at analysis + ADR-0169-style rooting OR an arena-thread of the literal into both backends (the doc rode NsFilterEntry as a plain string; a map needs a Value). Mechanise via phase15_ns_docstring e2e when landed."
3929
category: "clj-parity"
@@ -6526,6 +6516,18 @@ discharged:
65266516
barrier: "DISCHARGED (was: pending_recur_buf reentrancy)."
65276517
last_reviewed: "2026-07-07"
65286518

6519+
- id: "D-555"
6520+
status: "DISCHARGED 2026-07-07 — the check_vm_parity sweep is CLEAN (0 failing: corpus 3903/3903 + every e2e on -Dbackend=tree_walk). Finding disposition: (1) watch_recursion SEGFAULT -> ADR-0157 guard ported to tree_walk; (2) gc_torture + (3) symbol_metadata + (4) interleave_large -> the D-556 persist-analysis-roots root fix (+ conservative stack scan backstop); (5) dynamic_var guard + (6) require_fs missing/cycle -> the vm LOST SourceLocation on op_push_binding_frame / op_require raises (bare .{}) — wired instr_loc (the per-instruction line/col the chunk already carries) so the caret renders; e2e expectations updated to print.zig'''s intended form (message + caret = 2 occurrences). CADENCE mechanized: ci_gate.sh's FULL tier (nightly schedule + manual dispatch) now runs the sweep as step 3/3 — an oracle-only regression is caught within a day, no human in the loop. ORIGINAL: six findings from the sweep'''s first run (D-553)."
6521+
category: "runtime"
6522+
barrier: "DISCHARGED (was: 6 tree_walk-only e2e failures)."
6523+
last_reviewed: "2026-07-07"
6524+
6525+
- id: "D-556"
6526+
status: "DISCHARGED 2026-07-07 — confirmed by the CLEAN check_vm_parity sweep (0 failing) + the original rush-hour repro passing at default AND 1MB thresholds on tree_walk. Root cause + fix narrative preserved in this row'''s history and private/notes/D556-gc-persist-roots.md: fn-body Node literals (heap-lifetime analyzer arena) lost their AnalysisFrame roots at form end -> endAnalysisPersist (gc.persisted_analysis_roots, walked after permanent_roots) at every production seam; PLUS the conservative C-stack scan + setjmp register spill (Boehm idiom) as the structural backstop for tree_walk C-stack intermediates; PLUS 7 explicit A6/A6b brackets. The user-filed integer-overflow panic was this class (a swept string'''s corrupted len overflowing Writer.drain arithmetic). RESIDUAL (recorded, not blocking): registered-WORKER-thread stacks are not conservatively scanned (the agent+tree_walk area, D-418-adjacent)."
6527+
category: "runtime"
6528+
barrier: "DISCHARGED (was: transient overflow panic, repro lost-then-found)."
6529+
last_reviewed: "2026-07-07"
6530+
65296531
conventions: |-
65306532
- Structure: two top-level lists — `active:` and `discharged:`. Active entries carry id / status / category / barrier (+ optional quality_floor / last_reviewed); discharged entries carry id / discharged_at / resolution. Edit as YAML (no Markdown table to align).
65316533
- New debt: grep `.dev/debt.yaml` for class-overlap before adding (per `.claude/rules/debt_dedup.md`).

‎scripts/ci_gate.sh‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,15 @@ echo "[ci_gate] (1/2) zig fmt --check src/"
3838
zig fmt --check src/
3939

4040
if [ "${CLJW_CI_FULL:-0}" = "1" ]; then
41-
echo "[ci_gate] (2/2) FULL gate: test/run_all.sh --serial-e2e"
41+
echo "[ci_gate] (2/3) FULL gate: test/run_all.sh --serial-e2e"
4242
bash test/run_all.sh --serial-e2e
43+
# D-555: the NON-default-backend sweep — corpus + every e2e on the
44+
# tree_walk (F-012 oracle) build, so an oracle-only regression (GC
45+
# rooting, backend-divergent eval defects) cannot hide behind the
46+
# vm-default gate. Nightly/dispatch-only: it rebuilds the binary twice
47+
# (tree_walk, then the default restore) — too heavy per-push.
48+
echo "[ci_gate] (3/3) non-default-backend sweep: scripts/check_vm_parity.sh"
49+
bash scripts/check_vm_parity.sh
4350
else
4451
echo "[ci_gate] (2/2) fast CORE: test/run_all.sh --smoke"
4552
bash test/run_all.sh --smoke

‎src/eval/backend/vm.zig‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1125,7 +1125,9 @@ inline fn stepOnce(
11251125
rt.gpa.destroy(frame);
11261126
var name_buf: [512]u8 = undefined;
11271127
const qualified = std.fmt.bufPrint(&name_buf, "{s}/{s}", .{ var_ptr.ns.name, var_ptr.name }) catch var_ptr.name;
1128-
return error_catalog.raise(.binding_target_not_dynamic, .{}, .{ .@"var" = qualified });
1128+
// D-555: carry the instruction's compiled loc so the caret
1129+
// renderer fires (tree_walk parity; was a bare `.{}`).
1130+
return error_catalog.raise(.binding_target_not_dynamic, instr_loc, .{ .@"var" = qualified });
11291131
}
11301132
frame.bindings.put(rt.gpa, var_ptr, val) catch {
11311133
frame.bindings.deinit(rt.gpa);
@@ -1256,7 +1258,8 @@ inline fn stepOnce(
12561258
const ns_name = string_mod.asString(name_val);
12571259
// ADR-0163 D-516: one load path (loaded_libs-keyed + bytecode-region
12581260
// lazy load), not the mappings.count proxy + source-only inline copy.
1259-
_ = try loader.loadOrFindNs(rt, env, ns_name, .{});
1261+
// D-555: instr_loc so a lib_not_found renders the caret (tree_walk parity).
1262+
_ = try loader.loadOrFindNs(rt, env, ns_name, instr_loc);
12601263
if (sp >= stack.len)
12611264
return raiseInternal("vm: operand stack overflow");
12621265
stack[sp] = Value.nil_val;
@@ -1273,7 +1276,8 @@ inline fn stepOnce(
12731276
const spec = chunk.libspecs[instr.operand];
12741277
// ADR-0163 D-516: one load path (loaded_libs-keyed + bytecode-region
12751278
// lazy load), not the mappings.count proxy + source-only inline copy.
1276-
const target_ns = try loader.loadOrFindNs(rt, env, spec.ns_name, .{});
1279+
// D-555: instr_loc so a lib_not_found renders the caret (tree_walk parity).
1280+
const target_ns = try loader.loadOrFindNs(rt, env, spec.ns_name, instr_loc);
12771281
const here = env.current_ns orelse
12781282
return error_catalog.raise(.current_namespace_missing, .{}, .{ .sym = spec.ns_name });
12791283
if (spec.alias) |alias_name| {

‎test/e2e/phase15_dynamic_var.sh‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,10 @@ assert_eq 'bind' "$("$BIN" -e '(do (def ^:dynamic *x* 1) (binding [*x* 2] *x*
1414
assert_eq 'restore' "$("$BIN" -e '(do (def ^:dynamic *x* 1) [(binding [*x* 2] *x*) *x*])' 2>&1 | tail -1)" '[2 1]'
1515
assert_eq 'nested' "$("$BIN" -e '(do (def ^:dynamic *y* 10) (binding [*y* 20] (binding [*y* 30] *y*)))' 2>&1 | tail -1)" '30'
1616
assert_eq 'fn-sees' "$("$BIN" -e '(do (def ^:dynamic *a* :root) (defn pa [] *a*) [(binding [*a* :inner] (pa)) (pa)])' 2>&1 | tail -1)" '[:inner :root]'
17-
# non-dynamic var still rejects binding (the guard still fires)
18-
assert_eq 'guard' "$("$BIN" -e '(do (def plain 1) (binding [plain 2] plain))' 2>&1 | grep -c 'non-dynamic')" '1'
17+
# non-dynamic var still rejects binding (the guard still fires). Expect 2
18+
# occurrences: the message line + the source-window caret repeats it
19+
# (print.zig's intended form; D-555 restored the vm's loc fidelity here).
20+
assert_eq 'guard' "$("$BIN" -e '(do (def plain 1) (binding [plain 2] plain))' 2>&1 | grep -c 'non-dynamic')" '2'
1921

2022
# --- standard core version / flag vars (cljw targets the 1.12 surface) ---
2123
assert_eq 'cv_minor' "$("$BIN" -e '(:minor *clojure-version*)' 2>&1 | tail -1)" '12'

‎test/e2e/phase15_require_fs.sh‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,9 @@ assert_eq 'env-path' "$(CLJW_PATH="$CP" "$BIN" -e '(require (quote [demo.math :a
1919
# capstone: a disk test-ns (requires clojure.test + demo.math) + run its suite
2020
assert_eq 'run-suite' "$("$BIN" -cp "$CP" -e '(require (quote demo.math-test)) (let [s (clojure.test/run-tests (quote demo.math-test))] [(:pass s) (:fail s)])' 2>&1 | tail -1)" '[4 0]'
2121
# missing lib -> lib_not_found
22-
assert_eq 'missing' "$("$BIN" -cp "$CP" -e '(require (quote no.such.lib))' 2>&1 | grep -c 'Could not locate')" '1'
22+
# 2 occurrences: message line + the source-window caret (print.zig's
23+
# intended form; D-555 restored the vm's loc fidelity on op_require).
24+
assert_eq 'missing' "$("$BIN" -cp "$CP" -e '(require (quote no.such.lib))' 2>&1 | grep -c 'Could not locate')" '2'
2325
# embedded clojure.string NOT shadowed by a stray on-disk file
2426
SHADOW="$(mktemp -d)"; mkdir -p "$SHADOW/clojure"; printf '(ns clojure.string) (def upper-case :HIJACKED)\n' > "$SHADOW/clojure/string.clj"
2527
assert_eq 'no-shadow' "$("$BIN" -cp "$SHADOW" -e '(clojure.string/upper-case "hi")' 2>&1 | tail -1)" '"HI"'
@@ -28,7 +30,8 @@ rm -rf "$SHADOW"
2830
CYC="$(mktemp -d)"; mkdir -p "$CYC/cyc"
2931
printf '(ns cyc.a (:require [cyc.b]))\n' > "$CYC/cyc/a.clj"
3032
printf '(ns cyc.b (:require [cyc.a]))\n' > "$CYC/cyc/b.clj"
31-
assert_eq 'cycle' "$("$BIN" -cp "$CYC" -e '(require (quote cyc.a))' 2>&1 | grep -c 'Cyclic load dependency')" '1'
33+
# 2 occurrences: message + caret (same D-555 loc-fidelity form as 'missing').
34+
assert_eq 'cycle' "$("$BIN" -cp "$CYC" -e '(require (quote cyc.a))' 2>&1 | grep -c 'Cyclic load dependency')" '2'
3235
rm -rf "$CYC"
3336

3437
echo "OK — phase15_require_fs (6 cases) green"

0 commit comments

Comments
 (0)