Skip to content

Commit faea9ae

Browse files
committed
fix(tree_walk): D-557 — evalRecur reentrancy (nested lazy-for recur clobbered the pending buf)
Smell-audited: 1: the fix keeps the threadlocal (the loop-frame drain contract is unchanged) and only moves arg EVALUATION to a local ArgsFrame-rooted buffer — the scratch is now written solely in the no-eval window between RecurSignaled and the matching drain, where no nested recur can fire. evalRecur evaluated args directly into the threadlocal pending_recur_buf, but an arg's eval can force a lazy seq whose `for`-lowered iterator loop RECURS (the :when-miss arm emits `(recur (rest gxs))`) — the inner recur clobbered the outer's already-evaluated args. Symptom: the cw-arcade two-list BFS solver's queue slot received a stray :delta Long (`-nth on Long`); fired only when a :when miss occurred during recur-arg evaluation (visited-heavy real workloads), which is why synthetic twins passed. The vm is immune (recur args ride the operand stack). Verified: the solver repro (direct + shadow-renamed + no-reverse variants + the full 3-test suite) green on tree_walk; corpus lazy_eval +2 distilled goldens (clj-byte-matched) — re-verified per-gate on vm (corpus_regression) and per-sweep on tree_walk (check_vm_parity).
1 parent 9b0321b commit faea9ae

3 files changed

Lines changed: 38 additions & 26 deletions

File tree

‎.dev/debt.yaml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,6 @@
2424
# md-table-align no longer pads it to ~800 KB of whitespace.
2525

2626
active:
27-
- id: "D-557"
28-
status: "NARROWED 2026-07-07 (same session): the SHADOW is innocent (renaming front->front2 still fails); reverse is innocent (dropping it still fails); wrapping the for in `vec` FIXES it -> the defect is in the LAZY `for` seq being held across recur iterations and consumed TWICE ((into back nexts) then (map first nexts)) on the tree_walk backend — (first front) later yields a Long, i.e. a lazy-seq cell or chunk is corrupted/mis-linked. Synthetic twins (int states, plain + map-destructure for-bindings, two-list queue, 3000 iters) do NOT reproduce — the real repro needs the cw-arcade board/puzzles shape (map-of-maps states as HAMT-set keys + legal-moves itself built from a for). GC-independent (fails with collect disabled); vm green. REPRO: scratchpad rhmin (solver.clj = the 2026-07-07 two-list version), tree_walk build, cljw -cp src run_direct.clj -> `-nth on Long` at solver.clj:48. Variants preserved: solver_v3 (shadow renamed, fails), v5 (no reverse, fails), v7 (vec-wrapped for, PASSES). NEXT: diff the tree_walk lazy-seq/chunk realization path for a double-walk-across-recur defect; distill with the real board shape; land as a diff-oracle case. ORIGINAL: found during D-556 verification."
29-
category: "runtime"
30-
barrier: "none — deterministic repro preserved at private/d557_repro (gitignored): tree_walk build + `cljw -cp src run_direct.clj` fails; run_v7.clj (vec-wrapped for) passes. Minimize with the real board shape -> fix the tree_walk lazy-for double-consumption defect -> diff-oracle case."
31-
last_reviewed: "2026-07-07"
3227
- id: "D-556"
3328
status: "ROOT CAUSE FOUND + FIXED 2026-07-07 (the deftest-BFS integer-overflow panic + the D-555 GC family). The swept-string probe showed the FIRST post-bootstrap collect sweeping foundation strings (\"map\", \"clojure.core\", \"*in*\") on the tree_walk backend: fn-body Node literals live in the heap-lifetime analyzer arena, but the ADR-0169 AnalysisFrame bracket DROPPED its roots at form end — correct only for Nodes that die with their form; a (defn f …) body Node survives via Function.body and its literal pool was unrooted from the first collect on (the vm masks the EXECUTING chunk via the A1 pool root; a not-yet-executed fn had the same latent hole). FIX: root_set.endAnalysisPersist — production seams (driver x2, loader, repl, nrepl, evaluator oracle) persist their bracket roots into gc.persisted_analysis_roots (a GcHeap field, walked after permanent_roots; same lifetime the Nodes already have). ALSO landed in the same arc: the conservative C-stack scan (mark_sweep.conservativeStackScan: stack words from the ADR-0157 anchor + a setjmp register spill (Boehm GC_push_regs idiom), validated against a per-collect live-set hash, pinned — sound under non-moving mark-sweep) as the STRUCTURAL backstop for tree_walk's C-stack intermediates, and the 7 explicit A6/A6b brackets. VERIFIED: the original repro (rush-hour solver-test, pre-edit sources preserved in the scratchpad rhmin copy) passes 3/3 at BOTH default and 1MB thresholds on tree_walk; sym-meta GC e2e green; vm untouched. REMAINING out of scope: registered-WORKER-thread stacks are not conservatively scanned (parked at alloc-prologue; agent+tree_walk is the known D-418-adjacent area)."
3429
category: "runtime"
@@ -6525,6 +6520,12 @@ discharged:
65256520
category: "tooling"
65266521
barrier: "DISCHARGED (was: repurpose the idle probe)."
65276522
last_reviewed: "2026-07-06"
6523+
- id: "D-557"
6524+
status: "DISCHARGED 2026-07-07 (same session as filed). ROOT CAUSE: evalRecur evaluated its args DIRECTLY into the threadlocal pending_recur_buf — but an arg's eval can force a lazy seq whose `for`-lowered iterator loop RECURS (the :when-miss arm emits `(recur (rest gxs))`), and that INNER recur clobbers the same threadlocal, corrupting the outer recur's already-evaluated args (the two-list-BFS `-nth on Long`: a stray :delta Long landed in the queue slot). Fired only when a :when miss occurred during recur-arg evaluation — why the visited-heavy real solver hit it while synthetic twins passed; the vm is immune (recur args ride the operand stack). FIX: evaluate into a LOCAL buffer (ArgsFrame-rooted per D-556) and memcpy into the threadlocal only in the no-eval window right before RecurSignaled — the scratch is now touched only between signal and the matching loop frame's drain, where no nested recur can fire. VERIFIED: the cw-arcade two-list solver (direct + v3 shadow-renamed + v5 no-reverse variants + the full 3-test suite) all green on tree_walk; corpus lazy_eval +2 clj-byte-matched goldens (a distilled recur-arg-forces-:when-miss-for shape) — re-verified on the vm by every gate's corpus_regression AND on tree_walk by the check_vm_parity sweep (the double net). Diagnosis chain preserved in private/d557_repro + private/notes/D556-gc-persist-roots.md."
6525+
category: "runtime"
6526+
barrier: "DISCHARGED (was: pending_recur_buf reentrancy)."
6527+
last_reviewed: "2026-07-07"
6528+
65286529
conventions: |-
65296530
- Structure: two top-level lists — `active:` and `discharged:`. Active entries carry id / status / category / barrier (+ optional quality_floor / last_reviewed); discharged entries carry id / discharged_at / resolution. Edit as YAML (no Markdown table to align).
65306531
- New debt: grep `.dev/debt.yaml` for class-overlap before adding (per `.claude/rules/debt_dedup.md`).

‎src/eval/backend/tree_walk.zig‎

Lines changed: 28 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -440,15 +440,15 @@ pub fn allocFunctionFromSerialized(
440440

441441
// --- Top-level eval ---
442442

443-
/// ADR-0157 2a (tree_walk arm): per-thread native-stack guard state, the
444-
/// sibling of vm.zig's. The tree-walk evaluator recurses NATIVE frames per
445-
/// Node, so a deep user recursion (or a self-triggering watch's
446-
/// notify→swap!→notify chain) overflows the native stack → SIGSEGV where
447-
/// the VM raises the catchable stack_overflow. Anchor to the shallowest
448-
/// `eval` entry seen on this thread and raise once the REAL bytes consumed
449-
/// exceed the same 6 MiB budget. Per-node cost is a threadlocal read + two
450-
/// compares — negligible even for the oracle backend.
451-
threadlocal var stack_base: usize = 0;
443+
/// ADR-0157 2a (tree_walk arm): per-thread native-stack guard, the sibling of
444+
/// vm.zig's. The tree-walk evaluator recurses NATIVE frames per Node, so a
445+
/// deep user recursion (or a self-triggering watch's notify→swap!→notify
446+
/// chain) overflows the native stack → SIGSEGV where the VM raises the
447+
/// catchable stack_overflow. Anchor to the shallowest `eval` entry seen on
448+
/// this thread and raise once the REAL bytes consumed exceed the same 6 MiB
449+
/// budget. The anchor LIVES in root_set (`conservative_stack_top`) — it
450+
/// doubles as the D-556 conservative-scan upper bound, one anchor for both
451+
/// concerns. Per-node cost is a threadlocal read + two compares.
452452
const STACK_BUDGET_BYTES: usize = 6 * 1024 * 1024;
453453

454454
/// Evaluate one Node into a Value. `locals` is the slot array owned
@@ -461,9 +461,9 @@ pub fn eval(
461461
) anyerror!Value {
462462
{
463463
const sp = @frameAddress();
464-
if (stack_base == 0 or sp > stack_base) {
465-
stack_base = sp; // (re-)anchor to the shallowest (highest) entry seen
466-
} else if (stack_base - sp > STACK_BUDGET_BYTES) {
464+
if (root_set.conservative_stack_top == 0 or sp > root_set.conservative_stack_top) {
465+
root_set.conservative_stack_top = sp; // (re-)anchor to the shallowest (highest) entry seen
466+
} else if (root_set.conservative_stack_top - sp > STACK_BUDGET_BYTES) {
467467
return error_catalog.raise(.stack_overflow, .{}, .{ .max = STACK_BUDGET_BYTES });
468468
}
469469
}
@@ -1030,21 +1030,28 @@ fn evalRecur(rt: *Runtime, env: *Env, locals: []Value, n: node_mod.RecurNode) an
10301030
return error_catalog.raise(.recur_args_exceed_buffer, n.loc, .{ .got = n.args.len, .max = pending_recur_buf.len });
10311031
// Evaluate **all** args before mutating any slot — otherwise a
10321032
// recur arg that referenced an earlier binding would see the
1033-
// partially-rewritten frame. Stash them in the threadlocal scratch
1034-
// so the matching loop frame can drain them once the unwind
1035-
// returns control.
1033+
// partially-rewritten frame.
1034+
//
1035+
// D-557 REENTRANCY: evaluate into a LOCAL buffer, not the threadlocal
1036+
// scratch — an arg's eval can force a lazy seq whose iterator loop
1037+
// recurs (the `for` lowering's `:when`-miss arm), and that INNER recur
1038+
// writes the same threadlocal, clobbering the outer recur's
1039+
// already-evaluated args (the two-list-BFS `-nth on Long` corruption;
1040+
// vm is immune — its recur args ride the operand stack). The
1041+
// threadlocal is only written in the no-eval window between here and
1042+
// the matching loop frame's drain, where no nested recur can fire.
1043+
//
10361044
// GC-ROOT: A6b — already-evaluated recur values are reachable only
1037-
// through the scratch while a LATER arg's eval allocates (D-556; the
1038-
// hottest case: a BFS loop/recur carrying fresh vectors/maps). The
1039-
// RecurSignaled unwind + slot writeback allocate nothing, so the frame
1040-
// popping at return is safe.
1045+
// through this buffer while a LATER arg's eval allocates (D-556).
1046+
var local_buf: [MAX_LOCALS]Value = undefined;
10411047
var af: ArgsFrame = undefined;
1042-
af.push(&pending_recur_buf);
1048+
af.push(&local_buf);
10431049
defer af.pop();
10441050
for (n.args, 0..) |*a, i| {
1045-
pending_recur_buf[i] = try eval(rt, env, locals, a);
1051+
local_buf[i] = try eval(rt, env, locals, a);
10461052
af.publish(i + 1);
10471053
}
1054+
@memcpy(pending_recur_buf[0..n.args.len], local_buf[0..n.args.len]);
10481055
pending_recur_len = @intCast(n.args.len);
10491056
return error.RecurSignaled;
10501057
}

‎test/diff/clj_corpus/lazy_eval.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,7 @@
2828
;;=> [0 1 2 3]
2929
(realized? (lazy-seq [1]))
3030
;;=> false
31+
(loop [acc [] xs [1 2 3 4]] (if (empty? xs) acc (recur (into acc (for [y [1 2] :when (odd? y)] [y (first xs)])) (rest xs))))
32+
;;=> [[1 1] [1 2] [1 3] [1 4]]
33+
(loop [front (list [:s []]) back () n 0] (if (or (> n 20) (nil? (first front))) n (let [[st p] (first front) nexts (for [d [1 2] :when (= d 1)] [d (conj p st)])] (recur (rest front) (into back nexts) (inc n)))))
34+
;;=> 1

0 commit comments

Comments
 (0)