Skip to content

Commit 6abb9ee

Browse files
committed
fix(gc): D-556 root cause — persist analysis roots past their bracket + conservative tree_walk stack scan
Smell-audited: 2: the whack-a-mole arc itself was the smell — three structural layers landed instead of more per-site brackets: (1) the ROOT CAUSE, (2) a conservative backstop, (3) the D-555 sweep as the standing detection net. D-557 (a separate, GC-independent tree_walk shadow/slot bug the verification uncovered) is filed with a deterministic repro. ROOT CAUSE (the deftest-BFS "integer overflow" panic + most of the D-555 family): fn-body Node literals live in the heap-lifetime analyzer arena, but the ADR-0169 AnalysisFrame DROPPED its roots at form end — correct only for Nodes that die with their form. A (defn f ...) body Node survives via Function.body, so its literal pool (name strings, quoted collections — the swept-string probe showed "map"/"clojure.core"/"*in*" being swept) was unrooted from the first post-definition collect on. The vm masks the EXECUTING chunk via the A1 pool root; a not-yet-executed fn's pool had the same latent hole on BOTH backends. FIX: root_set.endAnalysisPersist — production seams (driver x2, loader, repl, nrepl, evaluator oracle) persist bracket roots into gc.persisted_analysis_roots (a GcHeap field walked after permanent_roots; the same lifetime the Nodes already have). Test fixtures keep the dropping endAnalysis. STRUCTURAL BACKSTOP: mark_sweep.conservativeStackScan — at collect, walk the collecting thread's native stack from the ADR-0157 anchor (now shared: root_set.conservative_stack_top) + a setjmp register spill (Boehm GC_push_regs idiom), decode words via the heapHeader membrane (zero-address pre-filter), validate against a per-collect live-set hash, pin matches. Sound under non-moving mark-sweep; tree_walk-only (the vm never sets the anchor). VERIFIED: the original rush-hour repro passes 3/3 at default AND 1MB thresholds on tree_walk; sym-meta GC e2e green; vm smoke green (gc_torture step included).
1 parent c935e26 commit 6abb9ee

9 files changed

Lines changed: 123 additions & 9 deletions

File tree

‎.dev/debt.yaml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,16 @@
2424
# md-table-align no longer pads it to ~800 KB of whitespace.
2525

2626
active:
27+
- id: "D-557"
28+
status: "open (2026-07-07, found DURING the D-556 verification — a SEPARATE, GC-independent tree_walk bug). The rewritten cw-arcade rush-hour solver (two-list amortized queue: 3-arity self-call chain + loop [front back visited] + (recur (reverse back) () visited) + a let that destructures [state path] (first front) and SHADOWS front with (rest front)) fails on the tree_walk backend with `-nth on Long` at the destructure line — (first front) returns a Long, i.e. a slot/shadow resolution defect. DETERMINISTIC; vm green on the identical code; CLJW_GC_THRESHOLD_MB=100000 (GC off) still fails -> NOT a rooting issue. Direct call fails too (no clojure.test needed). Simple synthetic shadow shapes pass — the trigger needs the real solver's combination (multi-arity self-call + reverse/list queue + same-name shadow + for). REPRO: scratchpad rhmin copy (or cw-arcade apps/rush-hour post-2026-07-07-edit sources), tree_walk build, cljw -cp src run_direct.clj. Fix by minimizing from the real solver; land the distilled case as a diff-oracle case (the F-012 net SHOULD have caught this — its absence is the gap D-555's cadence work addresses)."
29+
category: "runtime"
30+
barrier: "none — deterministic repro in hand (tree_walk build + the rhmin copy). Minimize -> analyzer slot-allocation fix -> diff case."
31+
last_reviewed: "2026-07-07"
2732
- id: "D-556"
28-
status: "RE-NARROWED 2026-07-06 (late session): the panic REPRODUCES — the split observation is explained: the repro binary was a -Dbackend=tree_walk build (the --version fingerprint shows \", tree_walk\"); the vm default never panics. ROOT CLASS confirmed by bisection: CLJW_GC_THRESHOLD_MB=100000 (auto-collect off) -> 3/3 tests pass; default threshold -> deterministic panic; 1MB threshold -> an EARLIER, cleaner corruption (\"+: expected number, got nil\" inside board.clj occupied). So: tree_walk holds evaluation intermediates ONLY on the C stack, and a mid-eval ADR-0164 auto-collect sweeps them — the corrupted-string usize overflow in Writer.drain is the downstream symptom. SEVEN real unrooted sites were found+fixed this session (A6 locals: callMethodImpl + driver.evalForm + evaluator oracle; A6b progressive-publish ArgsFrame: evalCall callee+args, ctor/instance/static interop args, vector/map/set literal accumulators, evalRecur pending buf, evalBinding pass-1 inits) — symbol-meta GC loss is FIXED — but the rush-hour repro STILL panics: at least one more unrooted path remains (the 1MB trace points at a closure/lazy intermediate in a some-fn/for chain). FINISHED-FORM DECISION RECORDED: stop the whack-a-mole — implement a CONSERVATIVE C-STACK SCAN for the tree_walk backend: at collect, walk the native stack words from the ADR-0157 stack_base anchor (already threadlocal!) down to the collector frame, decode each word as a potential NaN-boxed heap Value, validate the pointer against a per-collect hash set of live objects (built in one O(live) pass over the live list), and pin matches. Non-moving mark-sweep makes conservative pinning sound (false positives just delay a free). ~100-150 lines in root_set/mark_sweep + a tree_walk-registered stack-range provider. This makes ALL C-stack intermediates safe structurally and lets the explicit-bracket sites become optimizations rather than correctness requirements. Repro: cw-arcade rush-hour solver-test on a tree_walk build (see the private handover note); ALSO reachable via bash scripts/check_vm_parity.sh."
33+
status: "ROOT CAUSE FOUND + FIXED 2026-07-07 (the deftest-BFS integer-overflow panic + the D-555 GC family). The swept-string probe showed the FIRST post-bootstrap collect sweeping foundation strings (\"map\", \"clojure.core\", \"*in*\") on the tree_walk backend: fn-body Node literals live in the heap-lifetime analyzer arena, but the ADR-0169 AnalysisFrame bracket DROPPED its roots at form end — correct only for Nodes that die with their form; a (defn f …) body Node survives via Function.body and its literal pool was unrooted from the first collect on (the vm masks the EXECUTING chunk via the A1 pool root; a not-yet-executed fn had the same latent hole). FIX: root_set.endAnalysisPersist — production seams (driver x2, loader, repl, nrepl, evaluator oracle) persist their bracket roots into gc.persisted_analysis_roots (a GcHeap field, walked after permanent_roots; same lifetime the Nodes already have). ALSO landed in the same arc: the conservative C-stack scan (mark_sweep.conservativeStackScan: stack words from the ADR-0157 anchor + a setjmp register spill (Boehm GC_push_regs idiom), validated against a per-collect live-set hash, pinned — sound under non-moving mark-sweep) as the STRUCTURAL backstop for tree_walk's C-stack intermediates, and the 7 explicit A6/A6b brackets. VERIFIED: the original repro (rush-hour solver-test, pre-edit sources preserved in the scratchpad rhmin copy) passes 3/3 at BOTH default and 1MB thresholds on tree_walk; sym-meta GC e2e green; vm untouched. REMAINING out of scope: registered-WORKER-thread stacks are not conservatively scanned (parked at alloc-prologue; agent+tree_walk is the known D-418-adjacent area)."
2934
category: "runtime"
30-
barrier: "none — live deterministic repro (tree_walk build + cw-arcade rush-hour solver-test; vm default unaffected). NEXT UNIT = the conservative C-stack scan (the finished form; design in status). The 7 explicit-bracket fixes landed 2026-07-06 remain correct and useful regardless."
31-
last_reviewed: "2026-07-06"
35+
barrier: "FIXED for the reported class — keep open until the D-555 re-sweep (check_vm_parity.sh) confirms which of the 6 findings cleared, then discharge with the sweep numbers."
36+
last_reviewed: "2026-07-07"
3237
- id: "D-555"
3338
status: "RE-NARROWED 2026-07-06 (same day): finding (1) watch_recursion SEGFAULT is FIXED — the ADR-0157 native-stack guard (threadlocal stack_base anchor + 6MiB budget) existed ONLY in vm.zig; the tree_walk arm is now ported (per-node check in eval; a threadlocal read + two compares). SIGSEGV -> catchable stack_overflow, all watch/validator/reducer reentry cases green on BOTH backends. The e2e 500-deep legit-recursion cases were vm-shaped: tree_walk pays ~30KB native per HOF step / ~13KB per direct step (ReleaseSafe), so 500 PHYSICALLY exceeds the 6MiB budget (pre-guard: raw SIGSEGV) — cases (6)/(7) are now backend-aware (100/300 deep on tree_walk; --version gained a \", tree_walk\" marker for detection, vm banner unchanged). FORWARD: raising tree_walk depth to clj-like levels needs build.zig stack_size + an ADR-0157 budget amendment (depth-2) — take up if the oracle depth becomes limiting. REMAINING (4 findings): (2) phase16_gc_torture dies; (3) phase14_symbol_metadata sym_meta_gc_survives nil-want-1 (GC rooting gap, tree_walk-only); (4) phase14_seq_helpers2 interleave_large got-empty (possibly the SAME budget physics on a deep lazy realize — triage vs a rooting bug); (5) phase15_dynamic_var guard + (6) phase15_require_fs missing, both got-2-want-1 (exit-code shaped). ORIGINAL: six e2e fail on -Dbackend=tree_walk (D-553 first sweep), corpus green, F-012 oracle green."
3439
category: "runtime"

‎src/app/nrepl.zig‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -309,7 +309,7 @@ fn replyEval(
309309
// every exit of this loop iteration.
310310
var af: root_set.AnalysisFrame = undefined;
311311
root_set.beginAnalysis(&af, rt.gc.infra);
312-
defer root_set.endAnalysis(&af);
312+
defer root_set.endAnalysisPersist(&af, &rt.gc);
313313
const node = analyzeForm(arena, rt, env, null, form, macro_table) catch |err| {
314314
try replyError(arena, w, @errorName(err), session_id, id_val);
315315
continue;

‎src/app/repl.zig‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ fn evalOneLine(
223223
// D-430: per-form analysis bracket (roots literals through eval).
224224
var af: root_set.AnalysisFrame = undefined;
225225
root_set.beginAnalysis(&af, rt.gc.infra);
226-
defer root_set.endAnalysis(&af);
226+
defer root_set.endAnalysisPersist(&af, &rt.gc);
227227
const node = try analyzeForm(arena, rt, env, null, form, macro_table);
228228
var locals: [driver.MAX_LOCALS]Value = [_]Value{.nil_val} ** driver.MAX_LOCALS;
229229
const result = try driver.evalForm(rt, env, &locals, arena, node);

‎src/eval/driver.zig‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ pub fn runEnvelope(rt: *Runtime, env: *Env, arena: std.mem.Allocator, payload: [
6565
// constants, not the whole envelope's).
6666
var af: root_set.AnalysisFrame = undefined;
6767
root_set.beginAnalysis(&af, rt.gc.infra);
68-
defer root_set.endAnalysis(&af);
68+
defer root_set.endAnalysisPersist(&af, &rt.gc);
6969
var chunk = try serialize.deserializeChunk(arena, rt, env, chunk_bytes);
7070
_ = try vm.eval(rt, env, &locals, &chunk);
7171
}
@@ -154,7 +154,7 @@ pub fn evalTopLevelForm(
154154
// covers tree_walk Node constants, which have no EvalFrame pool).
155155
var af: root_set.AnalysisFrame = undefined;
156156
root_set.beginAnalysis(&af, rt.gc.infra);
157-
defer root_set.endAnalysis(&af);
157+
defer root_set.endAnalysisPersist(&af, &rt.gc);
158158
const node = try analyzer.analyze(arena, rt, env, null, form, table);
159159
return evalForm(rt, env, locals, arena, node);
160160
}

‎src/eval/evaluator.zig‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ fn evalTopLevelInBackend(
103103
// twin of driver.evalTopLevelForm's bracket.
104104
var af: root_set.AnalysisFrame = undefined;
105105
root_set.beginAnalysis(&af, rt.gc.infra);
106-
defer root_set.endAnalysis(&af);
106+
defer root_set.endAnalysisPersist(&af, &rt.gc);
107107
const node = try analyzer.analyze(arena, rt, env, null, form, table);
108108
var locals: [256]Value = [_]Value{.nil_val} ** 256;
109109
return switch (backend) {

‎src/eval/loader.zig‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ fn loadTopLevelForm(
9696
// driver.evalTopLevelForm's (kept open through eval + the sink recompile).
9797
var af: root_set.AnalysisFrame = undefined;
9898
root_set.beginAnalysis(&af, rt.gc.infra);
99-
defer root_set.endAnalysis(&af);
99+
defer root_set.endAnalysisPersist(&af, &rt.gc);
100100
const node = try analyzeForm(arena, rt, env, null, form, macro_table);
101101
var locals: [driver.MAX_LOCALS]Value = [_]Value{.nil_val} ** driver.MAX_LOCALS;
102102
_ = try driver.evalForm(rt, env, &locals, arena, node);

‎src/runtime/gc/gc_heap.zig‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,13 @@ pub const GcHeap = struct {
170170
/// first match. The root walker yields each entry's
171171
/// `Value.heapHeader()` (skipping immediates).
172172
permanent_roots: std.ArrayList(Value) = .empty,
173+
/// D-556: analysis-produced Values persisted past their bracket — the
174+
/// literal pool of every analyzed form whose Nodes outlive the form (fn
175+
/// bodies held by `Function.body` in the heap-lifetime analyzer arena).
176+
/// Appended by `root_set.endAnalysisPersist`; walked right after
177+
/// `permanent_roots`. Grows with the arena — the same lifetime the
178+
/// Nodes themselves already have.
179+
persisted_analysis_roots: std.ArrayList(Value) = .empty,
173180
/// Process-lifetime mark-waypoints (D-251): headers of `trackHeap`'d
174181
/// objects (`Function` / `ProtocolDescriptor` / `TypeDescriptorRef`) that
175182
/// are NOT in `allocations` and so are never swept. The mark bit doubles as
@@ -270,6 +277,7 @@ pub const GcHeap = struct {
270277
}
271278
self.allocations.deinit(self.infra);
272279
self.permanent_roots.deinit(self.infra);
280+
self.persisted_analysis_roots.deinit(self.infra);
273281
self.persistent_marks.deinit(self.infra);
274282
self.free_pools.deinit(self.infra);
275283
self.mark_worklist.deinit(self.infra);

‎src/runtime/gc/mark_sweep.zig‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,13 @@ pub fn collect(gc: *GcHeap, ctx: root_set_mod.WalkContext) void {
211211
// ...and every registered WORKER thread's transaction (parked at a safepoint
212212
// during STW, so its tx is quiescent). Empty registry in single-thread → no-op.
213213
root_set_mod.markRegisteredTxs(@ptrCast(gc), &markWorkerTx);
214+
// D-556: conservative native-stack scan (tree_walk backend only — the vm's
215+
// operand stack is the A1 root). Pins every stack word that decodes to a
216+
// live heap Value, covering evaluation intermediates the explicit brackets
217+
// miss. Runs on the COLLECTING thread's stack (workers are parked at their
218+
// alloc-prologue, where their intermediates are bracket-covered or a known
219+
// D-556 residual).
220+
conservativeStackScan(gc);
214221
// All roots are shaded gray; blacken transitively (ADR-0028 amendment 3).
215222
drainGray(gc);
216223
sweep(gc);
@@ -225,6 +232,57 @@ pub fn collect(gc: *GcHeap, ctx: root_set_mod.WalkContext) void {
225232
gc.stats.collect_count += 1;
226233
}
227234

235+
/// D-556: conservatively scan the collecting thread's native stack for live
236+
/// heap Values (tree_walk backend). Range: from the collector's own frame
237+
/// (deep) up to `root_set.conservative_stack_top` (the shallowest tree_walk
238+
/// eval entry; 0 = tree_walk never ran on this thread → skip). Each 8-byte
239+
/// word is decoded via the `Value.heapHeader()` membrane (tag check only, no
240+
/// deref) and pinned IFF the pointer matches a live allocation record — a
241+
/// stale/garbage word that happens to look like a heap tag can only pin a
242+
/// still-live object (false positive = one delayed free), never deref a
243+
/// dangling pointer. The live-pointer set is built in one pass over the
244+
/// `allocations` side-table using `gc.infra` (never `gc.alloc` — collect must
245+
/// not re-enter the heap).
246+
/// libc setjmp — used ONLY as a register spill (Boehm GC's `GC_push_regs`
247+
/// idiom): it writes every callee-saved register into the buffer, so a heap
248+
/// Value living ONLY in a callee-saved register (never spilled on the paths
249+
/// the collector happened to take) becomes scannable. We never longjmp.
250+
extern fn setjmp(env: *anyopaque) c_int;
251+
252+
/// Decode one candidate word and pin it if it is a live heap Value.
253+
inline fn pinIfLiveWord(gc: *GcHeap, live: *const std.AutoHashMapUnmanaged(usize, void), word: u64) void {
254+
const Value = @import("../value/value.zig").Value;
255+
const nb = @import("../value/nan_box.zig");
256+
// Pre-filter a zero address: `heapHeader`'s decodePtr would cast it to a
257+
// null pointer (a ReleaseSafe panic) — a heap-TAGGED word with address
258+
// bits 0 is garbage, never a real Value.
259+
if (word & nb.NB_ADDR_SHIFTED_MASK == 0) return;
260+
const v: Value = @enumFromInt(word);
261+
const hdr = v.heapHeader() orelse return;
262+
if (live.contains(@intFromPtr(hdr))) mark(gc, hdr);
263+
}
264+
265+
fn conservativeStackScan(gc: *GcHeap) void {
266+
const top = root_set_mod.conservative_stack_top;
267+
if (top == 0) return;
268+
const bottom = @frameAddress();
269+
if (bottom >= top) return;
270+
var live: std.AutoHashMapUnmanaged(usize, void) = .empty;
271+
defer live.deinit(gc.infra);
272+
live.ensureTotalCapacity(gc.infra, @intCast(gc.allocations.items.len)) catch return;
273+
for (gc.allocations.items) |rec| live.putAssumeCapacity(@intFromPtr(rec.header), {});
274+
// Registers first (see `setjmp` above): 128 u64s comfortably hold any
275+
// ABI's jmp_buf; unwritten tail words stay zero and are filtered.
276+
var jb: [128]u64 align(16) = [_]u64{0} ** 128;
277+
_ = setjmp(@ptrCast(&jb));
278+
for (jb) |word| pinIfLiveWord(gc, &live, word);
279+
// Then the native stack from the collector's frame up to the anchor.
280+
var addr = std.mem.alignForward(usize, bottom, @alignOf(u64));
281+
while (addr + @sizeOf(u64) <= top) : (addr += @sizeOf(u64)) {
282+
pinIfLiveWord(gc, &live, @as(*const u64, @ptrFromInt(addr)).*);
283+
}
284+
}
285+
228286
/// Stop-the-world collect (ADR-0090 Alt B / D-244 #4): pause every other
229287
/// registered worker at a safe point (its `alloc`-prologue park or the VM
230288
/// back-edge poll), run a full `collect` over the union root set (every parked

‎src/runtime/gc/root_set.zig‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,19 @@ pub const EvalFrame = struct {
124124
/// a `ThreadGcContext.eval_frame_slot` pointer (union walk).
125125
pub threadlocal var eval_frame_head: ?*EvalFrame = null;
126126

127+
/// D-556: conservative C-stack scan anchor for the tree_walk backend. The
128+
/// tree-walk evaluator holds evaluation intermediates in NATIVE stack slots
129+
/// (locals arrays, argument buffers, accumulator temporaries) — the explicit
130+
/// EvalFrame brackets cover the known hot sites, but per-site bracketing is
131+
/// not exhaustively provable, so the collector ALSO conservatively scans the
132+
/// native stack from this anchor (the shallowest tree_walk `eval` entry —
133+
/// the same value as its ADR-0157 budget anchor) down to the collector's own
134+
/// frame, pinning any word that decodes to a live heap Value. Sound under
135+
/// the non-moving mark-sweep (a false positive only delays one free). Zero
136+
/// (the default) = no tree_walk eval on this thread → scan skipped; the vm
137+
/// backend never sets it (its operand stack is the A1 root).
138+
pub threadlocal var conservative_stack_top: usize = 0;
139+
127140
/// Analysis-roots frame (D-430 root cause; EvalFrame's analysis-side
128141
/// sibling): GC Values produced BEFORE any frame executes — literal strings /
129142
/// quoted data (`analyzer.makeConstant` / `analyzeQuote`), compiler-alloc'd
@@ -176,6 +189,26 @@ pub fn endAnalysis(frame: *AnalysisFrame) void {
176189
frame.roots.deinit(frame.alloc);
177190
}
178191

192+
/// D-556: close an analysis bracket by PERSISTING its roots into the heap's
193+
/// `persisted_analysis_roots` (walked right after `permanent_roots`) instead
194+
/// of dropping them. The analyzer arena — and every `*Node` it holds,
195+
/// including fn bodies that survive the defining form via `Function.body` —
196+
/// lives as long as the heap, so the GC Values embedded in those Nodes
197+
/// (`.constant` literals, name strings, quoted collections) must stay rooted
198+
/// for the same lifetime. `endAnalysis`'s drop-on-close is correct only for
199+
/// Nodes that die with their form (test fixtures); a `(defn f …)` body Node
200+
/// survives, and its literals were swept at the first post-definition
201+
/// collect — the tree_walk bootstrap-string corruption class (the vm masks
202+
/// the EXECUTING chunk via the A1 pool root, but a not-yet-executed fn's
203+
/// pool had the same hole). An append failure leaks the frame's list in
204+
/// place — still reachable beats swept.
205+
pub fn endAnalysisPersist(frame: *AnalysisFrame, gc: *GcHeap) void {
206+
std.debug.assert(analysis_frame_head == frame);
207+
analysis_frame_head = frame.parent;
208+
gc.persisted_analysis_roots.appendSlice(gc.infra, frame.roots.items) catch return;
209+
frame.roots.deinit(frame.alloc);
210+
}
211+
179212
/// Root a Value produced during analysis/compile/deserialize until the
180213
/// owning bracket closes. Immediates / infra-lifetime values (var_ref, ns)
181214
/// are filtered here so the registry holds only real heap objects.
@@ -488,6 +521,9 @@ pub const RootIterator = struct {
488521

489522
pub const PermanentRootsCursor = struct {
490523
idx: usize = 0,
524+
/// D-556: cursor into `persisted_analysis_roots` (walked after the
525+
/// gc's own permanent_roots).
526+
persisted_idx: usize = 0,
491527
};
492528

493529
pub fn next(self: *RootIterator) ?*HeapHeader {
@@ -698,6 +734,13 @@ pub const RootIterator = struct {
698734
c.idx += 1;
699735
if (v.heapHeader()) |hdr| return hdr;
700736
}
737+
// D-556: then the persisted analysis roots (fn-body literals whose
738+
// Nodes live in the heap-lifetime analyzer arena).
739+
while (c.persisted_idx < self.ctx.gc.persisted_analysis_roots.items.len) {
740+
const v = self.ctx.gc.persisted_analysis_roots.items[c.persisted_idx];
741+
c.persisted_idx += 1;
742+
if (v.heapHeader()) |hdr| return hdr;
743+
}
701744
return null;
702745
}
703746
};

0 commit comments

Comments
 (0)