Skip to content

Commit 7a36f1b

Browse files
committed
fix(agent): D-418 send/await fabrication-window GC race — root the action vector + await promise
The recurring low-core agent_conj corruption (`[2 nil]` / `[#<promise> 2]` / `[#<fn> [#<promise>]]`, tracked D-418/D-258/D-548) was the send/await ENQUEUE WINDOW. On the JVM a constructed Action is automatically GC-reachable so there is no window; cljw's manual mark-sweep had one — `sendFn` built the `[f & args]` action vector (and `awaitFn` the completion promise) as a Zig LOCAL that stayed unrooted until `enqueueDirect` appended it to the off-heap queue (its traceGc root). A collect in that window swept the freshly-built value, so the drainer read recycled memory. Serialization (single-drainer + `draining` flag) was NEVER the bug — the Agent.java survey confirms it is the correct design for a manual-GC runtime (private/notes/D418-agent-race-survey.md). Fix: sendFn + awaitFn publish the action / promise on an EvalFrame across the whole enqueue (gc_rooting.md §A A6/A7). Deterministic reproducer: agent.zig `tortureCollectInWindow` injects a STW collect into the exact enqueue window under CLJW_GC_TORTURE_ALLOC (the "def→cgc→use" fault-injection idiom, USENIX Sec'21) — so the cross-thread race is a single-thread deterministic UAF (pre-fix agent_conj HUNG on a swept promise, exit 124; post-fix 10/10 [1 2]). Guard: gc_torture.sh gains a deterministic alloc-torture agent block AND drops the ncpu>=4 workaround on the back-edge block (both run everywhere now). A nested-cross-agent-send variant surfaced a distinct `@memcpy arguments alias` corruption → new row D-559. D-418 DISCHARGED; D-258/D-548 updated; D-559 opened. Smell-audited: 1: bug fix via the established EvalFrame GC-rooting idiom (gc_rooting.md §C) + a torture-gated fault injection mirroring gc_heap's alloc-torture; no new design. Verified: RED (exit 124 hang) → GREEN (10/10 deterministic) + full gc_torture.sh 80 PASS + smoke green.
1 parent 93e3bc3 commit 7a36f1b

5 files changed

Lines changed: 140 additions & 30 deletions

File tree

‎.dev/debt.yaml‎

Lines changed: 35 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,25 @@
2424
# md-table-align no longer pads it to ~800 KB of whitespace.
2525

2626
active:
27+
- id: "D-559"
28+
status: |-
29+
open (2026-07-13, surfaced while building the D-418 deterministic guard). A
30+
NESTED CROSS-AGENT send corrupts under alloc-torture: a watch on agent A that
31+
`(send B …)`s, then `(await A) (await B)`, panics `@memcpy arguments alias`
32+
(exit 134, ~1/10 under CLJW_GC_TORTURE_ALLOC=1). Repro:
33+
`CLJW_GC_TORTURE_ALLOC=1 cljw -e '(let [log (agent []) a (agent 0)] (add-watch a :k (fn [k r o n] (send log conj n))) (send a inc) (send a inc) (await a) (await log) @log)'`
34+
(clj-correct value is [1 2 2] — the await barrier's no-op watch fires once more).
35+
DISTINCT from D-418 (that was the send/await fabrication window, now fixed +
36+
the 4 single-agent cases are 10/10 clean): this is the TWO-DRAINER +
37+
nested_pending path — a nested send built/held on drainer A while it runs, and
38+
the log agent's own `conj` builder, interact with the main-thread window collect.
39+
`@memcpy arguments alias` = a vector op memcpy'd overlapping regions → a
40+
swept/recycled tail buffer aliasing. Suspect: the nested_pending held action
41+
(pinned) vs the drainer's conj intermediate rooting across the injected collect,
42+
or a two-agent drain ordering that lets a collect see an unrooted builder node.
43+
category: "concurrency"
44+
barrier: "none — deterministically reproducible (~1/10 under alloc-torture, the D-418 window-collect injection makes it surface). Needs a focused trace of the nested_pending handoff + the log-agent conj builder rooting under a concurrent collect; next unit after the D-418 discharge. NOT ncpu-gated away (per user: a wrong-result race is a bug, not an accepted flake)."
45+
last_reviewed: "2026-07-13"
2746
- id: "D-522"
2847
status: |-
2948
open (2026-06-24, user-directed; ADR-0166 public-ization sweep). Comment
@@ -377,7 +396,7 @@ active:
377396
gains a second concurrent writer.
378397
last_reviewed: "2026-06-15"
379398
- id: "D-258"
380-
status: "recorded (2026-06-06; RECURRED 2026-06-07 twice — during the D-280d2 gate (`[#<fn_val> 2]`) and the D-281 gate (`[#builtin 2]`), both 12/12 standalone PASS; RECURRED again 2026-06-19 during the java.time campaign-completion gate (`[#<promise> 2]`, a promise leaked into the agent's `[1 2]`), confirmed load-flaky by a 4× standalone re-run (3 PASS / 1 FAIL), unrelated to the java.time work which touches no agent/GC/threading; re-gate green each time) — `phase16_gc_torture` agent_conj flakes under gate load (the leaked object varies: `[#builtin 2]` / `[#<fn_val> 2]` / `[#<promise> 2]` vs `[1 2]`); standalone PASS → load-induced, not a regression. Confirms the dormant multi-thread-torture race (D-244 #4). RECURRED again 2026-06-20 during the B/C-sweep full gate (`[#<fn> [#<promise>]]`), under heavy session load (parallel gates + subagents + clj probes); 3× standalone re-run all PASS — same load-flake, NOT from the D-442 agent-surface work (which touches no drainer/GC path). RECURRED again 2026-06-25 during the D-526 interop-static-sweep full gate, this time on a DIFFERENT test — `phase14_future_promise_delay` SIGABRT'd (exit 134) after the delay sub-tests, in the future/promise/cancel section; standalone 5/5 PASS, and an 8× parallel-load stress reproduced it (3/8 exit 134, 5/8 PASS), confirming the same dormant D-244 #4 race (now observed in the future/promise path, not just gc_torture agent_conj); NOT from the interop work (adds only Long/Integer/Math/Character/Double/Boolean static methods — no threading/GC/eval change). Re-gate alone → green (394/0). RECURRED again 2026-06-25 during the easiest-first debt-drain batch (bytes?/Serializable/BigDecimal/BigInteger/file_io) — `agent_conj` → `[#<promise> 2]` under the `-P8` parallel pool, TWICE in a row; standalone 2/2 PASS, and `--serial-e2e` (the canonical full-gate mode, no parallel fan-out) → green (failed: 0). Same D-244 #4 / D-418 race; NONE of the batch touches agent/promise/GC/eval. Lesson reinforced: the load-flake reproduces under `-P8` even WITHOUT a concurrent build — the canonical full gate is `--serial-e2e`, not the parallel default."
399+
status: "recorded (2026-06-06; RECURRED 2026-06-07 twice — during the D-280d2 gate (`[#<fn_val> 2]`) and the D-281 gate (`[#builtin 2]`), both 12/12 standalone PASS; RECURRED again 2026-06-19 during the java.time campaign-completion gate (`[#<promise> 2]`, a promise leaked into the agent's `[1 2]`), confirmed load-flaky by a 4× standalone re-run (3 PASS / 1 FAIL), unrelated to the java.time work which touches no agent/GC/threading; re-gate green each time) — `phase16_gc_torture` agent_conj flakes under gate load (the leaked object varies: `[#builtin 2]` / `[#<fn_val> 2]` / `[#<promise> 2]` vs `[1 2]`); standalone PASS → load-induced, not a regression. Confirms the dormant multi-thread-torture race (D-244 #4). RECURRED again 2026-06-20 during the B/C-sweep full gate (`[#<fn> [#<promise>]]`), under heavy session load (parallel gates + subagents + clj probes); 3× standalone re-run all PASS — same load-flake, NOT from the D-442 agent-surface work (which touches no drainer/GC path). RECURRED again 2026-06-25 during the D-526 interop-static-sweep full gate, this time on a DIFFERENT test — `phase14_future_promise_delay` SIGABRT'd (exit 134) after the delay sub-tests, in the future/promise/cancel section; standalone 5/5 PASS, and an 8× parallel-load stress reproduced it (3/8 exit 134, 5/8 PASS), confirming the same dormant D-244 #4 race (now observed in the future/promise path, not just gc_torture agent_conj); NOT from the interop work (adds only Long/Integer/Math/Character/Double/Boolean static methods — no threading/GC/eval change). Re-gate alone → green (394/0). RECURRED again 2026-06-25 during the easiest-first debt-drain batch (bytes?/Serializable/BigDecimal/BigInteger/file_io) — `agent_conj` → `[#<promise> 2]` under the `-P8` parallel pool, TWICE in a row; standalone 2/2 PASS, and `--serial-e2e` (the canonical full-gate mode, no parallel fan-out) → green (failed: 0). Same D-244 #4 / D-418 race; NONE of the batch touches agent/promise/GC/eval. Lesson reinforced: the load-flake reproduces under `-P8` even WITHOUT a concurrent build — the canonical full gate is `--serial-e2e`, not the parallel default. ROOT CAUSE FIXED 2026-07-13: the recurring `agent_conj` flake was the D-418 send/await FABRICATION-WINDOW race (unrooted action vector / completion promise across the enqueue) — DISCHARGED (see D-418), deterministically reproduced via the alloc-torture window-collect injection + fixed by EvalFrame-rooting in primitive/agent.zig; the ncpu>=4 gate on the gc_torture agent block is REMOVED. Residual future/promise/delay low-core exposures stay under D-548; a newly-found nested-cross-agent corruption is D-559. This row is retained as the historical flake-recurrence log."
381400
category: "concurrency"
382401
barrier: |-
383402
During the cljw.http.server gate (2026-06-06, `--serial-e2e`), the
@@ -1004,11 +1023,6 @@ active:
10041023
deprecated). Cross-ref ADR-0135 (design SSOT), ADR-0099 (core spike), F-001,
10051024
ROADMAP §1.2 axis 2 + §8.2. Re-check the zwasm freeze each Phase boundary.
10061025
last_reviewed: "2026-07-02"
1007-
- id: "D-418"
1008-
status: "open (2026-06-13, surfaced as a parallel-gate flaky in e2e_phase16_gc_torture). `agent_conj` — `(let [a (agent [])] (send a conj 1) (send a conj 2) (await a) @a)` — intermittently returns `[#<promise> 2]` instead of `[1 2]` UNDER PARALLEL GATE LOAD (CPU contention from the full run_all.sh e2e fan-out). A `#<promise>` (the agent's internal send-completion / await signal) leaks into the agent's value in place of the sent arg `1`. Deterministic-green in isolation (6/6) and standalone (`bash test/e2e/phase16_gc_torture.sh` ALL PASS), so it is a genuine concurrency race in the agent send-action path (src/runtime/agent.zig + send/await), NOT a deftype/remap issue — it is independent of the D-417 change that exposed it (D-417 is macro-expansion-time only; this is runtime). Same load-sensitivity class as memory `gate_parallel_e2e_timeout` / `tool_channel_corrupts_under_load`. Finished form: the send action's arg/value handoff must not race with the await/promise signaling — trace where a `.promise` Value reaches the agent state slot. recall trigger: reproduce by running the full gate under load (or `e2e_phase16_gc_torture` concurrently with other e2e); fix the agent send/await value handoff."
1009-
category: "concurrency"
1010-
barrier: "blocked-by: a reliable reproducer. 2026-06-13 attempts: 200 iterations in-process = 0 mismatches; 3000 iterations under 6x `yes` CPU load = 0 mismatches. So the race needs the gate's SPECIFIC contention (≈16 parallel cljw processes + the e2e harness preempting the agent worker vs main thread vs a cross-thread GC collect), not mere in-process concurrency or coarse CPU load. LEAD for the next attempt: agent.zig docstring flags the `send` FABRICATION WINDOW (the `[f & args]` action vector is a Zig local, not yet GC-rooted, before the queue lock — GC-ROOT note ~:35) — a cross-thread collect during that window is the prime suspect for how `await`'s completion Promise (a heap value) displaces the first action's arg. NOTE: the leaked arg `1` is an IMMEDIATE (not heap), so the corruption is at the action-VECTOR level (the vector's slot/memory), not the int — re-read fabrication-window rooting + the await barrier-action injection order. Fix only with a reproducer in hand (a rare race fixed blind cannot be verified)."
1011-
last_reviewed: "2026-06-13"
10121026
- id: "D-435"
10131027
status: "open (2026-06-14, user-flagged via D-434). The dual-backend differential oracle (ADR-0005; src/lang/diff_test.zig's Fixture) is the project's correctness oracle for VM↔TreeWalk parity, BUT the Fixture is a MINIMAL runtime: it does `primitive.registerAll` + macro registration only, NOT the core.clj bootstrap. So `*out*`, `print`, `with-out-str`, and the whole clojure.core .clj surface are absent — any FULL-RUNTIME form cannot be dual-backend-diff-tested there. Today such behavior is verified single-backend (clj_corpus via `cljw -e`, which runs ONE backend) + structural arguments ('both backends call the same shared helper'). That is a real oracle-coverage HOLE: full-runtime dual-backend parity is UNVERIFIED except by hand-argument. Surfaced when D-434 (*out* writer-interop) could not get a diff case — the workaround (remove the diff test, lean on clj_corpus) is the bad smell the user caught. FINISHED FORM: a diff Fixture that bootstraps core.clj once (shared/cached so per-test cost stays low) OR a separate full-runtime dual-backend harness (run a form through both backends with the real runtime + compare), so ANY form — not just primitives — is parity-checked. Part of the D-436 大整理 epic."
10141028
category: "testing"
@@ -1056,6 +1070,16 @@ active:
10561070
passed), the OPEN D-418/D-258 send/await race surfacing at 3 vCPUs. All three
10571071
sites are ncpu>=4-gated in the scripts (SKIP cites this row) so the nightly
10581072
stays green; results-correctness cases still run everywhere.
1073+
UPDATE 2026-07-13: exposure (c) — the gc_torture agent_conj corruption — is
1074+
DISCHARGED as D-418 (the send/await FABRICATION-WINDOW race: an unrooted action
1075+
vector / completion promise across the enqueue; root-caused + deterministically
1076+
reproduced via the alloc-torture window-collect injection + fixed by
1077+
EvalFrame-rooting in primitive/agent.zig). The gc_torture agent block's ncpu>=4
1078+
gate is REMOVED — it runs everywhere now, incl. the 3-vCPU macOS runner. The
1079+
OTHER two exposures stay OPEN here: (a) e2e_phase14_future_promise_delay SIGABRT
1080+
+ (b) e2e_phase14_parallel_seq pmap wall-clock assert — both still ncpu>=4-gated
1081+
pending the delay-realise / future-spawn trace. A nested-cross-agent-send variant
1082+
found while fixing D-418 is D-559.
10591083
category: "runtime"
10601084
barrier: |-
10611085
needs a low-core repro environment (3-vCPU VM or cpuset-limited linux host) +
@@ -2298,6 +2322,11 @@ standing:
22982322
barrier: "open after the D-440 arc completes (R4 + R5 must first rewrite the bulk of 'Phase N' citations to gap-area terms). Premature adoption = ~70 dangling citations with no 1:1 redirect. Trigger: D-440 R4/R5 done + a user nod to re-architect the planning axis (it is depth-4, the project's organizing metaphor)."
22992323
last_reviewed: "2026-06-15"
23002324
discharged:
2325+
- id: "D-418"
2326+
status: "DISCHARGED 2026-07-13 (user-directed — 'race だから失敗して良い性質ではない'). The send/await FABRICATION-WINDOW race — `agent_conj` intermittently returned `[2 nil]` / `[#<promise> 2]` / `[#<fn> [#<promise>]]` (the leaked object varied) under low-core / parallel-gate scheduling. ROOT CAUSE (confirmed by the Agent.java survey, private/notes/D418-agent-race-survey.md): on the JVM a constructed Action is automatically GC-reachable, so there is NO enqueue window; cljw's manual mark-sweep had one — `sendFn` built the `[f & args]` action vector (and `awaitFn` the completion promise) as a Zig LOCAL that was unrooted until `enqueueDirect` appended it to the off-heap queue (its traceGc root). A collect in that window swept the freshly-built vector, so the drainer read recycled memory. FIX (primitive/agent.zig): sendFn + awaitFn now publish the action / promise on an EvalFrame across the whole enqueue [.dev/gc_rooting.md §C]. REPRODUCER (the 'def→cgc→use' fault-injection idiom, USENIX Sec'21 use-after-compacting-GC): agent.zig `tortureCollectInWindow` injects a STW collect into the exact enqueue window under CLJW_GC_TORTURE_ALLOC — so the cross-thread race is now a SINGLE-THREAD DETERMINISTIC UAF (pre-fix `agent_conj` HUNG on a swept promise, exit 124; post-fix 10/10 [1 2]). Mechanised guard: e2e phase16_gc_torture.sh agent block — the ncpu>=4 gate is REMOVED (workaround retired), + a new deterministic alloc-torture agent block (send/conj/drain/sendoff). The nested-CROSS-agent-send case surfaced a SECOND, distinct corruption (`@memcpy arguments alias`, exit 134) → new row D-559. Serialization itself (single-drainer + `draining` flag) was NEVER the bug — the survey confirms it is the correct design for a manual-GC runtime."
2327+
category: "concurrency"
2328+
barrier: "none — DISCHARGED. Deterministically reproduced + fixed + guarded."
2329+
last_reviewed: "2026-07-13"
23012330
- id: "D-545"
23022331
status: "DISCHARGED 2026-07-01 (user-directed audit). Exhaustive audit + cleanup of stale `ClojureWasmFromScratch` (working-dir name) + `cw-from-scratch` (merged branch) references ahead of the user's planned dir move (delete this dir; checkout `main` into `~/Documents/MyProducts/ClojureWasm`). git-grep audit of the tracked tree + the memory dir. FIXED (active scaffolding): CLAUDE.md Identity/Context rewritten (working dir = ClojureWasm, branch = main, no more 'keeps from-scratch name' narrative); ROADMAP §13 push-rules (cw-from-scratch→main; the stale '❌ push to main' inversion removed) + the working-dir diagram note + reference table; scripts/run_remote_ubuntu.sh REMOTE_DIR default → ClojureWasm; .dev/{reference_clones,ubuntunote_setup,orbstack_setup,structure_plan,v0_v1_feature_parity,decisions/0049}; docs/works/ladder.md; .claude/{rules/no_copy_from_v1,rules/textbook_survey,skills/audit_scaffolding/CHECKS,skills/code_learning_doc/TEMPLATE_*} (editable now that auto-mode is on). v0-reference reframed per user: cw v0 = git tag `v0.5.0` reached via `git worktree add ../cw-v0 v0.5.0` (NOT a live `~/Documents/MyProducts/ClojureWasm/` path — that becomes the working dir; the `-s ours` merge kept the redesign tree). D-538's residual (personal additionalDirectories → gitignored settings.local.json) done in the same pass. PRESERVED as history (user decision): docs/ja/archive/** + .dev/archive/** (frozen relics, ADR-0025), .dev/debt.yaml discharged-row history, project_facts.md verbatim user quotes, decisions/0006 narrative, docs/ja/README archive-era label, and `ClojureWasmFromScratch_v1_ref` (a separate valid clone). `zwasm_from_scratch` untouched (separate live repo). Descriptive 'from-scratch runtime' (clojure_vs/RELEASE_METRICS) kept (= 'built from scratch', not the dir name). USER ACTION FLAGGED: the harness memory dir is path-keyed (`-Users-…-ClojureWasmFromScratch`); after the move it must be relocated to `-Users-…-ClojureWasm` or the project memories are lost (6 memory files carry historical `cw-from-scratch` context, left as-is)."
23032332
category: "docs"
@@ -6365,19 +6394,16 @@ discharged:
63656394
category: "runtime"
63666395
barrier: "DISCHARGED (was: two-lowering drift risk — now corpus-netted)."
63676396
last_reviewed: "2026-07-07"
6368-
63696397
- id: "D-241"
63706398
status: "DISCHARGED 2026-07-07. The last coverage gap closed: installBaselineBindings now thread-binds 12 standard dynamic vars — the original 10 plus *assert* and *math-context* (both had dynamic roots but were missing from the baseline frame, so top-level `(set! *assert* false)` raised where the JVM REPL accepts it; clojure.main's with-bindings set binds both). Verified live ([false 7]) + corpus thread_bindings +2 clj-matched goldens (set! and binding forms). HISTORY: the ADR-0096 substance (baseline frame + bound-fn capture/replay both ways) landed 2026-06-05; *assert*/*math-context* roots landed by 2026-07-02; this cycle completed the frame membership. The remaining clj baseline names (*compile-path*, *command-line-args*, *1/*2/*3/*e) are features cljw does not carry as settable state — out of scope, not residual."
63716399
category: "clj-parity"
63726400
barrier: "DISCHARGED (was: baseline-frame coverage)."
63736401
last_reviewed: "2026-07-07"
6374-
63756402
- id: "D-460"
63766403
status: "DISCHARGED 2026-07-09 — sorted collections now work as hash-map KEYS / hash-set ELEMENTS, value-compared across ALL impls (was: `(contains? #{(sorted-set 1 2)} (sorted-set 2 1))` => cljw false / clj true; the rt-free keyEqValue path routed only array/hash impls to contentEq and valueHash had NO sorted arms, so a sorted coll missed the bucket AND the in-bucket compare). Fix took barrier option (a) but WITHOUT the anticipated default-comparator restriction: the walk compares STRUCTURALLY (equal.keyEqValue over an in-order rb-tree walk; the comparator is never invoked), so custom-comparator sorted colls work as keys too. Landed: sorted.contentHashMap/contentHashSet (same fold formulas as map.contentHash/set.contentHash so equal colls share a bucket), sorted.getByValue/setContainsByValue (rt-free O(n) linear walk — rare path), sorted.forEachElem + set.forEachElem walkers; equal.keyEqValue map/set arms widened to isAnyMapTag/isAnySetTag (hash-backed pairs keep the fast HAMT path, any sorted participant takes anyMapKeyEq/anySetKeyEq), valueHash gained .sorted_map/.sorted_set content arms. Mechanised per the barrier: corpus sorted_as_key (15 clj-matched goldens: cross-impl contains?/get both directions, sorted-set-by/sorted-map-by custom comparators, hash parity, into-#{} dedup across impls, nested sorted-in-sorted vs plain, negative cases)."
63776404
category: "clj-parity"
63786405
barrier: "DISCHARGED (was: rt-free constraint on keyEqValue — resolved by structural walk, no rt needed)."
63796406
last_reviewed: "2026-07-09"
6380-
63816407
conventions: |-
63826408
- Structure: two top-level lists — `active:` and `discharged:`. Active entries carry id / status / category / barrier (+ optional quality_floor / last_reviewed); discharged entries carry id / discharged_at / resolution. Edit as YAML (no Markdown table to align).
63836409
- New debt: grep `.dev/debt.yaml` for class-overlap before adding (per `.claude/rules/debt_dedup.md`).

0 commit comments

Comments
 (0)