Skip to content

Commit a7d23c2

Browse files
committed
feat(meta): D-305 — builtin :arglists/:doc on clojure.core vars (cider eldoc complete)
Smell-audited: 1: the generated file uses (resolve 'sym) not (var sym) — a Var VALUE in the literal pool is not bytecode-serializable — and the fill fn preserves existing keys so a defn-attached :arglists is never clobbered; docstring text is reproduced upstream text under the EPL variant-1 header. scripts/extract_core_meta.sh extracts :arglists/:doc from the JVM oracle, filters to cljw-interned vars, and generates clojure/core_meta.clj (241 vars) as alter-meta! fill forms; wired as an eager bundled ns. Landing it exposed D-558 (previous commit). e2e phase14_metadata +3 — (:arglists (meta #'map)) byte-matches clj's 5 arities. D-305 + D-558 discharged.
1 parent cf6aad1 commit a7d23c2

5 files changed

Lines changed: 348 additions & 19 deletions

File tree

‎.dev/debt.yaml‎

Lines changed: 12 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,6 @@
2424
# md-table-align no longer pads it to ~800 KB of whitespace.
2525

2626
active:
27-
- id: "D-558"
28-
status: "open (2026-07-07, found while landing D-305's core_meta.clj). ADDING A NEW ~60KB BUNDLED .clj (243 forms: ns + def + 241 alter-meta! forms) BREAKS THE AOT BOOTSTRAP with a memory-corruption signature. Repro: generate clojure/core_meta.clj (scripts preserved in the session scratchpad: extract_core_meta.sh + core_meta_gen_final.clj), register in bootstrap.zig FILES + EAGER_NS, build -> startup fails `<bootstrap>: error: UnknownValueTag`. INVESTIGATION LOG (all verified): (1) the cache_gen BLOB IS CORRECT — hand-decoded: region 'clojure.core-meta' chunk1 = [op_ns_with_refer_clojure, op_ret] with the name string constant, chunk2 = def fill, chunk3 = the first alter-meta!; (2) replay is sequential (chunk1 deser+eval, chunk2, then FAILS at chunk3's constants read: `fill` var_ref -> env.findNs('clojure.core-meta') == null); (3) the ns REGISTRY at that moment holds a GARBLED key (ReleaseSafe: 17 bytes of garbage where 'clojure.core-meta' should be; Debug build: the key is ABSENT entirely) — findOrCreateNs DUPES its key, so the GC String holding the ns name was ALREADY corrupted when chunk1's op_ns_with_refer_clojure eval'd it; (4) GC-INDEPENDENT: fails identically with CLJW_GC_THRESHOLD_MB=100000 (threshold read at GcHeap.init, before bootstrap — verified); Debug build raises NO DebugAllocator UAF -> the corruption is INSIDE the GC pool (infra allocator clean); (5) SIZE-DEPENDENT: 225 alter-meta! rows OK, 230 NG, every SUBRANGE (0-120/120-180/180-210/210-225/225-241) individually OK -> cumulative-volume trigger, not a specific form. HYPOTHESES REMAINING: string interning table rehash/free with shared buffers; free-pool reuse without collect; a deserialize-side buffer aliasing the region blob. NEXT: instrument string_collection.alloc for 'clojure.core-meta'-content writes; check the intern table's ownership on rehash; try the same 243-form payload as a LAZY (non-eager) region."
29-
category: "runtime"
30-
barrier: "correctness floor — PREEMPTS (memory corruption in the AOT bootstrap path; currently latent because no bundled file is that large, but any future bundled-lib growth trips it)."
31-
last_reviewed: "2026-07-07"
3227
- id: "D-547"
3328
status: |-
3429
open (2026-07-01, NARROWED same day). Enforcement-honesty sweep — surfaced by the
@@ -1029,16 +1024,6 @@ active:
10291024
- small, lets the ns load. ADR-level (which path). Cross-ref ladder
10301025
rung 6, D-281/ADR-0103 (host_inert supertype precedent for inert-accept).
10311026
last_reviewed: "2026-07-02"
1032-
- id: "D-305"
1033-
status: "BLOCKED by D-558 (2026-07-07): the implementation is DONE and preserved (scratchpad extract_core_meta.sh generates clojure/core_meta.clj — 241 vars' :arglists/:doc from the JVM oracle, resolve-based alter-meta! forms, EPL variant-1 header; bootstrap wiring = FILES + EAGER_NS entries), but registering the ~60KB file trips D-558's AOT-bootstrap memory corruption. Land after D-558. now (2026-06-07) — Stage 1.4 var-metadata Slice 3. Slice 1 LANDED (mechanical keys :name/:ns/:macro/:dynamic/:private synthesized on read in metaFn's .var_ref arm; :ns opaque = AD-021). Slice 3 = builtin :arglists/:doc: Zig-builtin primitives (map/filter/reduce/…) + plain-`def` core vars have NO :arglists/:doc on `(meta #'x)` (clj has both). defn-defined vars already get :arglists (the defn macro attaches it). Unblocks full cider eldoc/info on builtins (campaign Stage 1.4)."
1034-
category: "runtime"
1035-
barrier: |-
1036-
`(:arglists (meta #'map))` / `(:doc (meta #'map))` → nil for Zig-builtin
1037-
primitives + plain-`def` core vars (clj has both). Barrier dissolves when a
1038-
definition-derived arglists/doc SSOT (F-013) — a build-time/comptime table
1039-
extracted from JVM core.clj, NOT compat_tiers.yaml (no arglists/doc fields)
1040-
and NOT inline-per-builtin (F-013 ad-hoc smell) — is projected by
1041-
synthVarMeta. Cross-ref private/notes/stage14-var-metadata-survey.md, F-013.
10421027
- id: "D-339"
10431028
status: "open — security (SE-3, HTTP server slowloris). The cljw.http.server accept loop is serial + blocking with NO socket read/idle/header deadline (server.zig:63-71); one client that opens a connection and never completes the request head wedges the WHOLE server (binds 0.0.0.0, internet-facing in a container). The 8 MB body cap guards volume, not time. High severity (one packet DoSes a public deploy)."
10441029
category: "security"
@@ -6474,6 +6459,18 @@ discharged:
64746459
category: "clj-parity"
64756460
barrier: "DISCHARGED (was: format %t family unimplemented)."
64766461
last_reviewed: "2026-07-07"
6462+
- id: "D-558"
6463+
status: "DISCHARGED 2026-07-07 (same session as filed — root cause found via targeted probes). The corruption was NOT at runtime: `[str] alloc` + `[op_ns]` probes showed the garbled ns name was READ FROM THE BLOB itself — cache_gen had baked swept bytes into bootstrap_core.cljc. ROOT CAUSE = the D-556 class in the BUILD tool: builder.zig's AOT loops used the DROP bracket (endAnalysis) while the compiled chunks' constants are serialized AFTER the loop ends — a build-time collect past the 4MB threshold swept the constant Strings and serializeChunk wrote the garbage (size-dependent: ~225 forms was where cache_gen's heap crossed the threshold; every SUBRANGE passed because smaller inputs never collected). My earlier hand-decode 'verified' a STALE blob from a prior partial build — lesson: verify the artifact the failing binary actually embeds. FIX: builder.zig's four production seams (buildEnvelope, buildBootstrapEnvelope, buildMainEnvelope's require closure, tryRunEmbedded's -e tail) now endAnalysisPersist, completing the D-556 seam sweep (driver/loader/repl/nrepl/evaluator landed then; builder was missed). VERIFIED: the 62KB core_meta.clj (the trigger) loads clean; (meta #'map) full."
6464+
category: "runtime"
6465+
barrier: "DISCHARGED (was: bundled-file-size AOT corruption)."
6466+
last_reviewed: "2026-07-07"
6467+
6468+
- id: "D-305"
6469+
status: "DISCHARGED 2026-07-07 (Stage 1.4 var-metadata Slice 3 — the campaign's last slice). Builtin + plain-def clojure.core vars now carry :arglists/:doc: scripts/extract_core_meta.sh extracts both from the JVM oracle (ns-publics 'clojure.core), filters to the vars cljw interns, and generates src/lang/clj/clojure/core_meta.clj (241 vars, ~62KB, EPL variant-1 header — docstring text is reproduced upstream text) as alter-meta! fill forms ((resolve 'sym) not (var sym) — a Var VALUE in the literal pool is not bytecode-serializable; existing keys win, so a defn-attached :arglists is never clobbered). Wired as an EAGER bundled ns (bootstrap.zig FILES + EAGER_NS). Landing it exposed + fixed D-558 (the build-tool persist gap). e2e phase14_metadata +3 (core_arglists byte-matches clj's 5 map arities / core_doc / defn_meta_wins). Unblocks full cider eldoc/info on builtins. Slice 2 (:ns opaque rendering) = AD-021, unchanged."
6470+
category: "clj-parity"
6471+
barrier: "DISCHARGED (was: blocked by D-558; both landed together)."
6472+
last_reviewed: "2026-07-07"
6473+
64776474
conventions: |-
64786475
- Structure: two top-level lists — `active:` and `discharged:`. Active entries carry id / status / category / barrier (+ optional quality_floor / last_reviewed); discharged entries carry id / discharged_at / resolution. Edit as YAML (no Markdown table to align).
64796476
- New debt: grep `.dev/debt.yaml` for class-overlap before adding (per `.claude/rules/debt_dedup.md`).

‎scripts/extract_core_meta.sh‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
#!/usr/bin/env bash
2+
# scripts/extract_core_meta.sh — regenerate src/lang/clj/clojure/core_meta.clj
3+
# (D-305): :arglists/:doc for clojure.core vars, extracted from the JVM oracle
4+
# and filtered to the vars cljw actually interns. Run on demand after the core
5+
# surface grows; needs `clj` + a built zig-out/bin/cljw.
6+
set -euo pipefail
7+
cd "$(dirname "$0")/.."
8+
TMP=$(mktemp -d)
9+
trap 'rm -rf "$TMP"' EXIT
10+
11+
# 1. JVM oracle: every public clojure.core var's [sym arglists doc].
12+
timeout 60 clj -J-Xmx2g -M -e '
13+
(doseq [[sym v] (sort-by key (ns-publics (quote clojure.core)))]
14+
(let [m (meta v)]
15+
(when (or (:arglists m) (:doc m))
16+
(prn [sym (:arglists m) (:doc m)]))))' > "$TMP/raw.edn"
17+
18+
# 2. cljw truth: the interned public core vars.
19+
./zig-out/bin/cljw -e '(doseq [[s _] (ns-publics (quote clojure.core))] (println (name s)))' > "$TMP/cljw_vars.txt"
20+
21+
# 3. Intersect + emit.
22+
timeout 60 clj -J-Xmx2g -M -e "
23+
(require '[clojure.edn :as edn] '[clojure.java.io :as io] '[clojure.string :as str])
24+
(let [cljw-vars (-> (slurp \"$TMP/cljw_vars.txt\") str/split-lines set)
25+
rows (->> (line-seq (io/reader \"$TMP/raw.edn\"))
26+
(map edn/read-string)
27+
(filter (fn [[s _ _]] (contains? cljw-vars (name s)))))]
28+
(println \";; SPDX-License-Identifier: EPL-2.0\")
29+
(println \";;\")
30+
(println \";; Copyright (c) Rich Hickey. All rights reserved.\")
31+
(println \";; The use and distribution terms for this software are covered by the\")
32+
(println \";; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php).\")
33+
(println \";; By using this software in any fashion, you are agreeing to be bound by the\")
34+
(println \";; terms of this license. You must not remove this notice, or any other, from\")
35+
(println \";; this software.\")
36+
(println \";;\")
37+
(println \";; core_meta.clj — :arglists/:doc for clojure.core vars; docstring text\")
38+
(println \";; reproduced verbatim from Clojure's core sources. Reproduced in\")
39+
(println \";; ClojureWasm; redistributed under EPL-2.0 per EPL-1.0 §7. ClojureWasm\")
40+
(println \";; changes (c) the ClojureWasm authors.\")
41+
(println \";; GENERATED by scripts/extract_core_meta.sh — do not hand-edit.\")
42+
(println)
43+
(println \"(ns clojure.core-meta)\")
44+
(println)
45+
(println \";; Existing keys win: a var's own meta (a defn-attached :arglists) is\")
46+
(println \";; preserved; only the missing keys are filled from the oracle table.\")
47+
(println \"(def ^:private fill (fn [m mm] (merge mm m)))\")
48+
(println)
49+
(doseq [[s args doc] rows]
50+
(let [mm (cond-> {}
51+
args (assoc :arglists (list 'quote args))
52+
doc (assoc :doc doc))]
53+
;; (resolve 'sym), NOT (var sym): a Var VALUE in the literal pool is
54+
;; not bytecode-serializable (AOT cache_gen), a quoted symbol is.
55+
(prn (list 'alter-meta! (list 'resolve (list 'quote (symbol \"clojure.core\" (name s)))) 'fill mm))))
56+
(println)
57+
(println \";; generated:\" (count rows) \"vars\"))" > src/lang/clj/clojure/core_meta.clj
58+
tail -1 src/lang/clj/clojure/core_meta.clj

‎src/lang/bootstrap.zig‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,9 +68,10 @@ pub const FileEntry = struct {
6868
/// either). `clojure.spec.alpha`'s own deps (spec.gen.alpha / core.specs.alpha) load
6969
/// transitively when its eager region's `(:require …)` runs. Run in FILES order.
7070
pub const EAGER_NS = std.StaticStringMap(void).initComptime(.{
71-
.{"clojure.core"}, .{"clojure.string"}, .{"clojure.walk"},
72-
.{"clojure.edn"}, .{"clojure.java.io"}, .{"clojure.core.protocols"},
73-
.{"clojure.uuid"}, .{"clojure.instant"}, .{"clojure.spec.alpha"},
71+
.{"clojure.core"}, .{"clojure.string"}, .{"clojure.walk"},
72+
.{"clojure.edn"}, .{"clojure.java.io"}, .{"clojure.core.protocols"},
73+
.{"clojure.uuid"}, .{"clojure.instant"}, .{"clojure.spec.alpha"},
74+
.{"clojure.core-meta"},
7475
// cljw-SPECIFIC (not in clj's auto-set): the Wasm-component `:require` desugar
7576
// (ADR-0135 am1) emits `(cljw.wasm/require-component-libspec …)` which is resolved
7677
// at ANALYZE time — before its own `(require 'cljw.wasm)` prelude EVALs — so the
@@ -153,6 +154,7 @@ pub const FILES: []const FileEntry = &.{
153154
// clojure.test's `report` multimethod, like clojure.test.tap). Loads after
154155
// clojure.test (FILES[10]). Verbatim upstream. Appended last.
155156
.{ .label = "<clojure.test.junit>", .source = @embedFile("clj/clojure/test/junit.clj") },
157+
.{ .label = "<clojure.core-meta>", .source = @embedFile("clj/clojure/core_meta.clj") },
156158
};
157159

158160
/// First file's source — exposed so `main.zig`'s renderer can fall

0 commit comments

Comments
 (0)