Skip to content

Commit 5faffa5

Browse files
BuddhiLWBuddhiLW
committed
test(suites): convert the collection-key and chunked-seq surfaces
Smell-audited: 1: routine ports; one ledger repoint. phase14_collection_keys.sh 20 spawns -> suites/collection_keys_test.clj phase14_chunked_seq.sh 19 spawns -> suites/chunked_seq_test.clj .dev/optimizations.md cited `phase14_chunked_seq.sh` as verification for the chunk-preserving work; it now names the pin set and where it lives. The chunked suite's header states WHY its counts are 1 / 32 / 33 / 65 rather than round numbers: the chunk size is 32, so those straddle the boundary (partial, exact, one over, two-plus-one). The property is that chunking changes only how often the lazy machinery runs, never what is produced — and that it does not over-realise, which is what the take-5-from-a-million and take-5-from-infinite cases defend. 39 more spawns leave the gate; 781 for the arc across 24 scripts. Suite tier 29 suites / 229 tests / 1246 assertions / 0 failures. e2e 389 scripts (from 410). Gates: e2e_reach, runner_reach 68 units, e2e_dup. Smoke green 12/0.
1 parent 600090c commit 5faffa5

6 files changed

Lines changed: 101 additions & 107 deletions

File tree

‎.dev/optimizations.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
| O-001 | `runtime/collection/range.zig` + call sites | `(range a b s)` as a lazy cons-seq (one cons + lazy_seq per element) | Compact `.range` value `{start,end,step,count}`: O(1) count/nth, tight-loop reduce, chunked-cons `seq` | No per-element alloc on count/nth/reduce; 1 alloc/32 on walk | `phase14_range_indexed.sh` + diff oracle vs `clj` | D-163 / D-168 |
4747
| O-002 | `higher_order.zig::reduceFn` (`.vector` arm) | `reduce` over a vector via `seqFn` → `vectorToList` (N-element eager cons list), then walk via first/next | Index-walk: `vector.nth(coll, i)` in a tight `i` loop, honouring `reduced` | No N-element intermediate cons list; `(reduce f bigvec)` / `(into to bigvec)` went O(n) alloc → O(1). Measured `(reduce + (vec (range 1e6)))` 182s → fast | `phase14_*` reduce e2e + diff oracle vs `clj` | D-163 |
4848
| O-003 | `vector.zig::fromSlice` + `transient/transient_vector.zig::toPersistent` + `core.clj` `into`/`vec` | `persistent!` rebuilds the persistent vector via N persistent `conj`s (O(n log n)); `into`/`vec` = `(reduce conj …)`, also N persistent conjs | Bulk `fromSlice` builds the HAMT trie bottom-up from the transient's flat buffer in O(n) (32-element leaves → interiors grouped 32-at-a-time → root; last ≤32 = tail); `into`/`vec` route editable targets (vector/hash-map/hash-set, NOT sorted/nil/list) through `transient`/`conj!`/`persistent!` | `persistent!` O(n log n) → O(n); `into`/`vec` build O(n) over a flat buffer + one O(n) trie conversion, vs N persistent conjs. Measured `(count (vec (range 1e6)))` 121s → 2.4s; `(reduce + (vec (range 1e6)))` 123s → 2.5s | `vector.zig` boundary unit test (n ∈ {0,1,31,32,33,63,64,65,1023,1024,1025,1e5}: `fromSlice` == conj-built, same shift/tail/root) + diff oracle vs `clj` (into/vec over vector/map/set/sorted/nil/list + meta) | D-180 |
49-
| O-004 | `core.clj` `map`/`filter`/`keep` 2-arg + `higher_order.zig::reduceFn` chunked arm + `sequence.zig::countFn` chunk drain + `chunked_cons.zig` chunk-builder | `(map f coll)` / `(filter pred coll)` build a meta-less lazy-seq walked one element per `nextFn` — each step allocs a cons + lazy_seq node and tree-walks the `.clj` body (~408µs/element measured) | Chunk-preserving: when the source is chunked (range seq / chunked map), transform a whole 32-chunk per thunk into a fresh chunk (JVM `chunk-cons` shape); `reduce`/`count` drain a chunk per step. Fill loop stays in `.clj` (a tree-walk loop is ~2µs/iter, negligible vs the 408µs amortised) | The ~408µs/element lazy-seq machinery is paid once per 32 elements, not per element. Measured `(count (map inc (range 1e5)))` 41.3s → 2.8s (~15x); `(reduce + (map inc (range 1e5)))` → 2.4s. (Residual is the per-element `f` vtable call ≈ 2µs — D-133's, not this cycle's.) | `phase14_chunked_seq.sh` (chunk-boundary count 1/32/33/65/1000 + reduce/nth/last/=/lazy-take) + diff oracle vs `clj` | D-163 / ADR-0065 |
49+
| O-004 | `core.clj` `map`/`filter`/`keep` 2-arg + `higher_order.zig::reduceFn` chunked arm + `sequence.zig::countFn` chunk drain + `chunked_cons.zig` chunk-builder | `(map f coll)` / `(filter pred coll)` build a meta-less lazy-seq walked one element per `nextFn` — each step allocs a cons + lazy_seq node and tree-walks the `.clj` body (~408µs/element measured) | Chunk-preserving: when the source is chunked (range seq / chunked map), transform a whole 32-chunk per thunk into a fresh chunk (JVM `chunk-cons` shape); `reduce`/`count` drain a chunk per step. Fill loop stays in `.clj` (a tree-walk loop is ~2µs/iter, negligible vs the 408µs amortised) | The ~408µs/element lazy-seq machinery is paid once per 32 elements, not per element. Measured `(count (map inc (range 1e5)))` 41.3s → 2.8s (~15x); `(reduce + (map inc (range 1e5)))` → 2.4s. (Residual is the per-element `f` vtable call ≈ 2µs — D-133's, not this cycle's.) | the chunk-boundary pin set, landed as `phase14_chunked_seq.sh` and since migrated to `test/clj/suites/chunked_seq_test.clj` (count 1/32/33/65/1000 + reduce/nth/last/=/lazy-take) + diff oracle vs `clj` | D-163 / ADR-0065 |
5050
| O-005 **RETIRED 2026-06-11** | `eval/analyzer/{analyzer,bindings}.zig` + `eval/backend/tree_walk.zig::callMethodImpl` | Every TreeWalk fn call nil-inits the full `[MAX_LOCALS=256]Value` (~2 KB) call frame | (reverted) leave `[frame_size..256)` `undefined`, nil-init only the analyser frame high-water | **RETIRED — introduced a GC UAF.** `callMethodImpl`'s `locals` is ALSO the VM's frame: on the default VM backend it is handed to `vm.eval`, which publishes the WHOLE slice as a GC root (`gc_frame.locals`). The `undefined` tail was traced as Values under a `CLJW_GC_TORTURE` collect → SIGSEGV (deterministic repro: a deep call fills high stack slots, then a shallow fn runs under torture). The bound-the-slice fix then surfaced that `frame_high_water` *under-counts* the VM's slot needs for a fn containing a nested `fn*` (`slot_out_of_range`). 12 % on one microbench was not worth the correctness risk + the slot-accounting subtlety. The full-256 nil-init is restored. | repro `deep`+`shallow` under `CLJW_GC_TORTURE=1` (139→0 after revert) + full gate `e2e_phase16_gc_torture` | D-163 |
5151
| O-007 | `lang/primitive/higher_order.zig::sortNaturalFn` (`-sort-natural` leaf) + `core.clj` `sort` | `(sort coll)` ran the `.clj` `-msort` merge sort: per recursion level `(vec (take mid v))` + `(vec (drop mid v))`, and `-merge-sorted` does `(first a)`/`(rest a)`/`(conj acc …)`/`(empty? …)` + a `compare` call through the eval machinery per element | Default order copies the vector into a flat `[]Value` buffer, runs `std.mem.sort` (stable block sort) calling `valueCompare` directly (no eval reentry → no GC safepoint mid-sort → no frame rooting needed), and rebuilds via `vector.fromSlice` (O-003). Custom-comparator `(sort comp coll)` / `sort-by` stay on `.clj` `-msort` (a user comparator re-enters eval) | Eliminates the O(n log n) `.clj` take/drop/vec/rest/conj churn + per-comparison eval reentry. Measured `36_sort` bench (5×`(reduce + (take 100 (sort (vec (range 5000 0 -1)))))`) 0.39s → ~0.00s (startup-only) ReleaseFast | `test/diff/clj_corpus/sort.txt` (17 cases vs `clj`: empty/single/dups/mixed int·float ties/strings/keywords/nested vectors/custom comp/sort-by stability) + `zig build test` + `zig build lint` | D-163 |
5252
| O-008 | `build.zig` (`exe_mod.strip`) — **binary-size axis** | `b.installArtifact(exe)` shipped an UNSTRIPPED binary (the ~5.7K-symbol table rode in every release `cljw`); there was no packaging step to strip it, so the `bench/RELEASE_METRICS.md` "stripped" headline was aspirational, not the actual install | `.strip = optimize != .Debug` — every non-Debug build (ReleaseSafe / ReleaseFast / ReleaseSmall) strips at link time; Debug stays unstripped for `lldb`. cljw renders error traces from its own runtime `StackFrame` stack, not native symbols, so stripping costs no user diagnostics | Installed ReleaseSafe `cljw` 3.93 MB → **3.39 MB** (~400 KB / ~10% off the shipped artifact); ReleaseSmall also stripped (a further CLI `strip` floors it at 1.63 MB — Zig link-strip is less aggressive than CLI `strip` only for the ReleaseSmall layout) | `bash bench/release_metrics.sh` (3.24 MB on-disk ReleaseSafe) + smoke (stripped ReleaseSafe e2e: corpus 2289/2289, build_cljw pass) + `cljw -e '(sort [3 1 2])'` on the stripped binary | — |

‎test/clj/suites/chunked_seq_test.clj‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
;; Chunk-preserving map / filter / keep + chunked reduce / count
2+
;; (§9.2.S D-163 / ADR-0065).
3+
;;
4+
;; map/filter/keep over a chunked source (a range seq) emit a `chunked_cons`
5+
;; in the JVM chunk-cons shape, so the per-element lazy-seq machinery is
6+
;; amortised 32x; reduce and count drain a whole chunk per step.
7+
;;
8+
;; The chunk size is 32, so the counts below straddle the boundary on purpose:
9+
;; 1 (partial), 32 (exact), 33 (one over), 65 (two chunks plus one).
10+
;; Chunking must never change WHAT is produced, only how often the lazy
11+
;; machinery runs, and it must not over-realise a lazy source.
12+
;;
13+
;; Migrated from test/e2e/phase14_chunked_seq.sh; each assertion keeps its
14+
;; bash case name.
15+
(ns suites.chunked-seq-test
16+
(:require [clojure.test :refer [deftest is testing]]))
17+
18+
(deftest counts-straddle-the-chunk-boundary
19+
(is (= 1 (count (map inc (range 1))))) ; cnt_map_1
20+
(is (= 32 (count (map inc (range 32))))) ; cnt_map_32
21+
(is (= 33 (count (map inc (range 33))))) ; cnt_map_33
22+
(is (= 65 (count (map inc (range 65))))) ; cnt_map_65
23+
(is (= 1000 (count (map inc (range 1000)))))) ; cnt_map_1000
24+
25+
(deftest filter-keep-and-remove-chunk-too
26+
(is (= 500 (count (filter even? (range 1000))))) ; cnt_filt_1000
27+
(is (= 32 (count (keep (fn* [x] (if (even? x) x nil)) (range 64))))) ; cnt_keep
28+
(is (= 32 (count (remove even? (range 64)))))) ; cnt_remove
29+
30+
(deftest reduce-drains-a-chunk-per-step
31+
(is (= 5050 (reduce + (map inc (range 100))))) ; red_map
32+
(is (= 2500 (reduce + 0 (map inc (filter even? (range 100))))))) ; red_nested
33+
34+
(deftest indexed-access-and-equality-are-unchanged
35+
(is (= 51 (nth (map inc (range 100)) 50))) ; nth_map
36+
(is (= 100 (last (map inc (range 100))))) ; last_map
37+
(is (true? (= (map inc (range 40)) (range 1 41))))) ; eq_map_range
38+
39+
;; chunking must not over-realise: taking 5 from a million-element source, or
40+
;; from an INFINITE one, must still terminate
41+
(deftest chunking-does-not-defeat-laziness
42+
(is (= [1 2 3 4 5] (into [] (take 5 (map inc (range 1000000)))))) ; lazy_map_take
43+
(is (= [1 3 5 7 9] (into [] (take 5 (filter odd? (range)))))) ; lazy_filt_take
44+
(is (= 33 (count (into [] (map inc (range 33))))))) ; into_map_33
45+
46+
;; a chunked_cons still PRINTS as an ordinary seq
47+
(deftest a-chunked-cons-prints-as-a-seq
48+
(is (= "(0 1 2)" (str (seq (range 3))))) ; print_chunked_cons
49+
(is (= "(1 2 3 4)" (str (rest (range 5))))) ; print_cc_rest
50+
(is (= "[(0 1 2)]" (pr-str [(seq (range 3))])))) ; print_cc_nested

0 commit comments

Comments
 (0)