Skip to content

Commit 600090c

Browse files
committed
test(suites): convert the lazy-seq and list-count surfaces
Smell-audited: 1: routine ports; one ledger repoint. phase14_lazy_seq_cycle3.sh 22 spawns -> suites/lazy_seq_test.clj list_count_exact.sh 22 spawns -> suites/list_count_test.clj .dev/optimizations.md O-057 cited `e2e list_count_exact` as its verification; it now names the pin set and where it lives. Nothing enforces that path. Both suites keep the property their script was actually defending, stated in the header rather than left implicit: - lazy_seq: every case touching an infinite producer stays BOUNDED by take/first, exactly as the script had it — the point is that concat / mapcat / drop / (range) are lazy, so an unbounded realisation would hang rather than fail. - list_count: O-057 is only sound because COUNT_UNKNOWN marks the chains whose stored count would be a PREFIX length presented as a total. So the property is exactness on BOTH paths, not speed — a mixed chain must never report its pure prefix. 44 more spawns leave the gate; 742 for the arc across 22 scripts. Suite tier 27 suites / 217 tests / 1207 assertions / 0 failures. e2e 391 scripts (from 410). Gates: e2e_reach, runner_reach 68 units, e2e_dup, accepted_divergences 60 pinned.
1 parent 8a9e492 commit 600090c

6 files changed

Lines changed: 99 additions & 132 deletions

File tree

‎.dev/optimizations.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@
106106
| O-060 | `runtime/collection/string_seq.zig` (NEW, `.string_seq` A13 — a day-1 reservation with no producer) + `lang/primitive/sequence.zig` (`seqFn`/`countFn`/`firstFn`/`restFn`/`nextFn` + rev-walk `.string` arm → view) + `.string_seq` arms across `equal.zig` (isSequential via the interface_membership SSOT / isCountable / seqLen / valueHash / isSeqKeyValue) / `print.zig` (`printStringSeq`, ports-threaded) / `higher_order.zig` (fuse-base + chunkClassify) / `clojure_lang_method.zig` (native-collection arms) / `lazy_seq.zig` / `metadata.zig` / `meta.zig` / `runtime.zig` (GC trace registration) | `(seq s)` built an EAGER n-cell codepoint cons chain by asking `charset.codepointAt(s, i)` per element, and that restarts a `Utf8Iterator` from byte 0 every call — n calls at O(i) each = **O(n²)** overall. Measured cljw eager path: 2,000 chars 27 ms → 146,670 chars **89,910 ms**; downstream, bb-mcp paid ~79 s/boot asking `(seq value)` as a truthiness test on a 146,673-char nREPL response | A byte-offset VIEW satisfying the seq protocol: `StringSeq{backing, offset}`, two words. The cursor is a BYTE offset, not a codepoint index (an index would reintroduce the byte-0 walk): `first` decodes in place at `offset`, `rest` advances by the width it just read, `count` is a single forward byte scan (UTF-8 gives no length arithmetic — unlike the vector view's subtraction — but the scan allocates nothing and is no worse than `(count s)`). Consumers (`printStringSeq`, both `equal.zig` cursors) take the byte walk too rather than `nth(i)` (`nth` on UTF-8 is O(i) however the view is built, so the index-walk shape copied from array_seq would rebuild the quadratic inside the fix). `decodeAt` is total (malformed byte → width-1 raw), so a walk always advances — a view can never wedge or loop. Slot from the F-004 A13 day-1 reservation (already declared across interface_membership + class_name displayClassName), NO `heap_tag.unallocated` slot spent — the same position `.array_seq` (O-058) was in | **O(n²) → O(n).** NEW cljw ReleaseSafe end-to-end (incl. process startup + O(n) string build): `(count (seq (str-of-n "x")))` n=100k **0.13 s**, n=500k **0.60 s** (linear); the eager path's 89,910 ms at 146k chars is now sub-second (**>10,000× at that size**). clj JVM isolates the same seq+count to ~0 ms (StringSeq view + JIT); cljw's view is likewise O(1)/step, its residual wall time being the O(n) string construction, not the seq walk. Broad win: every hand-written `(seq s)` / `doseq`/`for` over a string and every `(seq value)` truthiness probe on a large string | `zig build -Dwasm -Doptimize=ReleaseSafe` (string-seq binary compiles) + corpus `string_seq.txt` (33 exprs vs clj, `33/33 reproduced`) + the 47-assertion observable-surface pin set, landed as e2e `string_seq_view.sh` and since migrated verbatim to `test/clj/suites/string_seq_view_test.clj` (vs clj: prints-as-seq, seq?/sequential?/coll?/not-string/not-vector, class=StringSeq, UTF-8 count/first/rest/nth/last, second/fnext/nthrest, `=`/hash with list AND vector + map-key + set-dedup + length-mismatch loses, vec/into/apply-str/reverse/take/drop/cons/concat, map/filter/reduce, empty→nil, scale count 20k + utf8 10k) | D-179, O-058, F-004, `feat/string-seq-view` |
107107
| O-059 | `runtime/collection/sub_vector.zig` (NEW, `.sub_vector` D11) + `lang/clj/clojure/core.clj` `subvec` (fast-path) + `lang/primitive/collection.zig` (`__subvec` prim + `conjOne`/`getFn`/`nthFn`/`assocFn`/`containsQFn`/`__vector-pop`/`isEntrySeqTag`) + `.sub_vector` arms across `interface_membership.zig` (16 arrays) / `sequence.zig` / `core.zig` / `equal.zig` / `print.zig` / `compare.zig` / `lookup.zig` + `tree_walk.zig` / `meta.zig` / `metadata.zig` / `class_name.zig` / `lazy_seq.zig` / `walk.zig` / `json.zig` / `higher_order.zig` / `intrinsic.zig` / `serialize.zig` / `clojure_lang_method.zig` / `multimethod.zig` / `transient_array_map.zig` / `array_seq.zig` (backing arm). `vector.subvec`→`copyRange` (materializing; LBQ compaction) | `clojure.core/subvec` on a vector was `(into [] (take (- end start) (drop start v)))` — an eager O(n) rebuild of a whole fresh trie per call, the LAST catastrophic collection gap (measured 7480x slower than clj). JVM never copies: `subvec` returns an `APersistentVector$SubVector` sharing the parent's root+tail (D-044/D-583). The Zig `vector.subvec` (eager) was reachable only from LBQ compaction, which genuinely wants a copy — so the view had to be a NEW representation, not a rewrite of it | A new REPRESENTATION satisfying the existing IPersistentVector protocol: `SubVector{start,end,parent,meta}`, 32 bytes fixed whatever the span. `count = end-start` (O(1)), `nth(i) = parent.nth(start+i)`. JVM `SubVector` semantics: FLATTEN on nest (parent is always a plain `.vector`), never-empty (empty range → EMPTY plain vector), write ops stay SubVectors via `parent.assocN` (`conj`/`assoc`/`pop`, structural sharing — never materialize). Slot came from the F-004 `box`→`sub_vector` amendment (user-selected, ADR-0179), 64-slot layout unchanged. Interface membership matches JVM ground truth exactly: `.sub_vector` joins every array `.vector` is in EXCEPT `EDITABLE_TAGS` (no transient) and `IREDUCEINIT_TAGS` (SubVector is not IReduceInit) — verified live against clj 1.11.1. seq/rest/next route through the O-058 `.array_seq` view (its backing switch gained a `.sub_vector` arm), so a subvec's seq is O(1) too. `=`/hash reuse the equal.zig `.vec` cursor by init'ing it with the parent + absolute [start,end) indices — zero new cursor code | O(n) alloc + trie rebuild -> O(1). Measured 2000 subvec ops on a 16000-element vector, ReleaseSafe: **9215 -> 0.77 ms (~11967x)**. Reference JVM clj 1.11.1, same box/probe: 1.06 ms — cljw was ~8693x SLOWER than clj and is now at PARITY (slightly faster). Broad win: `subvec` is the idiomatic vector slice, and it now shares structure instead of rebuilding | `zig build test -Dwasm` ×2 (VM + tree_walk diff oracle, all Values equal) + new `sub_vector.zig` units (layout, flatten-on-nest, never-empty, conj/assoc/pop parent-untouched, with-meta) + the 52-assertion observable-surface pin set (vector?/hash-eq/map-key/set-dedup/compare/into-{}/walk/nested/structural-sharing/meta/class=SubVector/empty+full-range/OOB-throws), landed as e2e `subvec_view.sh` and since migrated verbatim to `test/clj/suites/subvec_view_test.clj` + corpus `subvec.txt` (31 exprs) + `phase14_subvec.sh` (bounds) still green | D-583, D-044, O-058, ADR-0179, F-004, `private/notes/subvec-seqview-design.md` |
108108
| O-058 | `runtime/collection/array_seq.zig` (NEW, `.array_seq` A14) + `lang/primitive/sequence.zig` (`seqFn`/`restFn`/`nextFn`/`countFn`/`firstFn` + the three seq-walk helpers) + `.array_seq` arms in `equal.zig` / `print.zig` / `higher_order.zig` / `lazy_seq.zig` / `collection.zig` (`nthFn`) / `clojure_lang_method.zig` | `(seq v)` / `(rest v)` / `(next v)` on a vector built an EAGER `PersistentList` copy (`vectorToList` / `vectorTailAsList`): n `consHeap` allocations AND n trie descents per call. Clojure never copies — `PersistentVector$ChunkedSeq` is a VIEW holding `(vector, index)`. `.dev/bench/README.md` ranked this #2 and called SubVector (#1) its prerequisite, on the reading that a chunked seq's `next` must be "the vector minus its first 32 elements". It is not: `next` of a seq is a SEQ, and clj's own answer is a ChunkedSeq over the SAME vector at a higher index. So #2 never depended on #1, and the `heap_tag` census question #1 raises does not gate it | A new REPRESENTATION satisfying the existing seq protocol (OCP — no consumer changes, no `if` at call sites): `ArraySeq{backing, index, meta}`, 32 bytes, fixed size whatever it spans. `first` = one indexed read, `rest` = the same view at `index+1`, `count` = `backingCount - index`, `nth` = one indexed read (clj WALKS here — ChunkedSeq is Counted, not Indexed). Invariant: `make` answers nil when the view would be empty, mirroring `.range`, which is what makes `first` total. Costs NO slot from `heap_tag.unallocated`: A14 `.array_seq` was a day-1 reservation already declared across every `interface_membership.zig` protocol table and already named in `class_name.zig` but had no producer — `equal.zig`'s header literally anticipated "when a producer mints those tags". It also REPAIRS a latent census inaccuracy: A14 was classified boxed-in-production while nothing produced it | O(n) alloc + O(n log32 n) -> O(1) for all three. Measured 2000 ops on a 16000-element vector, ReleaseSafe: **`seq` 2123 -> 0.36 ms (~5897x), `rest` 2253 -> 0.39 ms (~5777x), `next` 2164 -> 0.38 ms (~5695x)**. Reference JVM clj 1.12.4, same box and probe: 0.60 / 0.87 / 0.99 ms — cljw was 2184-3549x SLOWER than clj on these and is now 1.7-2.6x FASTER. Broad win: this is what every hand-written `loop`/`recur` seq walk and every library seq walk pays (the idiomatic `reduce`/`doseq`/`into` paths already bypassed `seq` via O-002/O-004/O-032, which is why no single benchmark regressed on it) | `zig build test -Dwasm -Doptimize=ReleaseSafe` x2 (VM + tree_walk diff oracle) + new array_seq.zig units (layout, never-empty invariant, count-at-offset, rest/drop termination, saturating drop) + new the 76-assertion observable-surface pin set, landed as e2e `vector_seq_view.sh` and since split — 75 expression pins migrated to `test/clj/suites/vector_seq_view_test.clj`, the GC-torture rooting case kept in the e2e because CLJW_GC_TORTURE is read at process start. Pins: print-as-seq, `=`/`hash`/map-key interop with list AND vector, length-mismatch must lose, count exact at every offset, nth + default + OOB + negative, empty/one-element boundaries, next-walk visits each exactly once, 5000-element multi-level trie, nested collections) + `CLJW_GC_TORTURE=1` on the view (proves the trace fn reaches `backing`) + `phase16_gc_torture` | D-044, O-056, O-057, ADR-0179, F-004, `.dev/bench/README.md` |
109-
| O-057 | `runtime/collection/list.zig` (`COUNT_UNKNOWN` + `countPrepending` + `exactCountOf`) + `lang/primitive/sequence.zig` `countFn` (`.list`/`.cons`/`.lazy_seq` arm) | `(count lst)` WALKED every `.list`, even though `Cons` already stores an O(1) `count`. The stored field was distrusted for a real reason: a `.list` cell may head a MIXED chain whose `rest` leaves `.list` — `(cons x (map …))`, `(conj (range 3) 99)` — and there `1 + countOf(tail)` records the `.list` PREFIX length as if it were the total. The prior comment deferred the fast path to the F-004 `.list`/`.cons` tag split (D-178) | Make the stored count SELF-VALIDATING rather than splitting the tag: `countPrepending` writes `COUNT_UNKNOWN` (`maxInt(u32)`) when the tail is a non-`.list` seq or itself unknown, so unknown-ness propagates head-ward and the field is never a prefix presented as a total. `exactCountOf` returns `?u32`; `countFn` takes the O(1) path when it is non-null and keeps the existing walk otherwise. Sentinel chosen so every existing `countOf(v) > 0` / `== 0` emptiness test keeps its meaning (a cons cell always holds ≥1 element) | O(n) -> O(1) for a pure list. Measured 2000 `count` ops on a 16000-element list, ReleaseSafe: **380.15 -> 0.19 ms (~2006x)**. Reference JVM clj 1.12.4, same box and probe: 0.18 ms — cljw was ~2711x slower than clj on this op and is now at PARITY. Mixed chains keep the walk, so they keep their exact answer | Mixed-chain probes all still exact (`(cons 1 (map inc [1 2 3]))` 4, `(conj (range 3) 99)` 4, `(cons 0 (seq [1 2 3]))` 4, nested `(cons 1 (cons 2 (map inc [1 2 3])))` 5, pure `(list 1 2 3)` 3, `(list)` 0) + `(= (cons 1 (lazy-seq [2 3])) (list 1 2 3))` still true (the equal.zig `isCountable` exclusion of `.list` is what keeps `COUNT_UNKNOWN` out of the length short-circuit) + list.zig units + e2e `list_count_exact` + corpus | D-178, D-044, `.dev/bench/` |
109+
| O-057 | `runtime/collection/list.zig` (`COUNT_UNKNOWN` + `countPrepending` + `exactCountOf`) + `lang/primitive/sequence.zig` `countFn` (`.list`/`.cons`/`.lazy_seq` arm) | `(count lst)` WALKED every `.list`, even though `Cons` already stores an O(1) `count`. The stored field was distrusted for a real reason: a `.list` cell may head a MIXED chain whose `rest` leaves `.list` — `(cons x (map …))`, `(conj (range 3) 99)` — and there `1 + countOf(tail)` records the `.list` PREFIX length as if it were the total. The prior comment deferred the fast path to the F-004 `.list`/`.cons` tag split (D-178) | Make the stored count SELF-VALIDATING rather than splitting the tag: `countPrepending` writes `COUNT_UNKNOWN` (`maxInt(u32)`) when the tail is a non-`.list` seq or itself unknown, so unknown-ness propagates head-ward and the field is never a prefix presented as a total. `exactCountOf` returns `?u32`; `countFn` takes the O(1) path when it is non-null and keeps the existing walk otherwise. Sentinel chosen so every existing `countOf(v) > 0` / `== 0` emptiness test keeps its meaning (a cons cell always holds ≥1 element) | O(n) -> O(1) for a pure list. Measured 2000 `count` ops on a 16000-element list, ReleaseSafe: **380.15 -> 0.19 ms (~2006x)**. Reference JVM clj 1.12.4, same box and probe: 0.18 ms — cljw was ~2711x slower than clj on this op and is now at PARITY. Mixed chains keep the walk, so they keep their exact answer | Mixed-chain probes all still exact (`(cons 1 (map inc [1 2 3]))` 4, `(conj (range 3) 99)` 4, `(cons 0 (seq [1 2 3]))` 4, nested `(cons 1 (cons 2 (map inc [1 2 3])))` 5, pure `(list 1 2 3)` 3, `(list)` 0) + `(= (cons 1 (lazy-seq [2 3])) (list 1 2 3))` still true (the equal.zig `isCountable` exclusion of `.list` is what keeps `COUNT_UNKNOWN` out of the length short-circuit) + list.zig units + the 22 exactness pins (landed as e2e `list_count_exact`, since migrated to `test/clj/suites/list_count_test.clj`) + corpus | D-178, D-044, `.dev/bench/` |
110110
| O-055 | `eval/backend/vm.zig` (eval loop) + `eval/backend/tree_walk.zig` (`evalLoop` + fn-tail-recur loop) — budget-poll TLV hoist (D-450 lever 1) | The per-op / per-back-edge budget poll read the `threadlocal eval_budget_mod.current` each iteration. On macOS every threadlocal access is a dyld `_tlv_get_addr` CALL (D-571 moved the budget off the near-free Runtime field into the threadlocal, silently voiding D-386's "polls are near-free" verdict); `sample` on the rush-hour BFS showed `_tlv_get_addr` ≈ 7% top-of-stack, part of it this poll | Read `current` ONCE into a local at eval/loop entry; the per-op poll unwraps the local. Semantics preserved because every writer of `current` is stack-disciplined around a host call that re-enters eval (`with-budget` installs → invokes via callFn = a NEW eval invocation that re-reads at its entry → restores before returning to the hoisted loop); worker adopt/release bracket the worker's own eval | Same-session A/B on the rush-hour BFS headline (`(g/generate 7 :medium)`, M4 Pro, ReleaseSafe): median A 16.86 s (8 runs) → B 16.48 s (10 runs), **−2.3%**. The remaining TLV cost sits under inlined `stepOnce` (error-info per-call trace push/pop candidates — D-450 lever 2) | `zig build test -Dwasm` ×2 (diff oracle + the ADR-0125 dual-backend budget kill test) + `budget_thread_ownership` + `phase16_eval_budget` e2e green on the hoisted binary + smoke | D-450, D-571, D-386, ADR-0125 |
111111
| O-053 | `runtime/collection/value_sort.zig` (new) + Arrays/Collections/higher_order sort sites + 3 tiny-N insertion swaps — **binary-size axis** (ADR-0172 L6) | Every `std.mem.sort(T, items, ctx, less)` call monomorphizes ~25KB of block sort per (element, context, comparator) triple — 9 call sites had accreted ~224KB / 93 symbols; three of them sort `[]Value` with site-local comparators, and three more sort tiny-N arrays (txn ref set / fn arities / bencode dict keys) where a full block sort buys nothing | ONE shared instantiation: `value_sort.sort(items, ctx: *anyopaque, less: fn-pointer)` — the three Value sites pass their context opaquely (cast back inside the comparator); the three tiny-N cold sites drop to `std.sort.insertion` (~1KB each). Hot index-permutation sort (O-010 sortByKeysFn) keeps its dedicated instantiation | Shipped ReleaseSafe **7,073,080 → 6,974,584 B (−98.5KB)**; sort perf NEUTRAL (50k-element `(sort xs)` ×10: 1.00±0.03 hyperfine A/B — the indirect call is invisible next to valueCompare) | `zig build test -Dwasm` ×2 + new value_sort unit + probes: `(sort [3 1 2])`, string sort, `java.util.Collections/sort` + smoke | ADR-0172 (L6), O-010, D-515 |
112112
| O-052 | `build.zig` (`exe_mod.unwind_tables`) — **binary-size axis** (ADR-0172 L1) | Release builds carried `__eh_frame` + `__unwind_info` (749 KB, ~8% of the 9,469,816 B shipped binary) for native stack unwinding — but the shipped binary is STRIPPED (O-008), and a stripped Zig binary already prints "stack tracing is disabled" on panic (probed 2026-07-16 with a forced OOB panic, identical output with/without tables), so the tables bought nothing user-visible | `.unwind_tables = if (optimize != .Debug and !profile) .none else null` — mirrors the O-008 strip predicate; Debug + `-Dprofile` keep tables for lldb/Instruments. cljw renders Clojure errors from its own StackFrame stack; arm64 keeps frame pointers for debugger frame-walks | Shipped ReleaseSafe `cljw` **9,469,816 → 8,731,096 B (−738,720 B / −7.8%)**; `__eh_frame`/`__unwind_info` gone from `size -m` | smoke (units + diff oracle + e2e core on the table-less ReleaseSafe binary) + `size_claims` gate + panic-output probe pre/post identical | ADR-0172 (L1), O-008, D-515 |

‎test/clj/suites/lazy_seq_test.clj‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
;; Lazy-seq Layer-2 cycle 3 (Phase 14 §9.16 row 14.13.5, ADR-0054 D2/D4).
2+
;;
3+
;; concat / mapcat / drop are lazy `.clj` (the -drop-eager leaf is gone), the
4+
;; 0-arg `(range)` is an infinite lazy seq, and `=` force-walks a `.lazy_seq`
5+
;; operand (the equal.zig sequential arm).
6+
;;
7+
;; Migrated from test/e2e/phase14_lazy_seq_cycle3.sh; each assertion keeps its
8+
;; bash case name. Every case that touches an infinite producer is BOUNDED by
9+
;; take/first, as it was in the script.
10+
(ns suites.lazy-seq-test
11+
(:require [clojure.test :refer [deftest is testing]]))
12+
13+
(deftest zero-arg-range-is-infinite-and-lazy
14+
(is (= 0 (first (range)))) ; range0_first
15+
(is (= [0 1 2 3 4] (into [] (take 5 (range)))))) ; range0_take
16+
17+
(deftest drop-is-lazy
18+
(is (= '(3 4 5) (drop 2 [1 2 3 4 5]))) ; drop_print
19+
(is (= [3 4 5] (into [] (drop 2 [1 2 3 4 5])))) ; drop_into
20+
(is (= 7 (first (drop 0 [7 8])))) ; drop_zero
21+
(testing "dropping past the end yields empty, not an error"
22+
(is (= [] (into [] (drop 9 [1 2 3]))))) ; drop_overrun
23+
(testing "and it composes with an infinite source"
24+
(is (= 100 (first (drop 100 (range))))))) ; drop_inf_first
25+
26+
(deftest concat-and-mapcat-are-lazy
27+
(is (= '(1 2 3 4) (concat [1 2] [3 4]))) ; concat_print
28+
(is (= 9 (first (concat [] [9])))) ; concat_first
29+
(is (= [1 2 0 1 2] (into [] (take 5 (concat [1 2] (range)))))) ; concat_inf
30+
(is (= '(1 1 2 2 3 3) (mapcat (fn* [x] [x x]) [1 2 3]))) ; mapcat_print
31+
(is (= [0 0 1 1] (into [] (take 4 (mapcat (fn* [x] [x x]) (range))))))) ; mapcat_inf
32+
33+
;; `=` must force-walk a lazy operand on either side, and against any
34+
;; sequential representation
35+
(deftest equality-forces-a-lazy-operand
36+
(is (true? (= (map inc [1 2 3]) (list 2 3 4)))) ; eq_lazy_list
37+
(is (true? (= (list 2 3 4) (map inc [1 2 3])))) ; eq_list_lazy
38+
(is (true? (= (map inc [1 2]) [2 3]))) ; eq_lazy_vec
39+
(is (true? (= (map inc [1 2]) (map inc [1 2])))) ; eq_lazy_lazy
40+
(testing "length still discriminates in both directions"
41+
(is (false? (= (map inc [1 2 3]) (list 2 3)))) ; eq_lazy_short
42+
(is (false? (= (map inc [1 2]) (list 2 3 4))))) ; eq_lazy_long
43+
(testing "across the lazy producers"
44+
(is (true? (= (filter odd? [1 2 3 4 5]) (list 1 3 5)))) ; eq_filter_list
45+
(is (true? (= (drop 2 [1 2 3 4]) (list 3 4)))) ; eq_drop_list
46+
(is (true? (= (concat [1] [2 3]) (list 1 2 3))))) ; eq_concat_list
47+
(testing "a non-sequential operand is false, not an error"
48+
(is (false? (= (map inc [1 2]) 5))))) ; eq_lazy_num

0 commit comments

Comments
 (0)