Skip to content

Commit 767d346

Browse files
Prithvi-Webclaude
andcommitted
docs(handoff): T10 is built (9f59e5c, 5945a96) and S2 is complete; the harness fix (bbb93a8) and the counting cancel test (8fc8884) are in; the 10-loop run is owed on mains power; the owner is asked to push
- LATEST: T10a pushed and CI run 36282411813 read (macOS, Linux and pt-BR green; Windows' two failures are T9's worker-thread probe on its first Windows run, now RISKS R96); T10b, the harness fix and the counting cancel test; the incident of 27 Sep (a whole-suite run under the app's binary opened the app; TreeMap's files untouched, Chromium's state beside them changed; reported); T10's gate, with the 10-loop run owed on mains power (on battery the Automatic budget is Eco, whose background priority gets no CPU under the whole suite and ten loops); T11 ported in a scratch worktree, to commit once T10's CI is green. - The plan: S2 done; T10 [x] with its gate and what is owed. - RISKS R96 (MEDIUM): on Windows a process that loads the addon in a worker thread ends without a word; the harness now says how. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 8fc8884 commit 767d346

3 files changed

Lines changed: 47 additions & 7 deletions

File tree

‎NEXT_SESSION_PROMPT.md‎

Lines changed: 38 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,42 @@ Copy everything below the line into a fresh session started in
1414

1515
**This block is the live state. Update it after every task.** Everything below the next `---` is the older hand-over and is background. (Rewritten ~10:45 UTC; the earlier, longer version of this block is in git history — `git log -p NEXT_SESSION_PROMPT.md`.)
1616

17-
### LATEST (26 Sep 2026, ~23:50 UTC, session 38ea1b2d) — RESUME HERE
18-
19-
**Newest (26 Sep, ~23:50 UTC): T9c BUILT `8047340` (R92 fixed; native contract 0.4.0) and T10's first step `c910f6a`.**
17+
### LATEST (27 Sep 2026, ~01:15 UTC, session 38ea1b2d) — RESUME HERE
18+
19+
**Newest (27 Sep, ~01:15 UTC): T10 BUILT — `9f59e5c` (T10a, pushed) and `5945a96` (T10b). S2 is complete. Two fixes on top: `bbb93a8` and `8fc8884`. The owner is asked to push.**
20+
- **The owner pushed `9f59e5c` (T10a). CI run 36282411813:** macOS, Linux and Linux pt-BR GREEN. Windows: 2 failures in `tests/benchMemoryPath.test.ts`, the worker-thread probe ("the measuring process wrote no result: ", nothing on stderr) and the matrix command that includes it.
21+
- That probe ran on Windows for the first time: until `1afc943`, the worker could not compile there.
22+
- It passes on macOS and Linux. Its process dies, or exits quietly, before writing its result, and the harness could not say which.
23+
- **`bbb93a8` makes it say:** the exit code or signal and stderr (`runMeasuringProcess`, `noResultReason`); a worker that exits unanswered (`firstAnswer`; before, such an exit left the promise pending and the process ended with code 0); and one stderr line between reading the version and terminating the worker.
24+
- **Read the next Windows run's message:** it names where the process dies. Fix from that (RISKS R96); T18's full-pass runner loads the addon in a worker too.
25+
- **INCIDENT (27 Sep, ~00:21–00:30 UTC): I opened the owner's installed app, twice.**
26+
- How: a gate leg ran the WHOLE suite under the app's binary as Node (`ELECTRON_RUN_AS_NODE=1`). There, `process.execPath` is the app, and the harness's 'node' runtime spawned it with that variable removed. A packaged app runs its own bundle, window included.
27+
- Stopped by killing the processes; nothing else runs from it.
28+
- No TreeMap data file changed (newest still 24 Sep). Chromium's state in `~/Library/Application Support/TreeMap` did: `Preferences`, Local/Session Storage, one `blob_storage` folder, GPU/Dawn caches, 33 cache entries.
29+
- Reported to the owner, with a recommendation to leave those files; the owner has not answered.
30+
- Fixed in `bbb93a8` (`memoryPathExecutable` refuses the 'node' runtime inside Electron). `ci-sim.sh` `LEG=node20` now refuses without `NODE20_FILES`. Memory note `electron-as-node-spawns-open-the-app`.
31+
- The full-suite Electron leg was never valid anyway: better-sqlite3 is built for Node 24's ABI.
32+
- **T10a:** `WalkError::Ceiling`; `scanPoll`'s `ceiling`; native contract 0.5.0; 4 mutants.
33+
- **T10b:** every native scan walks in memory mode (`nativeStorageFor`). A tree past the store's room (6,187,500 rows, or its names' bytes) is walked again on the native columns path, with the engine's own ceiling sentence in `engineReason`. The count the scan shows never goes back.
34+
- Tests: `nativeMemoryDefault` (5); the pause test reads the store's stats. 9 mutants.
35+
- The review caught one wrong claim, a computed row count for a tree that meets the names' ceiling first.
36+
- The native golden leg now runs through the memory path.
37+
- **Decided:** the walk's default numbering stays Discovery (the columns path is the oracle).
38+
- **R95 (MEDIUM):** past the room, a tree is walked twice until T12–T14.
39+
- `native/README.md`'s "tm-node contains no unsafe" (false since T8b) is corrected.
40+
- **T10's gate on the T10b candidate:**
41+
- default: cargo 550/0, blocks 237/0, npm 3,227 / 3,213 / 0 fail / 14 skipped (the extra skip is the known silent-watch one);
42+
- pt-BR: 3,227 / 3,215 / 0 / 12;
43+
- 4 busy loops: 3,227 / 3,214 / 0 / 13.
44+
- **10 busy loops, first run:** 2 failures in `reclaimScoreProvider` (a fixture scan past the 120 s hang guard). This Mac is on battery, so Auto = Eco, at Background QoS, and my own Rust builds ran meanwhile (load average 141): Eco walks got no CPU (the known starvation in `nativeEngine.test.ts`'s pause-test note).
45+
- **A/B** of that test alone under 10 loops on battery: T10a 3 s, T10b 1 s, both pass.
46+
- **Rerun, nothing else running:** stopped after 8 min: on battery the whole suite plus ten loops (load average 165) starved every scan-based test — `facts`, `humanScale`, `benchSynthetic`, `benchScanHold`, `reclaimScoreProvider` past their hang guards; T10a and T10b alike by the single-test A/B. **Owed: the 10-loop run on mains power** (Balanced), before T11 is committed.
47+
- **`8fc8884`:** tm-walk's cancel test raced the clock twice. It now holds `d0`'s listing open until the cancel and counts: at most one listing begins after it (renamed `cancel_while_directories_are_being_listed_stops_every_worker_at_its_next_folder`). The mutant where the cancel reaches no worker — all three of the walk's paths gone: the queue closed by the cancel, the per-folder check, the driver's close — is red with "98 listings began after the cancel". Three live-disk tests still hold wall-clock bounds, without a race against the walk's end; noted, not changed.
48+
- **Electron spot check of T10b** (`LEG=node20 NODE20_FILES=` the five native-scan files, where every scan now takes `storeTakeInto`): 83 tests, 81 pass, 1 skipped, 1 fail — "plain Node lets an array be the store's own memory", a plain-Node fact false inside Electron by design.
49+
- **T11 is ported onto T10b** in `$S/wt-t11` (not on main): tm-store `build.rs`/`lib.rs` conflicts resolved, the table's check in `check_options` with a new sink test, and the three option builders written since pass an empty table; docs written there. Commit only after T10's CI is green on all four legs.
50+
- **Local `main` adds `5945a96`, `bbb93a8`, `8fc8884` and this handoff over `9f59e5c`. The owner is asked to push; re-arm the watcher with base `9f59e5c`.**
51+
52+
**Earlier (26 Sep, ~23:50 UTC): T9c BUILT `8047340` (R92 fixed; native contract 0.4.0) and T10's first step `c910f6a`.**
2053
- **T9c:** where Electron refuses external buffers, `storeTakeInto` fills arrays JavaScript made, on libuv's pool, with 0 bytes on the JS thread.
2154
- New: `externalBuffersAllowed`, `storeShape` and `NativeStore.handOver`; `takeNativeStore` chooses the path, and `runNativeWalk` and the harness use it.
2255
- Electron's main thread during the hand-over at 5M: 7.5 ms, from ~33. About 2 ms is the fill itself; the rest is a V8 full GC for the ~320 MB of arrays arriving.
@@ -71,8 +104,8 @@ Copy everything below the line into a fresh session started in
71104
2. ~~T9b~~: done, `c577f06`.
72105
3. ~~T9c~~: done, `8047340`.
73106
4. ~~T9's harness re-measures~~: done, `c910f6a` (`T_mem` = 5M in both runtimes).
74-
5. T10: the scanner passes `storage: 'memory'`; full npm test under 4 and 10 busy loops; the golden leg; push; CI green.
75-
6. T11: commit the parked cloud rule table.
107+
5. ~~T10~~: done, `9f59e5c` + `5945a96` (S2 complete). Owed: the 10-loop run on mains power, and all four CI legs green on the push of `5945a96`–`8fc8884` (Windows: read the harness's new no-result sentence and fix R96 from it).
108+
6. T11: ported in `$S/wt-t11`; rebase it onto `main`, gate, and commit once T10's CI is green on all four legs.
76109
7. T12: now also position paths and `totalsFinal`.
77110
8. T13: now also the link log's disk runs; it needs Q1.
78111
9. T14–T23, then Phases 5–8.

‎docs/engine/RISKS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,3 +156,4 @@ The risks of the S3–S5 design in `docs/superpowers/plans/2026-09-18-phase4-sto
156156
| R93 | **Sending the first tree costs +680 MB in plain Node and +325–410 MB in Electron**, not the 150 MB budgeted (measured by T9, 26 Sep 2026). The SSE `complete` frame at 250,000 nodes is 81–86 MB of JSON, because every node carries its full path. To send it, the server builds the pruned tree's objects, one string of the whole frame, and the socket's UTF-8 copy of that string. The cost does not grow with the scan, so it sets the peak of every mode: no memory-mode size meets 700 MB in plain Node, Electron meets it only up to 2M, and aggregate's 400 MB ceiling is out of reach. This is today's code, in every engine. The renderer parses the same frame; its memory is not measured | HIGH | `T_mem` is kept at 5M (plan §S.11 Q17, an engineering decision the owner may overrule) until T9's harness measures again at T10 | **Fixed by T9b** (26 Sep 2026): the tree is sent from the store in chunks, byte for byte today's, so the send adds +49 MB in plain Node at 5M and nothing in Electron (peaks 538 and 555 MB, from 1,156 and 878). The renderer's memory is still not measured |
157157
| R94 | **Linux's transparent huge pages made a column's untouched pages resident** (found by CI run 36275360027, 26 Sep 2026). Linux merges neighbouring anonymous mappings into one region and backs it with 2 MiB pages, so a huge page faulted in for one column covered the untouched tail of the column beside it: 0.5–1.1 MB of a name pool past its room on the Linux legs, beyond the test's 2 MiB allowance. The sink never touched those pages: on macOS nothing is resident past any column's room. Whether it happens depends on where the kernel places the mappings (run 36276205704 did not see it) | LOW | Memory only, a few MB a scan on Linux; no answer is wrong | **Fixed by `b08395f`** (26 Sep 2026): every anonymous mapping advises `MADV_NOHUGEPAGE` on Linux, and the residency test holds each column to its room exactly — its 2 MiB allowance is gone, and a byte written 64 KiB past the room is red |
158158
| R95 | **A tree past the memory store's room is walked twice** (T10b, 26 Sep 2026). Every native scan now starts in memory mode, because nothing projects a scan's size yet (T12's chooser). A walk that reaches the store's room — 6,187,500 rows, or the bytes reserved for their names — ends at its ceiling, and the scanner walks the tree again on the native columns path, which every native scan took before T10. So such a scan takes the first walk's time longer than it did before T10. The first walk's store is freed with its handle before the second walk starts, so the peak memory is the columns path's, as before, and the count the scan shows holds rather than going back | MEDIUM | Time only, and only past 6.19M entries; `engineReason` says why, and nothing falls back to the legacy walker | Fixed by T12–T14: T12's chooser starts such a tree in the right mode from the previous scan's size, and the conversions carry a walk past the room on instead of starting again |
159+
| R96 | **On Windows, a process that loads the addon in a worker thread ends without a word** (CI run 36282411813, 27 Sep 2026). T9's worker-thread probe ran on Windows for the first time once `1afc943` let its worker compile there: its measuring process wrote no result and nothing to stderr, while the same probe passes on macOS and Linux. The harness could not say whether the process crashed or exited quietly: the probe waited only for the worker's message or error, so a worker that exited unanswered left the promise pending and the process ending with code 0. §S.5.7's full-pass runner (T18) loads the addon in a worker too | MEDIUM | Only the measuring tool is affected today; the app loads the addon on its main thread alone. The harness now says how a measuring process ended (exit code or signal, and stderr), a worker that exits unanswered is reported with its code, and the probe writes one stderr line between reading the version and terminating the worker | Diagnosed by the next Windows run; the fix follows from where the process dies, before T18 |

‎docs/superpowers/plans/2026-09-18-phase4-storage.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
| --- | --- | --- |
99
| Plan written | 18 Sep 2026 | this file |
1010
| S1 `tm-store`: the finalized store built in Rust, in the packed store's own layout | **done 24 Sep 2026**, commit `2415d6c`, committed after Phase 3's CI went green (run 36069755180): 41 tests, 89 mutants red, clippy on three targets, rustdoc `-D warnings`. The six-lens review's 29 CONFIRMED findings are all fixed (index mix-ups pinned with walks whose order differs from the store's; cloud × hard-link order; side tables refused unless strictly ascending, then binary-searched; `sparse_terms` so Node sums `sparseBytes` in the ingest's order past 2^53), and a second review of the fix round confirmed 6 more, repaired | `native/treemap-core/crates/tm-store/`; `docs/superpowers/plans/2026-09-24-phase4-s1-review-findings.md` |
11-
| S2 `PackedScanStore.fromColumns` + the native engine reads the Rust store through views (no ingest copy) | **Node side done 24 Sep 2026**, commit `38469ee`: `PackedScanStore.adoptColumns` (the `fromColumns` above) adopts a finalized tree's columns in place, held to the object-store oracle on 96 random trees; 20 mutants, 19 red, the survivor explained in the code. The rest of S2 is T1–T10 (§S.9); T10 commits S2 | `src/services/scanStore.ts`; `tests/packedStoreAdopt.test.ts`; `tests/fixtures/storeFuzz.ts` |
11+
| S2 `PackedScanStore.fromColumns` + the native engine reads the Rust store through views (no ingest copy) | **done 27 Sep 2026** with T10 (`9f59e5c`, `5945a96`): every native scan walks through the memory store. The Node side was done 24 Sep 2026, commit `38469ee`: `PackedScanStore.adoptColumns` (the `fromColumns` above) adopts a finalized tree's columns in place, held to the object-store oracle on 96 random trees; 20 mutants, 19 red, the survivor explained in the code. T1–T10 built the rest (§S.9) | `src/services/scanStore.ts`; `tests/packedStoreAdopt.test.ts`; `tests/fixtures/storeFuzz.ts` |
1212
| S3–S5 as designed (§S below) | **designed 24 Sep 2026.** T0 (documents only) is written; its gate (§S.9) is the owner's answers to Q1–Q5 and Q7 (§S.11): ~~Q1, Q3 and Q5 wait for the owner~~ Q1 and Q5 wait for the owner (Q3 decided by the owner 25 Sep 2026: the 10M row is met by spill, memory mode up to 5M), and Q2, Q4 and Q7 are engineering decisions awaiting the owner's confirmation. T1–T6, T6b, T6c, T7 and T8 built 25–26 Sep 2026; T9 measured 26 Sep 2026, and its findings added T9b and T9c, built 26 Sep 2026 with T10; T11–T23 are not built | §S below |
1313
| S3 spill mode: columns written with `write()` ~~then mapped~~ and read with `pread()` (P4-5a), the free-space and same-volume rules, cleanup | not started: T11–T17 (T17 commits S3); the FullPassRunner is S3b, T18 | |
1414
| S4 aggregate-only mode: ~~directory rows~~ rows kept by the β rule (P4-7a), running totals, top-K, the notice naming what is off | not started: T12 (the AggregateState, which spill runs too), T19 (commits S4), T20 (Windows large mode) | |
@@ -860,7 +860,7 @@ House rules as in Phases 2–3. Tests count events; nothing times the wall clock
860860
- the rest is a full garbage collection V8 runs because some 320 MB of arrays arrive in a heap that holds the server (4.8 ms, seen with `--trace-gc`). It does not grow with the store, and the old path paid it too, on top of the copy.
861861

862862
Electron's 5M peak is 547 MB, from 555.
863-
- [ ] **T10. Switch production memory mode to the stream path** (`numbering = Blocks`). `ingestColumns`, `take()` and `build()` stay as oracles.
863+
- [x] **T10. Switch production memory mode to the stream path** (`numbering = Blocks`). `ingestColumns`, `take()` and `build()` stay as oracles.
864864
- **First (added by T9, 26 Sep 2026):** after T9b and T9c, T9's harness measures again in both runtimes and sets each runtime's `T_mem`, and `nativeMemory.ts`' provisional sizes follow it.
865865
- **The first step is done (26 Sep 2026).**
866866
- **The record:** `bench/baselines/memory-path/darwin-arm64.json`, from `8047340`. It was taken at load 10–11 on 8 cores, so its timings are loaded ones; its memory is not.
@@ -880,6 +880,12 @@ House rules as in Phases 2–3. Tests count events; nothing times the wall clock
880880
- **Decided: the walk's default numbering stays Discovery.** The app's scans walk under Blocks, because the memory path sets it (`feed_only`). The default is the columns path's, which is the oracle and the walk again past the room, and an oracle on the older, independent numbering is the stronger check. So T6's "until T10 makes Blocks the default" does not happen, and CI keeps running the tm-walk suite under both numberings.
881881
- **The cost:** until T12–T14, a tree past the room is walked twice (RISKS R95).
882882
- **Green gate:** full `npm test` under 4 and 10 busy loops; commit **S2**; the owner pushes; CI green on all four legs before T11.
883+
- **The gate, 27 Sep 2026.**
884+
- T10a (`9f59e5c`): cargo 550/0, blocks 237/0, npm 3,222 / 0 fail; pushed.
885+
- T10b's candidate (`5945a96`): default 3,227 / 0 fail, pt-BR 3,227 / 0 fail, 4 busy loops 3,227 / 0 fail.
886+
- **The 10-loop run is owed.** This Mac was on battery, so the Automatic budget ran every scan in Eco, at macOS's background priority. With the whole suite and ten busy loops (load average 141–165), macOS gave those scans no CPU. Scan-based tests in `reclaimScoreProvider`, `facts`, `humanScale`, `benchSynthetic` and `benchScanHold` waited past their hang guards, as designed (a starved walk waits). One such test alone under ten loops on battery passes in 3 s on T10a and 1 s on T10b. The run is repeated on mains power before T11.
887+
- In Electron's Node, the five native-scan test files, where every scan now takes `storeTakeInto`: 81 of 83 pass, 1 skipped, and 1 fails. The failure asserts a plain-Node fact that is false in Electron by design.
888+
- **CI run 36282411813 (T10a):** macOS, Linux and pt-BR green. On Windows, T9's worker-thread probe failed where it had never run before: RISKS R96; `bbb93a8` makes the harness say how its process ended.
883889
- [ ] **T11. The cloud rule table.** The regexes in `cloudFolders.ts` (lines 15-20) become built from `CLOUD_RULES`, with a Rust matcher over the same table.
884890
- **Tests first:** the old regex against the table over a path corpus (case variants, a trailing `.icloud` against `.icloud` inside a name, Windows separators, U+212A, U+0130, U+017F); the same fixture file through the Rust matcher; the lowercase-to-ASCII code point set pinned in Node 24 and Electron 31.
885891
- **Mutants:** a case-sensitive match; the `$` anchor dropped.

0 commit comments

Comments
 (0)