|
9 | 9 | | Plan written | 18 Sep 2026 | this file | |
10 | 10 | | S1 `tm-store`: the finalized store built in Rust, in the packed store's own layout | **done 24 Sep 2026**, commit `2415d6c`, committed after Phase 3's CI went green (run 36069755180): 41 tests, 89 mutants red, clippy on three targets, rustdoc `-D warnings`. The six-lens review's 29 CONFIRMED findings are all fixed (index mix-ups pinned with walks whose order differs from the store's; cloud × hard-link order; side tables refused unless strictly ascending, then binary-searched; `sparse_terms` so Node sums `sparseBytes` in the ingest's order past 2^53), and a second review of the fix round confirmed 6 more, repaired | `native/treemap-core/crates/tm-store/`; `docs/superpowers/plans/2026-09-24-phase4-s1-review-findings.md` | |
11 | 11 | | S2 `PackedScanStore.fromColumns` + the native engine reads the Rust store through views (no ingest copy) | **done 27 Sep 2026** with T10 (`9f59e5c`, `5945a96`): every native scan walks through the memory store. The Node side was done 24 Sep 2026, commit `38469ee`: `PackedScanStore.adoptColumns` (the `fromColumns` above) adopts a finalized tree's columns in place, held to the object-store oracle on 96 random trees; 20 mutants, 19 red, the survivor explained in the code. T1–T10 built the rest (§S.9) | `src/services/scanStore.ts`; `tests/packedStoreAdopt.test.ts`; `tests/fixtures/storeFuzz.ts` | |
12 | | -| S3–S5 as designed (§S below) | **designed 24 Sep 2026.** T0 (documents only) is written; its gate (§S.9) is the owner's answers to Q1–Q5 and Q7 (§S.11): ~~Q1, Q3 and Q5 wait for the owner~~ ~~Q1 and Q5 wait for the owner~~ Q5 waits for the owner (Q3 decided by the owner 25 Sep 2026: the 10M row is met by spill, memory mode up to 5M; Q1 decided by the owner 28 Sep 2026: yes, TreeMap may `unlink` its own spill files inside `<appData>/scan-spill`), and Q2, Q4 and Q7 are engineering decisions awaiting the owner's confirmation. T1–T6, T6b, T6c, T7 and T8 built 25–26 Sep 2026; T9 measured 26 Sep 2026, and its findings added T9b and T9c, built 26 Sep 2026 with T10; T11 and T12a–T12e built 25–27 Sep 2026 (T12e all but its `cargo mutants` run), T15a (the selection ports) and T13a–T13c (the spill file, `spill_plan` and the ledger, the sweep and the confined remover) 28 Sep 2026; T13d, T14, T15b and T16–T23 are not built | §S below | |
| 12 | +| S3–S5 as designed (§S below) | **designed 24 Sep 2026.** T0 (documents only) is written; its gate (§S.9) is the owner's answers to Q1–Q5 and Q7 (§S.11): ~~Q1, Q3 and Q5 wait for the owner~~ ~~Q1 and Q5 wait for the owner~~ ~~Q5 waits for the owner~~ (Q3 decided by the owner 25 Sep 2026: the 10M row is met by spill, memory mode up to 5M; Q1 decided by the owner 28 Sep 2026: yes, TreeMap may `unlink` its own spill files inside `<appData>/scan-spill`; Q5, Q6, Q10 and Q13 decided by the owner 28 Sep 2026: the proposals are taken, §S.11), and Q2, Q4 and Q7 are engineering decisions awaiting the owner's confirmation. T1–T6, T6b, T6c, T7 and T8 built 25–26 Sep 2026; T9 measured 26 Sep 2026, and its findings added T9b and T9c, built 26 Sep 2026 with T10; T11 and T12a–T12e built 25–27 Sep 2026 (T12e all but its `cargo mutants` run), T15a (the selection ports) and T13a–T13c (the spill file, `spill_plan` and the ledger, the sweep and the confined remover) 28 Sep 2026; T13d, T14, T15b and T16–T23 are not built | §S below | |
13 | 13 | | S3 spill mode: columns written with `write()` ~~then mapped~~ and read with `pread()` (P4-5a), the free-space and same-volume rules, cleanup | in progress: T11 built; T13a–T13c (the spill file, `spill_plan` and the ledger, the sweep and the confined remover) and T15a (the selection ports, `select_prune` and `select_treemap`) built 28 Sep 2026; T13d, T14, T15b, T16 and T17 not built (T17 commits S3); the FullPassRunner is S3b, T18 | `native/treemap-core/crates/tm-store/src/spill.rs`, `spill/plan.rs`; `src/services/spillSweep.ts`; `tm-store/src/select.rs`; `tests/fixtures/selectOracle.ts` | |
14 | 14 | | S4 aggregate-only mode: ~~directory rows~~ rows kept by the β rule (P4-7a), running totals, top-K, the notice naming what is off | in progress: T12 (the AggregateState, which spill runs too) built as T12a–T12e 27 Sep 2026, all but T12e's `cargo mutants` run; T19 (commits S4) and T20 (Windows large mode) not built | `tm-store/src/aggregate/` | |
15 | 15 | | S5 the synthetic-source gate: 100M entries from a scripted lister through the walk and the store, peak RSS measured by the harness | not started: T3 (the `SyntheticLister`, moved forward), T21 (bench plumbing), T22 (the gate) | | |
@@ -1416,29 +1416,29 @@ Each question's status was recorded on 24 Sep 2026 (T0). |
1416 | 1416 | 4. **Q4 — additive API fields.** Is `FileNode.omitted` (aggregate only) plus a `notKept` answer on `/nodes` and the facts acceptable under §11.1, which allows additions only? |
1417 | 1417 | - **Status: engineering decision, 24 Sep 2026, the owner may overrule.** As proposed: `FileNode.omitted` is added in aggregate only, and `/nodes` and the facts answer `notKept`. Both are additions; the memory-mode goldens do not change. |
1418 | 1418 | 5. **Q5 — Live mode and container expansion.** Should they be off in spill and aggregate for Phase 4? |
1419 | | - - **Status: engineering proposal, pending the owner.** Live mode and container expansion are off in spill and aggregate, and the notice names them (P4-6a, §S.7). |
| 1419 | + - **Status: decided by the owner on 28 Sep 2026 — the proposal is taken.** ~~Engineering proposal, pending the owner.~~ Live mode and container expansion are off in spill and aggregate, and the notice names them (P4-6a, §S.7). |
1420 | 1420 | 6. **Q6 — the FullPassRunner.** Must it land inside Phase 4 (S3b, proposed)? And may the per-node-state features stay off in spill until they are ported to native code? |
1421 | | - - **Status: engineering proposal, pending the owner.** The FullPassRunner lands inside Phase 4 (T18, commit S3b). The features whose JavaScript state grows per node stay off in spill until they are ported to native code (§S.5.7). |
| 1421 | + - **Status: decided by the owner on 28 Sep 2026 — the proposal is taken.** ~~Engineering proposal, pending the owner.~~ The FullPassRunner lands inside Phase 4 (T18, commit S3b). The features whose JavaScript state grows per node stay off in spill until they are ported to native code (§S.5.7). |
1422 | 1422 | 7. **Q7 — the cloud rule.** Is one table shared by `cloudFolders.ts` and Rust acceptable, with memory mode keeping P4-3's Node pass? |
1423 | 1423 | - **Status: engineering decision, 24 Sep 2026, the owner may overrule.** One table (`CLOUD_RULES`, T11) builds the regexes in `cloudFolders.ts` and feeds the Rust matcher; memory mode keeps P4-3's Node pass (P4-3a). |
1424 | 1424 | 8. **Q8 — the queue.** Do you accept the hybrid FIFO→LIFO queue (`Q_MAX` 65,536 in memory mode, 4,096 in the large modes), with a re-bench? |
1425 | 1425 | - **Status: engineering decision, 24 Sep 2026, the owner may overrule.** The hybrid FIFO→LIFO queue, `Q_MAX` 65,536 in memory mode and 4,096 in the large modes (P4-13), re-benched against the Phase 3 baselines (T22: within 10%). |
1426 | 1426 | 9. **Q9 — the hard-link winner.** Keep today's breadth-first-first member (proposed, byte-identical)? Or adopt the digest's smallest-path rule, which changes native output for families that span folders? |
1427 | 1427 | - **Status: engineering decision, 24 Sep 2026, the owner may overrule.** Today's breadth-first-first member stays the winner (P4-2a), so native output does not change. The digest's smallest-path normalisation (DESIGN §16 item 6) is untouched. |
1428 | 1428 | 10. **Q10 — no legacy fallback above `T_mem`.** A native failure there fails the scan with its reason (P4-14). How does that interact with step 2's open question about the native scan falling back under full CPU load? |
1429 | | - - **Status: engineering proposal, pending the owner.** No legacy fallback in spill or aggregate or above `T_mem` (P4-14): a native failure there fails the scan with its reason. Step 2's question was answered by the owner on 24 Sep 2026 (commit `298d752`, RISKS R56): a native walk's 30 s stall clock counts only time in which the machine had CPU to spare. Under P4-14, a walk that rule cancels above `T_mem` fails the scan with that reason instead of falling back. |
| 1429 | + - **Status: decided by the owner on 28 Sep 2026 — the proposal is taken.** ~~Engineering proposal, pending the owner.~~ No legacy fallback in spill or aggregate or above `T_mem` (P4-14): a native failure there fails the scan with its reason. Step 2's question was answered by the owner on 24 Sep 2026 (commit `298d752`, RISKS R56): a native walk's 30 s stall clock counts only time in which the machine had CPU to spare. Under P4-14, a walk that rule cancels above `T_mem` fails the scan with that reason instead of falling back. |
1430 | 1430 | 11. **Q11 — prune's giant-folder behaviour (R-S10).** Keep it, or stop before a folder whose children would pass `maxNodes` by more than X and mark it `pruned`? That second option is a JSON change only for such folders. |
1431 | 1431 | - **Status: engineering decision, 24 Sep 2026, the owner may overrule.** Phase 4 keeps today's behaviour: `select_prune` reproduces `pruneStore` exactly, every child of a popped folder included (§S.5.5), so no JSON changes. R-S10 (RISKS R82) stays open, and the gate's big-listing variant keeps that folder small in bytes. |
1432 | 1432 | 12. **Q12 — aggregate's weaker lists.** Is `exact:false` with a reason acceptable, rather than refusing the list? |
1433 | 1433 | - **Status: engineering decision, 24 Sep 2026, the owner may overrule.** An aggregate answer the seal cannot prove exact carries `exact: false` with the reason instead of being refused (P4-16). In spill such an answer is recomputed (§S.5.6). |
1434 | 1434 | 13. **Q13 — Windows timing.** Must Windows large mode (T20) land inside Phase 4? The DoD's "Tier C" wording implies it does. |
1435 | | - - **Status: engineering proposal, pending the owner.** Windows large mode (T20) lands inside Phase 4, proven at CI-fixture scale on the Windows leg. |
| 1435 | + - **Status: decided by the owner on 28 Sep 2026 — the proposal is taken.** ~~Engineering proposal, pending the owner.~~ Windows large mode (T20) lands inside Phase 4, proven at CI-fixture scale on the Windows leg. |
1436 | 1436 | 14. **Q14 — which shapes gate.** Does the developer shape (15% folders) gate, with the wide, dirheavy, links, windows and candidates variants gated too? Or should some only be recorded? |
1437 | 1437 | - **Status: engineering decision, 24 Sep 2026, the owner may overrule.** The developer shape gates, and so does every §S.8 variant. |
1438 | 1438 | 15. **Q15 — the storage endpoint.** Is a separate `/api/scan/:id/storage` acceptable, so the `sseComplete` golden stays unchanged? |
1439 | 1439 | - **Status: engineering decision, 24 Sep 2026, the owner may overrule.** A separate `GET /api/scan/:id/storage`, so the `sseComplete` golden stays unchanged (§S.6.5). |
1440 | 1440 | 16. **Q16 — E0 in the app itself.** T9 measured E0 in the installed app's binary run as Node (107 MB). Run that way it does not start Chromium's browser side, which the app's main process also runs. May the installed TreeMap be opened once, for about a minute, with a throwaway data folder and nothing of the owner's, so that its main process's memory can be read? |
1441 | | - - **Status: asked 26 Sep 2026, pending the owner; recommended yes.** Until then, E0 is a lower bound. |
| 1441 | + - **Status: decided by the owner on 28 Sep 2026: yes, once.** ~~Asked 26 Sep 2026, pending the owner; recommended yes.~~ The installed app is opened once, for about a minute, with a throwaway profile (a temporary `TREEMAP_DATA_DIR` and Chromium `--user-data-dir`), only while the owner is not using TreeMap. Until that measurement is recorded, E0 is a lower bound. |
1442 | 1442 | 17. **Q17 — T9's gate.** 5M passes 700 MB in both runtimes, because sending the first tree costs +680 MB (Node) and +325–410 MB (Electron) whatever the scan's size (T9's entry). The two ways forward: |
1443 | 1443 | - keep `T_mem` at 5M in both runtimes, and build T9b (the tree sent from the store, byte for byte today's) and T9c (Electron's hand-over off the JavaScript thread) before T10, measuring again after them; |
1444 | 1444 | - or lower `T_mem` now: Electron to 2M, while Node has no size that meets 700 MB. |
|
0 commit comments