fix(query): a node waiting on the query walk's stack holds its folder's path as the walk joins paths onto it - pieces never read - not the folder's node, whose own path a path: or in: term had read whole, so a deep chain whose every folder holds a file beside its subfolder costs memory in proportion to its depth, not its square (CI4 review F1.2)

The fix before this one let a node drop its folder once its own path was
built, which covered a bare chain only. The walk pushes a folder's
children in order and takes the last first, so a file listed before its
subfolder waits on the stack while everything under the folder is
walked. It held the folder's node, to build its own path from the
folder's, and with it the folder's own path - which a path: or in: term
had read, and reading a path copies it whole. Down a chain where every
folder holds such a file that is a whole copy of every folder's path:
185.6 MB at 8,000 deep (23 KB a folder, for path:leaf and for
in:/deep/d0 name:leaf alike), and Node 20 under a 2 GB heap aborted at
30,000 deep ("Ineffective mark-compacts near heap limit"). The Highlight
box asks on every pause in typing, and in: is a common term.

A node now holds a FolderPath: its folder's path as the walk joins its
nodes' paths onto it, made once per folder as its children are pushed.
While the walk runs it is handed only to childPath, which reads none of
it, so a chain of them stays the pieces they were joined from. A folder's
own path, the one terms and sort ties read, is a second string while the
walk runs; once the walk is over nothing waits, and a folder's path is
the one its children join onto, so each folder's path is still built once
for the page (G07's counted tests hold: a size query builds no more than
the 319 paths its hits need; with the two kept apart after the walk too,
it builds 468).

Measured (Node 20.20.2, 2 GB heap, the heap after a full collection when
the deep path is built), 8,000 deep: a file beside every folder, path:leaf
236 bytes a folder, in: 241, name:leaf 2; a bare chain, path:leaf 61. At
30,000 deep with a file beside every folder, path:leaf and in: finish
(134 and 143 MB resident). A deep chain's path is built from pieces, so
reading the deepest one walks every piece: slower than copying a flat
one, and only on depths no real tree has.

Tests (tests/queryWalkWork.test.ts):
- "a file waiting beside every folder ...": path:leaf,
  in:/deep/d0 name:leaf and name:leaf, one subtest each, down that
  8,000-deep chain - the one file found, its path exact and built once,
  under 2 KiB a folder held. RED before the fix: 185.6 / 185.5 MB.
- "a sort tie that reads a kept folder's path while its file still
  waits": folders sized by hand two at a time, type:dir, one kept, largest
  first - each odd folder ties with the even one kept above it, so the
  kept folder's path is read after its children were pushed. RED before
  the fix (185.5 MB); proves the "once the walk is over" gate.
- The heap is now also read as the deepest folder's path is joined (the
  last step of a chain joined in one go), so the existing CI4 test still
  fails if the store's childPath reads the parent path again (name:leaf
  184.0 MB).

Mutants, each RED for the right reason, restored and checked by SHA-256:
a folder's children join onto the path its readers read (path:leaf
184.2 MB); its path shared while the walk runs (tie: 93.6 MB, every
second folder); never shared, or the walk never marked over (paths built:
468 for 319 needed); a node's own path not kept (built twice, not once);
folders joined in the wrong order (the path is wrong); the store reading
the parent path (name:leaf 184.0 MB); size ties not broken by path (the
tie's path never built); the scanned folder not counted (total). Fixture
checks: the file listed after its folder; folders not paired by size.

Not changed: a kept hit, and a candidate of a query that looks facts up,
hold their own paths - bounded by how many are kept (the page, or the
20,000-candidate cap), not by what waits on the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Author: Prithvi-Web

src/services/query/execute.ts

@@ -413,20 +413,25 @@ class QueryNode implements EvalNode {
   readonly childCount: number | undefined;
   private nameRead: string | undefined;
   private pathRead: string | undefined;
+  /** What this folder's children join their paths onto: made as they are pushed. */
+  private within: FolderPath | null = null;
 
   constructor(
     private readonly store: ScanStore,
     readonly id: number,
     /**
-     * Held only to build this node's path, and let go once it is built (CI4):
-     * kept after, a node held every folder above it alive, and each folder's
-     * path with it — a whole copy once a path term had read it — so a walk
-     * down a deep chain held the square of its depth. Null for the root, and
-     * for a node whose path is known.
+     * The path of the folder this node is in, as its nodes join theirs onto
+     * it — never the folder's node (CI4 review F1.2). A node waiting on the
+     * walk's stack holds this; holding the folder's node instead held the
+     * folder's own path, which a path term had read whole, so a file listed
+     * before its subfolder kept that copy alive while the walk went down —
+     * the square of the depth down a chain where every folder holds one.
+     * Null for the root.
      */
-    private folder: QueryNode | null,
+    private readonly folder: FolderPath | null,
     readonly depth: number,
     childlessKnown: boolean,
+    private readonly walk: WalkProgress,
   ) {
     this.isDir = store.isDir(id);
     this.size = store.size(id);
@@ -439,24 +444,71 @@ class QueryNode implements EvalNode {
   }
 
   /**
-   * Built down from the nearest folder whose path is known, never by asking
-   * the folder's own getter: a query that reads no path while it walks leaves
-   * every folder's unbuilt, and a stack frame per level of a 30k-deep chain
-   * overflows (review F1.2; the store's own rule, tests/packedStore.test.ts).
+   * Built from its folder's path, which is built down from the nearest
+   * folder whose path is known, never by asking a folder's own getter: a
+   * query that reads no path while it walks leaves every folder's unbuilt,
+   * and a stack frame per level of a 30k-deep chain overflows (review F1.2;
+   * the store's own rule, tests/packedStore.test.ts).
+   *
+   * While the walk runs, a folder's path and the one its children join onto
+   * are two strings, though they spell the same path: whatever reads this one
+   * copies it whole, and that copy must not be what the nodes waiting in the
+   * folder hold. Once the walk is over nothing waits, and a folder's path is
+   * the one its children join onto, so a folder among the hits shares one
+   * string with the hits beneath it and each folder's path is still built
+   * once for the page (G07 review F2.3, F3.1).
    */
   get path(): string {
     if (this.pathRead !== undefined) return this.pathRead;
-    const unbuilt: QueryNode[] = [];
-    let known: QueryNode = this;
-    while (known.pathRead === undefined && known.folder !== null) {
-      unbuilt.push(known);
-      known = known.folder;
+    if (this.walk.over && this.within !== null) return (this.pathRead = this.within.path(this.store));
+    return (this.pathRead = this.folder === null ? this.store.rootPath : this.store.childPath(this.id, this.folder.path(this.store)));
+  }
+
+  /** The folder its children's paths join onto — made once, as the walk pushes them. */
+  folderForChildren(): FolderPath {
+    return (this.within ??= this.folder === null
+      ? new FolderPath(this.id, null, this.store.rootPath)
+      : new FolderPath(this.id, this.folder, undefined));
+  }
+}
+
+/** Shared by every node of one walk: whether it is over, so nothing waits on its stack any more. */
+interface WalkProgress {
+  over: boolean;
+}
+
+/**
+ * A folder's path as the paths of the nodes in it are joined onto it (CI4
+ * review F1.2). While the walk runs it is handed only to `childPath`, which
+ * reads none of it, so it stays the pieces it was joined from — its folder's,
+ * shared, and its own name with a separator — and a chain of them costs a few
+ * bytes a folder however deep it goes. Nothing that reads a path is handed
+ * it until the walk is over, because reading copies a path whole, and every
+ * node waiting on the walk's stack holds its folder's: a folder's path read
+ * whole there would stay whole as long as anything in that folder waited.
+ */
+class FolderPath {
+  constructor(
+    private readonly id: number,
+    /** The folder this one is in, until this one's path is joined. */
+    private folder: FolderPath | null,
+    private joined: string | undefined,
+  ) {}
+
+  /** Joined down from the nearest folder whose path is, without a stack frame per folder (review F1.2). */
+  path(store: ScanStore): string {
+    if (this.joined !== undefined) return this.joined;
+    const unjoined: FolderPath[] = [];
+    let known: FolderPath = this;
+    while (known.joined === undefined) {
+      unjoined.push(known);
+      known = known.folder as FolderPath; // the root's is joined from the start
     }
-    let path = (known.pathRead ??= this.store.rootPath); // known, or the root
-    for (let i = unbuilt.length - 1; i >= 0; i--) {
-      const node = unbuilt[i];
-      path = node.pathRead = this.store.childPath(node.id, path);
-      node.folder = null;
+    let path = known.joined;
+    for (let i = unjoined.length - 1; i >= 0; i--) {
+      const folder = unjoined[i];
+      path = folder.joined = store.childPath(folder.id, path);
+      folder.folder = null;
     }
     return path;
   }
@@ -490,14 +542,15 @@ class PassOneWalk implements SlicedWalk {
   readonly doubted = new WeakSet<EvalNode>();
   readonly kept: { offer(item: QueryNode): void; sorted(): QueryNode[] };
   private readonly stack: QueryNode[];
+  private readonly progress: WalkProgress = { over: false };
 
   constructor(private readonly store: ScanStore, private readonly ctx: PassOneContext) {
     this.kept = firstK<QueryNode>(ctx.keep, ctx.sort);
-    this.stack = [new QueryNode(store, store.rootId, null, 0, ctx.childlessKnown)];
+    this.stack = [new QueryNode(store, store.rootId, null, 0, ctx.childlessKnown, this.progress)];
   }
 
   step(budget: number): boolean {
-    const { store, stack, ctx } = this;
+    const { store, stack, ctx, progress } = this;
     for (let left = budget; stack.length > 0; left--) {
       if (left <= 0) return false;
       if (ctx.signal.aborted) { this.aborted = true; return true; }
@@ -522,11 +575,18 @@ class PassOneWalk implements SlicedWalk {
 
       if (node.isDir) {
         const depth = node.depth + 1;
-        store.forEachChild(node.id, (child) => { stack.push(new QueryNode(store, child, node, depth, ctx.childlessKnown)); });
+        store.forEachChild(node.id, (child) => {
+          stack.push(new QueryNode(store, child, node.folderForChildren(), depth, ctx.childlessKnown, progress));
+        });
       }
     }
     return true;
   }
+
+  /** Called once the walk is over, finished or stopped: nothing waits on its stack any more. */
+  end(): void {
+    this.progress.over = true;
+  }
 }
 
 /**
@@ -613,6 +673,7 @@ export async function executeAgainstScan(
     opts.slices ?? {},
     QUERY_SLICE_NODES,
   );
+  pass.end();
   const { candidates, kept, doubted, examined, capped } = pass;
   let { matched, undecided, childlessUndecided, aborted } = pass;
 

tests/queryWalkWork.test.ts

@@ -269,11 +269,18 @@ test('G07: a tree that changes during a slice\'s turn is walked again from the s
   assert.ok(!out.hits.some((h) => h.path === victim), 'the deleted file is not among the hits');
 });
 
-/** A completed scan of `depth` folders, one inside the next under /deep, with one file at the bottom. */
-function deepChain(depth: number): { scan: ScanResult; store: PackedScanStore; leaf: number; leafPath: string } {
+/**
+ * A completed scan of `depth` folders, one inside the next under /deep, with
+ * one file at the bottom. With `fileBeside`, every folder on the way also
+ * holds a file, f<i>.bin, listed before its subfolder.
+ */
+function deepChain(depth: number, fileBeside = false): { scan: ScanResult; store: PackedScanStore; leaf: number; leafPath: string } {
   const store = new PackedScanStore('/deep', '/', { name: 'deep', isDir: true, size: 0, modifiedAt: 1, isHidden: false });
   let parent = store.rootId;
-  for (let i = 0; i < depth; i++) parent = store.addNode(parent, { name: `d${i}`, isDir: true, size: 0, modifiedAt: 1, isHidden: false });
+  for (let i = 0; i < depth; i++) {
+    if (fileBeside) store.addNode(parent, { name: `f${i}.bin`, isDir: false, size: 1, modifiedAt: 1, isHidden: false, extension: 'bin' });
+    parent = store.addNode(parent, { name: `d${i}`, isDir: true, size: 0, modifiedAt: 1, isHidden: false });
+  }
   store.addNode(parent, { name: 'leaf.bin', isDir: false, size: 7, modifiedAt: 1, isHidden: false, extension: 'bin' });
   store.finalize();
   store.sumSizes();
@@ -283,7 +290,10 @@ function deepChain(depth: number): { scan: ScanResult; store: PackedScanStore; l
   scan.engine = 'walker';
   // Found by walking down after finalize, which may renumber.
   let leaf = store.rootId;
-  while (store.isDir(leaf)) leaf = store.childIds(leaf)[0];
+  while (store.isDir(leaf)) {
+    const kids = store.childIds(leaf);
+    leaf = kids.find((k) => store.isDir(k)) ?? kids[0];
+  }
   let deepest = '/deep';
   for (let i = 0; i < depth; i++) deepest += `/d${i}`;
   return { scan, store, leaf, leafPath: `${deepest}/leaf.bin` };
@@ -305,13 +315,65 @@ test('G07 review F1.2: a hit at the bottom of a 30k-deep chain is answered, its
 
 /**
  * What a deep hit's path may hold while it is built: a few hundred bytes per
- * folder on its way down (the node the walk holds, and the folder's share of
- * a path joined from its folder's) — never a whole copy of every folder's
- * path, which costs half the deep path's length per folder: 26 KB at 8,000
- * folders deep.
+ * folder on its way down (the nodes the walk holds — a file waiting beside
+ * the folder too — and the folder's share of a path joined from its
+ * folder's) — never a whole copy of every folder's path, which costs half the
+ * deep path's length per folder: 23 KB at 8,000 folders deep.
  */
 const HELD_PER_FOLDER = 2048;
 
+/**
+ * Run `q` and read the heap each time `probe`'s path is built, and each time
+ * a path of a node in `watch` is — the most the walk holds at once is at the
+ * bottom: as the deepest node's path is built, and as the deepest folder's is
+ * joined, the last of a chain joined in one go — after a full collection: a
+ * byte count, never a time. Answers the most held, and how often `probe`'s
+ * path was built.
+ */
+async function heldAsPathIsBuilt(
+  scan: ScanResult, store: PackedScanStore, probe: number, q: string, gc: () => void, opts: { limit?: number; watch?: number[] } = {},
+): Promise<{ out: QueryOutcome; probes: number; held: number }> {
+  const limit = opts.limit ?? 1000;
+  const watched = new Set([probe, ...(opts.watch ?? [])]);
+  await run(scan, q, { limit }); // whatever the first query loads is not the walk's
+  let probes = 0;
+  let held = -1;
+  const base = heldBytes(gc);
+  scan.store = new Proxy(store, {
+    get(target, prop, receiver) {
+      const value = Reflect.get(target, prop, receiver);
+      if (typeof value !== 'function') return value;
+      if (prop !== 'childPath') return (value as (...a: unknown[]) => unknown).bind(target);
+      return (child: number, parentPath: string): string => {
+        const built = (value as (c: number, p: string) => string).call(target, child, parentPath);
+        if (child === probe) probes++;
+        if (watched.has(child)) held = Math.max(held, heldBytes(gc) - base);
+        return built;
+      };
+    },
+  }) as ScanStore;
+  const out = await run(scan, q, { limit });
+  scan.store = store;
+  return { out, probes, held };
+}
+
+const heldMessage = (q: string, held: number, depth: number, deepPath: string): string =>
+  `${q}: ${(held / 1e6).toFixed(1)} MB held as the deep path is built, ${Math.round(held / depth)} bytes per folder of ${depth} (at most ${HELD_PER_FOLDER}; a whole copy of every folder's path is about ${Math.round(deepPath.length / 2)})`;
+
+/** The deep chain's one file found by `q`, its path exactly right and built once, and the walk's heap held to a few hundred bytes a folder as it is. */
+async function heldAsTheDeepPathIsBuilt(depth: number, fileBeside: boolean, q: string, gc: () => void): Promise<void> {
+  const { scan, store, leaf, leafPath } = deepChain(depth, fileBeside);
+  if (fileBeside) {
+    const top = store.childIds(store.rootId);
+    assert.ok(top.length === 2 && !store.isDir(top[0]) && store.isDir(top[1]), 'each folder lists its file before its subfolder');
+  }
+  const { out, probes, held } = await heldAsPathIsBuilt(scan, store, leaf, q, gc, { watch: [store.parent(leaf)] });
+  assert.equal(out.total, 1, `${q}: the one file matches`);
+  assert.equal(out.hits[0].path, leafPath, `${q}: its path, all ${depth} folders of it`);
+  assert.equal(probes, 1, `${q}: the deep path is built once, from its folder's`);
+  assert.ok(held < HELD_PER_FOLDER * depth, heldMessage(q, held, depth, leafPath));
+}
+
 test('CI4: a deep hit costs memory in proportion to its depth, not its square — with a path term or without, no folder on its way down keeps a whole copy of its path', async () => {
   // CI run 39095045009: the 30k-deep chain above ran macOS's 2 GB heap out
   // (it passed on this Mac's 4 GB). Each folder's path, built from its
@@ -320,34 +382,63 @@ test('CI4: a deep hit costs memory in proportion to its depth, not its square 
   // copy of its own path: the depth squared, about 2.7 GB at 30k. A path
   // term reads every node's path whole anyway, and a folder kept reachable
   // from below after its path was built then held that copy alive.
-  // Measured where the deep path is built: the most the walk holds at once.
-  const DEPTH = 8_000;
   const gc = collector();
-  for (const q of ['name:leaf', 'path:leaf']) {
-    const { scan, store, leaf, leafPath } = deepChain(DEPTH);
-    await run(scan, q); // whatever the first query loads is not the walk's
-    let probes = 0;
-    let held = -1;
-    const base = heldBytes(gc);
-    scan.store = new Proxy(store, {
-      get(target, prop, receiver) {
-        const value = Reflect.get(target, prop, receiver);
-        if (typeof value !== 'function') return value;
-        if (prop !== 'childPath') return (value as (...a: unknown[]) => unknown).bind(target);
-        return (child: number, parentPath: string): string => {
-          const built = (value as (c: number, p: string) => string).call(target, child, parentPath);
-          if (child === leaf) { probes++; held = heldBytes(gc) - base; }
-          return built;
-        };
-      },
-    }) as ScanStore;
-    const out = await run(scan, q);
-    assert.equal(out.total, 1, `${q}: the one file matches`);
-    assert.equal(out.hits[0].path, leafPath, `${q}: its path, all ${DEPTH} folders of it`);
-    assert.equal(probes, 1, `${q}: the deep path is built once, from its folder's`);
-    assert.ok(
-      held < HELD_PER_FOLDER * DEPTH,
-      `${q}: ${(held / 1e6).toFixed(1)} MB held as the deep path is built, ${Math.round(held / DEPTH)} bytes per folder of ${DEPTH} (at most ${HELD_PER_FOLDER}; a whole copy of every folder's path is about ${Math.round(leafPath.length / 2)})`,
-    );
+  for (const q of ['name:leaf', 'path:leaf']) await heldAsTheDeepPathIsBuilt(8_000, false, q, gc);
+});
+
+test('CI4 review F1.2: a file waiting beside every folder of a deep chain keeps no folder\'s whole path alive — with a path term or an in: term reading every path, or neither', async (t) => {
+  // Every folder holds a file listed before its subfolder. The walk pushes a
+  // folder's children in order and takes the last first, so each file waits
+  // on its stack while everything under its folder is walked, and it needs
+  // its folder's path for its own. It held the folder's node for that, and
+  // with it the folder's path, which a path: or in: term had read whole: the
+  // depth squared again — 185 MB at 8,000 deep, and a 2 GB heap run out at
+  // 30,000 — though the chain without the files held a few bytes a folder.
+  const gc = collector();
+  for (const q of ['path:leaf', 'in:/deep/d0 name:leaf', 'name:leaf']) {
+    await t.test(q, () => heldAsTheDeepPathIsBuilt(8_000, true, q, gc));
+  }
+});
+
+/**
+ * deepChain(depth, true) with its sizes set by hand, two folders at a time:
+ * d0 and d1 hold 1 byte, d2 and d3 hold 2, and so on; every file holds none.
+ */
+function pairedChain(depth: number): { scan: ScanResult; store: PackedScanStore; deepest: number; keptPath: string } {
+  const store = new PackedScanStore('/deep', '/', { name: 'deep', isDir: true, size: 0, modifiedAt: 1, isHidden: false });
+  let parent = store.rootId;
+  for (let i = 0; i < depth; i++) {
+    store.addNode(parent, { name: `f${i}.bin`, isDir: false, size: 0, modifiedAt: 1, isHidden: false, extension: 'bin' });
+    parent = store.addNode(parent, { name: `d${i}`, isDir: true, size: Math.floor(i / 2) + 1, modifiedAt: 1, isHidden: false });
   }
+  store.addNode(parent, { name: 'leaf.bin', isDir: false, size: 0, modifiedAt: 1, isHidden: false, extension: 'bin' });
+  store.finalize();
+  const scan = createScanRecord('/deep');
+  scan.store = store;
+  scan.status = 'complete';
+  scan.engine = 'walker';
+  let deepest = store.rootId;
+  for (let next = store.rootId; next !== -1; next = store.childIds(next).find((k) => store.isDir(k)) ?? -1) deepest = next;
+  let keptPath = '/deep';
+  for (let i = 0; i < depth - 1; i++) keptPath += `/d${i}`;
+  return { scan, store, deepest, keptPath };
+}
+
+test('CI4 review F1.2: a sort tie that reads a kept folder\'s path while its file still waits leaves the path that file holds in pieces', async () => {
+  // Folders only, the largest kept, one: each even folder is kept as it is
+  // met, being larger than the one kept before, and the odd folder inside it
+  // is the same size — a tie, broken by path, so the kept folder's path is
+  // read after its children were pushed, while its file still waits. Were
+  // its own path and the one its children join onto one string while the
+  // walk runs, that read would copy the joined one whole, and every second
+  // folder's would stay whole on the stack: the depth squared.
+  const DEPTH = 8_000; // even, so the deepest folder ties with the one above it as it is met
+  const gc = collector();
+  const { scan, store, deepest, keptPath } = pairedChain(DEPTH);
+  assert.equal(store.size(deepest), DEPTH / 2, 'the deepest folder is the size of the one above it');
+  const { out, probes, held } = await heldAsPathIsBuilt(scan, store, deepest, 'type:dir', gc, { limit: 1 });
+  assert.equal(out.total, DEPTH + 1, 'every folder matches, the scanned one too');
+  assert.deepEqual(out.hits.map((h) => h.path), [keptPath], 'the largest, the tie broken by path: the folder above the deepest');
+  assert.equal(probes, 1, 'the deepest folder\'s path is built once, for its tie');
+  assert.ok(held < HELD_PER_FOLDER * DEPTH, heldMessage('type:dir', held, DEPTH, keptPath));
 });