merge: v2.5.4 — V-memory embeds once per refresh (perf)

Author: copeus

.claude-plugin/marketplace.json

@@ -9,7 +9,7 @@
     {
       "name": "superpowers-v",
       "description": "Compound V for Superpowers: triple parallel pre-flight (code archaeology + domain-expert + Context7 library validator), disjoint partitioning, manifest-driven multi-backend dispatch (Claude + Codex + Antigravity + Cursor), git-diff scope enforcement, crash-resumable runs, adaptive tier-based routing, epic mode, V-memory local-first semantic+lexical recall over docs/superpowers (opt-in pure-python embeddings + a deterministic recall→action bridge), and batched parallel dispatch (Opus default, narrow Sonnet exception)",
-      "version": "2.5.3",
+      "version": "2.5.4",
       "source": "./",
       "author": {
         "name": "Oleg",

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "superpowers-v",
   "description": "Compound V for Superpowers: triple parallel pre-flight (code archaeology + domain-expert advisor + library/doc validator via Context7), disjoint file partitioning, manifest-driven multi-backend dispatch (Claude + headless Codex + Antigravity + Cursor workers), git-diff scope enforcement, crash-resumable runs, adaptive tier-based routing, epic mode for multi-feature builds, V-memory local-first semantic+lexical recall over docs/superpowers (opt-in pure-python embeddings + a deterministic recall\u2192action bridge), and batched parallel dispatch (Opus default, Sonnet for narrow junior-task carve-out), plus /v:onboard — a project-onboarding command that builds a citation-verified knowledge base + AGENTS.md/CLAUDE.md bridge behind a human gate. Auto-intercepts brainstorming \u2192 writing-plans \u2192 execution transitions.",
-  "version": "2.5.3",
+  "version": "2.5.4",
   "author": {
     "name": "Oleg",
     "email": "copeus@gmail.com"

CHANGELOG.md

@@ -4,6 +4,11 @@ All notable changes to **superpowers-v (Compound V)** are documented here.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project uses semantic versioning.
 
+## [2.5.4] — 2026-07-05
+
+### Performance
+- **V-memory DENSE refresh now loads the embedding model *once*, not per file.** The refresh embedded per file — `reindex_file` invoked the isolated-venv embedder subprocess once per file, and each subprocess rebuilt the ONNX `InferenceSession`, so `N` files meant `N` model loads (the reason the first full pass over `docs/superpowers/**` was slow). `cmd_refresh` now uses a new `reindex_batch` that chunks all to-index files, flattens their chunks into **one** embedder call, and slices the vectors back per file — **one model load per refresh**. The FTS5-only (embeddings-off) path is unchanged and it stays **degrade-safe** (a failed batch persists `NULL` embeddings → FTS5-only; the CORE lexical lane is never affected). Selftest injects a **call-counting fake embedder** proving the single call + correct per-file vector slicing + degrade — no network/model needed. **Codex cross-model verification: ACCURATE** on all five claims with `file:line` evidence (single call, offset slicing with no off-by-one, empty-corpus skips the model load, degrade-safe `NULL` fallback, atomic persistence preserved).
+
 ## [2.5.3] — 2026-07-05
 
 ### Added — `npx autoskills` recommender for `/v:onboard`

docs/superpowers/specs/2026-07-05-v2.5.4-embed-batch-design.md

@@ -0,0 +1,71 @@
+# Load the embedding model once per refresh (design, v2.5.4)
+
+**Status:** approved for build (2026-07-05). A performance fix for the V-memory DENSE lane.
+
+## 1. What it is
+The V-memory refresh currently embeds **per file** — `reindex_file` calls the isolated-venv
+embedder subprocess once per file
+([`compound-v-memory.py:546-547`](../../scripts/compound-v-memory.py:546)), and each subprocess
+re-loads the ONNX model ([`:380`](../../scripts/compound-v-memory.py:380)). For `N` files that is
+`N` model loads — the whole reason the first full pass over `docs/superpowers/**` is slow. This
+change makes the refresh embed **all chunks of all files in a single embedder call** → **one model
+load per refresh**.
+
+## 2. Why
+The embedder runs in an out-of-repo venv **as a subprocess** (dependency isolation + degrade-safe),
+so the ONNX `InferenceSession` is (re)built on every invocation. Batching all chunks into one call
+turns `N` model loads into **1** — a large, one-time speedup on the first full pass, with no change
+to isolation or degrade-safety. (A persistent embedder daemon was considered and rejected as
+overkill for a docs corpus — one call holds all chunk texts + 384-dim vectors in memory, trivially.)
+
+## 3. Components
+
+### 3.1 Extract the write path — `_persist_chunks(conn, root, rel, chunks, vecs)`
+Pull the atomic transaction out of `reindex_file` (delete old chunks → insert new chunks with
+`vecs[i]` blobs → upsert `indexed_files`) into a helper. `reindex_file` keeps its current behavior
+(chunk → per-file embed → `_persist_chunks`) and remains the **FTS5-only / no-embedder** path and a
+safe fallback. The existing per-chunk guard `if vecs is not None and i < len(vecs)` moves into the
+helper — a short/None vec list still degrades to `NULL` embeddings, never crashes.
+
+### 3.2 Batched embed — `reindex_batch(conn, root, rels, embedder)`
+1. Chunk every `rel` (`chunk_file`), collecting `[(rel, chunks), …]`.
+2. Flatten all chunk texts into ONE list, preserving order.
+3. `all_vecs = embedder(flat_texts)` — **exactly one** embedder call (⇒ one subprocess ⇒ one model
+   load). `embedder` is the same `lambda texts: embed_texts(...)`.
+4. Slice `all_vecs` back per file by a running offset and call `_persist_chunks` for each.
+5. Degrade-safe: `all_vecs is None` (embed failed) ⇒ every file persists with `NULL` embeddings
+   (FTS5-only) — the same fallback as today, batched.
+
+### 3.3 Wire into `cmd_refresh`
+Where the refresh loops `to_index` ([`:631-633`](../../scripts/compound-v-memory.py:631)): when
+`embedder is not None`, call `reindex_batch(conn, root, to_index, embedder)`; when it is `None`
+(FTS5-only), keep the existing per-file `reindex_file` loop unchanged.
+
+## 4. Invariants (non-negotiable)
+1. **Exactly one embedder call per refresh** when embedding (the whole point) — asserted by a
+   call-counting fake embedder in the selftest.
+2. **Correct vector→file mapping** — each file's chunks get *their* vectors (offset slicing),
+   verified against a fake embedder that returns index-encoding vectors.
+3. **Degrade-safe unchanged** — embedder `None` (off) uses the per-file path; a failed batch
+   (`all_vecs is None`) persists `NULL` embeddings (FTS5-only). Never crash the refresh.
+4. **FTS5-only path untouched** — no behavior change when embeddings are off/unbootstrapped.
+5. **Incrementality preserved** — `to_index` is still the changed/missing-vector set; batching only
+   changes *how* those files are embedded, not *which*.
+
+## 5. Verification
+- **`compound-v-memory.py --selftest`** gains cases with an **injected fake embedder**: a 3-file
+  batch ⇒ the fake is called **once** with all texts; each file's stored vectors match its own
+  chunks (offset slicing correct); a fake returning `None` ⇒ chunks stored with `NULL` embeddings.
+  No real venv/model needed (the perf property is tested structurally).
+- **Existing selftests stay green** (FTS5 path unchanged).
+- **Codex cross-model verification** of the batching/slicing + degrade logic.
+- Full regression + lint + CI version-lockstep green.
+
+## 6. Out of scope
+- A persistent embedder daemon / streaming protocol (overkill for a docs corpus).
+- Sub-batching a huge corpus into several subprocess calls (would defeat "one model load"; not
+  needed at this scale).
+- Any change to the isolation model, the bootstrap step, or the FTS5 (CORE) lane.
+
+## 7. Version
+**v2.5.4** (patch — a DENSE-lane performance fix; `plugin.json` + `marketplace.json` in lockstep).

scripts/compound-v-memory.py

@@ -538,19 +538,17 @@ def release_lock(fd):
 # --------------------------------------------------------------------------- #
 # refresh / indexing
 # --------------------------------------------------------------------------- #
-def reindex_file(conn, root, rel, embedder):
-    """Atomically replace one file's chunks (+ optional embeddings). Triggers sync FTS."""
+def _persist_chunks(conn, root, rel, chunks, vecs):
+    """Atomically replace one file's chunks (+ optional embeddings) and update indexed_files;
+    triggers the sync FTS. A None/short `vecs` (or a None element) degrades that chunk to a NULL
+    embedding — never crashes. Returns the chunk count."""
     abspath = os.path.join(root, rel)
-    chunks = chunk_file(abspath, rel)
-    vecs = None
-    if embedder is not None and chunks:
-        vecs = embedder([c["text"] for c in chunks])
     conn.execute("BEGIN IMMEDIATE")
     try:
         conn.execute("DELETE FROM chunks WHERE path=?", (rel,))
         for i, c in enumerate(chunks):
             blob = None
-            if vecs is not None and i < len(vecs):
+            if vecs is not None and i < len(vecs) and vecs[i] is not None:
                 blob = json.dumps(vecs[i]).encode("utf-8")
             conn.execute(
                 "INSERT INTO chunks(path,chunk_index,heading,text,doc_type,date,embedding) "
@@ -570,6 +568,36 @@ def reindex_file(conn, root, rel, embedder):
     return len(chunks)
 
 
+def reindex_file(conn, root, rel, embedder):
+    """Per-file (re)index: chunk -> optional per-file embed -> persist. Used when embeddings are
+    OFF (FTS5-only) and as a fallback. When many files are embedded at once, cmd_refresh uses
+    reindex_batch so the isolated-venv embedder loads the model ONCE, not once per file."""
+    abspath = os.path.join(root, rel)
+    chunks = chunk_file(abspath, rel)
+    vecs = None
+    if embedder is not None and chunks:
+        vecs = embedder([c["text"] for c in chunks])
+    return _persist_chunks(conn, root, rel, chunks, vecs)
+
+
+def reindex_batch(conn, root, rels, embedder):
+    """Re-index many files, embedding ALL their chunks in a SINGLE embedder call — so the isolated
+    venv embedder loads the ONNX model ONCE per refresh instead of once per file. Chunks are
+    flattened in order, embedded together, then the vectors are sliced back per file. Degrade-safe:
+    a None result (embed failed) persists every file with NULL embeddings (FTS5-only). Returns the
+    number of files processed."""
+    per_file = [(rel, chunk_file(os.path.join(root, rel), rel)) for rel in rels]
+    flat = [c["text"] for _, chunks in per_file for c in chunks]
+    all_vecs = embedder(flat) if (embedder is not None and flat) else None
+    offset = 0
+    for rel, chunks in per_file:
+        n = len(chunks)
+        vecs = all_vecs[offset:offset + n] if all_vecs is not None else None
+        offset += n
+        _persist_chunks(conn, root, rel, chunks, vecs)
+    return len(per_file)
+
+
 def _now() -> str:
     return time.strftime("%Y-%m-%dT%H:%M:%S")
 
@@ -627,10 +655,15 @@ def cmd_refresh(args) -> int:
                     if f in missing and f not in to_index:
                         to_index.append(f)
 
-        n_idx = 0
-        for f in to_index:
-            n_idx += 1
-            reindex_file(conn, root, f, embedder)
+        # Embed ALL files' chunks in ONE embedder call (one model load per refresh); the per-file
+        # path stays for the FTS5-only case (embedder is None).
+        if embedder is not None:
+            n_idx = reindex_batch(conn, root, to_index, embedder)
+        else:
+            n_idx = 0
+            for f in to_index:
+                n_idx += 1
+                reindex_file(conn, root, f, embedder)
         for p in removed:
             conn.execute("BEGIN IMMEDIATE")
             conn.execute("DELETE FROM chunks WHERE path=?", (p,))
@@ -1027,6 +1060,29 @@ class A2:
         cmd_refresh(A2())
         check("incremental stable", conn.execute("SELECT COUNT(*) FROM chunks").fetchone()[0] == before)
 
+        # --- v2.5.4: reindex_batch embeds ALL files' chunks in ONE embedder call (model loads once)
+        root = find_repo_root(tmp)
+        rels = [r[0] for r in conn.execute("SELECT path FROM indexed_files")]
+
+        def _enc(t):  # content-dependent scalar so a mis-sliced vector would not match its chunk
+            return float(sum(t.encode("utf-8")) % 1000000)
+        _calls = {"n": 0}
+
+        def _fake_embed(texts):
+            _calls["n"] += 1
+            return [[_enc(t)] for t in texts]
+        reindex_batch(conn, root, rels, _fake_embed)
+        check("reindex_batch: ONE embedder call for many files (model loaded once)",
+              _calls["n"] == 1 and len(rels) >= 2)
+        _rows = list(conn.execute("SELECT text, embedding FROM chunks WHERE embedding IS NOT NULL"))
+        check("reindex_batch: each chunk's vector matches its own text (slicing correct)",
+              len(_rows) > 0 and all(json.loads(bytes(e).decode()) == [_enc(t)] for t, e in _rows))
+        reindex_batch(conn, root, rels, lambda texts: None)   # failed embed -> degrade
+        _tot = conn.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
+        _nul = conn.execute("SELECT COUNT(*) FROM chunks WHERE embedding IS NULL").fetchone()[0]
+        check("reindex_batch: failed embed degrades to NULL (FTS5-only), no crash",
+              _tot > 0 and _nul == _tot)
+
         # lock: a held lock makes a second acquire a no-op (separate open file descriptions)
         fd = acquire_lock(paths["lock"])
         fd2 = acquire_lock(paths["lock"])