fix(embed): shrink over-context items; honour the positional project dir (self-host loop, wave two) (#405)

* fix(embed): shrink over-context items; honour the positional project dir

Two defects surfaced by embedding the first portfolio child
(games/the-survival):

- embedFiles rejected (Failed++) any item the provider answered 500 to, but
  the local text budget (1000 runes) is a heuristic and dense content exceeds
  the model's token context even at the cap (real Godot/TypeScript: 523
  tokens in 1000 runes). Now a provider-rejected item is HALVED and retried
  (up to 3 shrinks); successes after a shrink count in Report.Truncations;
  Failed is reserved for items that fail at every size. The all-failed abort
  still fires for a genuinely dead provider (TestRunAllFailedProvider-
  AbortsLoudly), and the repo's own 14 over-context items now embed (Coverage
  1.0).

- leankg-embed's verbs (run/full/export/import/status) stopped Go-flag parsing
  at the first non-flag arg, so a trailing positional project dir was silently
  dropped and the command embedded the CWD's store instead — the same footgun
  #370 fixed for index, reborn in the other binary (hit dogfooding: 'run
  /abs/child' wrote into the wrong project). bindPositional binds one
  positional to --project (explicit --project wins; >1 is a hard error).

* docs(prd): record wave two of the self-host loop — shrink retry, S3 first parent, embed positional fix

games/ registered T0 (zero eager), child the-survival embedded to Coverage
1.0 after the retry fix, portfolio fan-out attributed per-child with no
restart. Tracker: FR-SELF-02/03 IN_PROGRESS with evidence; module-to-root
relocation held as #403.

Author: linhdmn

docs/prd-task-tracker.md

@@ -1,6 +1,6 @@
 # LeanKG Task Tracker
 
-**Last synced:** 2026-09-14 — v4.11.2 **selfhost-validated wave**: S1 executed for real (dynamic HTTP server + `llama-server` bge-small sidecar over this repo's own index: 9,025 elements / 8,989 vectors / coverage 0.9984; all five gates green, memory survives restart). **FR-SELF-01 → DONE, FR-SELF-02 → IN_PROGRESS.** Four dogfood defects fixed failing-first: local embed text budget (chunker v2), per-item provider-failure fallback (a poison element no longer aborts a run), `leankg gc` implemented to make doctor's own hint real (5,810 dangling edges purged live; orphan check FAIL→PASS), held-lock doctor probe + PID stamping, + two test-hermeticity fixes. Test-layer policy codified (AGENTS + PRD §6): unit in-process+fast, real calls env-gated, **benchmarks never in CI**. Evidence: [§v4.11.2](prd.md#v4112-selfhost-validated--the-self-host-ran-on-itself-every-defect-it-found-is-fixed-2026-09-14).
+**Last synced:** 2026-09-14 — v4.11.2 wave two (same branch as the S1 record): the loop's first S3 step ran live — `games/` parent registered T0 (zero eager), child `the-survival` indexed+embedded (132/132, Coverage 1.0), portfolio fan-out attributed per-child, vanished demo registrations pruned. Two more dogfood defects fixed failing-first: dense content exceeded the 1000-rune budget (523 tokens/1000 runes on Godot/TS) → `embedFiles` now shrink-retries per item (repo's own 14 poison items embedded too → **Coverage 1.0 everywhere**); and `leankg-embed <verb> <dir>` silently ignored the positional dir (embedded the CWD store — the #370 footgun in the other binary) → `bindPositional` for all five verbs. **FR-SELF-01 DONE, FR-SELF-02 running (6 defects fixed), FR-SELF-03 running (first parent+child live).** Module-to-root relocation scoped and HELD as issue #403 (owner: after the parallel refactor waves land).
 **Previous syncs:** v4.11.1 ship-surface (the deploy/brand/module work FR-SELF-01 was gated on — Docker/Render service, `go/vX.Y.Z` mirror, `serve --ui` `/health`); v4.11.0 self-host dogfood loop anchored (§3.10 + M10 + runbook, FR-SELF-01..04 added).
 **SoT pairing:** narrative + ACs live in [`docs/prd.md`](prd.md); statuses live here.
 **Status legend:** `IN_PROGRESS` (being worked now) · `TODO` (backlog, ordered) · `DONE` (implemented + verified) · `BLOCKED` (needs external input) · `WONT_DO` (explicitly cancelled).
@@ -11,10 +11,10 @@
 
 | Status | Count |
 |--------|------:|
-| IN_PROGRESS | 2 (FR-ZCP-01; **FR-SELF-02** — the dogfood loop is running: first wave = 4 engine fixes from the v4.11.2 self-host run) |
-| TODO | 38 (9 live + 26 carry-forward + FR-GO-DASH #371 + FR-SELF-03..04) |
+| IN_PROGRESS | 3 (FR-ZCP-01; **FR-SELF-02** — 6 engine defects found+fixed by the loop; **FR-SELF-03** — first parent `games/` + child `the-survival` live) |
+| TODO | 37 (9 live + 26 carry-forward + FR-GO-DASH #371 + FR-SELF-04) |
 | DONE | 10 (FR-SELF-01: self-host bootstrapped + all five S1 gates verified live, v4.11.2) |
-| Open work | 42 (36 archived-inventory + 3 Go-rewrite slices pending PR #370 merge + FR-GO-DASH #371 + 2 remaining M10 loop items; FR-SELF-02 IN_PROGRESS, FR-SELF-01 DONE) |
+| Open work | 42 (36 archived-inventory + 3 Go-rewrite slices pending PR #370 merge + FR-GO-DASH #371 + FR-SELF-04; FR-SELF-01 DONE, FR-SELF-02/03 IN_PROGRESS) — relocation of the Go module to the repo root tracked as #403 (held) |
 
 **Inventory note (ID-level accounting):** the archived tracker holds **40 open inventory items** (35 master-table `NOT_DONE`/`PENDING`/`PARTIAL`/`OPEN` IDs + 5 `FR-HEA-*` section-table rows). All 40 are accounted for below: FR IDs appear as named rows; each paired `US-*` tracks with its FR (the archive itself pairs them `US-X / FR-X` as one work item); `FR-ZG-01..05` + `US-ZG-01..05` + `FR-B05` are superseded inside the live `FR-ZCP-*` rows (Supersedes column); `FR-HEA-05` is DONE (v4.0.0 §1 cutover). `FR-ZCP-09/10/11/12/13` are **new in v4.1.x–v4.3.0** (no archive IDs). Row-level open work = 1 IN_PROGRESS + 9 live + 26 carry-forward = 36. (The 26 carry-forward rows cover 35 archived open IDs: 3 rows pair multiple US stories with their FR; the inventory not…
 
@@ -29,7 +29,7 @@
 | M7 — Embedding correctness | 1 | — | **DONE** (FR-ZCP-11 closed by #279: pinned catalog with 40-hex revisions and query/document prefixes, whole-identity `ModelStamp` incl. `chunker_version`, hard rebuild guard on both write paths **and** the read path, 3-signal detection, per-file atomic replace + truncation accounting, watcher reconciliation. Remaining: single-flight indexing — see §6b remainders) |
 | M8 — Measured simplicity | 1 | — | **DONE** (FR-ZCP-12 T1 error catalog c5b4b991; T2 TTFV gate #280 — measured 21.6 s cold against a 300 s budget in CI; T3 superseded by v4.3.1's CI-enforced one-tool invariant) |
 | M9 — Three tools + dual backend | 4 | — | **IN_PROGRESS** (FR-3T-01/02/03 DONE; FR-3T-04 live validation complete on this repo — 581 files, 9522 vectors, L1/L2/L3 verified; PR #284 merged (v4.3.x)) |
-| M10 — Self-host dogfood loop | 4 | — | **IN_PROGRESS** (FR-SELF-01 **DONE** v4.11.2 — self-host live: 9,025 elements / 8,989 vectors, all five S1 gates green; FR-SELF-02 running (4 dogfood defects fixed); FR-SELF-03..04 TODO behind it — see §M10 below) |
+| M10 — Self-host dogfood loop | 4 | — | **IN_PROGRESS** (FR-SELF-01 **DONE** v4.11.2; FR-SELF-02 running — 6 dogfood defects fixed; FR-SELF-03 running — first portfolio parent+child verified live (Coverage 1.0, per-child attribution, zero eager indexing); FR-SELF-04 is the steady state — see §M10 below) |
 | M-GO — Go engine rewrite (#365) | 3 | — | **IN_PROGRESS** (FR-GO-W1 core DONE on feat/go-rewrite: store/core/index/mcp/rest + live smoke; FR-GO-EMBED DONE: leankg-embed binary + stamp guards + NDJSON; FR-GO-MEM DONE: full-markdown memory + banks adapter; v4.6.0: ALL waves landed (W2 watcher/writer, W4 pgvector, W5 ConnectRPC+auth, session, graph verbs, goldens, benchmarks + executed Rust-vs-Go A/B REPORT) and the Rust tree REMOVED — deferred ledger in docs/prd.md) |
 | FR-GO-LANGS | Lazy language wave (v4.7.0): 13-language registry + tstree/astgrep/lsp tiers + java/kotlin/swift/objc/dart extractors | **DONE** on feat/go-rewrite (objc/dart tree-sitter grammar gap documented) |
 | FR-GO-DASH | #371: port the ui-v2 dashboard data API — legacy `/api/*` (11 endpoints) or rebuild ui-v2 against `/api/v1/*`; today the SPA fallback answers those calls with `index.html`, so the embedded dashboard loads no data | 2026-09-11 | **TODO** — ledger row `web api+ui` corrected to PARTIAL in v4.7.1 |
@@ -60,7 +60,7 @@
 |----|-------|----------|--------|----------------------|
 | FR-SELF-01 | Bootstrap the self-host: dynamic `leankg serve --http :9699 --rest :9700 --ui :9701 --memory --embed-provider local` over this repo (9700/9701 — the live onegw gateway owns 8080); index + embed (pinned `LEANKG_EMBED_*` identity) + memory live | **P0** | **DONE 2026-09-14 (v4.11.2)** — sidecar: `llama-server` bge-small-en-v1.5 f16 GGUF @ 127.0.0.1:8085; store: 9,025 elements / 19,662 edges / 8,989 vectors (coverage 0.9984, chunker-v2 stamp `ea104dace…`); gates verified live: `/health` ×3, L1/L2/L3 with provenance (L3 = cosine on real vectors), retain→recall survives restart, `doctor --deep` 0-fail after the wave's `leankg gc` run | exit criteria all met — evidence in [§v4.11.2](prd.md#v4112-selfhost-validated--the-self-host-ran-on-itself-every-defect-it-found-is-fixed-2026-09-14) |
 | FR-SELF-02 | Build LeanKG with LeanKG: every exported-symbol change informed by `impact`/`callers`, every close by tested-by/traceability, every session-open by `session_recall`; wrong answers filed as engine defects, fixed failing-test-first against this repo's data, re-indexed, re-asked | **P0** | **IN_PROGRESS 2026-09-14** — the loop is producing: first wave fixed 4 defects found by running the pipeline/doctor on this repo's own data (embed 512-token budget + per-item provider-failure fallback; `leankg gc` implemented to make doctor's own hint real — purged 5,810 dangling edges live; held-lock probe + PID stamps; sidecar-test port hermeticity) | PRs cite graph evidence; dogfood findings enter this tracker; zero regressions attributed to missing graph context |
-| FR-SELF-03 | Scale to nested-repo parents: add small parents one repo at a time via registry + `LEANKG_PROJECT_DIRS` (T0 manifest, T1 hot-set cap 8, zero eager indexing). Hard guard: never bulk-index the `freepeak` root (99,574 files, multi-GB store) | P1 | TODO (gate: S1+S2 smooth) | Portfolio fan-out attributes per child; per-child freshness honest; adding a repo = one register/index, no restart |
+| FR-SELF-03 | Scale to nested-repo parents: add small parents one repo at a time via registry + `LEANKG_PROJECT_DIRS` (T0 manifest, T1 hot-set cap 8, zero eager indexing). Hard guard: never bulk-index the `freepeak` root (99,574 files, multi-GB store) | P1 | **IN_PROGRESS 2026-09-14** — first increment verified: `games/` registered T0 (portfolio answered honestly `n/a` for the unindexed parent); `the-survival` child promoted (index 132 elements → embed Coverage 1.0 after the shrink-retry fix); `portfolio` query attributed hits to the child, no restart anywhere. Next increments: one child at a time; keep doctor honest | Portfolio fan-out attributes per child; per-child freshness honest; adding a repo = one register/index, no restart |
 | FR-SELF-04 | Keep building, keep fixing: the loop is the end state — every wave surfaces defects, engine fixes ship via release-please, this PRD + tracker stay the status ledger | **P0** | TODO (steady state after S1–S3) | Dogfood-found defect rate > 0 with fix rate keeping pace; no milestone regresses; self-host up whenever development happens |
 
 The deploy/brand/module surface FR-SELF-01 was gated on shipped as `REL-SHIP-01..04` (§Done).
@@ -139,7 +139,7 @@ The deploy/brand/module surface FR-SELF-01 was gated on shipped as `REL-SHIP-01.
 | REL-SHIP-05 | The deploy verified **live**, not just in a container: Render rebuilt from the restored Dockerfile after two consecutive `build_failed` deploys | `dep-dajp5slg1s2s73cln2o0` → `live`; <https://leankg.onrender.com> `/` 200 shell, `/health` `{"ok":true}`, `/favicon.svg` the new mark, `/api/index/status` → ~4.6k elements / ~14.7k relationships (rebuilt from the tree each image build, so the counts move), `/api/search` + `/api/graph/clusters` answering |
 | REL-SHIP-06 | `go/README.md` (the pkg.go.dev module page) rewritten — it was the W1 slice note: shipped backends listed as deferred, a nonexistent `internal/ports/`, "7 packages" against 54, a moved doc link, Rust described as still maintained | Every claim read back against the code; both `go install` lines run for real |
 
-*Last updated: 2026-09-14 (v4.11.1 — ship-surface wave: `REL-SHIP-01..04` closed (Go `Dockerfile` + Render rebuild, `serve --ui` `/health`, `go/vX.Y.Z` module tag + `go/v0.31.3`, brand mark + README badges/claims), which satisfies the FR-SELF-01 gate; running the self-host over this repo is the next action. Prior wave: v4.11.0 — self-host dogfood loop anchored, FR-SELF-01..04 / M10 added)*
+*Last updated: 2026-09-14 (v4.11.2 selfhost-validated + wave two: S1 all gates green, 6 dogfood defects fixed failing-first (embed budget/shrink, poison-item fallback, `leankg gc`, held-lock probe, sidecar-test + verb-positional hermeticity), Coverage 1.0 on both live stores, first S3 parent+child verified; module-to-root held as #403. Prior wave: v4.11.1 ship-surface, REL-SHIP-01..06)*
 
 ## Repo hygiene (non-PRD)
 

docs/prd.md

@@ -13,11 +13,11 @@
 
 ### v4.11.2-selfhost-validated — the self-host ran on itself; every defect it found is fixed (2026-09-14)
 
-**Trigger:** M10/S1 executed for real on this repository. The dynamic HTTP server came up over this checkout (`leankg serve --http :9699 --rest :9700 --ui :9701 --memory --embed-provider local`) with a pinned local embedder (`llama-server` serving `bge-small-en-v1.5` GGUF, 384-d). All five S1 gates passed; the run then surfaced four real engine defects, each fixed with a failing-first test.
+**Trigger:** M10/S1 executed for real on this repository. The dynamic HTTP server came up over this checkout (`leankg serve --http :9699 --rest :9700 --ui :9701 --memory --embed-provider local`) with a pinned local embedder (`llama-server` serving `bge-small-en-v1.5` GGUF, 384-d). All five S1 gates passed; the run (and the first S3 step) then surfaced six real engine defects, each fixed with a failing-first test.
 
 **S1 self-host — verified live, not inferred:**
 - **Index:** this repo reconciled into sqlite (`9,025 elements`, `19,662 relationships`, `807 files`); `doctor --deep` → `orphaned-relationships PASS`, `duplicate-names PASS`, `migrations 14/14`.
-- **Embed:** 8,989 vectors from the pinned catalog model; `ChunkerVersion 2`; the `Failed`/`Truncations`/`Coverage` report surfaced honestly (`Coverage 0.9984`, 14 over-context items counted, not fatal). Stamp `bge-small-en-v1.5-384|local|384|cosine|ea104dace…|2` written to `emb_stamp`.
+- **Embed:** 8,989 vectors from the pinned catalog model; `ChunkerVersion 2`; the `Failed`/`Truncations`/`Coverage` report surfaced honestly (first pass: `Coverage 0.9984`, 14 over-context items counted, not fatal — after this wave's shrink-retry they embedded too: **Coverage 1.0**). Stamp `bge-small-en-v1.5-384|local|384|cosine|ea104dace…|2` written to `emb_stamp`.
 - **Ladder:** L1 (`exact identifier match`), L2 (`FTS5 keyword match`), **L3 (`vector similarity (cosine)`)** all answered over REST `POST /api/v1/query` with `retrieval` provenance on the repo's own embeddings.
 - **Memory:** `session/retain` → `written=2`; `session/recall` round-trips; **survives a full server restart** (read back after re-launch).
 - **Health:** `/health` → `{"ok":true}` on MCP (9699), REST (9700), and UI (9701).
@@ -31,6 +31,10 @@
 | `doctor --deep` orphan hint told users to purge leftovers with `leankg gc` — **a verb that did not exist** (it named the deliberately-dropped Rust `gc.rs`) | a mass deletion (v4.10.1) leaves dangling edges that incremental index never revisits, with no repair path | implemented `leankg gc` over a new `store.DeleteOrphanRelationships` (both engines; watermark bumps only when rows go) — purged 5,810 edges live, orphan check FAIL→PASS |
 | `checkLeankgDir` warned on any `*.lock` file's mere **existence**; `embed.lock`/`watch.lock` are persistent kernel-released flocks | every successful embed left a permanent "stray lock file" WARN, and the hint to `cat <lock>` for a PID was a lie (the PID was never written) | check now probes **HELD** via `LOCK_EX|LOCK_NB` (idle token → PASS); embed/watch stamp the owning PID so the WARN names it |
 | `TestSidecarExitsDuringStartup` hardcoded port `18080` | a dev ssh tunnel answering `/health` there made "the sidecar died" a silent **false pass** | use `freeTCPPort()` (the rest of the file already did); `TestLeankgDir` likewise rewritten to assert idle→PASS / held→WARN / released→PASS |
+| Dense content still exceeded the 1000-rune budget (real Godot/TS: **523 tokens in 1000 runes**) — every item of the first portfolio child was rejected, and the all-failed guard misread that as "provider down" | `leankg-embed run games/the-survival` died 14/14 with status `failed` | `embedFiles`' per-item path now **halves and retries** (≤3 shrinks, successes counted in `Truncations`); the abort is reserved for a provider that fails at any size. Live: 132/132 embedded, Coverage 1.0 (64 via shrink); the repo's own 14 poison items embedded too → repo Coverage 1.0 |
+| `leankg-embed <verb> <dir>` **silently ignored** the positional dir (Go's flag package stops at the first non-flag arg) and embedded the **cwd** store — the #370 index-root footgun reborn in the other binary | `run /abs/child` quietly wrote vectors into the wrong project | `bindPositional` binds the first positional to `--project` for all five verbs (explicit `--project` wins; >1 positional is a hard error); `cmd/leankg-embed/main_test.go` pins it |
+
+**S3 first increment — verified on the loop's second real parent:** `games/` (6 nested repos, ~1.2k code files) registered as T0 with zero eager indexing (`elements=0`, store `missing`, portfolio answers `n/a` honestly); one child (`the-survival`) promoted to its own indexed+embedded store; a `portfolio` query over the live server attributed its hits to `the-survival` per-child with `freshness: fresh` — no restart at any step. Vanished fixture registrations pruned with `leankg projects --forget`. The relocation of the module to the repo root (pkg.go.dev root URL) is scoped and held as **#403** behind the parallel refactor waves.
 
 **Test-layer policy (owner decision, codified in `AGENTS.md` + §6):** unit tests stay in-process and fast (no real provider/network/sleeps — `go test ./...` is the gate); anything making a real embed/LLM/PG call is an integration/e2e test gated on env (`LEANKG_TEST_PG_URL`, live sidecar) or its own job (`ttfv`); **benchmarks never run in CI**. The live self-host run above is the integration tier, executed by hand and recorded here, not a CI unit test.
 

go/cmd/leankg-embed/main.go

@@ -22,6 +22,7 @@ import (
 	"os"
 	"os/signal"
 	"path/filepath"
+	"strings"
 	"syscall"
 
 	"github.com/FreePeak/LeanKG/go/internal/embed"
@@ -105,6 +106,29 @@ func envOr(key, fallback string) string {
 	return fallback
 }
 
+// bindPositional lets the project dir be given as a bare positional
+// (matching `leankg index <dir>`): Go's flag package STOPS parsing at the
+// first non-flag arg, so `leankg-embed full <dir>` previously left <dir>
+// unparsed and silently embedded the CWD's store instead. That footgun was
+// hit dogfooding the portfolio (run /abs/child embedded the wrong project).
+// The positional wins only when --project was not also passed; more than one
+// positional is a hard error, never a silent drop.
+func bindPositional(verb string, args []string, project *string) {
+	var positional []string
+	for _, a := range args {
+		if len(a) > 1 && a[0] == '-' {
+			break
+		}
+		positional = append(positional, a)
+	}
+	if len(positional) > 1 {
+		log.Fatalf("%s: too many arguments (%s)", verb, strings.Join(positional, " "))
+	}
+	if len(positional) == 1 && *project == "" {
+		*project = positional[0]
+	}
+}
+
 // lock takes the single-flight flock for the PROJECT dir (not the process
 // cwd): concurrent invocations against one project serialize; abandoned
 // locks release with the process.
@@ -131,6 +155,7 @@ func cmdRun(args []string, mode string) {
 	if err := fs.Parse(args); err != nil {
 		log.Fatal(err)
 	}
+	bindPositional(mode, fs.Args(), project)
 	unlock, err := lock(projectDir(*project))
 	if err != nil {
 		log.Fatal(err)
@@ -164,6 +189,7 @@ func cmdExport(args []string) {
 	if err := fs.Parse(args); err != nil {
 		log.Fatal(err)
 	}
+	bindPositional("export", fs.Args(), project)
 	if *modelID == "" {
 		log.Fatal("export requires --model")
 	}
@@ -197,6 +223,7 @@ func cmdImport(args []string) {
 	if err := fs.Parse(args); err != nil {
 		log.Fatal(err)
 	}
+	bindPositional("import", fs.Args(), project)
 	if *modelID == "" || *revision == "" || *dims == 0 {
 		log.Fatal("import requires --model, --revision, --dims")
 	}
@@ -233,6 +260,7 @@ func cmdStatus(args []string) {
 	if err := fs.Parse(args); err != nil {
 		log.Fatal(err)
 	}
+	bindPositional("status", fs.Args(), project)
 	st, err := openStore(*project)
 	if err != nil {
 		log.Fatalf("store: %v", err)

go/cmd/leankg-embed/main_test.go

@@ -0,0 +1,26 @@
+package main
+
+import "testing"
+
+func TestBindPositional(t *testing.T) {
+	// A bare positional selects the project (the `leankg index <dir>` convention).
+	p := ""
+	bindPositional("run", []string{"/abs/child"}, &p)
+	if p != "/abs/child" {
+		t.Fatalf("positional ignored: project=%q — the silent-cwd-store footgun", p)
+	}
+
+	// --project wins over the positional.
+	p = "--set"
+	bindPositional("run", []string{"/abs/child"}, &p)
+	if p != "--set" {
+		t.Fatalf("explicit project overwritten: %q", p)
+	}
+
+	// Flag-only args leave project untouched.
+	p = ""
+	bindPositional("run", []string{"--model", "m"}, &p)
+	if p != "" {
+		t.Fatalf("flag value consumed as positional: %q", p)
+	}
+}