task: Performance & memory on large repositories #593

Scope

Umbrella tracker for performance & memory on large repositories — indexing that never finishes, times out, or grows memory without bound on big codebases. Distinct from #390 (crashes/segfaults/memory-safety) and #391 (silent corruption): here indexing is too slow or too memory-hungry, not crashing or lying about success.

Sub-issues

• Linux: cbm_system_info / cbm_default_worker_count don't respect cgroup CPU/memory limits #363 — Linux: cbm_system_info / cbm_default_worker_count don't respect cgroup CPU/memory limits (container over-subscription)
• GLR ambiguity-merge is O(n²) for deeply-nested ambiguous grammars (e.g. Perl), even with the recursion-depth cap #471 — GLR ambiguity-merge is O(n²) for deeply-nested ambiguous grammars (e.g. Perl), even with the recursion-depth cap
• constant timeouts #474 — constant timeouts
• never finishes indexing large python project #524 — never finishes indexing large Python project
• Index failed in large repo #563 — index failed in large repo
• Memory leak: process grows to 50+ GB virtual memory over hours/days, crashes Windows #581 — memory leak: process grows to 50+ GB virtual memory over hours/days, crashes Windows

Related

• Dump-phase crash on large TS monorepo (v0.6.1, darwin-arm64) — pipeline completes through semantic_edges then terminates abnormally before gbuf.dump #317 — dump-phase crash on large TS monorepo (also tracked under stability task: Stability — crashes, segfaults, and memory safety during indexing (14 issues) #390)
• Feature request: --max-memory flag for RAM-constrained hosts #580 — --max-memory / CBM_MAX_MEMORY_MB budget override — tracked via PR feat(mem): CBM_MAX_MEMORY_MB explicit memory-budget override (#580) #586

Acceptance

1. Indexing of a large public repo completes within a bounded time and memory envelope (capture a baseline corpus).
2. Worker-count / memory budget honor container (cgroup) limits and the explicit override.
3. No unbounded growth over long-running sessions — reproduce the leak in a soak test before fixing.

Why one task

These share the indexing throughput + memory-budget infrastructure (worker pool sizing, GLR merge, dump phase, long-running watcher). Grouping them keeps the large-repo experience coherent.

Added 2026-06-29 (triage)

• [bug] query_graph hangs on whole-graph OPTIONAL MATCH (no execution timeout / degradation) on large graphs #601 — whole-graph OPTIONAL MATCH can hang without timeout/degradation
• Crash when calling query_graph #627 — query_graph crash on Windows under large optional-match style query
• [ui] Graph canvas is black on high-DPI/large displays: EffectComposer MSAA render target exceeds GPU allocation (glRenderbufferStorageMultisample too large) due to uncapped setPixelRatio(dpr) #648 — graph UI high-DPI render target allocation failure / black canvas

Added 2026-06-30 (triage)

• auto_index uses tens of GB of memory in a non-git folder containing code #713 — non-git auto_index can ignore auto_index_limit and walk huge trees
• py_lsp: py_eval_expr_type infinite recursion — Python missed in 9f981a1 depth-guard fix #720 — Python LSP expression type recursion can hang indexing at 100% CPU
• UI /api/layout drops connection for large graph when max_nodes is high #726 — graph UI /api/layout drops the connection for large layouts instead of degrading

Added 2026-07-09 (triage)

• Memory not released after indexing: 20GB+ RSS for 5MB of indexed data #832 — Windows: memory not released after indexing, 20GB+ RSS for 5MB indexed
• watcher: continuous auto-sync re-indexing churn on Windows (real driver of #832) #841 — watcher: continuous auto-sync re-indexing churn on Windows (real driver of Memory not released after indexing: 20GB+ RSS for 5MB of indexed data #832)
• Watcher can repeatedly re-index dirty repos and cause large disk write amplification #937 — watcher can repeatedly re-index dirty repos → large disk write amplification
• Artifact bootstrap is a no-op across clone paths (project name derived from absolute repo path) #885 — artifact bootstrap is a no-op across clone paths (project name derived from absolute repo path)

(#867 incremental-cost family has the active PR stack #868/#869/#870.)

Added 2026-08-03 (triage)

• synchronous=FULL + 64KB page_size under WAL causes severe write amplification during normal indexing (~733 writes/sec, ~1.2TB lifetime writes for a 27MB DB) #1419 — WAL/write amplification and effective SQLite pragma mismatch
• Daemon peak RSS scales ~linearly with query concurrency (29 MB idle -> 3.07 GB at 20 concurrent queries) and releases slowly #1362 — daemon peak RSS scales near-linearly with concurrent queries