Skip to content

bug: memory runaway (12.8GB+ RSS) during index_repository on root paths triggers OOM crash #1241

Open

Description

@kodexArg

Note: This issue was audited and submitted automatically by an AI agent on behalf of @kodexArg.

Summary

When running index_repository on a broad root directory (~ / /home/<user>), codebase-memory-mcp recursively traverses non-project system directories (Docker overlays Dockers/data/, Go module caches go/pkg/mod/, Steam runtimes, and local LLM assets), causing severe memory runaway (12.85 GiB RSS / 27.8 GiB VSZ) that triggers the system-wide Linux OOM killer.

Environment

  • Version: 0.8.1
  • OS: Linux (Debian Sid x86_64)
  • RAM: 14 GB Total

Empirical Evidence (Kernel OOM Log)

kernel: oom-kill:constraint=CONSTRAINT_NONE,global_oom,task=codebase-memory,pid=595953
kernel: Out of memory: Killed process 595953 (codebase-memory) total-vm:27850072kB, anon-rss:12855988kB

Root Cause

  1. Unbounded Traversal: index_repository parsed over 237,000 files without default exclusions for Docker overlay volumes, package caches, or system headers.
  2. Uncapped Heap Expansion: Multithreaded parsing outpaced memory cleanup, breaching the auto-calculated budget (budget_mb=7678) and consuming ~13 GB RAM until kernel termination.

Suggested Fix / Improvements

  • Add default ignore filters for system/container directories (overlay2/, Dockers/, go/pkg/mod/, .cache/).
  • Provide configurable RAM cap flags to restrict heap allocation during large indexing runs.

Activity

  1. added
    bugSomething isn't working
    ux/behaviorDisplay bugs, docs, adoption UX
    priority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.
    on Jul 24, 2026
  2. added this to the 0.9.1-rc milestone on Jul 24, 2026
  3. DeusData commented on Jul 24, 2026

    @DeusData
    Owner

    Thanks. This report is from v0.8.1, while 0.9.0 changed worker supervision and memory accounting, so we need one current measurement before attributing the same failure. Please retest 0.9.0 against one explicit project root rather than the whole home/root tree, with CBM_DIAGNOSTICS=1, and share the discovered-file count plus the final error and peak RSS. Current main already excludes common caches such as .cache, node_modules, and vendor; a remaining runaway on an explicit project is actionable under #593 and 0.9.1-rc.

  4. azizramdan commented on Jul 28, 2026

    @azizramdan

    I also experienced this problem even though it was not in the root path

    $ codebase-memory-mcp --version                       
    codebase-memory-mcp 0.9.0
    $ CBM_DIAGNOSTICS=1 codebase-memory-mcp cli --progress index_repository --repo-path /home/velo/projects/laravel-project
    level=info msg=mem.init budget_mb=11087 total_ram_mb=31678
    [1]    1137311 terminated  CBM_DIAGNOSTICS=1 codebase-memory-mcp cli --progress index_repository
    
    $ sudo dmesg -T | grep -i -E 'oom|killed process'
    [Tue Jul 28 17:56:03 2026] IPDL Background invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=100
    [Tue Jul 28 17:56:03 2026]  oom_kill_process.cold+0xa/0xb5
    [Tue Jul 28 17:56:03 2026] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
    [Tue Jul 28 17:56:03 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/ptyxis-spawn-1484b17c-3042-4650-b18f-b6633ff3963e.scope,task=codebase-memory,pid=1137312,uid=1000
    [Tue Jul 28 17:56:03 2026] Out of memory: Killed process 1137312 (codebase-memory) total-vm:35951240kB, anon-rss:23103436kB, file-rss:316kB, shmem-rss:0kB, UID:1000 pgtables:45604kB oom_score_adj:200
  5. DeusData commented on Jul 28, 2026

    @DeusData
    Owner

    Thanks for the 0.9.0 retest on an explicit Laravel project. This materially broadens the issue beyond indexing a home/root directory.

    The process reached about 23.1 GB RSS while mem.init reported an 11.1 GB budget. On current main that budget drives pressure signals and selected pipeline sizing/cleanup decisions; it is not a process-wide allocator ceiling, so this evidence shows those controls are not containing the affected path. The kernel OOM record also removes client-timeout ambiguity.

    This remains priority/high in 0.9.1-rc, now as a general explicit-project memory runaway rather than only an unsafe-root traversal case. Thank you for providing the exact kernel evidence.

  6. azizramdan commented on Aug 11, 2026

    @azizramdan

    Just updated to version 0.10.0, and it's working now for the same Laravel project repo

  7. EvilBeaver commented on Aug 12, 2026

    @EvilBeaver

    large monorepo (approx. 5Gb of source code) version 0.10.2, same issue, OOM Killer on 15Gb of RAM used

  8. DeusData commented on Aug 12, 2026

    @DeusData
    Owner

    Thank you @EvilBeaver. This is fresh evidence that the memory runaway persists on v0.10.2 for a large monorepo, after the earlier Laravel reporter saw success on v0.10.0.

    As already established in this thread, the current memory budget drives pressure signals and selected sizing/cleanup decisions; it is not a process-wide RSS ceiling. The OOM-killer result is therefore sufficient to keep this priority/high and active under #593. If readily available, a CBM_DIAGNOSTICS=1 trajectory plus approximate file count and language mix would help isolate the growing phase, but the issue is not waiting on that information.

  9. DeusData commented on Sep 26, 2026

    @DeusData
    Owner

    Thank you, @kodexArg, @azizramdan and @EvilBeaver! The kernel OOM records and the per-version retests showed us this is a general memory problem, not just a home-directory one.

    In v0.11.0, indexing allocates through a central memory core. Extraction slows down, spills to disk under the budget and stops cleanly instead of overshooting (PR #2202). The worker budget is also split across active jobs. On our benchmarks that cut peak memory by roughly 55–80% compared with v0.10.8. One remaining gap, in the phases after extraction, is being handled in a follow-up tracked in #2184.

    @EvilBeaver, could you retry your monorepo on v0.11.0 with CBM_DIAGNOSTICS=1? We'd like to know whether it now finishes, or stops with a budget message instead of being OOM-killed.

  10. DeusData commented on Oct 19, 2026

    @DeusData
    Owner

    Thank you again for the per-version and kernel OOM evidence. The last update requested the large-monorepo result on v0.11.0 and separated the remaining post-extraction work in #2184. I have no newer result from that workload to add; the earlier Laravel success is not proof for all large repositories.

  11. EvilBeaver commented on Oct 23, 2026

    @EvilBeaver

    @EvilBeaver, could you retry your monorepo on v0.11.0 with CBM_DIAGNOSTICS=1? We'd like to know whether it now finishes, or stops with a budget message instead of being OOM-killed.

    @DeusData ok, I'll try it.

  12. EvilBeaver commented on Oct 23, 2026

    @EvilBeaver

    @DeusData version 0.12.1 - full monorepo indexed without OOM.

  13. DeusData commented on Oct 25, 2026

    @DeusData
    Owner

    Thank you for the 0.12.1 retest: completing the full monorepo without OOM is the result we needed. The earlier Laravel success and this workload confirmation are now both recorded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.stability/performanceServer crashes, OOM, hangs, high CPU/memoryux/behaviorDisplay bugs, docs, adoption UX

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions