Skip to content

MCP worker: panic hook panics recursively and aborts with multi-GB core dumps; separately OOMs at 110 GB RSS #65

Description

@bartoszmajsak

Summary

Over a single day of normal MCP use on one repo (~1.4M LOC Go/Kubernetes project), infigraph-mcp 3.2.16 hit three distinct failure modes: two OOM kills at ~110 GB RSS, six segfaults, and repeated aborts that each dumped a multi-gigabyte core file into the current working directory.

The aborts are the easiest to fix and they make everything else worse, so I'll lead with those.

1. The panic hook panics, recursively, and turns every panic into an abort

install_panic_hook logs through mcp_log, which uses _eprint. When the write to stderr fails - which it does the moment the MCP pipe to the agent is gone - _eprint panics with failed printing to stderr. That panic re-enters the hook, which calls mcp_log again, which panics again:

#3  std::sys::pal::unix::abort_internal
#4  std::process::abort
#5  std::panicking::panic_with_hook
...
#10 std::io::stdio::_eprint
#11 infigraph_mcp::mcp_log
#12 infigraph_mcp::install_panic_hook::{closure#0}
#13 std::panicking::panic_with_hook
...
#18 std::io::stdio::_eprint
#19 infigraph_mcp::mcp_log

It only stops because Rust's panicked-while-panicking guard calls abort().

Two consequences:

  • Any recoverable panic in a worker becomes a hard abort.
  • Because the process aborts rather than exits, it dumps its entire heap. The two cores this left in my repo are 1.7 GB and 1.1 GB.

It also explains why ~/.infigraph/mcp.log contains no error lines at all across all of these crashes - the logging call is the thing that dies, so nothing is ever recorded. From the outside it just looks like the server vanished.

Suggested fix: the panic hook must not use eprintln!/_eprint. Write to a raw File/Stderr handle and discard the result, or route it to the log file with the error swallowed. A panic hook that can itself panic is never safe.

2. OOM: 8.7 TB virtual, 110 GB resident

Twice today, killed by the kernel OOM killer:

Aug 21 17:35:19 kernel: Out of memory: Killed process 490216 (infigraph-mcp)
  total-vm:8700240216kB, anon-rss:105359944kB, file-rss:2772kB, pgtables:542476kB

Aug 21 18:43:10 kernel: Out of memory: Killed process 695487 (infigraph-mcp)
  total-vm:8700240216kB, anon-rss:110447384kB, file-rss:3000kB, pgtables:537648kB

That is 8.7 TB of address space and ~110 GB resident in a single process, and the identical total-vm in both suggests one specific allocation path rather than gradual growth.

This is related to #46 but not the same thing. #46 describes orphaned watchers pinning ~100-425 MB each and accumulating; I see that too (below), but no amount of accumulation puts 8.7 TB of VA into one process.

3. Segfaults

Six today, all with the identical signature - a write to a near-null pointer, on a worker thread, inside libc:

Aug 21 13:56:43 kernel: infigraph-mcp[3931683]: segfault at 40 ip 00007f2d775b81e2 sp 00007713685fcd78 error 6 in libc.so.6[1b81e2,...]
Aug 21 15:58:25 kernel: infigraph-mcp[139315]:  segfault at 40 ip 00007f84eb7b81e2 sp 00007f84dd3fcf38 error 6 in libc.so.6[1b81e2,...]
Aug 21 17:29:48 kernel: infigraph-mcp[486288]: segfault at 40 ip 00007fc6ddfb81e2 sp 00007fc6d1ffcf38 error 6
Aug 21 17:30:39 kernel: infigraph-mcp[488537]: segfault at 40 ip 00007f08071b81e2 sp 00007f07f1ffcf38 error 6 in libc.so.6[1b81e2,...]
Aug 21 18:36:06 kernel: infigraph-mcp[688871]: segfault at 40 ip 00007f6c95db81e2 sp 00007f6c91bfbf38 error 6 in libc.so.6[1b81e2,...]
Aug 21 18:38:08 kernel: infigraph-mcp[693914]: segfault at 40 ip 00007f2f56db81e2 sp 00007f2f35ffcf38 error 6 in libc.so.6[1b81e2,...]

error 6 is a user-mode write to a not-present page, faulting address 0x40, always at the same offset into libc. Given the OOM behaviour above, an unchecked allocation result is a plausible common cause, but I haven't confirmed that.

4. Worker processes accumulate and never exit

12 live infigraph-mcp processes at the time of writing, the oldest 1h32m old:

    PID   RSS      VSZ     ELAPSED COMMAND
 533030 258336  3432384  01:32:42 infigraph-mcp
 739445  61344  3331740     22:12 infigraph-mcp
 761647  16852  3297968     13:53 infigraph-mcp
 739349  16848  3297968     22:12 infigraph-mcp
 721806  16688  3297968     27:03 infigraph-mcp
 ... 7 more

~/.infigraph/mcp.log shows a fresh server starting every ~500s, each one losing the lock race:

[...] WARN: Another MCP instance holds the lock — running without watchers
[...] INFO: MCP server started
[...] INFO: stdin loop exited

stdin loop exited is logged but the process stays resident. This looks like the same shape as #9, which was closed as macOS-specific - this is Linux.

5. Core dumps land in the user's working directory

Not an infigraph bug as such, but worth knowing: on a system where kernel.core_pattern is a relative path, the aborts in section 1 drop gigabyte-scale cores into whatever directory the worker happened to be in, which is the user's repo. I had 2.8 GB of core.<pid> files sitting next to my source. Fixing section 1 removes the abort, and with it these.

Environment

  • infigraph 3.2.16 (mise-installed binary), MCP server via Claude Code
  • CachyOS Linux, kernel 7.1.8-1-cachyos, x86_64
  • Repo under analysis: kserve/kserve (Go, ~1.4M LOC), indexed with embeddings (embeddings.bin 111 MB, graph 91 MB)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions