nph-zms sometimes consumes a lot of memory

[IgorA100]

For a long time I could not understand why memory goes into swap. Yes, there is not a lot of memory, but during normal work it is used by about 40-60%. When there is a lot of free memory, this problem is even more difficult to catch. But sometimes memory goes into swap. I found out that the nph-zms process is to blame. For some time, memory consumption by some nph-zms processes is small 100-120MB
But then some kind of nph-zms process appears, which very quickly begins to eat up all the memory. Moreover, it seems that the nph-zms process was already born with a large consumption of 1.1GB!!!
19:45:01 - 1163576
19:45:30 - 2614612
19:45:38 - 3042788
19:45:47 - 3450148
19:45:55 - 3851564
19:46:04 - 4297580
19:46:12 - 4737648
19:46:21 - 5136088
19:46:29 - 5576156
19:46:38 - 9061324
........
19:48:05 - 10562060
After that, within 5 seconds the nph-zms process was killed (I don’t know the reason for the death of the process)

I can’t say what leads to a catastrophic increase in memory consumption, because... this may occur 1-2 times a day, or maybe less often....

[connortechnology]

Findings

1. The ~1.1 GB baseline is expected and is MAP_LOCKED shared memory, not a leak.
   Monitor::connect() maps the monitor's /dev/shm segment with MAP_SHARED|MAP_LOCKED (src/zm_monitor.cpp:1099). The segment is image_buffer_count × image_size × 2 (capture ring + analysis ring), sized to the RGBA upper bound (:1006). For
   a 4K monitor that's ~33 MB/slot × 2 × ~20 slots ≈ 1.3 GB, faulted in and locked resident the moment zms attaches. So every nph-zms on a big monitor is "born" at ~1 GB of locked RSS — and because it's MAP_LOCKED it can't be swapped,
   which is very likely why "memory goes into swap during normal work" once several streams run at once. This is worth revisiting on its own (QUERY-purpose readers arguably shouldn't mlock the whole ring).

2. The common live path does not leak. Live view resolves to mode=jpeg with a connkey + buffer (web/includes/Monitor.php), which uses the disk swap buffer (SwapImage[] holds filenames, frames go to zmswap-*.jpg). sendFrame,
   prepareImage, EncodeJpeg, and the JPEG structs all reuse static/persistent buffers. A steady 50 MB/s leak here would hit everyone always, contradicting "sometimes."

3. Most probable ramp culprit — repeated reload of an in‑process event (source=event on a still-recording event). In EventStream::checkEventLoaded() (src/zm_eventstream.cpp:771), viewing an event whose EndDateTime is NULL calls
   loadEventData() every time playback reaches the end of the currently-known frames. Each reload:

• rebuilds event_data->frames, whose size scales with event length (a long continuous event = tens of MB of FrameData), and

• for mp4/passthrough events re-runs delete ffmpeg_input; new FFmpeg_Input; Open(...) on an ever-growing file (:395).

  When playback outruns recording this becomes a tight reload loop. Combined with the frame-interpolation/trailing-fill loops (:303, :361) that generate one FrameData per expected frame from Events.Frames/Delta — those are only
  bounded by DB values, so an incomplete event, a bad Delta, or a far-future EndDateTime can make a single rebuild explode. This matches "sometimes," "large + growing," and "killed."

What I can't determine from code alone

Which mode the offending process is in. The single most useful datum is the dying process's QUERY_STRING. Next time (or from logs) it's in /proc//environ (or cmdline), and /var/log/zm/zms_*.log will show whether it's monitor-MPEG
vs event replay.

[IgorA100]

1. The ~1.1 GB baseline is expected and is MAP_LOCKED shared memory, not a leak.

But why then is the RES memory on my two servers around 100MB (for 3-4K monitors) and not around 1GB? I think 100MB is a more accurate and realistic figure.

[connortechnology]

I couldn't reproduce this from a static read of the streaming code, so I've pushed diagnostics to help catch it in the act. Branch: 5006-zms-mem-diagnostics (commit 7cd4be5).

What it adds (all in nph-zms, all at Debug level):

• A one-shot summary of the request at startup, e.g.
  zms request: source=event mode=jpeg monitor=3 event=12345 ... buffer=... ttl=... rss=...KB
  This ties a runaway process to its exact stream type/parameters.
• A periodic RSS trace every 30s in both the monitor and event stream loops:
  • mem trace m3: rss=...KB frame_count=... buffer_level=... temp_count=... paused=... delayed=...
  • mem trace event: rss=...KB event=... frames=... frame_count=... curr_frame_id=... sent=...
    The event trace reports the frames-vector size, which scales with (re)loaded event length — the leading suspect is repeated reload of an in-process (still-recording) event rebuilding that vector.

To capture these you'll need Debug enabled for the offending stream — either raise the monitor/system log level, or kill -USR1 <pid> to bump a running zms. Then, next time a nph-zms balloons:

1. grep "zms request" /var/log/zm/zms_*.log around its start — tells us the mode (monitor jpeg/mpeg, event replay, fifo).
2. grep "mem trace" /var/log/zm/zms_*.log — shows the RSS ramp and whether frames/buffer_level climb in lockstep with RSS.
3. On a live offender: cat /proc/<pid>/environ | tr '\0' '\n' | grep QUERY_STRING for the exact URL.

That will confirm which path is responsible so the fix is targeted rather than speculative. Two questions in the meantime: are the runaways happening while viewing live montage/watch, or while reviewing recorded events? And what resolution / Image Buffer Size are the monitors involved?

[connortechnology]

This comment was written by Claude (Anthropic's Claude Code), working on behalf of @connortechnology.

Merged to master (bd733d6), so the diagnostics land in the next build — no separate branch to check out.

On your RES question: you're right, and I was wrong. ~100 MB is the correct healthy figure, not ~1 GB. My mistake was conflating VSZ (virtual address space) with RES/RSS (resident):

• mmap() of the SHM reserves the whole ring in the process's address space (VSZ ≈ the full image_buffer_count × slot × 2), but a file-backed mapping only becomes resident for pages actually faulted in. zms only reads the slot(s) it's currently serving, so RSS ≈ a few touched capture-ring slots. For a small ring at 3–4K that's ~100 MB — exactly what your screenshot shows. The analysis ring isn't touched at all for a normal jpeg stream, and MAP_LOCKED here is best-effort (it falls back to a plain MAP_SHARED map on EAGAIN), so it doesn't force the whole ring resident either.

So please disregard finding #1 — there is no expected ~1 GB baseline. That actually sharpens the picture: the process in your report was already ~10× normal at 1.1 GB when you first caught it, so the entire 1.1 GB → 10 GB trajectory is the leak, not "baseline + leak." The new rss= values will read ~100 MB on a healthy stream; anything climbing above that is the bug.

(RSS does creep up as the ring cycles and more slots get touched, but that plateaus at the ring size — it can't explain multi-GB growth.)

So the diagnostics still point the same way: grab the zms request: line to see which mode the runaway is, and the mem trace lines to see whether frames/buffer_level climb with rss. And the two earlier questions still stand — live montage/watch vs. reviewing recorded events, and the monitor resolution / Image Buffer Size.

[connortechnology]

This comment was written by Claude (Anthropic's Claude Code), working on behalf of @connortechnology.

One more thing worth checking: if memory serves these are 4K cameras, and a 4K raw frame is large (RGBA upper-bound ~33 MB/slot). A live stream reads frames as last_write_index cycles through the ring, so over one full cycle zms ends up touching every capture-ring slot — its RSS will climb toward image_buffer_count × slot_size and then plateau there. That plateau is expected and is not the leak (it can't exceed the ring size), but on a big-ring 4K monitor it can still be a substantial, legitimate chunk of RAM per live viewer.

So it's worth also looking at the RSS of the corresponding zmc process for that monitor — it writes all the slots and holds the same mmap, so it should sit at roughly the full ring size and gives us the baseline to compare zms against. If zms is sitting near zmc's footprint that's just the ring being fully touched; if it's climbing well past it, that's the bug we're hunting.

The mem trace rss= values will make this easy to see: a healthy live stream should ramp up to ~ring-size and flatten; the runaway keeps going.

[IgorA100]

Thanks, Isaac.
But I haven't been able to get the ZMS process to leak memory for a couple of days now. Perhaps I'm doing something wrong... I'll continue monitoring, but for now everything is stable and ZMS processes are consuming 90-130MB, which is probably normal.