object-cache-srv/src/saturation_monitor.rs samples a good set of pressure signals every 5 s — fetch-budget occupancy, in-flight entries, RAM-tier usage and entries, memory-budget occupancy, prefetch queue depth, NIC and foyer disk throughput — and emits them all as imetric!/fmetric! gauges. It takes no action on any of them.
The result is that the server has no mechanism to defend its own memory. On a production deployment (single 8 GiB im4gn.large) under a large partition rebuild, memory grew from a ~5.0 GB steady state to a ~6.9 GB RSS with system free memory down to ~103 MB, and the process was SIGKILLed by the kernel OOM killer. It repeated on a ~50 minute cycle. Throughout, every gauge the monitor emits stayed inside its configured bound — the growth was in transient fetch buffers (#1537) and allocator fragmentation (#1536), neither of which any budget was tracking. Nothing stood between growth and the OOM killer.
Suggested fix
Give the monitor a control path in addition to its reporting path — a high-water mark with graded response, sharing the shape of the sink's existing graded drop policy in telemetry-sink/src/http_event_sink.rs:
- stop admitting prefetch work (the queue already load-sheds by design, so this is the cheapest lever)
- shrink available fetch permits, throttling origin concurrency
- shed demand reads with 503 as the last resort
jemalloc_resident_bytes and process_resident_bytes are already exported (#1319) and are the right trigger signal — resident, not allocated, since the fragmentation gap is precisely what the OOM killer counts. The ceiling should be an explicit configured value rather than inferred from cgroup or host memory, so it stays correct on a shared box.
Related: #1206 added these gauges to locate bottlenecks; this issue is about acting on them.
object-cache-srv/src/saturation_monitor.rssamples a good set of pressure signals every 5 s — fetch-budget occupancy, in-flight entries, RAM-tier usage and entries, memory-budget occupancy, prefetch queue depth, NIC and foyer disk throughput — and emits them all asimetric!/fmetric!gauges. It takes no action on any of them.The result is that the server has no mechanism to defend its own memory. On a production deployment (single 8 GiB
im4gn.large) under a large partition rebuild, memory grew from a ~5.0 GB steady state to a ~6.9 GB RSS with system free memory down to ~103 MB, and the process was SIGKILLed by the kernel OOM killer. It repeated on a ~50 minute cycle. Throughout, every gauge the monitor emits stayed inside its configured bound — the growth was in transient fetch buffers (#1537) and allocator fragmentation (#1536), neither of which any budget was tracking. Nothing stood between growth and the OOM killer.Suggested fix
Give the monitor a control path in addition to its reporting path — a high-water mark with graded response, sharing the shape of the sink's existing graded drop policy in
telemetry-sink/src/http_event_sink.rs:jemalloc_resident_bytesandprocess_resident_bytesare already exported (#1319) and are the right trigger signal — resident, not allocated, since the fragmentation gap is precisely what the OOM killer counts. The ceiling should be an explicit configured value rather than inferred from cgroup or host memory, so it stays correct on a shared box.Related: #1206 added these gauges to locate bottlenecks; this issue is about acting on them.