Skip to content

Commit 26cd467

Browse files
fix(mem): use real /proc RSS on Linux (mimalloc current_rss undercounts, blinding backpressure)
Distilled from DeusData#776's 132460f. This project sets arena_eager_commit=0 + purge_decommits=1 + purge_delay=0 in cbm_mem_init to reduce upfront memory, so mimalloc's committed-page counter reads low. On Linux mi_process_info() never sets current_rss (vendored/mimalloc/src/prim/unix/prim.c only fills peak_rss), so current_rss defaults to that low committed counter — cbm_mem_rss() returned a few MB while true RSS was multiple GB, leaving cbm_mem_over_budget() backpressure, the memory ceiling, and the host RAM tier blind on Linux. Prefer os_rss() (/proc/self/statm) as the primary source on Linux; macOS and Windows are unchanged (their mi_process_info current_rss is accurate via task_info / GetProcessMemoryInfo). cbm_mem_peak_rss() is untouched. Complements DeusData#752 by making the RAM tiers actually bite on Linux. Reproduce-first: mem_rss_reflects_external_resident_memory pins mimalloc's committed counter low with a live mi_malloc, then grows true RSS via a raw 256MB mmap (invisible to mimalloc); unfixed Linux returns the ~few-MB committed counter (RED), fixed Linux returns /proc RSS (GREEN). macOS/Windows pass either way, so the RED manifests on the Linux CI leg. Co-authored-by: petercoxphoto <info@petercox.ie> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent a4e1f21 commit 26cd467

2 files changed

Lines changed: 93 additions & 1 deletion

File tree

‎src/foundation/mem.c‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,13 +151,37 @@ void cbm_mem_init(double ram_fraction) {
151151
}
152152

153153
size_t cbm_mem_rss(void) {
154+
#if defined(__linux__)
155+
/* Linux: mimalloc's _mi_prim_process_info() (vendored/mimalloc/src/prim/
156+
* unix/prim.c) never sets pinfo->current_rss on Linux — it only sets
157+
* peak_rss (from getrusage's ru_maxrss). current_rss therefore keeps
158+
* mi_process_info()'s default of pinfo.current_commit: mimalloc's OWN
159+
* committed-page counter, which this project deliberately tunes low via
160+
* mi_option_arena_eager_commit=0 + purge_decommits=1 + purge_delay=0
161+
* (cbm_mem_init) to reduce upfront memory. So on Linux "current_rss" is a
162+
* low-biased mimalloc-internal metric, not true RSS: under concurrent
163+
* large-file parsing it can read a few MB while real RSS is multiple GB,
164+
* silently blinding cbm_mem_over_budget()'s backpressure to real memory
165+
* pressure (small-but-nonzero, so the `current_rss > 0` guard below never
166+
* catches it). os_rss() reads /proc/self/statm — authoritative OS RSS,
167+
* unaffected by mimalloc's accounting — so it is the PRIMARY source on
168+
* Linux, not a last-resort fallback. macOS/Windows are unaffected:
169+
* mimalloc sets current_rss correctly there via task_info /
170+
* GetProcessMemoryInfo. */
171+
size_t proc_rss = os_rss();
172+
if (proc_rss > 0) {
173+
return proc_rss;
174+
}
175+
/* Extremely unlikely (/proc unavailable) — fall through to mimalloc. */
176+
#endif
154177
size_t current_rss = 0;
155178
size_t peak_rss = 0;
156179
mi_process_info(NULL, NULL, NULL, &current_rss, &peak_rss, NULL, NULL, NULL);
157180
if (current_rss > 0) {
158181
return current_rss;
159182
}
160-
/* Fallback for ASan builds (MI_OVERRIDE=0) */
183+
/* Fallback for ASan builds (MI_OVERRIDE=0) and any platform where
184+
* mimalloc's current_rss is unavailable/zero. */
161185
return os_rss();
162186
}
163187

‎tests/test_mem.c‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,10 @@
1616

1717
#include <stdatomic.h>
1818
#include <sys/stat.h>
19+
#include <mimalloc.h>
20+
#ifndef _WIN32
21+
#include <sys/mman.h>
22+
#endif
1923

2024
/* ASan detection — mimalloc MI_OVERRIDE=0 under ASan, mi_process_info
2125
* may return 0 for RSS. Tests that depend on accurate RSS must skip. */
@@ -183,6 +187,69 @@ TEST(mem_collect_no_crash) {
183187
PASS();
184188
}
185189

190+
/* Reproduce-first guard for the Linux cbm_mem_rss() undercount (distilled
191+
* from #776's 132460f5).
192+
*
193+
* On Linux, mimalloc's mi_process_info() never sets current_rss
194+
* (vendored/mimalloc/src/prim/unix/prim.c only fills peak_rss from
195+
* getrusage's ru_maxrss); current_rss silently keeps mi_process_info()'s
196+
* default of pinfo.current_commit — mimalloc's OWN committed-page counter
197+
* (stats.c:555). The UNFIXED cbm_mem_rss() returns that counter whenever it is
198+
* nonzero, so on Linux it reports mimalloc-committed bytes, NOT true RSS. The
199+
* FIXED code reads /proc/self/statm (os_rss) as the primary source → true RSS.
200+
*
201+
* The guard makes the two quantities DIVERGE deterministically:
202+
* 1. mi_malloc() a small block (kept live) so mimalloc's committed counter is
203+
* a small POSITIVE value — this both defeats the UNFIXED `current_rss > 0`
204+
* fallback guard AND pins the reported value low. mi_malloc always routes
205+
* through mimalloc regardless of MI_OVERRIDE, so this works in the ASan
206+
* test-runner (MI_OVERRIDE=0) too.
207+
* 2. Grow TRUE process RSS by ~256MB via a raw anonymous mmap — memory
208+
* mimalloc's committed counter never sees, but /proc/self/statm does.
209+
* On UNFIXED Linux, cbm_mem_rss() then returns the ~few-MB committed counter
210+
* (< 128MB) → this assertion FAILS (RED). On FIXED Linux it returns the /proc
211+
* RSS (>= 256MB) → GREEN.
212+
*
213+
* macOS/Windows set current_rss from task_info/GetProcessMemoryInfo, which DO
214+
* include the mapped+touched region, so cbm_mem_rss() is accurate there both
215+
* before and after the fix — this passes on those platforms either way. The
216+
* RED therefore manifests only on the Linux CI leg, which is exactly where the
217+
* production undercount bit (backpressure/ceiling blinded). */
218+
TEST(mem_rss_reflects_external_resident_memory) {
219+
cbm_mem_init(0.5);
220+
221+
/* (1) Pin mimalloc's committed-page counter to a small positive value. */
222+
const size_t warm = (size_t)1 * 1024 * 1024; /* 1 MB via mimalloc */
223+
void *mi_buf = mi_malloc(warm);
224+
ASSERT_NOT_NULL(mi_buf);
225+
memset(mi_buf, 0x11, warm);
226+
227+
const size_t region = (size_t)256 * 1024 * 1024; /* 256 MB true RSS */
228+
const size_t threshold = (size_t)128 * 1024 * 1024; /* generous half */
229+
230+
#ifdef _WIN32
231+
/* Windows current_rss (WorkingSetSize) is accurate; a plain resident
232+
* allocation is reflected regardless of allocator. No Linux undercount. */
233+
void *big = malloc(region);
234+
ASSERT_NOT_NULL(big);
235+
memset(big, 0x5A, region);
236+
size_t rss = cbm_mem_rss();
237+
ASSERT_GTE(rss, threshold);
238+
free(big);
239+
#else
240+
/* (2) Raw mmap bypasses mimalloc entirely: its committed counter does NOT
241+
* grow, but the true RSS does — this is what exposes the Linux undercount. */
242+
void *big = mmap(NULL, region, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
243+
ASSERT_TRUE(big != MAP_FAILED);
244+
memset(big, 0x5A, region); /* fault every page in → resident */
245+
size_t rss = cbm_mem_rss();
246+
ASSERT_GTE(rss, threshold);
247+
munmap(big, region);
248+
#endif
249+
mi_free(mi_buf);
250+
PASS();
251+
}
252+
186253
TEST(mem_collect_rss_still_positive) {
187254
cbm_mem_init(0.5);
188255
cbm_mem_collect();
@@ -803,6 +870,7 @@ SUITE(mem) {
803870
RUN_TEST(mem_rss_positive);
804871
RUN_TEST(mem_peak_rss_gte_rss);
805872
RUN_TEST(mem_rss_increases_after_alloc);
873+
RUN_TEST(mem_rss_reflects_external_resident_memory);
806874
RUN_TEST(mem_collect_no_crash);
807875
RUN_TEST(mem_collect_rss_still_positive);
808876
/* Memory pressure simulation */

0 commit comments

Comments
 (0)