Skip to content

Commit 4353119

Browse files
authored
Merge pull request DeusData#1666 from DeusData/fix/linux-arena-eager-commit
fix(mem): restore mimalloc's Linux arena-commit default (DeusData#1654)
2 parents 49d928b + 3491a8e commit 4353119

2 files changed

Lines changed: 56 additions & 0 deletions

File tree

‎src/foundation/mem.c‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,7 +291,36 @@ void cbm_mem_init_with_cap(double ram_fraction, size_t hard_cap_bytes) {
291291
}
292292
#endif
293293

294+
/* Lazy arena commit costs address-space FRAGMENTATION, and on Linux we now
295+
* pay it for every allocation in the process. mimalloc commits a range with
296+
* mprotect(PROT_READ|PROT_WRITE) over a sub-range of a PROT_NONE reservation
297+
* (prim/unix/prim.c), and each partial commit SPLITS the reserved VMA. While
298+
* mimalloc only served the bound populations (sqlite, tree_sitter) that was a
299+
* handful of mappings. Since #1360 routed ordinary malloc/new through
300+
* mimalloc on Linux, an index worker peaked at ~22k mappings against
301+
* v0.9.0's 10 (Go corpus, 18 workers). The count tracks CONCURRENCY, not
302+
* corpus size — sampled mid-run it was 999 at 1 worker, 8460 at 4 and 11965
303+
* at 18, while v0.9.0 stayed at 10 regardless — so every worker thread
304+
* fragments the address space independently.
305+
*
306+
* Two consequences, both reported as #1654 on a 96-CPU/376 GB host: the
307+
* mmap/mprotect churn serialises on the kernel's per-process mmap_lock (the
308+
* reporter saw 1.2% of extraction in 45 minutes, where v0.9.0 finished the
309+
* tree in ~13), and the VMA count climbs toward vm.max_map_count, after
310+
* which mmap fails for ANY size — hence mimalloc reporting it "cannot
311+
* allocate" 10 KB while `free -g` still showed 246 GB available.
312+
*
313+
* mimalloc's own default is 2, meaning "eager-commit arenas only on an OS
314+
* with overcommit (i.e. linux)" — precisely because commit is free there
315+
* until the pages are touched. Overriding it to 0 opted Linux out of the
316+
* default written for it. Restore the default on Linux; keep the lazy
317+
* setting everywhere else, where commit is NOT free and the upfront-memory
318+
* reason for it still holds (Windows especially — see #581). */
319+
#if defined(__linux__)
320+
mem_option_set_verified(mi_option_arena_eager_commit, 2, "arena_eager_commit");
321+
#else
294322
mem_option_set_verified(mi_option_arena_eager_commit, 0, "arena_eager_commit");
323+
#endif
295324
mem_option_set_verified(mi_option_purge_decommits, SKIP_ONE, "purge_decommits");
296325
mem_option_set_verified(mi_option_purge_delay, 0, "purge_delay"); /* immediate */
297326
/* v3 (#832): reclaim abandoned pages on ANY thread's free (=1), restoring the

‎tests/test_mem.c‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,32 @@
4141

4242
/* ── mem basic tests ──────────────────────────────────────────── */
4343

44+
/* Since #1360 routed ordinary malloc/new through mimalloc on Linux, the arena
45+
* policy governs EVERY allocation in the process, not just the bound
46+
* sqlite/tree_sitter populations. With lazy arena commit (0), mimalloc commits
47+
* sub-ranges via mprotect(PROT_READ|PROT_WRITE) over a PROT_NONE reservation,
48+
* and each partial commit SPLITS the reserved VMA: an index worker on the Go
49+
* corpus held ~22k mappings where v0.9.0 held 10, growing with worker count.
50+
* That is how #1654's 96-CPU host reached vm.max_map_count, after which mmap
51+
* fails for ANY size — 10 KB allocations failing while `free -g` still showed
52+
* 246 GB available. mimalloc's own default is 2, meaning "eager-commit arenas
53+
* only on an OS that overcommits (i.e. linux)", where commit is free until the
54+
* pages are touched; overriding it to 0 opted Linux out of the default written
55+
* for Linux. Measured: 22450 -> 17312 mappings, wall time and peak RSS
56+
* unchanged. Pin the platform split so the Linux default cannot be silently
57+
* opted out again — and keep the lazy setting where commit is NOT free
58+
* (Windows especially, see #581). */
59+
TEST(mem_arena_eager_commit_follows_platform_commit_cost) {
60+
cbm_mem_init(0.5);
61+
long eager = mi_option_get(mi_option_arena_eager_commit);
62+
#if defined(__linux__)
63+
ASSERT_EQ(eager, 2);
64+
#else
65+
ASSERT_EQ(eager, 0);
66+
#endif
67+
PASS();
68+
}
69+
4470
TEST(mem_rss_tracking) {
4571
cbm_mem_init(0.5);
4672

@@ -1253,6 +1279,7 @@ TEST(mem_map_attributes_a_known_allocation) {
12531279

12541280
SUITE(mem) {
12551281
/* mem API */
1282+
RUN_TEST(mem_arena_eager_commit_follows_platform_commit_cost);
12561283
RUN_TEST(mem_map_attributes_a_known_allocation);
12571284
RUN_TEST(mem_rss_tracking);
12581285
RUN_TEST(mem_collect_reclaims);

0 commit comments

Comments
 (0)