Repository navigation
Commit 3491a8e
committed
fix(mem): restore mimalloc's Linux arena-commit default (DeusData#1654)
Since DeusData#1360 routed ordinary malloc/new through mimalloc on Linux, the
arena policy governs every allocation in the process rather than just the
bound sqlite/tree_sitter populations. cbm sets arena_eager_commit=0, so
mimalloc commits sub-ranges with mprotect(PROT_READ|PROT_WRITE) over a
PROT_NONE reservation, and each partial commit SPLITS the reserved VMA.
Measured on the Go corpus, Linux arm64, shipped binaries:
v0.9.0 10 mappings, at ANY worker count
v0.10.5 ~22k mappings, peak; the count tracks CONCURRENCY
(999 at 1 worker, 8460 at 4, 11965 at 18)
Two consequences, both of which DeusData#1654 reported from a 96-CPU/376 GB host:
the mmap/mprotect churn serialises on the kernel's per-process mmap_lock,
and the VMA count climbs toward vm.max_map_count, after which mmap fails
for ANY size -- so mimalloc reported it could not allocate 10 KB while
`free -g` still showed 246 GB available.
mimalloc's own default for this option is 2, meaning "eager-commit arenas
only on an OS that overcommits (i.e. linux)", precisely because commit is
free there until pages are touched. Overriding it to 0 opted Linux out of
the default written for Linux. Restore it on Linux only; every other
platform keeps the lazy setting, where commit is NOT free and the
upfront-memory reason still holds (Windows especially, DeusData#581).
Measured effect, same corpus and host, baseline build vs this build:
mappings 22450 -> 17312 (-23%)
wall 92.4s -> 92.6s (unchanged)
peak RSS 19.14 -> 19.22 GB (unchanged)
This is a partial mitigation, not a cure: the remaining ~17k mappings are
individual 64 KB-3 MB extraction buffers, each taking its own mmap (the
worker reserves ~40 GB of address space for ~19 GB of RSS). Pooling those
is the durable fix and is deliberately left out of this change.
Guard: mem_arena_eager_commit_follows_platform_commit_cost pins the
platform split so the Linux default cannot be silently opted out again.
Reproduction and controlled 2x2 (only vm.max_map_count varied) are
recorded on DeusData#1654.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>1 parent 49d928b commit 3491a8e
2 files changed
Lines changed: 56 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
291 | 291 | | |
292 | 292 | | |
293 | 293 | | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
294 | 322 | | |
| 323 | + | |
295 | 324 | | |
296 | 325 | | |
297 | 326 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
44 | 70 | | |
45 | 71 | | |
46 | 72 | | |
| |||
1253 | 1279 | | |
1254 | 1280 | | |
1255 | 1281 | | |
| 1282 | + | |
1256 | 1283 | | |
1257 | 1284 | | |
1258 | 1285 | | |
| |||
0 commit comments