Skip to content

Commit da3258f

Browse files
committed
feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline
Memory in this project is now allocated through ONE core. This lands the core, the gate that keeps new code on it, and the instrumentation that lets an index run say which class of memory grew in which pass -- the three things the 2026-09-13 audit found missing. WHAT THE AUDIT FOUND There was no centralized allocation layer. foundation/mem.h was policy and measurement only (budget, RSS, over_budget, memory map, phase marks) with no alloc/free wrapper, so the budget could be observed after the fact and never enforced. Raw allocator calls in the tree: 4086 across 92 files. The extraction engine had already adopted arenas (1301 call sites); the graph buffer -- which holds the 35 GB peak of a kernel index -- used none, allocating every 64 B node, 48 B edge and every name/qualified_name/ properties_json individually. Two further facts made a self-accounting core the only workable design: 1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking the process-wide heap is a data race TSan caught on macOS, so it is deliberately not done, and an 18-worker index attributes almost nothing -- the rest lands in residual. 2. The mimalloc global override is ON for Linux/MinGW and permanently OFF for macOS (two-level namespace: this binary's free becomes mi_free while system libraries keep allocating from the system zone, and a pointer crossing that boundary aborts). On macOS ordinary malloc is served by the system allocator; the startup audit reports owned_classes=0/6. Any accounting that assumes mimalloc owns the pointer is blind on an entire platform. THE CORE (src/foundation/mem_core.{h,c}) cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index, extract, semantic, dump, store, other). Per-class live bytes, live blocks and peak, kept in atomics -- thread-safe by construction, identical on every platform, independent of which allocator serves malloc. libc semantics are preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL) allocates, a failed realloc leaves the block intact) so adoption is a mechanical rename and never a behaviour change. No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come from the platform usable-size query (malloc_size / malloc_usable_size / _msize), which is correct under either allocator and free. Where no query exists (BSD) the request size is charged, which understates -- the safe direction for a diagnostic. A mismatched class on free clamps at zero rather than wrapping, so a small drift can never masquerade as a colossal leak. Arena-backed allocators report in bulk through cbm_mem_class_add_external / remove_external so their memory appears in the same table; the extraction engine must not be rewritten to per-object allocation, that would undo the batching that keeps its allocation count low. THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci) Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect. 4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in baseline (scripts/memory-core-baseline.txt) records each file's count and the build goes red when any file grows, or a new file appears with any. Files only ever go down; the baseline line is lowered in the change that migrates the file, and --strict fails a stale baseline. Comments and string literals are stripped first, so prose that mentions malloc( cannot trip it -- the security audit already bit on exactly that with fork(. Proven both ways: one appended malloc turns it red naming the file (83 -> 85), restoring turns it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline. PHASE ATTRIBUTION (src/pipeline/pipeline.c) cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin and at every pass.timing site: parallel_extract, the six sequential passes, the eight predump passes, tests and dump_and_persist, with peaks reset per index. Both instruments existed in foundation/ and were wired only into MCP request handling, never into the index pipeline -- which is where the memory is. Locating the kernel peak this week required an external RSS sampler because nothing in-process could say which phase it was in. PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h) cbm_system_available_ram() -- macOS host_statistics64 (free + inactive + purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown, never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM available and false when unknown so nothing ever aborts on a guess. Landed UNWIRED: an earlier draft used them to let the over-budget latch press on whenever the system was not under pressure, and that permitted the very overshoot this work exists to end (Decision A's tests at test_pipeline.c:13481 and :13583 went red under it, correctly). They are the backstop for the next step, admission control, which keeps a run under budget before an allocation rather than measuring it after. mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci clean including the new gate. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 339b3f4 commit da3258f

11 files changed

Lines changed: 1053 additions & 1 deletion

File tree

‎Makefile.cbm‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -248,6 +248,7 @@ FOUNDATION_SRCS = \
248248
src/foundation/compat_fs.c \
249249
src/foundation/compat_regex.c \
250250
src/foundation/mem.c \
251+
src/foundation/mem_core.c \
251252
src/foundation/diagnostics.c \
252253
src/foundation/profile.c \
253254
src/foundation/dump_verify.c \
@@ -1188,8 +1189,17 @@ lint-no-suppress:
11881189
lint: lint-tidy lint-cppcheck lint-format lint-no-suppress
11891190
@echo "=== All linters passed ==="
11901191

1192+
# Memory-core linter: memory is allocated through src/foundation/mem_core.h,
1193+
# not through raw malloc/calloc/realloc/free/strdup. A checked-in baseline
1194+
# (scripts/memory-core-baseline.txt) records each file's raw-site count and the
1195+
# gate goes red the moment any file grows. Files only ever go down; lower the
1196+
# baseline line in the same change that migrates the file.
1197+
lint-memory-core:
1198+
@echo "=== memory-core linter ==="
1199+
@python3 scripts/lint-memory-core.py
1200+
11911201
# CI linters (no clang-tidy — platform-dependent, enforced locally via pre-commit)
1192-
lint-ci: lint-cppcheck lint-format lint-no-suppress
1202+
lint-ci: lint-cppcheck lint-format lint-no-suppress lint-memory-core
11931203
@echo "=== CI linters passed ==="
11941204

11951205
# ── Local memory-diagnostic lanes (not PR-CI gates by decision: the diag

‎scripts/lint-memory-core.py‎

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
#!/usr/bin/env python3
2+
"""
3+
lint-memory-core.py — the memory-core linter.
4+
5+
Memory in this project is allocated through ONE core (src/foundation/mem_core.h),
6+
not through scattered raw malloc/calloc/realloc/free/strdup. This linter is the
7+
gate that keeps that true.
8+
9+
WHY A RATCHET, NOT A BAN. An audit on 2026-09-13 found ~800 raw allocation
10+
sites in src/ and none of them can be migrated in one change. So the gate works
11+
like every other honest debt gate: a checked-in baseline records how many raw
12+
sites each file has TODAY, and the build goes red the moment any file has MORE
13+
than its baseline, or a file not in the baseline grows one. Files can only ever
14+
go down. When a file is migrated, its baseline line is lowered (or removed) in
15+
the same change -- and --strict turns "below baseline" into a failure too, so
16+
the baseline cannot silently rot behind the code.
17+
18+
WHAT COUNTS AS RAW. A call to malloc/calloc/realloc/free/strdup/strndup that is
19+
not part of a longer identifier. cbm_alloc, cbm_free, cbm_calloc, mi_malloc,
20+
cbm_arena_alloc and heap_strdup are all NOT matches: the character before the
21+
name is an identifier character. Comments and string literals are stripped
22+
first, so prose that mentions malloc( does not trip the gate -- the security
23+
audit already bit us once on exactly that with fork(.
24+
25+
EXEMPT. The core itself and the allocator plumbing it sits on:
26+
src/foundation/mem_core.c the route
27+
src/foundation/arena.c bulk allocator that reports INTO the core
28+
src/foundation/slab_alloc.c same
29+
src/foundation/mem.c policy/measurement, probes with malloc
30+
src/foundation/mem_override_*.c the --wrap / override shims
31+
src/foundation/compat*.c libc replacement surface
32+
internal/cbm/vendored/** not ours
33+
vendored/** not ours
34+
35+
Usage:
36+
lint-memory-core.py check against the baseline (CI)
37+
lint-memory-core.py --strict also fail when a file is BELOW baseline
38+
lint-memory-core.py --write-baseline regenerate scripts/memory-core-baseline.txt
39+
lint-memory-core.py --list print every raw site (file:line: call)
40+
41+
Exit 0 = clean. Exit 1 = a file grew. Exit 2 = usage/IO error.
42+
"""
43+
from __future__ import annotations
44+
45+
import argparse
46+
import pathlib
47+
import re
48+
import sys
49+
50+
ROOT = pathlib.Path(__file__).resolve().parent.parent
51+
BASELINE = ROOT / "scripts" / "memory-core-baseline.txt"
52+
SCAN_ROOTS = ("src", "internal/cbm")
53+
RAW = re.compile(r"(?<![A-Za-z0-9_])(malloc|calloc|realloc|free|strdup|strndup)\s*\(")
54+
55+
EXEMPT_EXACT = {
56+
"src/foundation/mem_core.c",
57+
"src/foundation/arena.c",
58+
"src/foundation/slab_alloc.c",
59+
"src/foundation/mem.c",
60+
}
61+
EXEMPT_PREFIX = (
62+
"src/foundation/mem_override_",
63+
"src/foundation/compat",
64+
"internal/cbm/vendored/",
65+
"vendored/",
66+
)
67+
68+
69+
def is_exempt(rel: str) -> bool:
70+
return rel in EXEMPT_EXACT or rel.startswith(EXEMPT_PREFIX)
71+
72+
73+
def strip_comments_and_strings(text: str) -> str:
74+
"""Blank out comments and string literals, preserving line structure so
75+
reported line numbers stay right. Character-class aware enough for C:
76+
handles escapes inside strings and does not treat // inside a string as a
77+
comment."""
78+
out = []
79+
i, n = 0, len(text)
80+
while i < n:
81+
c = text[i]
82+
nxt = text[i + 1] if i + 1 < n else ""
83+
if c == "/" and nxt == "*":
84+
j = text.find("*/", i + 2)
85+
j = n if j < 0 else j + 2
86+
out.append("".join(ch if ch == "\n" else " " for ch in text[i:j]))
87+
i = j
88+
elif c == "/" and nxt == "/":
89+
j = text.find("\n", i)
90+
j = n if j < 0 else j
91+
out.append(" " * (j - i))
92+
i = j
93+
elif c == '"' or c == "'":
94+
q = c
95+
j = i + 1
96+
while j < n and text[j] != q:
97+
if text[j] == "\\":
98+
j += 1
99+
if j < n and text[j] == "\n":
100+
break
101+
j += 1
102+
j = min(j + 1, n)
103+
out.append(q + " " * max(0, j - i - 2) + (q if j - i >= 2 else ""))
104+
i = j
105+
else:
106+
out.append(c)
107+
i += 1
108+
return "".join(out)
109+
110+
111+
def scan() -> dict[str, list[tuple[int, str]]]:
112+
hits: dict[str, list[tuple[int, str]]] = {}
113+
for root in SCAN_ROOTS:
114+
base = ROOT / root
115+
if not base.exists():
116+
continue
117+
for path in sorted(base.rglob("*")):
118+
if path.suffix not in (".c", ".h") or not path.is_file():
119+
continue
120+
rel = path.relative_to(ROOT).as_posix()
121+
if is_exempt(rel):
122+
continue
123+
text = strip_comments_and_strings(path.read_text(encoding="utf-8", errors="replace"))
124+
for lineno, line in enumerate(text.splitlines(), 1):
125+
for m in RAW.finditer(line):
126+
hits.setdefault(rel, []).append((lineno, m.group(1)))
127+
return hits
128+
129+
130+
def read_baseline() -> dict[str, int]:
131+
if not BASELINE.exists():
132+
return {}
133+
out: dict[str, int] = {}
134+
for raw in BASELINE.read_text(encoding="utf-8").splitlines():
135+
line = raw.strip()
136+
if not line or line.startswith("#"):
137+
continue
138+
rel, _, count = line.rpartition("\t")
139+
if not rel:
140+
continue
141+
try:
142+
out[rel] = int(count)
143+
except ValueError:
144+
continue
145+
return out
146+
147+
148+
def write_baseline(hits: dict[str, list]) -> None:
149+
lines = [
150+
"# memory-core-baseline.txt -- raw allocator call sites per file.",
151+
"# Generated by scripts/lint-memory-core.py --write-baseline.",
152+
"# A file may only ever go DOWN. Lower a line in the same change that",
153+
"# migrates the file to src/foundation/mem_core.h; never raise one.",
154+
"",
155+
]
156+
for rel in sorted(hits):
157+
lines.append(f"{rel}\t{len(hits[rel])}")
158+
BASELINE.write_text("\n".join(lines) + "\n", encoding="utf-8")
159+
160+
161+
def main() -> int:
162+
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
163+
ap.add_argument("--write-baseline", action="store_true")
164+
ap.add_argument("--strict", action="store_true", help="fail when a file is BELOW its baseline")
165+
ap.add_argument("--list", action="store_true", help="print every raw site")
166+
args = ap.parse_args()
167+
168+
hits = scan()
169+
total = sum(len(v) for v in hits.values())
170+
171+
if args.write_baseline:
172+
write_baseline(hits)
173+
print(f"memory-core baseline written: {len(hits)} files, {total} raw sites")
174+
return 0
175+
176+
if args.list:
177+
for rel in sorted(hits):
178+
for lineno, call in hits[rel]:
179+
print(f"{rel}:{lineno}: {call}(")
180+
print(f"-- {total} raw sites in {len(hits)} files")
181+
return 0
182+
183+
baseline = read_baseline()
184+
if not baseline:
185+
print(f"ERROR: no baseline at {BASELINE.relative_to(ROOT)}; run --write-baseline", file=sys.stderr)
186+
return 2
187+
188+
grew: list[str] = []
189+
shrank: list[str] = []
190+
for rel, sites in sorted(hits.items()):
191+
now = len(sites)
192+
allowed = baseline.get(rel, 0)
193+
if now > allowed:
194+
# Show the LAST sites: new code is usually appended, so these are the
195+
# likeliest culprits. The linter cannot know which sites are new
196+
# without a diff; --list prints them all.
197+
delta = now - allowed
198+
tail = ", ".join(f"{ln}:{c}" for ln, c in sites[-max(delta, 1):][-6:])
199+
grew.append(f" {rel}: grew by {delta} ({allowed} -> {now}); latest sites: {tail}"
200+
f" [run --list for all]")
201+
elif now < allowed:
202+
shrank.append(f" {rel}: {now} (baseline {allowed}) -- lower the baseline")
203+
for rel, allowed in sorted(baseline.items()):
204+
if rel not in hits and allowed > 0:
205+
shrank.append(f" {rel}: 0 (baseline {allowed}) -- remove the line")
206+
207+
if grew:
208+
print("memory-core linter FAILED: raw allocator use grew. Allocate through")
209+
print("src/foundation/mem_core.h (cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/")
210+
print("cbm_mem_strdup) instead of malloc/calloc/realloc/free/strdup.")
211+
print("\n".join(grew))
212+
if shrank:
213+
print("memory-core ratchet: these files improved; tighten scripts/memory-core-baseline.txt:")
214+
print("\n".join(shrank))
215+
if grew or (args.strict and shrank):
216+
return 1
217+
print(f"memory-core linter passed: {total} raw sites across {len(hits)} files, none grew")
218+
return 0
219+
220+
221+
if __name__ == "__main__":
222+
sys.exit(main())

‎scripts/memory-core-baseline.txt‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# memory-core-baseline.txt -- raw allocator call sites per file.
2+
# Generated by scripts/lint-memory-core.py --write-baseline.
3+
# A file may only ever go DOWN. Lower a line in the same change that
4+
# migrates the file to src/foundation/mem_core.h; never raise one.
5+
6+
internal/cbm/ac.c 20
7+
internal/cbm/arena.c 3
8+
internal/cbm/cbm.c 13
9+
internal/cbm/extract_defs.c 5
10+
internal/cbm/helpers.c 3
11+
internal/cbm/lsp/c_lsp.c 4
12+
internal/cbm/lsp/perl_lsp.c 21
13+
internal/cbm/lsp/py_lsp.c 2
14+
internal/cbm/macro_table.c 1
15+
internal/cbm/service_patterns.c 2
16+
internal/cbm/sqlite_writer.c 92
17+
src/cli/activation_transaction.c 80
18+
src/cli/agent_clients.c 73
19+
src/cli/agent_profiles.c 7
20+
src/cli/cli.c 173
21+
src/cli/client_adapter.c 3
22+
src/cli/config_edit_path.c 2
23+
src/cli/config_json_like.c 142
24+
src/cli/config_text_edit.c 72
25+
src/cli/config_toml_edit.c 85
26+
src/cli/config_yaml_edit.c 117
27+
src/cli/hook_augment.c 31
28+
src/cypher/cypher.c 132
29+
src/daemon/application.c 107
30+
src/daemon/bootstrap.c 13
31+
src/daemon/daemon.c 31
32+
src/daemon/frontend.c 12
33+
src/daemon/ipc.c 146
34+
src/daemon/project_lock.c 6
35+
src/daemon/runtime.c 45
36+
src/daemon/service.c 6
37+
src/daemon/version_cohort.c 9
38+
src/discover/discover.c 31
39+
src/discover/gitignore.c 12
40+
src/discover/userconfig.c 15
41+
src/foundation/diagnostics.c 7
42+
src/foundation/dyn_array.h 4
43+
src/foundation/hash_table.c 3
44+
src/foundation/lock_registry.c 18
45+
src/foundation/platform.c 11
46+
src/foundation/platform.h 5
47+
src/foundation/private_file_lock.c 56
48+
src/foundation/str_intern.c 7
49+
src/foundation/subprocess.c 24
50+
src/foundation/win_utf8.h 17
51+
src/foundation/workspace.c 3
52+
src/foundation/yaml.c 10
53+
src/git/git_context.c 18
54+
src/graph_buffer/graph_buffer.c 83
55+
src/main.c 32
56+
src/mcp/compact_out.c 39
57+
src/mcp/index_supervisor.c 8
58+
src/mcp/mcp.c 814
59+
src/pipeline/artifact.c 38
60+
src/pipeline/fqn.c 23
61+
src/pipeline/lsp_resolve.h 4
62+
src/pipeline/lsp_surface.c 5
63+
src/pipeline/pass_calls.c 23
64+
src/pipeline/pass_compile_commands.c 17
65+
src/pipeline/pass_complexity.c 15
66+
src/pipeline/pass_configlink.c 3
67+
src/pipeline/pass_cross_repo.c 8
68+
src/pipeline/pass_definitions.c 14
69+
src/pipeline/pass_enrichment.c 15
70+
src/pipeline/pass_ensemble_routing.c 10
71+
src/pipeline/pass_envscan.c 2
72+
src/pipeline/pass_githistory.c 27
73+
src/pipeline/pass_importance.c 4
74+
src/pipeline/pass_infrascan.c 2
75+
src/pipeline/pass_k8s.c 13
76+
src/pipeline/pass_lsp_cross.c 50
77+
src/pipeline/pass_parallel.c 31
78+
src/pipeline/pass_pkgmap.c 82
79+
src/pipeline/pass_semantic.c 13
80+
src/pipeline/pass_semantic_edges.c 48
81+
src/pipeline/pass_similarity.c 12
82+
src/pipeline/pass_tests.c 3
83+
src/pipeline/pass_usages.c 4
84+
src/pipeline/path_alias.c 42
85+
src/pipeline/pipeline.c 134
86+
src/pipeline/pipeline_delta.c 19
87+
src/pipeline/pipeline_incremental.c 144
88+
src/pipeline/registry.c 19
89+
src/pipeline/worker_pool.c 2
90+
src/semantic/semantic.c 54
91+
src/simhash/minhash.c 10
92+
src/store/store.c 358
93+
src/ui/config.c 12
94+
src/ui/http_server.c 31
95+
src/ui/httpd.c 21
96+
src/ui/layout3d.c 66
97+
src/watcher/watcher.c 33

‎src/foundation/mem.c‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -514,6 +514,22 @@ bool cbm_mem_over_budget(void) {
514514
return rss > g_budget;
515515
}
516516

517+
/* Reclaimable memory below this share of total RAM is where paging starts to
518+
* hurt, so it is the point at which pressing on stops being reasonable. */
519+
enum { MEM_PRESSURE_AVAIL_DIVISOR = 8 }; /* 12.5% of total RAM */
520+
521+
bool cbm_mem_system_under_pressure(void) {
522+
size_t available = cbm_system_available_ram();
523+
if (available == 0) {
524+
return false; /* platform cannot answer - do not abort on a guess */
525+
}
526+
cbm_system_info_t info = cbm_system_info();
527+
if (info.total_ram == 0) {
528+
return false;
529+
}
530+
return available < info.total_ram / MEM_PRESSURE_AVAIL_DIVISOR;
531+
}
532+
517533
size_t cbm_mem_worker_budget(int num_workers) {
518534
if (num_workers <= 0) {
519535
num_workers = SKIP_ONE;

‎src/foundation/mem.h‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,21 @@ bool cbm_mem_over_budget(void);
7575
/* Per-worker budget hint: budget / num_workers. */
7676
size_t cbm_mem_worker_budget(int num_workers);
7777

78+
/* True only when the SYSTEM is genuinely short of memory, not merely when this
79+
* process is over its advisory budget.
80+
*
81+
* The budget is a static fraction of TOTAL ram, so on a large host it refuses
82+
* work the machine can plainly do: measured 2026-09-13, the linux kernel needs
83+
* 31.75 GB of a 48 GB host against a 24 GB budget, and the previous release
84+
* completed the very same index by overshooting to 33.56 GB. Aborting on the
85+
* budget alone therefore turned a working index into a refusal.
86+
*
87+
* So the budget keeps its job as the BACKPRESSURE trigger (workers park, peers
88+
* return transients) and this answers the different question of whether giving
89+
* up is warranted. Returns false when availability is unknown: never abort on
90+
* a guess. */
91+
bool cbm_mem_system_under_pressure(void);
92+
7893
/* Return unused pages to the OS. Call between files to bound per-file peak. */
7994
void cbm_mem_collect(void);
8095

0 commit comments

Comments
 (0)