Skip to content

Commit 8c1a9d6

Browse files
committed
fix(extract): no clock decides what ends up in the graph
Indexing the same C# corpus twice on the same machine produced 5,794,336 edges and then 5,794,328. Every other edge type matched exactly, every semantic checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks), and the difference was ten WRITES edges out of one generated file. Three places in extraction decided graph content from a clock: * the unified AST walk stopped on a thread-CPU deadline; * a file whose parse spent more than half its budget was refused the per-file and cross-file LSP walks; * so was a file whose parse plus walk exceeded the budget. The first is the one that bit. Any file large enough to reach the deadline is cut at whatever node the clock lands on, and five generated .NET JIT tests reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one run and 2,574,336 in the next, out of 2,774,794. That file's tail is field assignments, so the two runs disagreed by exactly ten WRITES edges. The graph was a function of how fast the machine happened to be running, and nothing said so: result->walk_truncated was set and no code anywhere read it. The deadline existed because those files cost 18-48 us per visited node, and that turns out to be one fixable thing. csharp_callable_value_site() was the only language branch in call_reference_candidate_site() still climbing with ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning children at every level; every other language already climbs on the walk cursor. On expressions megabytes deep that was ~85% of the entire unified walk (sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant). It now takes the cursor and the parent the caller already resolved, and is_direct_argument_value_cursor() is split so a caller that has stepped the cursor onto the parent can continue from there rather than paying for it twice. With that fixed, the budgets are not needed to make indexing finish, so they are gone rather than made deterministic. A cutoff in nodes would at least be reproducible, but it still answers "what is in this repo?" with "as much as we felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants it; the default is no budget. Removing the two LSP-skip rules was checked against the file and the path they were added for (a 23 MB single-expression JIT test that once cost 354 s in the walk and then crashed the cross-file resolve): the whole 12k-file C# corpus twice with cross-file resolve, that file alone, and synthetic C# nested 10,000 levels deep — no crash, and past roughly that depth tree-sitter stops producing a usable tree, so a tree deep enough to endanger a recursive walk never reaches one. C# corpus, before -> after: edges 5,794,318 / 5,794,328 / 5,794,336 -> 5,794,873 every run truncated 5 files, unreported -> 0 time ~115 s -> 97-101 s Deterministic, about 550 edges more complete (the tails nobody was walking), and faster than the build that was cutting them off. All ten bench corpora now return identical nodes and edges across two runs. A truncated walk and a skipped LSP walk are now reported in the coverage report with how far the walk got ("1673216/4133946 nodes walked"), like a partial parse. A file the walk abandoned halfway used to be reported as fully indexed, which is a graph quietly lying about its coverage. Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence falls back to the root-descending parent lookup — 128 fallbacks without the fix and 0 with it. Its fixture parenthesises the argument on purpose: a bare Sink(Target) needs no climb at all, so it cannot tell the two walks apart and passed either way while the bug was in place. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent c7ccb47 commit 8c1a9d6

6 files changed

Lines changed: 308 additions & 96 deletions

File tree

‎internal/cbm/cbm.c‎

Lines changed: 85 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -225,17 +225,56 @@ static const char *cbm_string_read(void *payload, uint32_t byte, TSPoint point,
225225
* edge). A generous WALL ceiling stays as a backstop so a genuinely
226226
* stuck/spinning parse still terminates in bounded time. */
227227
#define CBM_PARSE_WALL_CEILING_FACTOR 12ULL /* ~60 s ceiling for the 5 s CPU budget */
228-
/* A parse that used more than 1/N of the per-file budget disqualifies the file
229-
* from the unbudgeted LSP walks (see cbm_extract_file_ex). */
230-
#define CBM_LSP_BUDGET_SHARE_DIV 2ULL
231-
/* The unified walk may spend this many parse budgets of thread CPU time: wide
232-
* enough for a 7,873-definition reference file (~10 s), tight enough to stop the
233-
* generated JIT tests (65-350 s). */
234-
#define CBM_WALK_BUDGET_FACTOR 6ULL
228+
/* The unified walk has NO budget: it walks every node of every file.
229+
*
230+
* It used to stop on a thread-CPU deadline, which made the GRAPH a function of
231+
* how fast the machine happened to be running. Two indexes of the same C#
232+
* corpus, same machine, minutes apart, stopped the walk of
233+
* src/tests/JIT/jit64/opt/cse/hugeSimpleExpr1.cs at node 2,660,352 and at node
234+
* 2,574,336 (measured 2026-09-19). That file's tail is all field assignments,
235+
* so the two runs disagreed by 10 WRITES edges: a graph that differed run to
236+
* run for a reason no user could see or reproduce. A budget in NODES would at
237+
* least have been reproducible, but it still answers "what is in this repo?"
238+
* with "depends how much we felt like reading" — so there is no budget at all.
239+
* Everything the walk does is a pure function of the tree, and now so is when
240+
* it stops: at the end.
241+
*
242+
* What the deadline was protecting against is real and is handled where it
243+
* belongs — in the cost per node, not in a clock. Generated blobs (2.7-8.2 M
244+
* node trees) used to cost 18-48 us per node because usage stamping called
245+
* ts_node_parent, which descends from the root; the walk now carries its own
246+
* parent chain, so a deep tree costs what a shallow one does.
247+
*
248+
* CBM_WALK_MAX_NODES reinstates a budget for an operator who needs one
249+
* (0 = the default, no budget). */
250+
#define CBM_WALK_MAX_NODES_DEFAULT 0u
251+
252+
/* Read fresh each call, like cbm_max_file_bytes: cheap next to a file walk, and
253+
* no memoized copy to go stale between runs in the same process. */
254+
static uint32_t cbm_walk_max_nodes(void) {
255+
const char *raw = getenv("CBM_WALK_MAX_NODES");
256+
if (raw && raw[0]) {
257+
errno = 0;
258+
char *end = NULL;
259+
unsigned long v = strtoul(raw, &end, 10);
260+
if (errno == 0 && end != raw && *end == '\0' && v <= UINT32_MAX) {
261+
return (uint32_t)v; /* 0 is a legitimate choice: no budget */
262+
}
263+
/* Unparseable / out-of-range → the default, never an accidental 0. */
264+
}
265+
return CBM_WALK_MAX_NODES_DEFAULT;
266+
}
235267

236268
typedef struct {
237269
uint64_t cpu_deadline_ns; // trip once this thread's CPU time passes it
238270
uint64_t wall_ceiling_ns; // hard wall backstop for a spinning/stuck parse
271+
/* Set by the callback when IT ended the parse. ts_parser_parse_with_options
272+
* returns NULL for several reasons and the budget is only one of them, but
273+
* the failure used to be reported as "parse timeout" whenever a budget was
274+
* configured at all. A 7-line .properties file was accused of timing out
275+
* (2026-09-19); whatever really happened to it, saying "timeout" sent the
276+
* next reader looking at the clock. */
277+
bool tripped;
239278
} CBMParseBudget;
240279

241280
#ifdef CBM_ENABLE_TEST_SEAMS
@@ -253,12 +292,17 @@ static CBM_TLS uint64_t tl_parse_wall_seam_offset_ns = 0;
253292
* parameter here would not match the opts.progress_callback assignment below. */
254293
// cppcheck-suppress constParameterCallback
255294
static bool cbm_timeout_cb(TSParseState *state) {
256-
const CBMParseBudget *budget = (const CBMParseBudget *)state->payload;
295+
CBMParseBudget *budget = (CBMParseBudget *)state->payload;
257296
uint64_t wall = now_ns();
258297
#ifdef CBM_ENABLE_TEST_SEAMS
259298
wall += tl_parse_wall_seam_offset_ns;
260299
#endif
261-
return cbm_thread_cpu_time_ns() > budget->cpu_deadline_ns || wall > budget->wall_ceiling_ns;
300+
bool over =
301+
cbm_thread_cpu_time_ns() > budget->cpu_deadline_ns || wall > budget->wall_ceiling_ns;
302+
if (over) {
303+
budget->tripped = true; /* so a NULL tree can name its real cause */
304+
}
305+
return over;
262306
}
263307

264308
// --- Thread-local parser pool ---
@@ -1882,7 +1926,6 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
18821926
ts_parser_reset(parser);
18831927

18841928
uint64_t t0 = now_ns();
1885-
uint64_t cpu_start_ns = cbm_thread_cpu_time_ns();
18861929

18871930
// Build string input + timeout options for parse_with_options
18881931
CBMStringInput str_input = {source, (uint32_t)source_len};
@@ -1922,40 +1965,47 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
19221965

19231966
if (!tree) {
19241967
result->has_error = true;
1925-
result->error_msg =
1926-
cbm_arena_strdup(a, timeout_micros > 0 ? "parse timeout" : "parse failed");
1968+
/* Only the budget's own callback may call this a timeout. Any other
1969+
* NULL is a parse failure, and saying so is the difference between a
1970+
* reader chasing the clock and a reader chasing the real cause. */
1971+
result->error_msg = cbm_arena_strdup(a, budget.tripped ? "parse timeout" : "parse failed");
19271972
cbm_index_mark_done(rel_path);
19281973
return result;
19291974
}
19301975

19311976
TSNode root = ts_tree_root_node(tree);
19321977

1933-
/* Parse-budget share. A file whose parse alone consumed more than
1934-
* 1/CBM_LSP_BUDGET_SHARE_DIV of its budget is too large for the per-file
1935-
* LSP walk that follows: the walk is superlinear in expression size and
1936-
* has no budget of its own (C#, a 23 MB single-expression JIT test: 354 s
1937-
* in the walk, then a crash in the cross-file resolve on the same tree,
1938-
* 2026-09-14 -- the parse used to time out at 5 s and hide both). The
1939-
* unified extractor's defs stay; the LSP refinement here and the
1940-
* cross-file resolve (cbm_pxc_dispatch_file) skip the file, logged. The
1941-
* budget is the same for every parser, so the rule is too. */
1942-
bool lsp_skipped = timeout_micros > 0 && (t1 - t0) * CBM_LSP_BUDGET_SHARE_DIV > budget_ns;
1978+
/* No file is disqualified from the LSP walks by how long its parse took.
1979+
*
1980+
* The rule used to be "a parse that spent more than half its budget means
1981+
* this tree is too heavy for the walks" — a clock deciding which files get
1982+
* type-aware refinement, so the same repo could come back with different
1983+
* graphs. It was added on 2026-09-14 against a 23 MB single-expression C#
1984+
* JIT test that cost 354 s in the per-file walk and then crashed the
1985+
* cross-file resolve.
1986+
*
1987+
* Removing it was checked against exactly that file and that path
1988+
* (2026-09-19): the whole 12k-file C# corpus twice, cross-file resolve
1989+
* included, and hugeexpr1.cs on its own — no crash, same 1,224,981 nodes /
1990+
* 5,794,873 edges as with the rule in place, same ~100 s. Synthetic C# with
1991+
* valid nesting 10,000 levels deep indexes fine; past roughly that depth
1992+
* tree-sitter itself stops producing a usable tree, so a tree deep enough
1993+
* to endanger a recursive walk never reaches one. The 354 s is gone for a
1994+
* different reason: the walk that spent it was C#'s own, climbing with
1995+
* ts_node_parent, and it now uses the cursor (extract_usages.c).
1996+
*
1997+
* CBM_TEST_LSP_SKIP_ON still names a file for the tests that pin the
1998+
* skipped-file behaviour itself. */
19431999
#ifdef CBM_ENABLE_TEST_SEAMS
19442000
{
19452001
const char *skip_on = getenv("CBM_TEST_LSP_SKIP_ON");
19462002
if (skip_on && skip_on[0] && rel_path && strstr(rel_path, skip_on)) {
1947-
lsp_skipped = true; /* the test names the file; no real timing involved */
2003+
result->lsp_skipped = true; /* the test names the file; no timing involved */
2004+
cbm_log_warn("extract.lsp.skipped", "reason", "test_seam", "path",
2005+
rel_path ? rel_path : "");
19482006
}
19492007
}
19502008
#endif
1951-
if (lsp_skipped) {
1952-
char parse_ms[CBM_SZ_32];
1953-
snprintf(parse_ms, sizeof(parse_ms), "%llu",
1954-
(unsigned long long)((t1 - t0) / CBM_NSEC_PER_MSEC));
1955-
cbm_log_warn("extract.lsp.skipped", "reason", "parse_budget", "parse_ms", parse_ms, "path",
1956-
rel_path ? rel_path : "");
1957-
result->lsp_skipped = true;
1958-
}
19592009

19602010
// Compute module QN. Java/Go derive the module from the CONTAINING
19612011
// DIRECTORY (package semantics) rather than baking the filename stem in,
@@ -1979,28 +2029,20 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
19792029
.root = root,
19802030
.macro_table = macro_table,
19812031
.return_type_table = return_type_table,
1982-
.walk_deadline_cpu_ns =
1983-
timeout_micros > 0 ? cbm_thread_cpu_time_ns() + budget_ns * CBM_WALK_BUDGET_FACTOR : 0,
2032+
.walk_budget_nodes = cbm_walk_max_nodes(),
19842033
};
19852034

19862035
// Run extractors: defs + imports use separate walks (unique recursion patterns),
19872036
// then a single unified cursor walk handles the remaining 7 extractors.
19882037
cbm_extract_definitions(&ctx);
19892038
cbm_extract_imports(&ctx);
19902039
cbm_extract_unified(&ctx);
2040+
result->tree_nodes = ts_node_descendant_count(root);
2041+
result->walk_nodes_visited = ctx.walk_nodes_visited;
19912042
if (ctx.walk_budget_exhausted) {
19922043
result->walk_truncated = true;
19932044
result->lsp_skipped = true;
1994-
cbm_log_warn("extract.walk.truncated", "reason", "cpu_budget", "path",
1995-
rel_path ? rel_path : "");
1996-
}
1997-
/* A file that spent the budget on parse plus walk is too heavy for the
1998-
* unbudgeted LSP walks as well (the C# JIT test files: 65-73 s each in
1999-
* the per-file walk after a parse under the share rule). */
2000-
if (!result->lsp_skipped && timeout_micros > 0 &&
2001-
cbm_thread_cpu_time_ns() - cpu_start_ns > budget_ns) {
2002-
result->lsp_skipped = true;
2003-
cbm_log_warn("extract.lsp.skipped", "reason", "file_budget", "path",
2045+
cbm_log_warn("extract.walk.truncated", "reason", "node_budget", "path",
20042046
rel_path ? rel_path : "");
20052047
}
20062048

‎internal/cbm/cbm.h‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -569,6 +569,12 @@ typedef struct CBMFileResult {
569569
* to that point are kept, the rest of the file is not walked. Implies
570570
* lsp_skipped. */
571571
bool walk_truncated;
572+
/* Size of this file's parse tree, and how much of it the unified walk got
573+
* through. Reported for a truncated or LSP-skipped file so the coverage
574+
* report says how much of it is missing, instead of leaving the gap
575+
* silent. */
576+
uint32_t tree_nodes;
577+
uint32_t walk_nodes_visited;
572578
CBMLanguage cached_lang; // language of cached tree (for parser selection)
573579

574580
// Retained source bytes — copied into `arena` by the parallel
@@ -659,14 +665,19 @@ typedef struct {
659665
* class-body variable def records which class declares it (parent_class)
660666
* without changing its module-level qualified name. NULL elsewhere. */
661667
const char *var_parent_class;
662-
/* Per-file walk budget (thread CPU time, ns; 0 = unbounded). The unified
663-
* cursor walk checks it every 1024 nodes and stops when it is spent, so
664-
* no single file can hold a worker for minutes: a 23 MB single-expression
665-
* C# test file cost 346 s in usage stamping alone (tree-sitter's
666-
* ts_node_parent descends from the root, quadratic on a deep tree;
667-
* 2026-09-14). What was extracted before the stop is kept. */
668-
uint64_t walk_deadline_cpu_ns;
668+
/* Per-file walk budget in VISITED NODES (0 = unbounded). The unified cursor
669+
* walk stops once it is spent, so no single file can hold a worker for
670+
* minutes: a 23 MB single-expression C# test file cost 346 s in usage
671+
* stamping alone (tree-sitter's ts_node_parent descends from the root,
672+
* quadratic on a deep tree; 2026-09-14). What was extracted before the stop
673+
* is kept, and the file is named in the coverage report. Counted in nodes
674+
* rather than CPU time so that the same file always stops at the same node
675+
* — see CBM_WALK_MAX_NODES_DEFAULT for what a clock did here. */
676+
uint32_t walk_budget_nodes;
669677
bool walk_budget_exhausted;
678+
/* How many nodes the unified walk actually visited (whether or not it ran
679+
* out of budget) — the measurement the budget has to be expressed in. */
680+
uint32_t walk_nodes_visited;
670681
} CBMExtractCtx;
671682

672683
// --- Public API ---

‎internal/cbm/extract_unified.c‎

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2600,31 +2600,31 @@ void cbm_extract_unified(CBMExtractCtx *ctx) {
26002600
uint32_t depth = 0;
26012601
uint32_t visited = 0;
26022602
#ifdef CBM_ENABLE_TEST_SEAMS
2603-
/* CBM_TEST_WALK_BUDGET_NODES=<n>: the budget is "spent" after n nodes,
2604-
* no real timing involved. */
2605-
uint32_t seam_budget_nodes = 0;
2603+
/* CBM_TEST_WALK_BUDGET_NODES=<n>: spend the budget after n nodes. Production
2604+
* now counts nodes too, so this seam only shortens the budget; it no longer
2605+
* has to stand in for a mechanism of its own. */
2606+
uint32_t budget_nodes = ctx->walk_budget_nodes;
26062607
{
26072608
const char *seam = getenv("CBM_TEST_WALK_BUDGET_NODES");
26082609
if (seam && seam[0]) {
2609-
seam_budget_nodes = (uint32_t)strtoul(seam, NULL, 10);
2610+
budget_nodes = (uint32_t)strtoul(seam, NULL, 10);
26102611
}
26112612
}
2613+
#else
2614+
const uint32_t budget_nodes = ctx->walk_budget_nodes;
26122615
#endif
26132616

26142617
for (;;) {
26152618
TSNode node = ts_tree_cursor_current_node(&cursor);
26162619
visited++;
2617-
if (ctx->walk_deadline_cpu_ns != 0 && (visited & 1023u) == 0 &&
2618-
cbm_thread_cpu_time_ns() > ctx->walk_deadline_cpu_ns) {
2620+
/* The budget is spent in nodes, so the walk of a given file always ends
2621+
* on the same node — on any machine, under any load. A CPU deadline
2622+
* here made the graph differ between two runs on one machine; see
2623+
* CBM_WALK_MAX_NODES_DEFAULT (cbm.c). */
2624+
if (budget_nodes != 0 && visited > budget_nodes) {
26192625
ctx->walk_budget_exhausted = true;
26202626
break;
26212627
}
2622-
#ifdef CBM_ENABLE_TEST_SEAMS
2623-
if (seam_budget_nodes != 0 && visited > seam_budget_nodes) {
2624-
ctx->walk_budget_exhausted = true;
2625-
break;
2626-
}
2627-
#endif
26282628
bool trivia = is_unified_trivia_node(node);
26292629
if (!trivia) {
26302630
/* Trivia consumes no semantic state. Scope expiry may be deferred
@@ -2678,6 +2678,7 @@ void cbm_extract_unified(CBMExtractCtx *ctx) {
26782678
}
26792679

26802680
cbm_finalize_lexical_usages(ctx, &state);
2681+
ctx->walk_nodes_visited = visited;
26812682
ts_tree_cursor_delete(&occurrence_cursor);
26822683
ts_tree_cursor_delete(&cursor);
26832684
}

0 commit comments

Comments
 (0)