Skip to content

Commit cb167b9

Browse files
fix(coverage): narrow parse-error ranges with the preprocessed parse (DeusData#963)
src/cli/cli.c reported an error range of 1-13047 — the whole file. The file indexed fine; the report was wrong. Three #ifndef _WIN32 blocks split a brace (two `if` headers, one closing brace), so the raw tree-sitter parse cannot resync at file scope, the root node becomes ERROR, and cbm.c takes its whole-file branch. The pipeline already parses these files a second time after preprocessing, and that parse is clean. The report just never consulted it. Build one byte per original line from the preprocessed pass, then cut each raw error range down to the runs of lines the second parse could not vouch for. Three rules, all found by running it and all load-bearing: - An expanded line only vouches for its original line when it HAS TEXT. The preprocessor emits a blank line where it dropped a branch; treating that blank as proof suppressed every C range in the suite. - Preprocessor directive lines (with backslash continuations) never count as missing code — the preprocessor consumes them, so the second parse can never vouch for one. Without this every #include block reported as a miss. Known cost: a #define the raw parse really dropped no longer shows up on its own. - A TOP-LEVEL macro invocation line never counts as vouched-for even when the expanded line parses clean. The macro can expand to a whole definition that the recovery walker deliberately refuses to adopt (DeusData#949), so a clean second parse there proves nothing. An in-body invocation is the benign DeusData#1071 case and is left to the existing macro subtraction. The order of the three coverage steps is now settled by where each one's evidence lives: recovery subtraction -> before the refinement; its evidence is a whole definition that STARTS inside the range, so it must be asked while the range still matches the construct the refinement -> middle DeusData#1071 macro rule -> after the refinement; its evidence is per-line, so a narrow range points at the call itself Measured on this repo: src/cli/cli.c goes from one whole-file range to 64 ranges over ~9.8% of the file, tests/test_cli.c from 48.6% to ~2.9%, src/cli/activation_transaction.c from 38% to 7.5%. What survives is honest — the biggest remaining ranges in cli.c are genuinely discarded #ifdef _WIN32 and #ifdef CBM_CLI_ENABLE_TEST_API blocks, absent from the graph on this platform. Both percentages above are floors, not measurements: cli.c and test_cli.c now land on exactly 64 ranges, which is CBM_MAX_ERROR_REGIONS. That cap drops regions with no signal, and a follow-up raises it and adds a truncation marker. Five tests, all red before the change: the range narrows to the dropped branch; lines the preprocessor explained are excluded; a range never starts or ends on a directive; real garbage beside a split brace stays flagged; a clean file stays unflagged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
1 parent fe85a6b commit cb167b9

2 files changed

Lines changed: 410 additions & 1 deletion

File tree

‎internal/cbm/cbm.c‎

Lines changed: 271 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -870,6 +870,130 @@ static void cbm_collect_error_regions(TSNode n, cbm_error_regions_t *acc, const
870870
}
871871
}
872872

873+
/* ── Phase 2 line map: what the preprocessed parse already explained ───────
874+
*
875+
* The raw parse is preprocessor-blind. When an #ifdef splits a brace it sees
876+
* both branches at once, the braces do not balance, and the ERROR node
877+
* swallows the whole construct — at file scope it swallows the whole FILE.
878+
* The second parse, on preprocessed source, does not have that problem: the
879+
* preprocessor already picked one branch, so that parse is clean.
880+
*
881+
* So we build one byte per ORIGINAL line and use it to cut the raw ranges
882+
* down to the lines the second parse cannot vouch for. Lines in the branch
883+
* the preprocessor threw away never appear in the second parse at all, so
884+
* they stay flagged — which is right, because they really are missing from
885+
* the graph.
886+
*
887+
* CBM_LINE_PP_PARSED — the preprocessed parse covered this original line and
888+
* found no error on it. Nothing here was dropped.
889+
* CBM_LINE_NO_CODE — the line is empty, is only a comment, or is a
890+
* preprocessor directive. A reported range must never
891+
* begin or end on one.
892+
*
893+
* Directives are in this set because the preprocessor
894+
* CONSUMES them: no directive line ever survives into
895+
* the expanded text, so the second parse can never
896+
* vouch for one, and treating that silence as a miss
897+
* would flag every #include block in the file. The
898+
* known cost is a #define that the raw parse really did
899+
* drop: it no longer shows up on its own. That trade is
900+
* deliberate — it removes far more noise than signal. */
901+
enum { CBM_LINE_PP_PARSED = 1u, CBM_LINE_NO_CODE = 2u };
902+
903+
/* True when the line's first non-blank character starts a preprocessor
904+
* directive. */
905+
static bool cbm_is_directive_line(const char *line, int len) {
906+
int i = 0;
907+
while (i < len && (line[i] == ' ' || line[i] == '\t')) {
908+
i++;
909+
}
910+
return i < len && line[i] == '#';
911+
}
912+
913+
/* True when the line ends with a backslash, so the directive carries on to
914+
* the next line. */
915+
static bool cbm_line_continues(const char *line, int len) {
916+
int end = len;
917+
while (end > 0 && (line[end - 1] == ' ' || line[end - 1] == '\t' || line[end - 1] == '\r')) {
918+
end--;
919+
}
920+
return end > 0 && line[end - 1] == '\\';
921+
}
922+
923+
/* Set CBM_LINE_NO_CODE on every line of `src` that holds no construct.
924+
* One pass over the file. Carries block-comment state across lines so a line
925+
* in the middle of a comment counts as no-code too. */
926+
static void cbm_mark_no_code_lines(const char *src, int src_len, uint8_t *map,
927+
uint32_t line_count) {
928+
bool in_block = false;
929+
bool in_directive = false;
930+
uint32_t line = 1;
931+
int i = 0;
932+
while (i <= src_len && line <= line_count) {
933+
int end = i;
934+
while (end < src_len && src[end] != '\n') {
935+
end++;
936+
}
937+
bool has_code = false;
938+
bool line_starts_in_block = in_block;
939+
for (int j = i; j < end; j++) {
940+
if (in_block) {
941+
if (src[j] == '*' && j + 1 < end && src[j + 1] == '/') {
942+
in_block = false;
943+
j++;
944+
}
945+
continue;
946+
}
947+
if (src[j] == '/' && j + 1 < end && src[j + 1] == '*') {
948+
in_block = true;
949+
j++;
950+
continue;
951+
}
952+
if (src[j] == '/' && j + 1 < end && src[j + 1] == '/') {
953+
break; /* rest of the line is a comment */
954+
}
955+
if (src[j] != ' ' && src[j] != '\t' && src[j] != '\r') {
956+
has_code = true;
957+
}
958+
}
959+
bool directive =
960+
!line_starts_in_block && (in_directive || cbm_is_directive_line(src + i, end - i));
961+
if (!has_code || directive) {
962+
map[line] |= CBM_LINE_NO_CODE;
963+
}
964+
in_directive = directive && cbm_line_continues(src + i, end - i);
965+
line++;
966+
i = end + 1;
967+
}
968+
}
969+
970+
/* Paint CBM_LINE_PP_PARSED for every original line the preprocessed parse
971+
* covered without an error on it.
972+
*
973+
* Step 1 marks the EXPANDED rows that sit under an ERROR/MISSING node.
974+
* Step 2 walks the expanded lines and, for each one that is unmarked, belongs
975+
* to the file itself (not an included header) and maps back to a real
976+
* original line, records that original line as parsed. */
977+
static void cbm_mark_pp_error_rows(TSNode n, uint8_t *rows, uint32_t row_count, const char *src,
978+
int src_len) {
979+
uint32_t k = ts_node_child_count(n);
980+
for (uint32_t i = 0; i < k; i++) {
981+
TSNode c = ts_node_child(n, i);
982+
if (ts_node_is_missing(c) || strcmp(ts_node_type(c), "ERROR") == 0) {
983+
if (cbm_is_eof_terminator_miss(c, src, src_len)) {
984+
continue; /* absent final newline only — nothing was dropped */
985+
}
986+
uint32_t s = ts_node_start_point(c).row + 1;
987+
uint32_t e = ts_node_end_point(c).row + 1;
988+
for (uint32_t r = s; r <= e && r <= row_count; r++) {
989+
rows[r] = 1;
990+
}
991+
} else if (ts_node_has_error(c)) {
992+
cbm_mark_pp_error_rows(c, rows, row_count, src, src_len);
993+
}
994+
}
995+
}
996+
873997
/* Recovery subtraction (#963): tree-sitter error recovery plus the
874998
* ERROR-descending def walker often still extract constructs INSIDE a failed
875999
* region (verified: a function in an #ifdef-split ERROR region and even a
@@ -1163,6 +1287,75 @@ static void cbm_subtract_macro_invocation_regions(cbm_error_regions_t *regs,
11631287
regs->count = kept;
11641288
}
11651289

1290+
/* Push [start, end] after trimming no-code lines off both ends. A run made
1291+
* only of directives, comments or blank lines disappears entirely — there was
1292+
* never a construct on it to lose. */
1293+
static void cbm_push_trimmed_run(cbm_error_regions_t *out, uint32_t start, uint32_t end,
1294+
const uint8_t *map, uint32_t line_count) {
1295+
while (start <= end && start <= line_count && (map[start] & CBM_LINE_NO_CODE)) {
1296+
start++;
1297+
}
1298+
while (end >= start && end <= line_count && (map[end] & CBM_LINE_NO_CODE)) {
1299+
end--;
1300+
}
1301+
if (start > end || out->count >= CBM_MAX_ERROR_REGIONS) {
1302+
return;
1303+
}
1304+
out->starts[out->count] = start;
1305+
out->ends[out->count] = end;
1306+
out->count++;
1307+
}
1308+
1309+
/* #949: a top-level macro invocation is the one place where a clean second
1310+
* parse proves nothing. The macro can expand to a whole definition, and the
1311+
* recovery walker deliberately refuses to adopt that definition because it is
1312+
* absent from the original span. So the expanded line parses fine while the
1313+
* construct really is missing from the graph, and the line must stay flagged.
1314+
* An invocation INSIDE a function body is the benign #1071 case and is left
1315+
* alone here — cbm_subtract_macro_invocation_regions handles it later. */
1316+
static bool cbm_line_is_toplevel_macro_call(const char *src, int src_len, uint32_t line,
1317+
const CBMDefArray *defs) {
1318+
return cbm_span_is_macro_invocation(src, src_len, line, line, defs) &&
1319+
!cbm_region_inside_callable(line, line, defs);
1320+
}
1321+
1322+
/* Cut every raw region down to the lines the preprocessed parse could not
1323+
* vouch for. Each region becomes zero or more smaller ranges: one per run of
1324+
* consecutive lines that the second parse did not cover cleanly.
1325+
*
1326+
* This is what collapses a whole-file range on a file whose only real problem
1327+
* is an #ifdef splitting a brace. It deliberately does NOT clear the region
1328+
* outright — the branch the preprocessor discarded is genuinely absent from
1329+
* the graph and must stay flagged. */
1330+
static void cbm_refine_regions_with_pp_lines(cbm_error_regions_t *regs, const uint8_t *map,
1331+
uint32_t line_count, const char *src, int src_len,
1332+
const CBMDefArray *defs) {
1333+
cbm_error_regions_t out = {{0}, {0}, 0};
1334+
for (int i = 0; i < regs->count; i++) {
1335+
uint32_t run_start = 0;
1336+
uint32_t run_end = 0;
1337+
uint32_t end = regs->ends[i] < line_count ? regs->ends[i] : line_count;
1338+
for (uint32_t line = regs->starts[i]; line <= end; line++) {
1339+
if ((map[line] & CBM_LINE_PP_PARSED) &&
1340+
!cbm_line_is_toplevel_macro_call(src, src_len, line, defs)) {
1341+
if (run_start != 0) {
1342+
cbm_push_trimmed_run(&out, run_start, run_end, map, line_count);
1343+
run_start = 0;
1344+
}
1345+
} else {
1346+
if (run_start == 0) {
1347+
run_start = line;
1348+
}
1349+
run_end = line;
1350+
}
1351+
}
1352+
if (run_start != 0) {
1353+
cbm_push_trimmed_run(&out, run_start, run_end, map, line_count);
1354+
}
1355+
}
1356+
*regs = out;
1357+
}
1358+
11661359
/* Serialize collected regions as "start-end,start-end,..." into the arena. */
11671360
static const char *cbm_error_ranges_str(CBMArena *a, const cbm_error_regions_t *regs) {
11681361
if (regs->count <= 0) {
@@ -1419,6 +1612,14 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
14191612
// metrics. Remember the boundary.
14201613
int orig_calls_count = result->calls.count;
14211614

1615+
/* Phase 2 line map, built by the second (preprocessed) pass below and read
1616+
* by the parse-coverage block near the end of this function. Stays NULL
1617+
* for every language that has no second pass, which leaves the coverage
1618+
* signal exactly as it was. Arena-allocated so it outlives the
1619+
* preprocessed source and its tree. */
1620+
uint8_t *pp_line_map = NULL;
1621+
uint32_t pp_line_map_lines = 0;
1622+
14221623
// Second pass: preprocess C/C++/CUDA and extract additional macro-hidden calls.
14231624
// Defs keep original-source line numbers; only CALLS are extracted from expanded source.
14241625
if (language == CBM_LANG_C || language == CBM_LANG_CPP || language == CBM_LANG_CUDA) {
@@ -1545,6 +1746,60 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
15451746
}
15461747
}
15471748

1749+
/* Build the original-line map before the expanded tree
1750+
* goes away. Skipped when the expanded parse is itself a
1751+
* total loss (root is ERROR), because then it vouches for
1752+
* nothing and there is no refinement to make. */
1753+
if (strcmp(ts_node_type(pp_root), "ERROR") != 0) {
1754+
uint32_t orig_lines = 1;
1755+
for (int ci = 0; ci < source_len; ci++) {
1756+
if (source[ci] == '\n') {
1757+
orig_lines++;
1758+
}
1759+
}
1760+
uint8_t *map =
1761+
(uint8_t *)cbm_arena_alloc(a, (size_t)orig_lines + 2);
1762+
int exp_lines = preprocessed->expanded_line_count;
1763+
uint8_t *bad_rows =
1764+
exp_lines > 0 ? (uint8_t *)calloc((size_t)exp_lines + 2, 1) : NULL;
1765+
if (map && bad_rows) {
1766+
memset(map, 0, (size_t)orig_lines + 2);
1767+
cbm_mark_no_code_lines(source, source_len, map, orig_lines);
1768+
cbm_mark_pp_error_rows(pp_root, bad_rows, (uint32_t)exp_lines,
1769+
expanded, expanded_len);
1770+
/* Walk the expanded text once. An expanded line
1771+
* only vouches for its original line when it
1772+
* actually HAS content: the preprocessor emits a
1773+
* blank line where it dropped a branch, and a
1774+
* blank line proves nothing about the code that
1775+
* used to be there. */
1776+
uint32_t eline = 1;
1777+
bool eline_has_text = false;
1778+
for (int ci = 0; ci <= expanded_len; ci++) {
1779+
if (ci < expanded_len && expanded[ci] != '\n') {
1780+
char ch = expanded[ci];
1781+
if (ch != ' ' && ch != '\t' && ch != '\r') {
1782+
eline_has_text = true;
1783+
}
1784+
continue;
1785+
}
1786+
if (eline_has_text && (int)eline <= exp_lines && !bad_rows[eline] &&
1787+
preprocessed->belongs_to_main_file[eline]) {
1788+
uint32_t orig =
1789+
preprocessed->original_line_by_expanded_line[eline];
1790+
if (orig >= 1 && orig <= orig_lines) {
1791+
map[orig] |= CBM_LINE_PP_PARSED;
1792+
}
1793+
}
1794+
eline++;
1795+
eline_has_text = false;
1796+
}
1797+
pp_line_map = map;
1798+
pp_line_map_lines = orig_lines;
1799+
}
1800+
free(bad_rows);
1801+
}
1802+
15481803
ts_tree_delete(pp_tree);
15491804
}
15501805
}
@@ -1666,9 +1921,24 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
16661921
} else {
16671922
cbm_collect_error_regions(root, &regs, source, source_len);
16681923
}
1924+
/* Recovery subtraction runs on the RAW ranges, before the Phase 2
1925+
* refinement below. Its evidence is a whole definition that starts
1926+
* inside the range, so it has to be asked while the range still
1927+
* matches the construct. Ask it after the refinement and an #ifdef
1928+
* splitting a brace inside a recovered function looks unrecovered:
1929+
* the refinement keeps only the discarded branch, the function starts
1930+
* above it, and the evidence falls outside the range. */
16691931
cbm_subtract_recovered_regions(&regs, &result->defs);
1932+
/* Phase 2: cut what is left down to the lines the preprocessed parse
1933+
* could not explain. */
1934+
if (pp_line_map) {
1935+
cbm_refine_regions_with_pp_lines(&regs, pp_line_map, pp_line_map_lines, source,
1936+
source_len, &result->defs);
1937+
}
16701938
/* #1071: don't flag a benign function-like-macro call (defined in-file)
1671-
* that tree-sitter can't parse without the preprocessor. */
1939+
* that tree-sitter can't parse without the preprocessor. Runs AFTER the
1940+
* refinement, because its evidence is per-line: a narrow range points at
1941+
* the call itself instead of the whole blob around it. */
16721942
cbm_subtract_macro_invocation_regions(&regs, &result->defs, source, source_len);
16731943
if (regs.count > 0) {
16741944
result->parse_incomplete = true;

0 commit comments

Comments
 (0)