Skip to content

Commit fa258d3

Browse files
committed
fix(extraction): parse every file as if its last line were terminated (DeusData#2078)
A file whose last byte is not "\n" leaves the construct on its final line unterminated for every grammar that treats the line ending as part of that construct. DeusData#1610/DeusData#1746 already excused the mild form, a ZERO-WIDTH MISSING newline at EOF. The severe form is a WIDTH-BEARING error that cannot be excused. In tree-sitter-markdown an opening fence, a bare list marker or an ATX heading as the last bytes costs the last line (`text\n` + fence -> parse_partial 2-2) or the whole file (a lone fence, `- ` -> parse_unusable). An unterminated one-line heading was lost silently: the root is ERROR, there is no Section and no flag. A Makefile whose last recipe line is unterminated lost the recipe. The same bytes plus one "\n" parse clean. The reporter confirmed this on 21 real files (43 -> 23 flagged after appending newlines). Supply the newline rather than excuse its absence. cbm_parse_source() feeds the parser one virtual "\n" at offset source_len (through the TSInput read callback, so the caller's buffer is never copied or written) and then applies one ts_tree_edit that deletes that byte again. tree-sitter's edit arithmetic clamps every node that reached into it back to the real end, bytes and row/column points both. So def line ranges, node text, call byte spans, parse-coverage ranges and the retained tree all see only real bytes and real lines. Empty files and files that already end in "\n" are parsed exactly as before. Every whole-file parse goes through the helper: the extraction parse, and the whole-file fallback re-parses in the C/C++, C#, Go, Java, Kotlin, PHP, Python, Rust and TS LSP passes. That keeps a spilled result (no retained tree) consistent with a cached one. Sub-range and synthetic parses (preprocessed C, Rust wrappers, Kotlin patches, injected Jinja/imports ranges) are unchanged. The two DeusData#1610/DeusData#1746 guards that pinned "an unterminated Makefile recipe is a width-bearing loss" now use a construct that is broken with or without a newline (an unclosed Python tuple at EOF). The recipe is no longer lost, so flagging it would be false. The new makefile_unterminated_recipe_is_parsed_issue2078 test pins that. Proof (this repo, vendored grammars excluded, before/after binaries): - unchanged tree, and the same tree with the final newline stripped from all 1,220 files: nodes, edges and parse_partial lists byte-identical before vs after; - EOF-class corpus (the repo plus each .md with a fence, list marker, heading or opening fence appended without a newline, plus an unterminated Makefile): parse_partial 122 -> 77, exactly the unchanged repo's list. The only graph deltas are the 45 dropped ::missed shadow File rows (+7 Folders, 52 CONTAINS edges) and 22 Section docstrings that now include their real last line. The other four constructs in DeusData#2078 (HTML bare `&`, bash heredoc + redirect + pipe, bash `$((10#$n))`, markdown empty `||` cell) are upstream grammar gaps. Reports for them are drafted separately. Refs DeusData#2078 (the EOF class; the remaining triggers are upstream grammar gaps) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent b05a68c commit fa258d3

12 files changed

Lines changed: 376 additions & 39 deletions

File tree

‎internal/cbm/cbm.c‎

Lines changed: 71 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -278,20 +278,80 @@ void cbm_channels_push(CBMChannelArray *arr, CBMArena *a, CBMChannel ch) {
278278
typedef struct {
279279
const char *string;
280280
uint32_t length;
281+
/* #2078: serve one "\n" at offset `length` -- see cbm_parse_source. */
282+
bool virtual_newline;
281283
} CBMStringInput;
282284

283285
static const char *cbm_string_read(void *payload, uint32_t byte, TSPoint point,
284286
uint32_t *bytes_read) {
285287
(void)point;
286288
CBMStringInput *self = (CBMStringInput *)payload;
287289
if (byte >= self->length) {
290+
if (self->virtual_newline && byte == self->length) {
291+
*bytes_read = 1;
292+
return "\n";
293+
}
288294
*bytes_read = 0;
289295
return "";
290296
}
291297
*bytes_read = self->length - byte;
292298
return self->string + byte;
293299
}
294300

301+
/* Where the parser stands after reading all `len` real bytes. */
302+
static TSPoint cbm_source_end_point(const char *source, uint32_t len) {
303+
TSPoint end = {0, 0};
304+
const char *p = source;
305+
const char *stop = source + len;
306+
const char *nl;
307+
while (p < stop && (nl = memchr(p, '\n', (size_t)(stop - p))) != NULL) {
308+
end.row++;
309+
p = nl + 1;
310+
}
311+
end.column = (uint32_t)(stop - p);
312+
return end;
313+
}
314+
315+
/* #2078: parse every file as if its last line were terminated.
316+
*
317+
* Many grammars treat the line terminator as part of the construct it ends --
318+
* a markdown fence, heading or list marker, a Makefile recipe, a Dockerfile
319+
* instruction. A file whose last byte is not "\n" leaves that construct
320+
* unterminated: at best a zero-width MISSING token (#1610/#1746 excuse those at
321+
* EOF), at worst a WIDTH-BEARING error that costs the whole last line or, for a
322+
* one-line file, the whole file, and sometimes a silent loss (an unterminated
323+
* markdown heading produced no Section and no flag). The same bytes plus one
324+
* "\n" parse clean, so the absent newline is supplied instead of excused.
325+
*
326+
* The parser reads one virtual "\n" past EOF; afterwards a single tree edit
327+
* deletes it again. tree-sitter's own edit arithmetic then clamps every node
328+
* that reached into the virtual byte back to the real end -- byte offsets AND
329+
* row/column points -- so every consumer (def line ranges, node text, call
330+
* byte spans, parse-coverage ranges, the retained tree the LSP passes reuse)
331+
* sees only real bytes and real lines; nothing downstream needs to know. The
332+
* caller's buffer is never copied or written.
333+
*
334+
* A file that is empty or already ends in "\n" is parsed exactly as before. */
335+
TSTree *cbm_parse_source(TSParser *parser, const char *source, uint32_t source_len,
336+
TSParseOptions opts) {
337+
CBMStringInput input = {source, source_len, source_len > 0 && source[source_len - 1] != '\n'};
338+
TSInput ts_input = {&input, cbm_string_read, TSInputEncodingUTF8, NULL};
339+
TSTree *tree = ts_parser_parse_with_options(parser, NULL, ts_input, opts);
340+
if (tree && input.virtual_newline) {
341+
TSPoint end = cbm_source_end_point(source, source_len);
342+
TSInputEdit drop_virtual_newline = {
343+
.start_byte = source_len,
344+
.old_end_byte = source_len + 1,
345+
.new_end_byte = source_len,
346+
.start_point = end,
347+
.old_end_point = {end.row + 1, 0},
348+
.new_end_point = end,
349+
};
350+
ts_tree_edit(tree, &drop_virtual_newline);
351+
}
352+
return tree;
353+
}
354+
295355
// --- Parse timeout callback ---
296356

297357
/* Budget for the tree-sitter progress callback. The PRIMARY gate is per-thread
@@ -1261,7 +1321,11 @@ static void cbm_error_regions_push(cbm_error_regions_t *acc, TSNode n) {
12611321
* #1746: the Dockerfile grammar places that zero-width missing newline before
12621322
* trailing whitespace rather than at raw EOF. Preserve the broad exact-EOF
12631323
* rule above; only extend it past blanks when the missing token is specifically
1264-
* a newline. */
1324+
* a newline.
1325+
*
1326+
* #2078: cbm_parse_source now supplies the absent final newline, so a MISSING
1327+
* newline at EOF no longer arises from it; the rule stays for the other
1328+
* zero-width terminators a grammar can leave at EOF. */
12651329
static bool cbm_is_blank_not_newline(char c) {
12661330
return c == ' ' || c == '\t' || c == '\v' || c == '\f' || c == '\r';
12671331
}
@@ -2221,15 +2285,7 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
22212285

22222286
uint64_t t0 = now_ns();
22232287

2224-
// Build string input + timeout options for parse_with_options
2225-
CBMStringInput str_input = {source, (uint32_t)source_len};
2226-
TSInput ts_input = {
2227-
&str_input,
2228-
cbm_string_read,
2229-
TSInputEncodingUTF8,
2230-
NULL,
2231-
};
2232-
2288+
// Timeout options for parse_with_options
22332289
TSParseOptions opts = {0};
22342290
CBMParseBudget budget = {0}; // cppcheck-suppress unreadVariable
22352291
uint64_t budget_ns = 0;
@@ -2257,12 +2313,14 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
22572313
/* #1735: a SQL data dump's literal-only INSERT rows carry no graph content
22582314
* but dominate its parse. Keep them out through included ranges; offsets
22592315
* and positions of everything kept are unchanged. The parser is
2260-
* thread-local and reused, so the ranges are cleared right after. */
2316+
* thread-local and reused, so the ranges are cleared right after. The last
2317+
* range ends at the real EOF, so the virtual final newline cbm_parse_source
2318+
* supplies (#2078) stays outside a ranged parse. */
22612319
CBMSqlKeptRanges sql_kept = {NULL, 0};
22622320
bool sql_ranged = language == CBM_LANG_SQL && cbm_sql_values_exclusion_on(rel_path) &&
22632321
cbm_sql_values_kept_ranges(source, (uint32_t)source_len, &sql_kept) &&
22642322
ts_parser_set_included_ranges(parser, sql_kept.items, sql_kept.count);
2265-
TSTree *tree = ts_parser_parse_with_options(parser, NULL, ts_input, opts);
2323+
TSTree *tree = cbm_parse_source(parser, source, (uint32_t)source_len, opts);
22662324
if (sql_ranged) {
22672325
(void)ts_parser_set_included_ranges(parser, NULL, 0);
22682326
}
@@ -2465,7 +2523,7 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
24652523
TSParser *pp_parser = get_thread_parser(ts_lang, language);
24662524
if (pp_parser) {
24672525
ts_parser_reset(pp_parser);
2468-
CBMStringInput pp_input = {expanded, (uint32_t)expanded_len};
2526+
CBMStringInput pp_input = {expanded, (uint32_t)expanded_len, false};
24692527
TSInput pp_ts_input = {
24702528
&pp_input,
24712529
cbm_string_read,

‎internal/cbm/cbm.h‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -819,6 +819,14 @@ void cbm_work_arena_keep_begin(void);
819819
/* Free the compaction scratch this thread kept (cbm_work_arena_release calls it). */
820820
void cbm_result_compact_release_thread(void);
821821

822+
/* Parse one whole file as if its last line ended with "\n" (#2078). The
823+
* parser sees the source plus one virtual newline when the last byte is not
824+
* already one; the returned tree is then clamped back to `source_len`, so no
825+
* node range, point or text reaches past the real bytes. Every whole-file parse
826+
* goes through here, so a retained tree and a fallback re-parse agree. */
827+
TSTree *cbm_parse_source(TSParser *parser, const char *source, uint32_t source_len,
828+
TSParseOptions opts);
829+
822830
// Extract all data from one file. Caller must call cbm_free_result().
823831
// source must remain valid for the duration of the call.
824832
// timeout_micros: per-file tree-sitter parse budget in microseconds of the

‎internal/cbm/lsp/c_lsp.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6101,7 +6101,7 @@ void cbm_run_c_lsp_cross_with_registry(CBMArena *arena, const char *source, int
61016101
return;
61026102
const TSLanguage *ts_lang = cpp_mode ? tree_sitter_cpp() : tree_sitter_c();
61036103
ts_parser_set_language(parser, ts_lang);
6104-
tree = ts_parser_parse_string(parser, NULL, source, source_len);
6104+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
61056105
ts_parser_delete(parser);
61066106
owns_tree = true;
61076107
if (!tree)
@@ -6151,7 +6151,7 @@ void cbm_run_c_lsp_cross(CBMArena *arena, const char *source, int source_len, co
61516151
return;
61526152
const TSLanguage *ts_lang = cpp_mode ? tree_sitter_cpp() : tree_sitter_c();
61536153
ts_parser_set_language(parser, ts_lang);
6154-
tree = ts_parser_parse_string(parser, NULL, source, source_len);
6154+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
61556155
ts_parser_delete(parser);
61566156
owns_tree = true;
61576157
if (!tree)

‎internal/cbm/lsp/cs_lsp.c‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3763,8 +3763,9 @@ void cbm_run_cs_lsp_cross_with_registry(CBMArena *arena, const char *source, int
37633763
if (!parser)
37643764
return;
37653765
ts_parser_set_language(parser, tree_sitter_c_sharp());
3766-
tree = ts_parser_parse_string(
3767-
parser, NULL, source, source_len > 0 ? (uint32_t)source_len : (uint32_t)strlen(source));
3766+
tree = cbm_parse_source(parser, source,
3767+
source_len > 0 ? (uint32_t)source_len : (uint32_t)strlen(source),
3768+
(TSParseOptions){0});
37683769
ts_parser_delete(parser);
37693770
owns = true;
37703771
}
@@ -3805,8 +3806,9 @@ void cbm_run_cs_lsp_cross(CBMArena *arena, const char *source, int source_len,
38053806
if (!parser)
38063807
return;
38073808
ts_parser_set_language(parser, tree_sitter_c_sharp());
3808-
tree = ts_parser_parse_string(
3809-
parser, NULL, source, source_len > 0 ? (uint32_t)source_len : (uint32_t)strlen(source));
3809+
tree = cbm_parse_source(parser, source,
3810+
source_len > 0 ? (uint32_t)source_len : (uint32_t)strlen(source),
3811+
(TSParseOptions){0});
38103812
ts_parser_delete(parser);
38113813
owns = true;
38123814
}

‎internal/cbm/lsp/go_lsp.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3224,7 +3224,7 @@ void cbm_run_go_lsp_cross(CBMArena *arena, const char *source, int source_len,
32243224
if (!parser)
32253225
return;
32263226
ts_parser_set_language(parser, tree_sitter_go());
3227-
tree = ts_parser_parse_string(parser, NULL, source, source_len);
3227+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
32283228
owns_tree = true;
32293229
if (!tree) {
32303230
ts_parser_delete(parser);
@@ -3630,7 +3630,7 @@ void cbm_run_go_lsp_cross_with_registry(CBMArena *arena, const char *source, int
36303630
if (!parser)
36313631
return;
36323632
ts_parser_set_language(parser, tree_sitter_go());
3633-
tree = ts_parser_parse_string(parser, NULL, source, source_len);
3633+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
36343634
owns_tree = true;
36353635
if (!tree) {
36363636
ts_parser_delete(parser);

‎internal/cbm/lsp/java_lsp.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3931,7 +3931,7 @@ void cbm_run_java_lsp_cross_with_registry(CBMArena *arena, CBMFileResult *result
39313931
return;
39323932
}
39333933
ts_parser_set_language(parser, tree_sitter_java());
3934-
tree = ts_parser_parse_string(parser, NULL, source, (uint32_t)source_len);
3934+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
39353935
ts_parser_delete(parser);
39363936
owns_tree = true;
39373937
}
@@ -3985,7 +3985,7 @@ void cbm_run_java_lsp_cross(CBMArena *arena, const char *source, int source_len,
39853985
if (!tree) {
39863986
TSParser *parser = ts_parser_new();
39873987
ts_parser_set_language(parser, tree_sitter_java());
3988-
tree = ts_parser_parse_string(parser, NULL, source, (uint32_t)source_len);
3988+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
39893989
ts_parser_delete(parser);
39903990
owns_tree = true;
39913991
}

‎internal/cbm/lsp/kotlin_lsp.c‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5415,7 +5415,7 @@ void cbm_run_kotlin_lsp_cross(CBMArena *arena, const char *source, int source_le
54155415
return;
54165416
}
54175417
ts_parser_set_language(parser, tree_sitter_kotlin());
5418-
tree = ts_parser_parse_string(parser, NULL, source, (uint32_t)source_len);
5418+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
54195419
ts_parser_delete(parser);
54205420
owns_tree = true;
54215421
}

‎internal/cbm/lsp/php_lsp.c‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4498,7 +4498,7 @@ void cbm_run_php_lsp_cross(CBMArena *arena, const char *source, int source_len,
44984498
if (!parser)
44994499
return;
45004500
ts_parser_set_language(parser, tree_sitter_php_only());
4501-
tree = ts_parser_parse_string(parser, NULL, source, (uint32_t)source_len);
4501+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
45024502
owns_tree = true;
45034503
if (!tree) {
45044504
ts_parser_delete(parser);

‎internal/cbm/lsp/py_lsp.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5110,7 +5110,7 @@ void cbm_run_py_lsp_cross(CBMArena *arena, const char *source, int source_len,
51105110
if (!parser)
51115111
return;
51125112
ts_parser_set_language(parser, tree_sitter_python());
5113-
tree = ts_parser_parse_string(parser, NULL, source, (uint32_t)source_len);
5113+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
51145114
owns_tree = true;
51155115
if (!tree) {
51165116
ts_parser_delete(parser);
@@ -5191,7 +5191,7 @@ void cbm_run_py_lsp_cross_with_registry(CBMArena *arena, const char *source, int
51915191
if (!parser)
51925192
return;
51935193
ts_parser_set_language(parser, tree_sitter_python());
5194-
tree = ts_parser_parse_string(parser, NULL, source, (uint32_t)source_len);
5194+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
51955195
owns_tree = true;
51965196
if (!tree) {
51975197
ts_parser_delete(parser);

‎internal/cbm/lsp/rust_lsp.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6377,7 +6377,7 @@ void cbm_run_rust_lsp_cross_with_registry(CBMArena *arena, const char *source, i
63776377
if (!parser)
63786378
return;
63796379
ts_parser_set_language(parser, tree_sitter_rust());
6380-
tree = ts_parser_parse_string(parser, NULL, source, source_len);
6380+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
63816381
owns_tree = true;
63826382
if (!tree) {
63836383
ts_parser_delete(parser);
@@ -6412,7 +6412,7 @@ void cbm_run_rust_lsp_cross_with_manifest(CBMArena *arena, const char *source, i
64126412
if (!parser)
64136413
return;
64146414
ts_parser_set_language(parser, tree_sitter_rust());
6415-
tree = ts_parser_parse_string(parser, NULL, source, source_len);
6415+
tree = cbm_parse_source(parser, source, (uint32_t)source_len, (TSParseOptions){0});
64166416
owns_tree = true;
64176417
if (!tree) {
64186418
ts_parser_delete(parser);

0 commit comments

Comments
 (0)