You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fa258d3
Browse filesBrowse the repository at this point in the historyBrowse files
fix(extraction): parse every file as if its last line were terminated (DeusData#2078)
A file whose last byte is not "\n" leaves the construct on its final line
unterminated for every grammar that treats the line ending as part of
that construct. DeusData#1610/DeusData#1746 already excused the mild form, a ZERO-WIDTH
MISSING newline at EOF. The severe form is a WIDTH-BEARING error that
cannot be excused. In tree-sitter-markdown an opening fence, a bare list
marker or an ATX heading as the last bytes costs the last line
(`text\n` + fence -> parse_partial 2-2) or the whole file (a lone fence,
`- ` -> parse_unusable). An unterminated one-line heading was lost
silently: the root is ERROR, there is no Section and no flag. A Makefile
whose last recipe line is unterminated lost the recipe. The same bytes
plus one "\n" parse clean. The reporter confirmed this on 21 real files
(43 -> 23 flagged after appending newlines).
Supply the newline rather than excuse its absence. cbm_parse_source()
feeds the parser one virtual "\n" at offset source_len (through the
TSInput read callback, so the caller's buffer is never copied or
written) and then applies one ts_tree_edit that deletes that byte
again. tree-sitter's edit arithmetic clamps every node that reached into
it back to the real end, bytes and row/column points both. So def line
ranges, node text, call byte spans, parse-coverage ranges and the
retained tree all see only real bytes and real lines. Empty files and
files that already end in "\n" are parsed exactly as before.
Every whole-file parse goes through the helper: the extraction parse,
and the whole-file fallback re-parses in the C/C++, C#, Go, Java,
Kotlin, PHP, Python, Rust and TS LSP passes. That keeps a spilled
result (no retained tree) consistent with a cached one. Sub-range and
synthetic parses (preprocessed C, Rust wrappers, Kotlin patches,
injected Jinja/imports ranges) are unchanged.
The two DeusData#1610/DeusData#1746 guards that pinned "an unterminated Makefile recipe
is a width-bearing loss" now use a construct that is broken with or
without a newline (an unclosed Python tuple at EOF). The recipe is no
longer lost, so flagging it would be false. The new
makefile_unterminated_recipe_is_parsed_issue2078 test pins that.
Proof (this repo, vendored grammars excluded, before/after binaries):
- unchanged tree, and the same tree with the final newline stripped from
all 1,220 files: nodes, edges and parse_partial lists byte-identical
before vs after;
- EOF-class corpus (the repo plus each .md with a fence, list marker,
heading or opening fence appended without a newline, plus an
unterminated Makefile): parse_partial 122 -> 77, exactly the
unchanged repo's list. The only graph deltas are the 45 dropped
::missed shadow File rows (+7 Folders, 52 CONTAINS edges) and 22
Section docstrings that now include their real last line.
The other four constructs in DeusData#2078 (HTML bare `&`, bash heredoc +
redirect + pipe, bash `$((10#$n))`, markdown empty `||` cell) are
upstream grammar gaps. Reports for them are drafted separately.
Refs DeusData#2078 (the EOF class; the remaining triggers are upstream grammar gaps)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
0 commit comments