Repository navigation
Commit c7ccb47
committed
fix(grammar): give the properties scanner per-parser state
Indexing a repo with .properties files in it produced a different graph from
one run to the next. Every few runs one small file - 253 bytes, valid, the same
file that had indexed fine a moment earlier - vanished from the graph, and the
coverage tree put raw-path File/Folder nodes where it had been. On the kotlin
corpus that showed up as node counts wandering between 33,486 and 33,494.
The vendored tree-sitter properties grammar keeps its external scanner's entire
state in a file-scope static:
static bool reached_eof = false;
bool ..._external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid) {
lexer->result_symbol = FAKE_EOL;
return reached_eof = !reached_eof && valid[FAKE_EOL] && lexer->eof(lexer);
}
That flag is how the scanner refuses to emit a second end-of-input FAKE_EOL,
and the grammar needs exactly one to close a file. One static is one object for
the whole process, so with several index workers parsing .properties files at
once, whichever thread reached EOF first set the flag and the next thread's
`!reached_eof` was false: its parse never received the FAKE_EOL it was waiting
for and sat at end-of-input asking for a token that would never arrive. The
per-file parse budget stopped it five seconds later and the file was dropped.
Measured on a 253-byte file: 1,060,630 progress callbacks, ~106 M parse
operations, byte offset pinned at EOF, has_error false - the parser was not
recovering from an error, it was waiting.
The state now lives in the payload from ..._external_scanner_create(), which is
what tree-sitter's create/destroy contract is for. The serialize/deserialize
wire format is unchanged: the state is still the returned length.
Proof, interleaved under identical load so both binaries meet the same machine,
40 runs each: unpatched 6 stalls, patched 0. Under load the kotlin corpus now
returns 33494/203579 ten times out of ten and java 693129/5523595 three times
out of three; both used to wander.
All 103 vendored scanners were audited for the same shape. This was the only
one that WRITES a process-wide static. haskell's debug_parse_metadata and
purescript's res_cont/res_fail are read-only - constants missing a const - and
are left alone.
Upstream 6310671b24d4 still carries the static, so MANIFEST.md records this in
the local-patch table and scripts/vendored-checksums.txt is repinned.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>1 parent 6f290e1 commit c7ccb47
3 files changed
Lines changed: 35 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
90 | 91 | | |
91 | 92 | | |
92 | 93 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
610 | 610 | | |
611 | 611 | | |
612 | 612 | | |
613 | | - | |
| 613 | + | |
614 | 614 | | |
615 | 615 | | |
616 | 616 | | |
| |||
0 commit comments