Skip to content

Commit c7ccb47

Browse files
committed
fix(grammar): give the properties scanner per-parser state
Indexing a repo with .properties files in it produced a different graph from one run to the next. Every few runs one small file - 253 bytes, valid, the same file that had indexed fine a moment earlier - vanished from the graph, and the coverage tree put raw-path File/Folder nodes where it had been. On the kotlin corpus that showed up as node counts wandering between 33,486 and 33,494. The vendored tree-sitter properties grammar keeps its external scanner's entire state in a file-scope static: static bool reached_eof = false; bool ..._external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid) { lexer->result_symbol = FAKE_EOL; return reached_eof = !reached_eof && valid[FAKE_EOL] && lexer->eof(lexer); } That flag is how the scanner refuses to emit a second end-of-input FAKE_EOL, and the grammar needs exactly one to close a file. One static is one object for the whole process, so with several index workers parsing .properties files at once, whichever thread reached EOF first set the flag and the next thread's `!reached_eof` was false: its parse never received the FAKE_EOL it was waiting for and sat at end-of-input asking for a token that would never arrive. The per-file parse budget stopped it five seconds later and the file was dropped. Measured on a 253-byte file: 1,060,630 progress callbacks, ~106 M parse operations, byte offset pinned at EOF, has_error false - the parser was not recovering from an error, it was waiting. The state now lives in the payload from ..._external_scanner_create(), which is what tree-sitter's create/destroy contract is for. The serialize/deserialize wire format is unchanged: the state is still the returned length. Proof, interleaved under identical load so both binaries meet the same machine, 40 runs each: unpatched 6 stalls, patched 0. Under load the kotlin corpus now returns 33494/203579 ten times out of ten and java 693129/5523595 three times out of three; both used to wander. All 103 vendored scanners were audited for the same shape. This was the only one that WRITES a process-wide static. haskell's debug_parse_metadata and purescript's res_cont/res_fail are read-only - constants missing a const - and are left alone. Upstream 6310671b24d4 still carries the static, so MANIFEST.md records this in the local-patch table and scripts/vendored-checksums.txt is repinned. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 6f290e1 commit c7ccb47

3 files changed

Lines changed: 35 additions & 10 deletions

File tree

‎internal/cbm/vendored/grammars/MANIFEST.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ row instead.
8787
| rescript | `rescript/scanner.c`, deserialize | guard `memcpy(state, buffer, n_bytes)` with `if (n_bytes > 0)` | UBSan: zero-length `memcpy` with a NULL `buffer` / `n_bytes == 0` on empty-state deserialize (formal UB, harmless). The sibling serialize copies a fixed `sizeof(ScannerState)` (always > 0, non-NULL src) and needs no guard. |
8888
| purescript | `purescript/scanner.c`, serialize | guard `memcpy(buffer, indents->data, to_copy)` with `if (to_copy > 0)` | UBSan: zero-length `memcpy` with a NULL/0-size source when the indent vector is empty (formal UB, harmless) |
8989
| plsql | `plsql/parser.c`, include | `#include <tree_sitter/parser.h>` → `#include "tree_sitter/parser.h"` | The older ABI-14 generator emits angle brackets; every other vendored grammar uses the quoted form, which resolves the per-grammar `tree_sitter/` header from the including file's directory |
90+
| properties | `properties/scanner.c`, whole file | move the `reached_eof` flag out of a file-scope `static bool` and into a per-parser payload allocated by `..._external_scanner_create()` | **Data race, and the only one of its kind in 103 vendored scanners.** That flag is how the scanner refuses to emit a second end-of-input `FAKE_EOL`, and upstream keeps it in ONE process-wide object. With several worker threads indexing `.properties` files at once, whichever reached EOF first set the flag, so another thread's `!reached_eof` was false and its parse never received the `FAKE_EOL` it needed to finish — it sat at end-of-input asking for a token that would never arrive. Measured 2026-09-19: ~106 M parse operations on a 253-byte file before the per-file budget stopped it, after which the file was dropped from the graph and java/kotlin indexes differed run to run. Interleaved A/B under identical load, 40 runs each: unpatched 6 stalls, patched 0. The serialize/deserialize wire format is unchanged (the state is still the returned length). Upstream `6310671b24d4` still carries the static, so a re-vendor must re-apply this |
9091
| swift | `swift/scanner.c`, `OP_SYMBOL_SUPPRESSOR` + `eat_operators` | `1UL <<` / `1 <<` → `1ULL <<` | UBSan: `1 << suppressor` shifts an `int` by up to `TOKEN_COUNT` bits, undefined once the index reaches 31, while the mask it feeds is `uint64_t`. `1UL << FAKE_TRY_BANG` is the same defect on Windows, where `unsigned long` is 32 bits and `FAKE_TRY_BANG` is 32; the CLANGARM64 leg runs UBSan in trap mode, so there it is an illegal instruction rather than a log line. Upstream already carries both changes: `fb63a7004f07` (2026-04-06, upstream #558) for `eat_operators`, `6ab8d1d74ebd` (2026-08-10) for the `OP_SYMBOL_SUPPRESSOR` entry. Our pin `8abb3e8b3325` (2026-03-20) predates both, so this is a backport rather than a local invention — a re-vendor past 2026-08-10 should delete this row, not re-apply it |
9192

9293
## Vendored from verified upstream

‎internal/cbm/vendored/grammars/properties/scanner.c‎

Lines changed: 33 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎scripts/vendored-checksums.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -610,7 +610,7 @@ fbce36bb615a015ea545373da62c178fe9e7e3a7b7de656455738e6f671aa652 internal/cbm/v
610610
c4b9482069f61a2a26a590181baa34f08eb6dcc57a138def578e6f9720c35ea5 internal/cbm/vendored/grammars/prisma/tree_sitter/parser.h
611611
3297073719030b7261afd78b7267931ccc03a27f7a65ca6fc644316180f71bf6 internal/cbm/vendored/grammars/properties/LICENSE
612612
98e06c3a7b3043d7641ec0b2c28c9f66e54341e343e1f553f994ae5a6f0c0452 internal/cbm/vendored/grammars/properties/parser.c
613-
3c00e7a8ec38662b05cb00858d2959b123b2b5c783eb3dc25c80b7577b4c27ac internal/cbm/vendored/grammars/properties/scanner.c
613+
a313549615512cf06b2ee55b080bfee1559860aaa1dfcab56e9af6010323aad8 internal/cbm/vendored/grammars/properties/scanner.c
614614
115a75d000bef9c70c4de7dfbb7f2a80b90cbb14ba265056dc7abb1ef3b9b6db internal/cbm/vendored/grammars/properties/tree_sitter/alloc.h
615615
02d139d44303b0b0b85a899839dbc1346c478e7318d1f2971fad2b38fd57cb73 internal/cbm/vendored/grammars/properties/tree_sitter/array.h
616616
c4b9482069f61a2a26a590181baa34f08eb6dcc57a138def578e6f9720c35ea5 internal/cbm/vendored/grammars/properties/tree_sitter/parser.h

0 commit comments

Comments
 (0)