You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cd368ce
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: internal/cbm/vendored/grammars/MANIFEST.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -50,7 +50,7 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r
50
50
newlines; the new files keep upstream's bytes exactly. Per-file SHA-256
51
51
values are pinned in `scripts/vendored-checksums.txt`. Regression coverage:
52
52
the `cs_*_issue1748` tests in `tests/test_parse_coverage.c`.
53
-
- **objectscript_udl / objectscript_routine** (added 2026-06-24): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) @ `a7ffcdf` — MIT, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c`/`scanner.c` are otherwise byte-for-byte upstream — on re-vendor, re-apply only that single include rename. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript).
53
+
- **objectscript_udl / objectscript_routine** (added 2026-06-24): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) @ `a7ffcdf` — MIT, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c` files remain byte-for-byte upstream. Each `scanner.c` also guards empty-state deserialization with `if (length > 0)` before `memcpy`, because Tree-sitter can pass a NULL buffer with zero length. On re-vendor, re-apply the include rename and this guard unless upstream already carries them. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript).
54
54
-**mojo** (added 2026-07-01): vendored from [lsh/tree-sitter-mojo](https://github.com/lsh/tree-sitter-mojo) @ `33193a99afe6` — MIT, ABI 15. Helix tracks `lsh/tree-sitter-mojo` as its Mojo grammar source, but the Helix-pinned commit (`3d7c53b8038f`) no longer resolves in the upstream repository after a force-push, so this vendor uses current upstream `main` rather than the stale registry SHA. Security review covered only the vendored C surface (`parser.c`, `scanner.c`, `tree_sitter/*.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored.
- **javascript + tsx** (forked 2026-09-25, #1736): **self-maintained forks** of the pins we already vendored — [tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript) @ `58404d8cf191` and [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `tsx` dialect @ `75b3874edb2d` (javascript base @ `3a837b6f3658`, the version upstream's package-lock pins, as for arkts). **Why:** upstream accepts a `&` inside a JSX string only before a space/digit or as a complete character reference, and never inside JSX text, so a plain URL query (`href="…?a=1&b=2"`, Google Fonts `&display=swap`) or `<p>Tom & Jerry</p>` produced an ERROR node — a `parse_partial` flag, plus false USAGE edges from identifiers the error recovery pulled out of the URL (on vercel/next.js `examples/`: 45 of 92 flagged files, 115 false cross-example USAGE edges). **The change** is one patch, `tools/tree-sitter-javascript/patches/jsx-lone-ampersand.patch`, applied to both javascript bases: a lone `&` that does not begin a complete `html_character_reference` is a one-character token (lexical precedence -1, so `&`/`&` still lex as references) aliased to the EXISTING `string_fragment` / `jsx_text` nodes, and the string-fragment regexes stop before a `&` that directly precedes the closing quote. Node types are unchanged (same type/named set; `node-types.json` only gains an empty `fields` map on those two entries). Without the patch both directories regenerate upstream exactly (javascript: byte-identical `parser.c` apart from the generator-version comment; tsx: identical state/symbol counts, ABI 14 → 15 from the newer CLI). **External scanners** are unchanged: `javascript/scanner.c` and `tsx/scanner.c` + `_common_scanner.h` stay the reviewed upstream bytes. Regenerate with `npx tree-sitter-cli@0.25.10 generate` in `tools/tree-sitter-{javascript,tsx}/` (ABI 15) and copy `src/parser.c` (tsx: also `src/tree_sitter/*.h`) here. **On an upstream bump:** re-fetch the new pins, re-apply the patch, regenerate, re-run `test/corpus/jsx_ampersand.txt` — and drop the fork if upstream fixes it. **LICENSE:** upstream's MIT text byte-identical (registered in the audit's `FORKS` map); our copyright for the patch lives in the fork `grammar.js` headers and `THIRD_PARTY.md`.
@@ -109,6 +109,8 @@ row instead.
109
109
| crystal |`crystal/scanner.c`, serialize | guard `memcpy(&buffer[offset], state->literals.contents, literal_content_size)` with `if (literal_content_size > 0)`| UBSan: zero-length `memcpy` with a NULL/0-size source on the empty-state serialize round-trip (formal UB, harmless) |
110
110
| rescript |`rescript/scanner.c`, deserialize | guard `memcpy(state, buffer, n_bytes)` with `if (n_bytes > 0)`| UBSan: zero-length `memcpy` with a NULL `buffer` / `n_bytes == 0` on empty-state deserialize (formal UB, harmless). The sibling serialize copies a fixed `sizeof(ScannerState)` (always > 0, non-NULL src) and needs no guard. |
111
111
| rescript | `rescript/scanner.c`, scan, `TEMPLATE_CHARS` branch | `if (valid_symbols[TEMPLATE_CHARS])` → `if (valid_symbols[TEMPLATE_CHARS] && !valid_symbols[NEWLINE])` | **Quadratic lexing in error recovery (#2176).** The only parse state whose external lex state enables both tokens is tree-sitter's ERROR state (`ts_lex_modes[0]`, external state 1; the in-template state 7 has no `_newline`), where every external token is marked valid and the lexer retries at each byte. From every byte the template loop ran to the next `` ` ``, `$`, `\` or NUL — the end of file when there is none — and returned false, so an unparseable stretch cost O(n²) inside lexing, where the parse budget's progress callback never runs. 50 KB of `0xFF` took 5.3 s, 100 KB 30 s; after: 3 ms, linear (lexer bytes pulled for 4 KB → 8 KB of `0xFF`: 8.8 M → 35.3 M before, 8.7 K → 17.4 K after, `complexity_rescript_error_recovery_lexing_is_linear`). Valid files parse to byte-identical trees; broken files recover more (a `let` after an unterminated expression is kept instead of swallowed). Upstream pin `43c2f1f35024` carries the unguarded branch |
112
+
| objectscript_routine |`objectscript_routine/scanner.c`, deserialize | guard `memcpy(payload, buffer, length)` with `if (length > 0)`| UBSan: Tree-sitter passes a NULL buffer and zero length for empty scanner state; skip the zero-length copy |
113
+
| objectscript_udl |`objectscript_udl/scanner.c`, deserialize | guard `memcpy(payload, buffer, length)` with `if (length > 0)`| UBSan: Tree-sitter passes a NULL buffer and zero length for empty scanner state; skip the zero-length copy |
112
114
| purescript |`purescript/scanner.c`, serialize | guard `memcpy(buffer, indents->data, to_copy)` with `if (to_copy > 0)`| UBSan: zero-length `memcpy` with a NULL/0-size source when the indent vector is empty (formal UB, harmless) |
113
115
| plsql |`plsql/parser.c`, include |`#include <tree_sitter/parser.h>` → `#include "tree_sitter/parser.h"`| The older ABI-14 generator emits angle brackets; every other vendored grammar uses the quoted form, which resolves the per-grammar `tree_sitter/` header from the including file's directory |
114
116
| properties | `properties/scanner.c`, whole file | move the `reached_eof` flag out of a file-scope `static bool` and into a per-parser payload allocated by `..._external_scanner_create()` | **Data race, and the only one of its kind in 103 vendored scanners.** That flag is how the scanner refuses to emit a second end-of-input `FAKE_EOL`, and upstream keeps it in ONE process-wide object. With several worker threads indexing `.properties` files at once, whichever reached EOF first set the flag, so another thread's `!reached_eof` was false and its parse never received the `FAKE_EOL` it needed to finish — it sat at end-of-input asking for a token that would never arrive. Measured 2026-09-19: ~106 M parse operations on a 253-byte file before the per-file budget stopped it, after which the file was dropped from the graph and java/kotlin indexes differed run to run. Interleaved A/B under identical load, 40 runs each: unpatched 6 stalls, patched 0. The serialize/deserialize wire format is unchanged (the state is still the returned length). Upstream `6310671b24d4` still carries the static, so a re-vendor must re-apply this |
0 commit comments