Repository navigation
[pull] main from DeusData:main - #162
Merged
Merged
Conversation
Create missing parent directories recursively and require fixture nodes before negative graph assertions. Cover exact bytes, deep paths and blocked parents. Fixes #2034 Signed-off-by: astandrik <astandrik@yandex-team.ru>
Signed-off-by: astandrik <astandrik@yandex-team.ru>
…check (#1735) cbm_subtract_macro_invocation_regions asked "is this region a call to a file-defined macro?" before "is it inside a function?". The first question (cbm_span_is_macro_invocation) walks the source from byte 0 to find the region's lines, so every file with parse-error regions paid regions x file bytes, whatever its language and whether or not it has a single function. On a multi-MB SQL dump that is up to 256 walks over the whole file. Both predicates are pure, so asking the cheap one first changes no answer: a region no function encloses never touches the source, and the regions that do share one cbm_line_offsets table built on first use. The table gives exactly the span the walk found; the walk stays as the fallback when the table cannot be allocated. #1071 behaviour is unchanged: same regions, same verdicts. Indexing this repository's src/ before and after gives identical graphs (7,339 nodes, 28,924 edges) and the same 46 parse_partial files with byte-identical ranges. Tests count the bytes read, never time them (test-seam counter cbm_test_macro_line_scan_bytes): - 40 top-level junk regions: 39,870 bytes read -> 0, still 40 regions - 60 in-body real errors: 85,725 bytes read -> 2,801 (one table = file size), still 60 regions Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of `INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \' escapes, which the grammar does not know and error-recovers from) until the 5 s parse budget ran out, and the whole file, tables included, was skipped as "parse timeout". The reporter's database/sql/world.sql was dropped this way. The rows contribute nothing to the graph. A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c) finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the first that consists only of literals: strings (standard '' and MySQL \' escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE, FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier or a quoted identifier stays in the parse, and so does one whose string spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges as the complement, so byte offsets and line/column positions of all kept text are unchanged. The ranges are cleared right after the parse because the parser is thread-local and reused. This is a syntax rule, not a work cap: the same file always gives the same ranges, and the parse budget is unchanged. Measured on generated mysqldump-shaped dumps, release build, same machine, index worker's own numbers (base = origin/main): dump extraction pipeline peak RSS graph 8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical 8 MB MySQL \' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical 32 MB MySQL \' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed 59 MB MySQL \' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed "identical" = every stored node and edge, Module end line included. For the 32/59 MB dumps the base run skipped the file as "parse timeout" (Module + 3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB graph except for the Module's end line. With standard escapes the extraction and the stored graph are identical to the full parse. With MySQL \' escapes the full parse's error recovery swallows rows next to a misread escape, a subquery row among them, and the cut parse keeps those rows. So it loses nothing the full parse found outside the dropped rows and recovers what the full parse lost. The only things the full parse extracted INSIDE the dropped rows were junk usages (DEFAULT and _binary read as identifiers, fragments of misread strings) that name nothing. Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the exclusion off for one file so the same source can be compared both ways): - scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions - extraction equals the full parse on a small dump (standard escapes); nothing lost outside dropped rows (MySQL escapes) - the subquery and function-call rows are still parsed - the tree does not grow when the row count doubles (a count, not a clock) - a 25 MB MySQL dump under the production budget is indexed, not "parse timeout" (asserts the outcome, not the time) - stored nodes and edges of a pipeline index equal the full parse's Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished indexing: `.res` maps to ReScript, so the resource was parsed as ReScript source and the definitions pass spun. Three defects, each fixed: 1. Quadratic usage walk (the hang). The ReScript `let` binding check (is_first_named_part_of, shared with Julia, Typst, Elm, PureScript and Nickel) climbed from every named leaf with ts_node_parent, which descends from the root and scans each level's children. A binary file parses into a flat error tree (~170k root children per MB), so each climb cost O(root children) and the file O(n^2): 1 MB spent 46 s in cbm_extract_unified, 4.7 MB never returned. The check now climbs the unified walk's own cursor (O(1) per hop), like the other occurrence classifiers; the node-based climb stays only for callers outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end. 2. Quadratic lexing in error recovery. tree-sitter retries the lexer at every byte of an unparseable stretch with every external token valid; the ReScript scanner's template-string loop then ran to the next backtick/$/backslash/NUL (end of file if none) and discarded the result. That cost O(n^2) inside lexing, where the parse budget's progress callback never runs, so such files were dropped by the 5 s CPU budget instead of parsed -- plain text too (100 KB of non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the template branch when NEWLINE is also valid, which only the ERROR state allows. Valid files parse to identical trees. Recorded in vendored/grammars/MANIFEST.md. 3. Binary content parsed as source. `.res` is also Godot's binary resource format and the Windows compiled-resource format. Following the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the first 4 KB and reports a file holding a NUL byte as unsupported; ReScript text never contains one. Tests (deterministic work counters, no clocks): - extraction::extract_rescript_let_bindings_use_the_walk_cursor -- zero root-descending parent hops at 128 and 1024 statements (before: 515 / 4099). - complexity::complexity_rescript_error_recovery_lexing_is_linear -- lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio must be linear (before 3.99, after 2.00). - language::lang_res_binary_resource_unsupported and lang_res_rescript_stays_rescript. Fixes #2176 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_registry_add takes the symbol's `name`, discards it, and derives the
bare-name lookup key from the QN's last dot segment. That makes the QN's tail
load-bearing for the by-name index every language shares, and for one grammar
the tail is not a name anybody writes. rust_cfg_qualified_name mints a
`#[cfg(test)]` twin as "proj.lib.add#cfg(test)"; simple_name() has no '#'
handling, so that function is filed under the whole string "add#cfg(test)",
which no bare `add` callee can reach, leaving a test-only twin of a resolved
function absent from the by-name index entirely.
The caller always knows the name, because it is the argument. So the index
takes a second key from it, only where the derived key carries a fence:
const char *derived = simple_name(qualified_name);
index_under_name(r, derived, owned_qn);
if (name && name[0] && strchr(derived, '#') && strcmp(name, derived) != 0) {
index_under_name(r, name, owned_qn);
}
The gate keys on the fence rather than on the file's language, and that is the
part worth reviewing. It makes "unchanged for every other grammar" checkable:
the derived key is written unconditionally exactly as before, and every by-name
lookup keys through simple_name(), which splits on '.' and "::" only. A passed
name that differs from the derived key by anything other than a fence is
therefore a key no lookup can reach, so adding it would only widen the bucket
the scorer walks. A grammar that mints no '#' sees no new key at all.
Two of the rows a language gate would have had to cover turn out not to reach
this function. Registry membership is decided by cbm_label_is_registry_symbol
at all three production callers (pass_definitions.c, pass_parallel.c,
pipeline_incremental.c), and it admits neither "Module" nor "Section". So a
file's Module node never registers, and neither does a Markdown heading. What
does register, under label "Class", is the dotted-name direction:
grammar def->name QN tail
HCL resource.aws_instance.web web
TOML tool.poetry.dependencies dependencies
INI tool.isort isort
These are why the derived key stays unconditional. `instance =
aws_instance.web.id` reaches the block through the tail `web`, so re-keying the
index on the passed name would drop that edge outright. The fence gate leaves
all three byte-identical.
`name` is NULL or empty only for callers that have no symbol name to give;
those have the derived key and nothing else.
tests/test_registry.c: registry_indexes_by_passed_name_not_qn_tail resolves a
bare `add` against a cfg-fenced QN and is the reproducer, failing without this
commit. registry_indexes_a_dotted_name_under_its_tail_too pins the unconditional
derived key for the HCL and Module shapes.
tests/test_pipeline.c: pipeline_hcl_block_reference_resolves_to_its_block indexes
a two-resource Terraform file end to end and asserts the surviving USAGE edge.
The latter two pass without this commit and are guards: what would break them is
re-keying the by-name index, and an HCL block name is dotted where its QN tail is
not, so they pin the shape that regression would take.
A throwaway probe walked tests/grammar_cases.h (one fixture per grammar, all 162)
comparing def->name with simple_name(def->qualified_name), then targeted fixtures
for the dotted shapes above. HCL is the only fixture-level mismatch among
registry-eligible labels; TOML and INI mismatch on targeted fixtures. XML
namespaced elements (`ns:root`) do not mismatch, because ':' is not a QN
separator.
Signed-off-by: Chad <4307099+BobbieBarker@users.noreply.github.com>
…onflict (#2277) A client refused because another CBM build owns the account daemon printed only "Close all CBM sessions and commands, then retry." The reporter had a 0.10.3 daemon kept alive by a running Codex session while starting 0.11.0 from a terminal, and nothing in the message said which process that was, that an agent session held it, or that `daemon stop` exists. The refusal itself is correct and stays unchanged: session-managed daemons serving live sessions must never be killed by a newer client, and cbm_daemon_conflict_format() is validated byte-for-byte across generations on the HELLO wire, so its text cannot change without breaking old/new daemon pairs. The fix adds a separate, client-side remedy sentence built from the cross-build STATUS probe (cbm_daemon_conflict_remedy_format): active daemon pid, version, permanent vs session-managed, "already stopping", the pids of the CBM sessions holding it, the likely cause (another editor or agent launched a different install; compare `which -a codebase-memory-mcp` with each client's MCP config) and `daemon stop` once no session uses it. When STATUS does not answer, the remedy points at `daemon status`. It is appended on every user-facing conflict exit: the MCP/CLI bootstrap path (after the older-permanent auto-drain policy declines), the one-shot CLI exact-build cohort refusal and the MCP client cohort refusal (the path the reporter hit). Hooks skip the extra probe to keep their fail-open budget. The probe uses the `daemon status` budget (3 s): the daemon hashes the ~300 MB peer image before answering, which overran the 1 s connect budget in the real-binary proof. No identity or admission check changes; STATUS is already the authenticated, cross-build observability op behind `daemon status`. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Upgrading from v0.10.8 to v0.11.0 on a machine with OpenCode reported all
three OpenCode subagent profiles as "preserved modified profile" plus an
op=agent_install error each, and the activation stopped ("one or more agent
configurations failed; the published/current executable was kept"), although
nobody had edited those files.
Root cause: #1933 (v0.11.0) added the tool_search / tool_search_regex
permissions to the rendered OpenCode profile. Install and uninstall recognise
a profile as ours only when it matches the current rendering or one of the
known released renderings (the other access mode, the v0.9.1-rc.1 Codex
shape, the pre-tier Verify file). The v0.10.8 OpenCode shape was not in that
set, so cbm_text_migrate_owned_document classified the untouched v0.10.8
files as user-modified. Reproduced end to end: install with the v0.10.8
release binary, then install with v0.11.0/main, into an isolated HOME; the
three profiles are 1729/1880/1910 bytes, the same sizes as in the report.
cbm_render_graph_profile_opencode_v0108() renders the pre-#1933 shape
(both access modes), and install/uninstall now list it as a released
rendering, so those files are upgraded in place and uninstall removes them
as owned. A profile the user actually edited is still preserved. The
released-list assembly that install and uninstall duplicated moves into one
helper, and the two legacy renderers share one implementation (cli.c drops
three raw free sites; the memory-core baseline is tightened 173 -> 170).
The Codex and OpenCode op=mcp_install errors in the same report do not
reproduce from a pristine v0.10.8 config and depend on the reporter's own
config.toml / opencode.json; they are not addressed here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…anner patch security-static (scripts/security-vendored.sh) failed on this branch: the #2176 fix patches internal/cbm/vendored/grammars/rescript/scanner.c (skip the template-string branch in the error-recovery state) and records that patch in vendored/grammars/MANIFEST.md, but the checked-in digests in scripts/vendored-checksums.txt still described the unpatched files. Both changes are intentional and reviewed in the parent commit; this records their SHA-256 via `scripts/security-vendored.sh --update`. Only the two affected lines change, and the integrity check passes again. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two CI failures, both in the tests this branch added: 1. Windows (MinGW clang64) did not compile the test runner: test_parse_coverage.c and test_pipeline.c called POSIX setenv/unsetenv, which the Windows CRT does not declare. They now use the repo's cbm_setenv/cbm_unsetenv from foundation/compat.h, as the other suites do. 2. sql_dump_of_many_megabytes_is_indexed_not_timed_out_issue1735 failed on the ubuntu-24.04-arm ASan leg with "parse timeout". It extracted a 26 MB dump under CBM_EXTRACT_BUDGET, which is CPU time, so the verdict depended on runner speed. It passed on the fast legs and tripped on the slow one, and the early FAIL return leaked the result (the LSan report in that log). The property behind "indexed, not timed out" is a count: parse work is set by the statements, not the rows. The test now extracts unbudgeted, asserts the outcome (the view and the subquery usage are in the graph), and bounds the tree by the same 330 statements with 4 rows each. Results are freed before any assertion. Binds to the fix: forcing the full parse for the big dump through the CBM_TEST_SQL_FULL_PARSE_ON seam fails the bound (5,158,222 tree nodes vs a bound of 52,872); with the row exclusion it passes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> # Conflicts: # tests/test_pipeline.c
fix(rescript): stop binary .res resources hanging the indexer (#2176)
fix(sql,coverage): make large SQL dumps cheap to parse (#1735)
fix(daemon): name the conflicting daemon and the way out on a build conflict (#2277)
fix(install): migrate v0.10.8 OpenCode agent profiles on upgrade (#2264)
…e-name fix(registry): reach a cfg-fenced function from its bare name
fix(tests): create nested pipeline fixtures reliably
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )