Skip to content

Commit 834e4f0

Browse files
authored
Merge branch 'main' into fix/issue-1682-dotted-ts-imports
2 parents 28d9892 + 997d087 commit 834e4f0

105 files changed

Lines changed: 14294 additions & 838 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitattributes‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,7 @@ internal/cbm/vendored/grammars/**/scanner.c linguist-generated=true diff=false
1111
# the git hooks are extensionless and need explicit entries. (#1272)
1212
scripts/git-hooks/commit-msg text eol=lf
1313
scripts/hooks/pre-commit text eol=lf
14+
15+
# Chialisp grammar corpus: one case asserts CRLF comment termination, so this
16+
# file must keep its exact bytes on every platform (no eol conversion). (#919)
17+
tools/tree-sitter-chialisp/test/corpus/comments.txt -text

‎Makefile.cbm‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -354,6 +354,7 @@ PIPELINE_SRCS = \
354354
src/pipeline/pass_similarity.c \
355355
src/pipeline/pass_semantic_edges.c \
356356
src/pipeline/pass_complexity.c \
357+
src/pipeline/pass_importance.c \
357358
src/pipeline/pass_cross_repo.c \
358359
src/pipeline/artifact.c \
359360
src/pipeline/pass_ensemble_routing.c \
@@ -560,7 +561,7 @@ TEST_DISCOVER_SRCS = \
560561

561562
TEST_GRAPH_BUFFER_SRCS = tests/test_graph_buffer.c
562563

563-
TEST_PIPELINE_SRCS = tests/test_registry.c tests/test_pipeline.c tests/test_cross_repo.c tests/test_fqn.c tests/test_route_canon.c tests/test_path_alias.c tests/test_configlink.c tests/test_infrascan.c tests/test_worker_pool.c tests/test_parallel.c tests/test_index_resilience.c tests/test_index_format.c tests/test_call_reference_contract.c tests/repro/repro_call_scope_usages.c tests/repro/repro_call_argument_usages.c tests/repro/repro_reference_precision.c tests/repro/repro_lexical_binding_precision.c tests/repro/repro_call_argument_matrix_a.c tests/repro/repro_call_argument_matrix_b.c tests/repro/repro_call_node_behaviors.c tests/repro/repro_language_registry.c tests/repro/repro_call_node_manifest.c tests/repro/repro_lsp_ordered_signatures.c tests/repro/repro_lsp_ordered_local.c tests/repro/repro_ts_overload_return_chains.c tests/repro/repro_harness_cleanup.c tests/repro/repro_runner_filter.c
564+
TEST_PIPELINE_SRCS = tests/test_registry.c tests/test_pipeline.c tests/test_importance.c tests/test_cross_repo.c tests/test_fqn.c tests/test_route_canon.c tests/test_path_alias.c tests/test_configlink.c tests/test_infrascan.c tests/test_worker_pool.c tests/test_parallel.c tests/test_index_resilience.c tests/test_index_format.c tests/test_call_reference_contract.c tests/repro/repro_call_scope_usages.c tests/repro/repro_call_argument_usages.c tests/repro/repro_reference_precision.c tests/repro/repro_lexical_binding_precision.c tests/repro/repro_call_argument_matrix_a.c tests/repro/repro_call_argument_matrix_b.c tests/repro/repro_call_node_behaviors.c tests/repro/repro_language_registry.c tests/repro/repro_call_node_manifest.c tests/repro/repro_lsp_ordered_signatures.c tests/repro/repro_lsp_ordered_local.c tests/repro/repro_ts_overload_return_chains.c tests/repro/repro_harness_cleanup.c tests/repro/repro_runner_filter.c
564565

565566
TEST_WATCHER_SRCS = tests/test_watcher.c
566567

‎README.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
55
[![CI](https://img.shields.io/github/actions/workflow/status/DeusData/codebase-memory-mcp/dry-run.yml?label=CI)](https://github.com/DeusData/codebase-memory-mcp/actions/workflows/dry-run.yml)
66
[![Tests](https://img.shields.io/badge/tests-6768_passing-brightgreen)](https://github.com/DeusData/codebase-memory-mcp)
7-
[![Languages](https://img.shields.io/badge/languages-161-orange)](https://github.com/DeusData/codebase-memory-mcp)
7+
[![Languages](https://img.shields.io/badge/languages-162-orange)](https://github.com/DeusData/codebase-memory-mcp)
88
[![Hybrid LSP](https://img.shields.io/badge/Hybrid_LSP-10_languages-blue)](#hybrid-lsp)
99
[![Agents](https://img.shields.io/badge/agent_surfaces-45-purple)](https://github.com/DeusData/codebase-memory-mcp)
1010
[![Pure C](https://img.shields.io/badge/pure_C-no_language_runtime-blue)](https://github.com/DeusData/codebase-memory-mcp)
@@ -16,7 +16,7 @@
1616

1717
**The fastest and most efficient code intelligence engine for AI coding agents.** Full-indexes an average repository in milliseconds, the Linux kernel (28M LOC, 75K files) in 3 minutes. Answers structural queries in under 1ms. Ships as a native executable with a small verified runtime-asset set for macOS, Linux, and Windows — download, run `install`, done.
1818

19-
High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 161 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 15 MCP tools. No language runtime, hosted service, or API key. Plug and play across 45 supported automatic/conditional client surfaces.
19+
High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 162 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 15 MCP tools. No language runtime, hosted service, or API key. Plug and play across 45 supported automatic/conditional client surfaces.
2020

2121
> **Research** — The design and benchmarks behind this project are described in the preprint [*Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP*](https://arxiv.org/abs/2603.27277) (arXiv:2603.27277). Evaluated across 31 real-world repositories: 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file exploration.
2222
@@ -32,7 +32,7 @@ High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-si
3232

3333
- **Extreme indexing speed** — Linux kernel (28M LOC, 75K files) in 3 minutes. RAM-first pipeline: LZ4 compression, in-memory SQLite, fused Aho-Corasick pattern matching. Memory released after indexing.
3434
- **Plug and play** — native executable plus authenticated release-owned assets for macOS (arm64/amd64), Linux (arm64/amd64), and Windows (amd64). The native install needs no Docker, language runtime, or API keys. Download → `install` → restart agent → done.
35-
- **161 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks.
35+
- **162 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks.
3636
- **120x fewer tokens** — 5 structural queries: ~3,400 tokens vs ~412,000 via file-by-file search. One graph query replaces dozens of grep/read cycles.
3737
- **45 supported automatic/conditional client surfaces** — `install` configures detected clients and safely activates conditional clients only when their documented platform, marker, or explicit existing config path is present. See [Multi-Agent Support](#multi-agent-support) for the complete matrix and manual/UI-only boundaries.
3838
- **Built-in graph visualization** — 3D interactive UI at `localhost:9749`, served from the binary itself.
@@ -289,6 +289,8 @@ Benchmarked on Apple M3 Pro:
289289

290290
**Token efficiency**: Five structural queries consumed ~3,400 tokens via codebase-memory-mcp versus ~412,000 tokens via file-by-file grep exploration — a **99.2% reduction**.
291291

292+
To measure comparable quality, latency, and agent-efficiency metrics on your own workload, see [Measuring quality, latency, and agent savings](docs/MEASURING_SAVINGS.md). Exact reproduction of the figures above requires the original inputs and raw artifacts.
293+
292294
## Troubleshooting & Diagnostics
293295

294296
codebase-memory-mcp runs **100% locally and collects no telemetry** — your code, queries, environment, and usage never leave your machine. That privacy guarantee also means that when you hit something we can't reproduce on our side (a slow memory climb over hours, a performance regression, a leak that only appears after days of real use), **we have no data at all unless you choose to send it.** Here is how to capture it yourself.
@@ -664,9 +666,11 @@ JSON arguments can also be piped on stdin, for tools that take arguments. A tool
664666
| `get_code_snippet` | Read source code for a function by qualified name. |
665667
| `get_architecture` | Codebase overview: languages, packages, routes, hotspots, clusters, ADR. |
666668
| `search_code` | Grep-like text search within indexed project files. |
667-
| `manage_adr` | CRUD for Architecture Decision Records. Query modes do not wait behind a same-project reindex; writes remain serialized. |
669+
| `manage_adr` | CRUD for Architecture Decision Records (`get` reads, `update` replaces the whole document, `set_sections` rewrites only the named sections and leaves every other byte untouched, `sections` lists headings). Query modes do not wait behind a same-project reindex; writes remain serialized. |
668670
| `ingest_traces` | Ingest runtime traces to validate HTTP_CALLS edges. |
669671

672+
`manage_adr(mode='set_sections')` writes one or more sections by name and splices them into the stored document, so text outside the named sections — including a preamble, code fences and section ordering — is preserved byte-for-byte. Any `## Heading` works, not just the conventional PURPOSE / STACK / ARCHITECTURE / PATTERNS / TRADEOFFS / PHILOSOPHY set; names match exactly, including case. Writing the same section twice is a no-op, so a retry after a lost response cannot duplicate content.
673+
670674
`manage_adr` query modes (`get` and `sections`) use the server's cached query store so they can proceed while a same-project reindex is running. If another process publishes a replacement store during reindexing, they can return the pre-publication ADR until idle eviction refreshes that cache. Updates remain serialized through the project mutation guard.
671675

672676
## Graph Data Model
@@ -796,14 +800,14 @@ codebase-memory-mcp ships a **lightweight C implementation of language type-reso
796800

797801
**Two-layer architecture:**
798802

799-
1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 161 languages. Extracts definitions, calls, imports.
803+
1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 162 languages. Extracts definitions, calls, imports.
800804
2. **Hybrid LSP pass** — type-aware, runs above the tree-sitter pass per-language. Refines call edges using the import graph plus a per-file or pre-built cross-file definition registry. Languages without a Hybrid LSP pass yet fall back to textual resolution, so you always get *some* answer.
801805

802806
The result is a knowledge graph accurate enough to drive `trace_path` across packages, inheritance hierarchies, and stdlib calls — without paying for a language server process per project.
803807

804808
## Language Support
805809

806-
161 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes):
810+
162 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes):
807811

808812
| Tier | Score | Languages |
809813
|------|-------|-----------|
@@ -829,7 +833,7 @@ src/
829833
traces/ Runtime trace ingestion
830834
ui/ Local HTTP server + verified external 3D-UI asset pack
831835
foundation/ Platform abstractions (threads, filesystem, logging, memory)
832-
internal/cbm/ Vendored tree-sitter grammars (161 languages) + AST extraction engine
836+
internal/cbm/ Vendored tree-sitter grammars (162 languages) + AST extraction engine
833837
```
834838

835839
## Security

‎THIRD_PARTY.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,15 @@ License summary:
5252
- `clojure` ([sogaiu/tree-sitter-clojure](https://github.com/sogaiu/tree-sitter-clojure)) is **CC0-1.0**;
5353
`fennel` is **CC0-1.0**; `jinja2` and `just` are **Apache-2.0**;
5454
`pine` is **ISC** (declared by its upstream).
55-
- The grammars authored in-house for this project (`cobol`, `form`, `janet`,
56-
`magma`, `protobuf`, `wolfram`) are **MIT** under the project's own license,
57-
(c) DeusData. Seven further grammars (`arkts`, `assembly`, `cfml`,
55+
- The grammars authored in-house for this project (`chialisp`, `cobol`, `form`,
56+
`janet`, `magma`, `protobuf`, `wolfram`) are **MIT** under the project's own
57+
license, (c) DeusData. Each ships the repository's own LICENSE, byte-identical
58+
to the root copy; they carry no third-party copyright because there is no
59+
third party. `chialisp` is a generic s-expression grammar for the Chia
60+
smart-coin language, written for this project because no usable public
61+
grammar exists; its source and corpus tests live in
62+
`tools/tree-sitter-chialisp/`.
63+
- Seven further grammars (`arkts`, `assembly`, `cfml`,
5864
`cfscript`, `dotenv`, `pine`, `qml`) are self-maintained forks that retain
5965
their original upstream authors' licenses — see the manifest for per-grammar
6066
provenance. `arkts` is a first-party derivative of

0 commit comments

Comments
 (0)