Skip to content

Commit 706bb2c

Browse files
DeusDatadarkverbito
andcommitted
feat(lang): add Chialisp support with a first-party tree-sitter grammar
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five reference files from chia-blockchain@main it produced 3-34 ERROR nodes each. Three defects account for all of them, and each desynchronised the rest of the file rather than losing one node: * comments were `/;.*\r\n/` -- CRLF-only, while real files are LF; * `(include foo.clib)` was rejected because of the dot in the filename; * `(defconstant NAME <expr>)` accepted only a primitive after the name. So the grammar is written here instead, in tools/tree-sitter-chialisp/, and files as first-party: its vendored LICENSE is the project's own, byte-identical to the repository root, carrying no third-party copyright. It is deliberately GENERIC -- source_file / list / symbol / string / number / hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head symbols; which lists are definitions is decided in extract_defs.c. A dialect that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs no regenerated parser. Acceptance: all five reference files parse with zero ERROR and zero MISSING nodes; the three defects above are pinned as tests. Extraction, and the four decisions in it: * `Constant` is admitted to the cross-file name registry (cbm_label_is_registry_symbol). Chialisp's condition codes live as `(defconstant ...)` in a shared .clib and are referenced by name from every puzzle; labelled but unregistered they are seeded nowhere, and roughly half of a Chialisp graph would be present as nodes yet never the target of an edge. All three registry-seeding sites route through that one predicate, so they cannot diverge. No other language emits `Constant`, so the blast radius is exactly Chialisp. Pinned by a two-file graph test that goes red when the label is removed from the predicate. * `export` / `namespace` are NOT definition heads. `(export foo)` names a function already defined in the same file, so treating it as a def head mints a second node for one symbol. They stay in the not-a-call filter. * A parameter list is a `list` too, but it binds rather than invokes. Without that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions. * `(include *standard-cl-26*)` is a dialect directive, not a file dependency; recording it would invent a dependency on a file that does not exist. Three helpers that the defs, calls and unified walks all need -- the quote-data gate, comment-skipping named-child access, and the Chialisp def-head set -- live in helpers.c rather than being copied per translation unit. They previously existed as byte-identical duplicates that had already drifted: extract_defs.c skipped comments when reading a def head while its mirror in extract_unified.c did not, so a comment between a head and its name desynchronised defs from call scope. Both now use the same accessor. Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87, first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165. Distilled from PR DeusData#919 by darkverbito, whose extraction logic -- the CLVM not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and the include/embed import handling -- is kept and is the substance of this change. The grammar that PR vendored is not used. Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 31b611a commit 706bb2c

51 files changed

Lines changed: 4136 additions & 511 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitattributes‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,7 @@ internal/cbm/vendored/grammars/**/scanner.c linguist-generated=true diff=false
1111
# the git hooks are extensionless and need explicit entries. (#1272)
1212
scripts/git-hooks/commit-msg text eol=lf
1313
scripts/hooks/pre-commit text eol=lf
14+
15+
# Chialisp grammar corpus: one case asserts CRLF comment termination, so this
16+
# file must keep its exact bytes on every platform (no eol conversion). (#919)
17+
tools/tree-sitter-chialisp/test/corpus/comments.txt -text

‎README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
55
[![CI](https://img.shields.io/github/actions/workflow/status/DeusData/codebase-memory-mcp/dry-run.yml?label=CI)](https://github.com/DeusData/codebase-memory-mcp/actions/workflows/dry-run.yml)
66
[![Tests](https://img.shields.io/badge/tests-6768_passing-brightgreen)](https://github.com/DeusData/codebase-memory-mcp)
7-
[![Languages](https://img.shields.io/badge/languages-161-orange)](https://github.com/DeusData/codebase-memory-mcp)
7+
[![Languages](https://img.shields.io/badge/languages-162-orange)](https://github.com/DeusData/codebase-memory-mcp)
88
[![Hybrid LSP](https://img.shields.io/badge/Hybrid_LSP-10_languages-blue)](#hybrid-lsp)
99
[![Agents](https://img.shields.io/badge/agent_surfaces-45-purple)](https://github.com/DeusData/codebase-memory-mcp)
1010
[![Pure C](https://img.shields.io/badge/pure_C-no_language_runtime-blue)](https://github.com/DeusData/codebase-memory-mcp)
@@ -16,7 +16,7 @@
1616

1717
**The fastest and most efficient code intelligence engine for AI coding agents.** Full-indexes an average repository in milliseconds, the Linux kernel (28M LOC, 75K files) in 3 minutes. Answers structural queries in under 1ms. Ships as a native executable with a small verified runtime-asset set for macOS, Linux, and Windows — download, run `install`, done.
1818

19-
High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 161 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 15 MCP tools. No language runtime, hosted service, or API key. Plug and play across 45 supported automatic/conditional client surfaces.
19+
High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 162 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 15 MCP tools. No language runtime, hosted service, or API key. Plug and play across 45 supported automatic/conditional client surfaces.
2020

2121
> **Research** — The design and benchmarks behind this project are described in the preprint [*Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP*](https://arxiv.org/abs/2603.27277) (arXiv:2603.27277). Evaluated across 31 real-world repositories: 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file exploration.
2222
@@ -32,7 +32,7 @@ High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-si
3232

3333
- **Extreme indexing speed** — Linux kernel (28M LOC, 75K files) in 3 minutes. RAM-first pipeline: LZ4 compression, in-memory SQLite, fused Aho-Corasick pattern matching. Memory released after indexing.
3434
- **Plug and play** — native executable plus authenticated release-owned assets for macOS (arm64/amd64), Linux (arm64/amd64), and Windows (amd64). The native install needs no Docker, language runtime, or API keys. Download → `install` → restart agent → done.
35-
- **161 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks.
35+
- **162 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks.
3636
- **120x fewer tokens** — 5 structural queries: ~3,400 tokens vs ~412,000 via file-by-file search. One graph query replaces dozens of grep/read cycles.
3737
- **45 supported automatic/conditional client surfaces** — `install` configures detected clients and safely activates conditional clients only when their documented platform, marker, or explicit existing config path is present. See [Multi-Agent Support](#multi-agent-support) for the complete matrix and manual/UI-only boundaries.
3838
- **Built-in graph visualization** — 3D interactive UI at `localhost:9749`, served from the binary itself.
@@ -796,14 +796,14 @@ codebase-memory-mcp ships a **lightweight C implementation of language type-reso
796796

797797
**Two-layer architecture:**
798798

799-
1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 161 languages. Extracts definitions, calls, imports.
799+
1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 162 languages. Extracts definitions, calls, imports.
800800
2. **Hybrid LSP pass** — type-aware, runs above the tree-sitter pass per-language. Refines call edges using the import graph plus a per-file or pre-built cross-file definition registry. Languages without a Hybrid LSP pass yet fall back to textual resolution, so you always get *some* answer.
801801

802802
The result is a knowledge graph accurate enough to drive `trace_path` across packages, inheritance hierarchies, and stdlib calls — without paying for a language server process per project.
803803

804804
## Language Support
805805

806-
161 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes):
806+
162 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes):
807807

808808
| Tier | Score | Languages |
809809
|------|-------|-----------|
@@ -829,7 +829,7 @@ src/
829829
traces/ Runtime trace ingestion
830830
ui/ Local HTTP server + verified external 3D-UI asset pack
831831
foundation/ Platform abstractions (threads, filesystem, logging, memory)
832-
internal/cbm/ Vendored tree-sitter grammars (161 languages) + AST extraction engine
832+
internal/cbm/ Vendored tree-sitter grammars (162 languages) + AST extraction engine
833833
```
834834

835835
## Security

‎THIRD_PARTY.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,15 @@ License summary:
5252
- `clojure` ([sogaiu/tree-sitter-clojure](https://github.com/sogaiu/tree-sitter-clojure)) is **CC0-1.0**;
5353
`fennel` is **CC0-1.0**; `jinja2` and `just` are **Apache-2.0**;
5454
`pine` is **ISC** (declared by its upstream).
55-
- The grammars authored in-house for this project (`cobol`, `form`, `janet`,
56-
`magma`, `protobuf`, `wolfram`) are **MIT** under the project's own license,
57-
(c) DeusData. Seven further grammars (`arkts`, `assembly`, `cfml`,
55+
- The grammars authored in-house for this project (`chialisp`, `cobol`, `form`,
56+
`janet`, `magma`, `protobuf`, `wolfram`) are **MIT** under the project's own
57+
license, (c) DeusData. Each ships the repository's own LICENSE, byte-identical
58+
to the root copy; they carry no third-party copyright because there is no
59+
third party. `chialisp` is a generic s-expression grammar for the Chia
60+
smart-coin language, written for this project because no usable public
61+
grammar exists; its source and corpus tests live in
62+
`tools/tree-sitter-chialisp/`.
63+
- Seven further grammars (`arkts`, `assembly`, `cfml`,
5864
`cfscript`, `dotenv`, `pine`, `qml`) are self-maintained forks that retain
5965
their original upstream authors' licenses — see the manifest for per-grammar
6066
provenance. `arkts` is a first-party derivative of

‎docs/index.html‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,15 @@
44
<meta charset="UTF-8">
55
<meta name="viewport" content="width=device-width, initial-scale=1.0">
66
<title>codebase-memory-mcp — Code Intelligence Knowledge Graph for AI Coding Agents</title>
7-
<meta name="description" content="codebase-memory-mcp is an open-source MCP server that indexes any codebase into a persistent knowledge graph so AI coding agents answer structural questions with ~120x fewer tokens. 161 languages, Hybrid LSP type resolution, local semantic vector search, code-clone detection, sub-1ms queries, Linux kernel indexed in 3 minutes. Native executable with verified assets and no language runtime. One command configures 45 automatic/conditional client surfaces.">
7+
<meta name="description" content="codebase-memory-mcp is an open-source MCP server that indexes any codebase into a persistent knowledge graph so AI coding agents answer structural questions with ~120x fewer tokens. 162 languages, Hybrid LSP type resolution, local semantic vector search, code-clone detection, sub-1ms queries, Linux kernel indexed in 3 minutes. Native executable with verified assets and no language runtime. One command configures 45 automatic/conditional client surfaces.">
88
<meta name="keywords" content="MCP server, code intelligence, knowledge graph, tree-sitter, Hybrid LSP, semantic code search, code embeddings, code clone detection, cross-repo analysis, data-flow analysis, Claude Code, Codex CLI, Gemini CLI, Cursor, Zed, code exploration, token reduction, call graph, dead code detection">
99
<meta name="author" content="DeusData">
1010
<meta name="robots" content="index, follow, max-image-preview:large, max-snippet:-1">
1111
<meta name="theme-color" content="#0d1117">
1212

1313
<!-- Open Graph -->
1414
<meta property="og:title" content="codebase-memory-mcp — Code Intelligence Knowledge Graph for AI Coding Agents">
15-
<meta property="og:description" content="Index any codebase into a persistent knowledge graph. AI agents answer structural questions with ~120x fewer tokens. 161 languages, Hybrid LSP, sub-1ms queries, native executable with verified assets.">
15+
<meta property="og:description" content="Index any codebase into a persistent knowledge graph. AI agents answer structural questions with ~120x fewer tokens. 162 languages, Hybrid LSP, sub-1ms queries, native executable with verified assets.">
1616
<meta property="og:type" content="website">
1717
<meta property="og:url" content="https://deusdata.github.io/codebase-memory-mcp/">
1818
<meta property="og:image" content="https://deusdata.github.io/codebase-memory-mcp/graph-ui-screenshot.png">
@@ -24,7 +24,7 @@
2424
<!-- Twitter Card -->
2525
<meta name="twitter:card" content="summary_large_image">
2626
<meta name="twitter:title" content="codebase-memory-mcp — Code Intelligence for AI Coding Agents">
27-
<meta name="twitter:description" content="Index any codebase into a knowledge graph. ~120x fewer tokens for AI code exploration. 161 languages, Hybrid LSP, sub-1ms queries, native executable with verified assets.">
27+
<meta name="twitter:description" content="Index any codebase into a knowledge graph. ~120x fewer tokens for AI code exploration. 162 languages, Hybrid LSP, sub-1ms queries, native executable with verified assets.">
2828
<meta name="twitter:image" content="https://deusdata.github.io/codebase-memory-mcp/graph-ui-screenshot.png">
2929

3030
<link rel="canonical" href="https://deusdata.github.io/codebase-memory-mcp/">
@@ -41,7 +41,7 @@
4141
"applicationSubCategory": "Model Context Protocol (MCP) server",
4242
"operatingSystem": "macOS, Linux, Windows",
4343
"softwareVersion": "0.8.1",
44-
"description": "An open-source MCP server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links, so AI coding agents answer structural questions with roughly 120x fewer tokens than file-by-file search. Parses 161 languages via tree-sitter with Hybrid LSP semantic type resolution for 9 language families. Ships as a native executable with authenticated release-owned assets; the native install needs no language runtime.",
44+
"description": "An open-source MCP server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links, so AI coding agents answer structural questions with roughly 120x fewer tokens than file-by-file search. Parses 162 languages via tree-sitter with Hybrid LSP semantic type resolution for 9 language families. Ships as a native executable with authenticated release-owned assets; the native install needs no language runtime.",
4545
"url": "https://deusdata.github.io/codebase-memory-mcp/",
4646
"downloadUrl": "https://github.com/DeusData/codebase-memory-mcp/releases/latest",
4747
"codeRepository": "https://github.com/DeusData/codebase-memory-mcp",
@@ -125,7 +125,7 @@
125125
"name": "What is codebase-memory-mcp?",
126126
"acceptedAnswer": {
127127
"@type": "Answer",
128-
"text": "codebase-memory-mcp is an open-source Model Context Protocol (MCP) server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. AI coding agents query that graph instead of reading files one by one, answering structural questions with roughly 120x fewer tokens. It parses 161 languages and ships as a native executable with authenticated release-owned assets; the native install needs no language runtime."
128+
"text": "codebase-memory-mcp is an open-source Model Context Protocol (MCP) server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. AI coding agents query that graph instead of reading files one by one, answering structural questions with roughly 120x fewer tokens. It parses 162 languages and ships as a native executable with authenticated release-owned assets; the native install needs no language runtime."
129129
}
130130
},
131131
{
@@ -141,7 +141,7 @@
141141
"name": "Which programming languages does codebase-memory-mcp support?",
142142
"acceptedAnswer": {
143143
"@type": "Answer",
144-
"text": "It supports 161 languages through vendored tree-sitter grammars compiled into the binary, including Python, Go, JavaScript, TypeScript, Rust, Java, C, C++, C#, PHP, Ruby, Kotlin, Swift, and many more. Nine language families — Python, TypeScript/JavaScript, PHP, C#, Go, C/C++, Java, Kotlin, and Rust — additionally get Hybrid LSP semantic type resolution."
144+
"text": "It supports 162 languages through vendored tree-sitter grammars compiled into the binary, including Python, Go, JavaScript, TypeScript, Rust, Java, C, C++, C#, PHP, Ruby, Kotlin, Swift, and many more. Nine language families — Python, TypeScript/JavaScript, PHP, C#, Go, C/C++, Java, Kotlin, and Rust — additionally get Hybrid LSP semantic type resolution."
145145
}
146146
},
147147
{
@@ -444,7 +444,7 @@ <h1>codebase-memory-mcp</h1>
444444
The fastest, most efficient code intelligence engine for AI coding agents. It indexes any
445445
repository into a persistent knowledge graph — full-indexing an average repo in seconds and the
446446
Linux kernel in 3 minutes — so your agent answers structural questions with ~120x fewer tokens.
447-
Tree-sitter parsing across 161 languages, Hybrid LSP type resolution, native executable with verified assets.
447+
Tree-sitter parsing across 162 languages, Hybrid LSP type resolution, native executable with verified assets.
448448
</p>
449449
<div>
450450
<div class="stat"><div class="number">~120x</div><div class="label">fewer tokens</div></div>
@@ -483,7 +483,7 @@ <h2>What is codebase-memory-mcp?</h2>
483483
Protocol (MCP)</a> server that indexes a codebase into a persistent knowledge graph of functions,
484484
classes, call chains, HTTP routes, and cross-service links. Instead of reading files one at a time,
485485
an AI coding agent queries the graph — answering structural questions with roughly 120x fewer
486-
tokens. It parses 161 languages and ships as a native executable with authenticated release-owned assets;
486+
tokens. It parses 162 languages and ships as a native executable with authenticated release-owned assets;
487487
the native install needs no language runtime.
488488
</p>
489489
<p class="muted">
@@ -606,7 +606,7 @@ <h3>Languages with full Hybrid LSP</h3>
606606
</tbody>
607607
</table>
608608
<p class="muted">
609-
The two-layer pipeline runs a fast syntactic tree-sitter pass for every one of the 161 languages,
609+
The two-layer pipeline runs a fast syntactic tree-sitter pass for every one of the 162 languages,
610610
then a type-aware Hybrid LSP pass on top for the families above. Languages without a Hybrid LSP pass
611611
yet fall back to textual resolution, so you always get an answer.
612612
</p>
@@ -699,7 +699,7 @@ <h2>How fast is it?</h2>
699699
<h2>Features</h2>
700700
<div class="features">
701701
<div class="feature">
702-
<h3>161 languages</h3>
702+
<h3>162 languages</h3>
703703
<p>Python, Go, JS, TS, TSX, Rust, Java, C++, C#, C, PHP, Ruby, Kotlin, Scala, Zig, Elixir, Haskell, OCaml, Swift, Dart, Lean 4, and many more via vendored tree-sitter grammars compiled into the binary.</p>
704704
</div>
705705
<div class="feature">

‎docs/llms.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# codebase-memory-mcp
22

3-
> An open-source Model Context Protocol (MCP) server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. AI coding agents query the graph instead of reading files one by one, answering structural questions with roughly 120x fewer tokens (~3,400 vs ~412,000 across five structural queries). Parses 161 languages via vendored tree-sitter grammars, with Hybrid LSP semantic type resolution for Python, TypeScript/JavaScript, PHP, C#, Go, C/C++, Java, Kotlin, and Rust. It also generates semantic graph edges (SEMANTICALLY_RELATED for vocabulary-mismatch matches, SIMILAR_TO for near-clone detection) and supports semantic vector search via bundled nomic-embed-code embeddings compiled into the native executable — no API key, no Ollama, no Docker. Ships a native executable with an authenticated integration asset; the UI variant adds one content-addressed frontend pack. All processing is local — the embeddings run on-device; there is no embedded LLM and no API key.
3+
> An open-source Model Context Protocol (MCP) server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. AI coding agents query the graph instead of reading files one by one, answering structural questions with roughly 120x fewer tokens (~3,400 vs ~412,000 across five structural queries). Parses 162 languages via vendored tree-sitter grammars, with Hybrid LSP semantic type resolution for Python, TypeScript/JavaScript, PHP, C#, Go, C/C++, Java, Kotlin, and Rust. It also generates semantic graph edges (SEMANTICALLY_RELATED for vocabulary-mismatch matches, SIMILAR_TO for near-clone detection) and supports semantic vector search via bundled nomic-embed-code embeddings compiled into the native executable — no API key, no Ollama, no Docker. Ships a native executable with an authenticated integration asset; the UI variant adds one content-addressed frontend pack. All processing is local — the embeddings run on-device; there is no embedded LLM and no API key.
44

55
## Key facts
66

‎internal/cbm/cbm.h‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -174,8 +174,9 @@ typedef enum {
174174
CBM_LANG_OBJECTSCRIPT_UDL, // InterSystems ObjectScript UDL (.cls class files)
175175
CBM_LANG_OBJECTSCRIPT_ROUTINE, // InterSystems ObjectScript routine (.mac/.int/.rtn/.inc)
176176
CBM_LANG_OBJECTSCRIPT_EXPORT, // InterSystems Studio Export XML (<Export generator="Cache">)
177-
CBM_LANG_ARKTS, // ArkTS (HarmonyOS/OpenHarmony .ets — TypeScript superset + ArkUI)
178-
CBM_LANG_PLSQL, // Oracle PL/SQL
177+
CBM_LANG_ARKTS, // ArkTS (HarmonyOS/OpenHarmony .ets — TypeScript superset + ArkUI)
178+
CBM_LANG_PLSQL, // Oracle PL/SQL
179+
CBM_LANG_CHIALISP, // Chialisp (.clsp/.clib/.clinc — Chia smart-coin s-expression language)
179180
CBM_LANG_COUNT
180181
} CBMLanguage;
181182

0 commit comments

Comments
 (0)