Skip to content

Commit a272663

Browse files
authored
Merge pull request DeusData#1888 from DeusData/feat/919-chialisp-distill
feat(lang): add Chialisp support with a first-party tree-sitter grammar
2 parents 5d96377 + 82c4067 commit a272663

51 files changed

Lines changed: 3749 additions & 72 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitattributes‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,7 @@ internal/cbm/vendored/grammars/**/scanner.c linguist-generated=true diff=false
1111
# the git hooks are extensionless and need explicit entries. (#1272)
1212
scripts/git-hooks/commit-msg text eol=lf
1313
scripts/hooks/pre-commit text eol=lf
14+
15+
# Chialisp grammar corpus: one case asserts CRLF comment termination, so this
16+
# file must keep its exact bytes on every platform (no eol conversion). (#919)
17+
tools/tree-sitter-chialisp/test/corpus/comments.txt -text

‎README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
55
[![CI](https://img.shields.io/github/actions/workflow/status/DeusData/codebase-memory-mcp/dry-run.yml?label=CI)](https://github.com/DeusData/codebase-memory-mcp/actions/workflows/dry-run.yml)
66
[![Tests](https://img.shields.io/badge/tests-6768_passing-brightgreen)](https://github.com/DeusData/codebase-memory-mcp)
7-
[![Languages](https://img.shields.io/badge/languages-161-orange)](https://github.com/DeusData/codebase-memory-mcp)
7+
[![Languages](https://img.shields.io/badge/languages-162-orange)](https://github.com/DeusData/codebase-memory-mcp)
88
[![Hybrid LSP](https://img.shields.io/badge/Hybrid_LSP-10_languages-blue)](#hybrid-lsp)
99
[![Agents](https://img.shields.io/badge/agent_surfaces-45-purple)](https://github.com/DeusData/codebase-memory-mcp)
1010
[![Pure C](https://img.shields.io/badge/pure_C-no_language_runtime-blue)](https://github.com/DeusData/codebase-memory-mcp)
@@ -16,7 +16,7 @@
1616

1717
**The fastest and most efficient code intelligence engine for AI coding agents.** Full-indexes an average repository in milliseconds, the Linux kernel (28M LOC, 75K files) in 3 minutes. Answers structural queries in under 1ms. Ships as a native executable with a small verified runtime-asset set for macOS, Linux, and Windows — download, run `install`, done.
1818

19-
High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 161 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 15 MCP tools. No language runtime, hosted service, or API key. Plug and play across 45 supported automatic/conditional client surfaces.
19+
High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 162 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 15 MCP tools. No language runtime, hosted service, or API key. Plug and play across 45 supported automatic/conditional client surfaces.
2020

2121
> **Research** — The design and benchmarks behind this project are described in the preprint [*Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP*](https://arxiv.org/abs/2603.27277) (arXiv:2603.27277). Evaluated across 31 real-world repositories: 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file exploration.
2222
@@ -32,7 +32,7 @@ High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-si
3232

3333
- **Extreme indexing speed** — Linux kernel (28M LOC, 75K files) in 3 minutes. RAM-first pipeline: LZ4 compression, in-memory SQLite, fused Aho-Corasick pattern matching. Memory released after indexing.
3434
- **Plug and play** — native executable plus authenticated release-owned assets for macOS (arm64/amd64), Linux (arm64/amd64), and Windows (amd64). The native install needs no Docker, language runtime, or API keys. Download → `install` → restart agent → done.
35-
- **161 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks.
35+
- **162 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks.
3636
- **120x fewer tokens** — 5 structural queries: ~3,400 tokens vs ~412,000 via file-by-file search. One graph query replaces dozens of grep/read cycles.
3737
- **45 supported automatic/conditional client surfaces** — `install` configures detected clients and safely activates conditional clients only when their documented platform, marker, or explicit existing config path is present. See [Multi-Agent Support](#multi-agent-support) for the complete matrix and manual/UI-only boundaries.
3838
- **Built-in graph visualization** — 3D interactive UI at `localhost:9749`, served from the binary itself.
@@ -798,14 +798,14 @@ codebase-memory-mcp ships a **lightweight C implementation of language type-reso
798798

799799
**Two-layer architecture:**
800800

801-
1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 161 languages. Extracts definitions, calls, imports.
801+
1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 162 languages. Extracts definitions, calls, imports.
802802
2. **Hybrid LSP pass** — type-aware, runs above the tree-sitter pass per-language. Refines call edges using the import graph plus a per-file or pre-built cross-file definition registry. Languages without a Hybrid LSP pass yet fall back to textual resolution, so you always get *some* answer.
803803

804804
The result is a knowledge graph accurate enough to drive `trace_path` across packages, inheritance hierarchies, and stdlib calls — without paying for a language server process per project.
805805

806806
## Language Support
807807

808-
161 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes):
808+
162 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes):
809809

810810
| Tier | Score | Languages |
811811
|------|-------|-----------|
@@ -831,7 +831,7 @@ src/
831831
traces/ Runtime trace ingestion
832832
ui/ Local HTTP server + verified external 3D-UI asset pack
833833
foundation/ Platform abstractions (threads, filesystem, logging, memory)
834-
internal/cbm/ Vendored tree-sitter grammars (161 languages) + AST extraction engine
834+
internal/cbm/ Vendored tree-sitter grammars (162 languages) + AST extraction engine
835835
```
836836

837837
## Security

‎THIRD_PARTY.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,15 @@ License summary:
5252
- `clojure` ([sogaiu/tree-sitter-clojure](https://github.com/sogaiu/tree-sitter-clojure)) is **CC0-1.0**;
5353
`fennel` is **CC0-1.0**; `jinja2` and `just` are **Apache-2.0**;
5454
`pine` is **ISC** (declared by its upstream).
55-
- The grammars authored in-house for this project (`cobol`, `form`, `janet`,
56-
`magma`, `protobuf`, `wolfram`) are **MIT** under the project's own license,
57-
(c) DeusData. Seven further grammars (`arkts`, `assembly`, `cfml`,
55+
- The grammars authored in-house for this project (`chialisp`, `cobol`, `form`,
56+
`janet`, `magma`, `protobuf`, `wolfram`) are **MIT** under the project's own
57+
license, (c) DeusData. Each ships the repository's own LICENSE, byte-identical
58+
to the root copy; they carry no third-party copyright because there is no
59+
third party. `chialisp` is a generic s-expression grammar for the Chia
60+
smart-coin language, written for this project because no usable public
61+
grammar exists; its source and corpus tests live in
62+
`tools/tree-sitter-chialisp/`.
63+
- Seven further grammars (`arkts`, `assembly`, `cfml`,
5864
`cfscript`, `dotenv`, `pine`, `qml`) are self-maintained forks that retain
5965
their original upstream authors' licenses — see the manifest for per-grammar
6066
provenance. `arkts` is a first-party derivative of

‎docs/index.html‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,15 @@
44
<meta charset="UTF-8">
55
<meta name="viewport" content="width=device-width, initial-scale=1.0">
66
<title>codebase-memory-mcp — Code Intelligence Knowledge Graph for AI Coding Agents</title>
7-
<meta name="description" content="codebase-memory-mcp is an open-source MCP server that indexes any codebase into a persistent knowledge graph so AI coding agents answer structural questions with ~120x fewer tokens. 161 languages, Hybrid LSP type resolution, local semantic vector search, code-clone detection, sub-1ms queries, Linux kernel indexed in 3 minutes. Native executable with verified assets and no language runtime. One command configures 45 automatic/conditional client surfaces.">
7+
<meta name="description" content="codebase-memory-mcp is an open-source MCP server that indexes any codebase into a persistent knowledge graph so AI coding agents answer structural questions with ~120x fewer tokens. 162 languages, Hybrid LSP type resolution, local semantic vector search, code-clone detection, sub-1ms queries, Linux kernel indexed in 3 minutes. Native executable with verified assets and no language runtime. One command configures 45 automatic/conditional client surfaces.">
88
<meta name="keywords" content="MCP server, code intelligence, knowledge graph, tree-sitter, Hybrid LSP, semantic code search, code embeddings, code clone detection, cross-repo analysis, data-flow analysis, Claude Code, Codex CLI, Gemini CLI, Cursor, Zed, code exploration, token reduction, call graph, dead code detection">
99
<meta name="author" content="DeusData">
1010
<meta name="robots" content="index, follow, max-image-preview:large, max-snippet:-1">
1111
<meta name="theme-color" content="#0d1117">
1212

1313
<!-- Open Graph -->
1414
<meta property="og:title" content="codebase-memory-mcp — Code Intelligence Knowledge Graph for AI Coding Agents">
15-
<meta property="og:description" content="Index any codebase into a persistent knowledge graph. AI agents answer structural questions with ~120x fewer tokens. 161 languages, Hybrid LSP, sub-1ms queries, native executable with verified assets.">
15+
<meta property="og:description" content="Index any codebase into a persistent knowledge graph. AI agents answer structural questions with ~120x fewer tokens. 162 languages, Hybrid LSP, sub-1ms queries, native executable with verified assets.">
1616
<meta property="og:type" content="website">
1717
<meta property="og:url" content="https://deusdata.github.io/codebase-memory-mcp/">
1818
<meta property="og:image" content="https://deusdata.github.io/codebase-memory-mcp/graph-ui-screenshot.png">
@@ -24,7 +24,7 @@
2424
<!-- Twitter Card -->
2525
<meta name="twitter:card" content="summary_large_image">
2626
<meta name="twitter:title" content="codebase-memory-mcp — Code Intelligence for AI Coding Agents">
27-
<meta name="twitter:description" content="Index any codebase into a knowledge graph. ~120x fewer tokens for AI code exploration. 161 languages, Hybrid LSP, sub-1ms queries, native executable with verified assets.">
27+
<meta name="twitter:description" content="Index any codebase into a knowledge graph. ~120x fewer tokens for AI code exploration. 162 languages, Hybrid LSP, sub-1ms queries, native executable with verified assets.">
2828
<meta name="twitter:image" content="https://deusdata.github.io/codebase-memory-mcp/graph-ui-screenshot.png">
2929

3030
<link rel="canonical" href="https://deusdata.github.io/codebase-memory-mcp/">
@@ -41,7 +41,7 @@
4141
"applicationSubCategory": "Model Context Protocol (MCP) server",
4242
"operatingSystem": "macOS, Linux, Windows",
4343
"softwareVersion": "0.8.1",
44-
"description": "An open-source MCP server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links, so AI coding agents answer structural questions with roughly 120x fewer tokens than file-by-file search. Parses 161 languages via tree-sitter with Hybrid LSP semantic type resolution for 9 language families. Ships as a native executable with authenticated release-owned assets; the native install needs no language runtime.",
44+
"description": "An open-source MCP server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links, so AI coding agents answer structural questions with roughly 120x fewer tokens than file-by-file search. Parses 162 languages via tree-sitter with Hybrid LSP semantic type resolution for 9 language families. Ships as a native executable with authenticated release-owned assets; the native install needs no language runtime.",
4545
"url": "https://deusdata.github.io/codebase-memory-mcp/",
4646
"downloadUrl": "https://github.com/DeusData/codebase-memory-mcp/releases/latest",
4747
"codeRepository": "https://github.com/DeusData/codebase-memory-mcp",
@@ -125,7 +125,7 @@
125125
"name": "What is codebase-memory-mcp?",
126126
"acceptedAnswer": {
127127
"@type": "Answer",
128-
"text": "codebase-memory-mcp is an open-source Model Context Protocol (MCP) server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. AI coding agents query that graph instead of reading files one by one, answering structural questions with roughly 120x fewer tokens. It parses 161 languages and ships as a native executable with authenticated release-owned assets; the native install needs no language runtime."
128+
"text": "codebase-memory-mcp is an open-source Model Context Protocol (MCP) server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. AI coding agents query that graph instead of reading files one by one, answering structural questions with roughly 120x fewer tokens. It parses 162 languages and ships as a native executable with authenticated release-owned assets; the native install needs no language runtime."
129129
}
130130
},
131131
{
@@ -141,7 +141,7 @@
141141
"name": "Which programming languages does codebase-memory-mcp support?",
142142
"acceptedAnswer": {
143143
"@type": "Answer",
144-
"text": "It supports 161 languages through vendored tree-sitter grammars compiled into the binary, including Python, Go, JavaScript, TypeScript, Rust, Java, C, C++, C#, PHP, Ruby, Kotlin, Swift, and many more. Nine language families — Python, TypeScript/JavaScript, PHP, C#, Go, C/C++, Java, Kotlin, and Rust — additionally get Hybrid LSP semantic type resolution."
144+
"text": "It supports 162 languages through vendored tree-sitter grammars compiled into the binary, including Python, Go, JavaScript, TypeScript, Rust, Java, C, C++, C#, PHP, Ruby, Kotlin, Swift, and many more. Nine language families — Python, TypeScript/JavaScript, PHP, C#, Go, C/C++, Java, Kotlin, and Rust — additionally get Hybrid LSP semantic type resolution."
145145
}
146146
},
147147
{
@@ -444,7 +444,7 @@ <h1>codebase-memory-mcp</h1>
444444
The fastest, most efficient code intelligence engine for AI coding agents. It indexes any
445445
repository into a persistent knowledge graph — full-indexing an average repo in seconds and the
446446
Linux kernel in 3 minutes — so your agent answers structural questions with ~120x fewer tokens.
447-
Tree-sitter parsing across 161 languages, Hybrid LSP type resolution, native executable with verified assets.
447+
Tree-sitter parsing across 162 languages, Hybrid LSP type resolution, native executable with verified assets.
448448
</p>
449449
<div>
450450
<div class="stat"><div class="number">~120x</div><div class="label">fewer tokens</div></div>
@@ -483,7 +483,7 @@ <h2>What is codebase-memory-mcp?</h2>
483483
Protocol (MCP)</a> server that indexes a codebase into a persistent knowledge graph of functions,
484484
classes, call chains, HTTP routes, and cross-service links. Instead of reading files one at a time,
485485
an AI coding agent queries the graph — answering structural questions with roughly 120x fewer
486-
tokens. It parses 161 languages and ships as a native executable with authenticated release-owned assets;
486+
tokens. It parses 162 languages and ships as a native executable with authenticated release-owned assets;
487487
the native install needs no language runtime.
488488
</p>
489489
<p class="muted">
@@ -606,7 +606,7 @@ <h3>Languages with full Hybrid LSP</h3>
606606
</tbody>
607607
</table>
608608
<p class="muted">
609-
The two-layer pipeline runs a fast syntactic tree-sitter pass for every one of the 161 languages,
609+
The two-layer pipeline runs a fast syntactic tree-sitter pass for every one of the 162 languages,
610610
then a type-aware Hybrid LSP pass on top for the families above. Languages without a Hybrid LSP pass
611611
yet fall back to textual resolution, so you always get an answer.
612612
</p>
@@ -699,7 +699,7 @@ <h2>How fast is it?</h2>
699699
<h2>Features</h2>
700700
<div class="features">
701701
<div class="feature">
702-
<h3>161 languages</h3>
702+
<h3>162 languages</h3>
703703
<p>Python, Go, JS, TS, TSX, Rust, Java, C++, C#, C, PHP, Ruby, Kotlin, Scala, Zig, Elixir, Haskell, OCaml, Swift, Dart, Lean 4, and many more via vendored tree-sitter grammars compiled into the binary.</p>
704704
</div>
705705
<div class="feature">

‎docs/llms.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# codebase-memory-mcp
22

3-
> An open-source Model Context Protocol (MCP) server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. AI coding agents query the graph instead of reading files one by one, answering structural questions with roughly 120x fewer tokens (~3,400 vs ~412,000 across five structural queries). Parses 161 languages via vendored tree-sitter grammars, with Hybrid LSP semantic type resolution for Python, TypeScript/JavaScript, PHP, C#, Go, C/C++, Java, Kotlin, and Rust. It also generates semantic graph edges (SEMANTICALLY_RELATED for vocabulary-mismatch matches, SIMILAR_TO for near-clone detection) and supports semantic vector search via bundled nomic-embed-code embeddings compiled into the native executable — no API key, no Ollama, no Docker. Ships a native executable with an authenticated integration asset; the UI variant adds one content-addressed frontend pack. All processing is local — the embeddings run on-device; there is no embedded LLM and no API key.
3+
> An open-source Model Context Protocol (MCP) server that indexes a codebase into a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. AI coding agents query the graph instead of reading files one by one, answering structural questions with roughly 120x fewer tokens (~3,400 vs ~412,000 across five structural queries). Parses 162 languages via vendored tree-sitter grammars, with Hybrid LSP semantic type resolution for Python, TypeScript/JavaScript, PHP, C#, Go, C/C++, Java, Kotlin, and Rust. It also generates semantic graph edges (SEMANTICALLY_RELATED for vocabulary-mismatch matches, SIMILAR_TO for near-clone detection) and supports semantic vector search via bundled nomic-embed-code embeddings compiled into the native executable — no API key, no Ollama, no Docker. Ships a native executable with an authenticated integration asset; the UI variant adds one content-addressed frontend pack. All processing is local — the embeddings run on-device; there is no embedded LLM and no API key.
44

55
## Key facts
66

‎internal/cbm/cbm.h‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -174,8 +174,9 @@ typedef enum {
174174
CBM_LANG_OBJECTSCRIPT_UDL, // InterSystems ObjectScript UDL (.cls class files)
175175
CBM_LANG_OBJECTSCRIPT_ROUTINE, // InterSystems ObjectScript routine (.mac/.int/.rtn/.inc)
176176
CBM_LANG_OBJECTSCRIPT_EXPORT, // InterSystems Studio Export XML (<Export generator="Cache">)
177-
CBM_LANG_ARKTS, // ArkTS (HarmonyOS/OpenHarmony .ets — TypeScript superset + ArkUI)
178-
CBM_LANG_PLSQL, // Oracle PL/SQL
177+
CBM_LANG_ARKTS, // ArkTS (HarmonyOS/OpenHarmony .ets — TypeScript superset + ArkUI)
178+
CBM_LANG_PLSQL, // Oracle PL/SQL
179+
CBM_LANG_CHIALISP, // Chialisp (.clsp/.clib/.clinc — Chia smart-coin s-expression language)
179180
CBM_LANG_COUNT
180181
} CBMLanguage;
181182

0 commit comments

Comments
 (0)