search_graph: non-ASCII queries return noise, and docstrings are never indexed

Summary

search_graph(query=...) returns unranked noise for any query written in a non-Latin script, and no amount of documentation helps, because docstrings are not part of the full-text index either. On a codebase documented in Russian (or Greek, or Japanese) the BM25 mode is effectively unusable.

Two independent causes, both small:

1. The query tokenizer drops every non-ASCII byte. bm25_build_match() (src/mcp/mcp.c) accepts only a-z, A-Z, 0-9 and _ as token bytes; everything else is a separator. A Cyrillic query is therefore tokenized to nothing, reaches FTS5 as an empty MATCH, and the caller gets whatever the fallback path returns.

2. Docstrings are not indexed. nodes_fts is declared as fts5(name, qualified_name, label, file_path, ...) in src/store/store.c. Identifiers in real projects are overwhelmingly English even when the team writes prose in another language, so the only text that could match a native-language query is exactly the text that never enters the index.

Reproduction

Index any project that has a non-English docstring, e.g.

def _validate_args(name: str, arguments: dict) -> str | None:
    """Провалидировать аргументы по схеме инструмента."""

codebase-memory-mcp cli search_graph --project <p> --query 'провалидировать аргументы схеме'

Expected: the documented function.
Actual (v0.9.0): unrelated nodes — in my case .env.example, a Coordinates class and a README section; the target function is nowhere in the results.

The English control query behaves as expected, which is what makes the failure easy to miss.

Suggested fix

Treat bytes >= 0x80 as token bytes in bm25_build_match(). No UTF-8 decoding is required — the point is only to keep multi-byte sequences intact and let the FTS5 unicode61 tokenizer apply the real word-boundary rules, which it already does correctly on both sides.
Add a docstring column to nodes_fts and populate it from json_extract(properties, '$.docstring') in both backfill paths (pipeline.c, pipeline_incremental.c). Existing databases carry the 4-column table, so the schema probe has to drop and recreate it; the table is contentless, so nothing is lost — the next backfill repopulates it from nodes.

Result after the fix

Same query, same project:

total: 1
search_mode: bm25
  ...server._validate_args  Function  src/yandex_biz_mcp/server.py  78-96  -18.89

The English query improves as well — validate arguments schema now ranks _validate_args first, where previously it returned noise: docstrings carry intent that identifiers alone do not.

I have a patch with a regression test ready and will open a PR referencing this issue.

Environment

v0.9.0 (release build, UI variant), macOS 15.5 arm64
Reproduced on several projects; the smallest is an 8-tool Python MCP server, 137 nodes

Patch attached below rather than as a PR: I opened #1403, then made my fork private, which closed the PR and left an outdated revision in it. Please do not take the diff from #1403 — it has a defect I found afterwards (described in that PR's last comment and fixed here).

Applies cleanly to main at d6be58e (git apply --check passes). Two files of implementation, one of tests, plus the two backfill call sites.

What it does

bm25_build_match() treats bytes >= 0x80 as token bytes, so a non-Latin query is no longer tokenized to nothing. No UTF-8 decoding — just keeping multi-byte sequences intact for the unicode61 tokenizer.
nodes_fts gains a docstring column, populated from json_extract(properties, '$.docstring') in both backfill paths.
The schema upgrade drops the old 4-column table and repopulates it in the same step. This is the part I got wrong the first time: leaving repopulation to the pipeline meant an already-indexed project with unchanged files never got its index back, so search returned nothing at all — worse than the bug being fixed.

Verified

Reproducing the upgrade sequence on real databases:

step	English query	non-ASCII query
index built by released 0.9.0 (4-column table)	works	—
first run of the patched binary	works	works
second run, no files changed	works	works

Full suite: 6824 passed, 1 failed, 4 skipped. The failure is tool_detect_changes_contained_commands_clean_up_error_and_success at ASSERT(merge_base_reported) — git merge-base reporting, untouched by this diff; it passes 3/3 in isolation and the same build alternated between 0 and 1 failures across full runs, so I read it as flaky. lint passed in your CI on this code.

Known limitation

The upgrade runs when the schema is created, not on read: upgrading the binary alone neither breaks nor fixes an existing database until the project is indexed again. I also tried self-healing an already-emptied index by probing the shadow table on open, but could not make it fire reliably across projects, so I left it out rather than ship behaviour I cannot explain.

No DCO sign-off on my side, so treat this as material to use as you see fit rather than a contribution ready to merge.

Patch (git format-patch)

From 83f257f0848ddc1d1a20624d16a22784ed37eb92 Mon Sep 17 00:00:00 2001
From: Andrey Shinkarenko <metatau@github.com>
Date: Sat, 1 Aug 2026 15:28:51 +0500
Subject: [PATCH] fix(search): index docstrings and accept non-ASCII query
 bytes
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

BM25 search was unusable for any project documented in a non-Latin
script, for two independent reasons.

bm25_build_match() treated only [a-zA-Z0-9_] as token bytes, so a
Cyrillic (or Greek, or CJK) query was tokenized to nothing, reached
FTS5 as an empty MATCH and returned unranked noise. Bytes >= 0x80 are
now token bytes: no UTF-8 decoding is needed here, only keeping
multi-byte sequences intact, since the unicode61 tokenizer already
applies the real word-boundary rules on both sides.

nodes_fts also indexed identifiers only. Identifiers are English in
practice even when the team writes prose in another language, so the
one field that could match a native-language query was the one field
never indexed. A docstring column is added and populated from
json_extract(properties, '$.docstring') in both backfill paths.
Databases created before this carry the 4-column table, which
CREATE ... IF NOT EXISTS would leave in place and every backfill would
then fail on, so the new column is probed and the table dropped when
missing — it is contentless, nothing is lost, the next backfill
repopulates it from nodes.

English search improves as a side effect: docstrings carry intent that
identifiers alone do not.

Fixes #1401

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 src/mcp/mcp.c                       | 18 +++++++--
 src/pipeline/pipeline.c             | 20 ++++++---
 src/pipeline/pipeline_incremental.c | 12 ++++--
 src/store/store.c                   | 37 ++++++++++++++++-
 tests/test_mcp.c                    | 63 +++++++++++++++++++++++++++++
 5 files changed, 136 insertions(+), 14 deletions(-)

diff --git a/src/mcp/mcp.c b/src/mcp/mcp.c
index d40571b3..d542d717 100644
--- a/src/mcp/mcp.c
+++ b/src/mcp/mcp.c
@@ -2769,6 +2769,18 @@ static sqlite3_destructor_type mcp_sqlite_transient(void) {
 }
 #define MCP_SQLITE_TRANSIENT (mcp_sqlite_transient())
 
+/* A byte that may sit inside a query token.  ASCII alphanumerics and '_' as
+ * before, plus every byte >= 0x80: those are UTF-8 lead/continuation bytes, so
+ * treating them as separators (the previous behaviour) silently dropped every
+ * non-Latin word from the query — a Cyrillic, Greek or CJK search reached the
+ * index as an empty match and returned unranked noise.  We do not need to
+ * decode UTF-8 here, only to keep multi-byte sequences intact: the FTS5
+ * unicode61 tokenizer applies the real word-boundary rules on both sides. */
+static int bm25_is_token_byte(unsigned char c) {
+    return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '_' ||
+           c >= 0x80;
+}
+
 static int bm25_build_match(const char *query, char *out, size_t out_size) {
     if (!query || !out || out_size < BM25_MIN_BUF) {
         return 0;
@@ -2777,16 +2789,14 @@ static int bm25_build_match(const char *query, char *out, size_t out_size) {
     int tokens = 0;
     const char *p = query;
     while (*p) {
-        while (*p && !((*p >= 'a' && *p <= 'z') || (*p >= 'A' && *p <= 'Z') ||
-                       (*p >= '0' && *p <= '9') || *p == '_')) {
+        while (*p && !bm25_is_token_byte((unsigned char)*p)) {
             p++;
         }
         if (!*p) {
             break;
         }
         const char *tok_start = p;
-        while (*p && ((*p >= 'a' && *p <= 'z') || (*p >= 'A' && *p <= 'Z') ||
-                      (*p >= '0' && *p <= '9') || *p == '_')) {
+        while (*p && bm25_is_token_byte((unsigned char)*p)) {
             p++;
         }
         size_t tok_len = (size_t)(p - tok_start);
diff --git a/src/pipeline/pipeline.c b/src/pipeline/pipeline.c
index 0c304195..9a33de95 100644
--- a/src/pipeline/pipeline.c
+++ b/src/pipeline/pipeline.c
@@ -1418,16 +1418,26 @@ static int dump_and_persist_hashes(cbm_pipeline_t *p, const cbm_file_info_t *fil
          * Contentless FTS5 requires the special 'delete-all' command instead of
          * DELETE FROM to wipe prior rows (there's no underlying content table).
          * Falls back to plain names if cbm_camel_split is unavailable (which
-         * shouldn't happen because we always register it, but we stay defensive). */
+         * shouldn't happen because we always register it, but we stay defensive).
+         *
+         * `docstring` is pulled out of the properties JSON so prose in comments
+         * is searchable, not just identifiers. It matters most for non-English
+         * codebases: identifiers are almost always English, so a query written
+         * in the team's own language matched nothing at all before. The
+         * unicode61 tokenizer already handles non-Latin scripts. */
         CBM_PROF_START(t_fts);
         cbm_store_exec(hash_store, "INSERT INTO nodes_fts(nodes_fts) VALUES('delete-all');");
         if (cbm_store_exec(hash_store,
-                           "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path) "
-                           "SELECT id, cbm_camel_split(name), qualified_name, label, file_path "
+                           "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path, "
+                           "docstring) "
+                           "SELECT id, cbm_camel_split(name), qualified_name, label, file_path, "
+                           "COALESCE(json_extract(properties, '$.docstring'), '') "
                            "FROM nodes;") != CBM_STORE_OK) {
             cbm_store_exec(hash_store,
-                           "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path) "
-                           "SELECT id, name, qualified_name, label, file_path FROM nodes;");
+                           "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path, "
+                           "docstring) "
+                           "SELECT id, name, qualified_name, label, file_path, "
+                           "COALESCE(json_extract(properties, '$.docstring'), '') FROM nodes;");
         }
         CBM_PROF_END("persist", "5_fts_backfill", t_fts);
 
diff --git a/src/pipeline/pipeline_incremental.c b/src/pipeline/pipeline_incremental.c
index 21740ba6..0f32abd6 100644
--- a/src/pipeline/pipeline_incremental.c
+++ b/src/pipeline/pipeline_incremental.c
@@ -690,12 +690,16 @@ static int dump_and_persist(cbm_gbuf_t *gbuf, const char *db_path, const char *p
         rc = CBM_STORE_ERR;
     }
     if (cbm_store_exec(hash_store,
-                       "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path) "
-                       "SELECT id, cbm_camel_split(name), qualified_name, label, file_path "
+                       "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path, "
+                       "docstring) "
+                       "SELECT id, cbm_camel_split(name), qualified_name, label, file_path, "
+                       "COALESCE(json_extract(properties, '$.docstring'), '') "
                        "FROM nodes;") != CBM_STORE_OK) {
         if (cbm_store_exec(hash_store,
-                           "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path) "
-                           "SELECT id, name, qualified_name, label, file_path FROM nodes;") !=
+                           "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path, "
+                           "docstring) "
+                           "SELECT id, name, qualified_name, label, file_path, "
+                           "COALESCE(json_extract(properties, '$.docstring'), '') FROM nodes;") !=
             CBM_STORE_OK) {
             rc = CBM_STORE_ERR;
         }
diff --git a/src/store/store.c b/src/store/store.c
index 4a032970..0a351f7e 100644
--- a/src/store/store.c
+++ b/src/store/store.c
@@ -337,15 +337,50 @@ static int init_schema(cbm_store_t *s) {
      * Fails silently if FTS5 is not compiled in (SQLITE_ENABLE_FTS5). */
     {
         char *fts_err = NULL;
+
+        /* Schema upgrade (#docstring-fts): databases created before the
+         * `docstring` column exists carry a 4-column nodes_fts, and
+         * CREATE ... IF NOT EXISTS would leave them that way — the backfill
+         * INSERT would then fail on every index run. The table is
+         * contentless, so dropping it loses no source data. Detect the old
+         * shape by probing the new column and drop only on failure. */
+        int fts_migrated = 0;
+        if (sqlite3_exec(s->db, "SELECT docstring FROM nodes_fts LIMIT 0;", NULL, NULL, NULL) !=
+            SQLITE_OK) {
+            if (sqlite3_exec(s->db, "DROP TABLE IF EXISTS nodes_fts;", NULL, NULL, NULL) ==
+                SQLITE_OK) {
+                fts_migrated = 1;
+            }
+        }
+
         int fts_rc = sqlite3_exec(s->db,
                                   "CREATE VIRTUAL TABLE IF NOT EXISTS nodes_fts USING fts5("
-                                  "  name, qualified_name, label, file_path,"
+                                  "  name, qualified_name, label, file_path, docstring,"
                                   "  content='',"
                                   "  tokenize='unicode61 remove_diacritics 2'"
                                   ");",
                                   NULL, NULL, &fts_err);
         if (fts_rc != SQLITE_OK && fts_err) {
             sqlite3_free(fts_err);
+            fts_err = NULL;
+        }
+
+        /* Repopulate immediately after an upgrade rather than waiting for the
+         * next backfill. An already-indexed project whose files have not
+         * changed never reaches the pipeline's backfill, so deferring would
+         * leave the index empty until something happens to change a file —
+         * search would return nothing at all in the meantime, for every
+         * query, not just the non-ASCII ones this change is about.
+         * cbm_camel_split is registered per connection later, so the recovery
+         * insert uses plain names; the next full index run rewrites the table
+         * with split tokens. */
+        if (fts_migrated && fts_rc == SQLITE_OK) {
+            sqlite3_exec(s->db,
+                         "INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path, "
+                         "docstring) "
+                         "SELECT id, name, qualified_name, label, file_path, "
+                         "COALESCE(json_extract(properties, '$.docstring'), '') FROM nodes;",
+                         NULL, NULL, NULL);
         }
     }
     return CBM_STORE_OK;
diff --git a/tests/test_mcp.c b/tests/test_mcp.c
index 36252b1f..63ba2977 100644
--- a/tests/test_mcp.c
+++ b/tests/test_mcp.c
@@ -2203,6 +2203,68 @@ TEST(tool_search_graph_query_honors_file_pattern_issue552) {
     PASS();
 }
 
+/* Identifiers are English almost everywhere, so a team that documents in its
+ * own language had no working full-text query: the docstring never reached the
+ * FTS index, and the query tokenizer dropped every non-ASCII byte on the way
+ * in.  Both halves are exercised here — a Cyrillic query must find the node
+ * whose only Cyrillic text lives in its docstring. */
+TEST(tool_search_graph_query_matches_non_ascii_docstring) {
+    cbm_mcp_server_t *srv = cbm_mcp_server_new(NULL);
+    ASSERT_NOT_NULL(srv);
+    cbm_store_t *st = cbm_mcp_server_store(srv);
+    ASSERT_NOT_NULL(st);
+
+    const char *proj = "non-ascii-doc";
+    cbm_mcp_server_set_project(srv, proj);
+    cbm_store_upsert_project(st, proj, "/tmp/non-ascii-doc");
+
+    cbm_node_t documented = {0};
+    documented.project = proj;
+    documented.label = "Function";
+    documented.name = "handle_payment";
+    documented.qualified_name = "non-ascii-doc.src.billing.handle_payment";
+    documented.file_path = "src/billing.c";
+    documented.start_line = 1;
+    documented.end_line = 9;
+    documented.properties_json = "{\"docstring\":\"Провести платёж по счёту\"}";
+    ASSERT_GT(cbm_store_upsert_node(st, &documented), 0);
+
+    cbm_node_t undocumented = {0};
+    undocumented.project = proj;
+    undocumented.label = "Function";
+    undocumented.name = "unrelated_helper";
+    undocumented.qualified_name = "non-ascii-doc.src.util.unrelated_helper";
+    undocumented.file_path = "src/util.c";
+    undocumented.start_line = 1;
+    undocumented.end_line = 3;
+    ASSERT_GT(cbm_store_upsert_node(st, &undocumented), 0);
+
+    cbm_store_exec(st, "INSERT INTO nodes_fts(nodes_fts) VALUES('delete-all');");
+    ASSERT_EQ(cbm_store_exec(st,
+                             "INSERT INTO nodes_fts(rowid, name, qualified_name, label, "
+                             "file_path, docstring) "
+                             "SELECT id, cbm_camel_split(name), qualified_name, label, file_path, "
+                             "COALESCE(json_extract(properties, '$.docstring'), '') "
+                             "FROM nodes;"),
+              CBM_STORE_OK);
+
+    char *resp = cbm_mcp_server_handle(
+        srv, "{\"jsonrpc\":\"2.0\",\"id\":553,\"method\":\"tools/call\","
+             "\"params\":{\"name\":\"search_graph\","
+             "\"arguments\":{\"project\":\"non-ascii-doc\",\"query\":\"платёж\",\"limit\":10}}}");
+    ASSERT_NOT_NULL(resp);
+    char *inner = extract_text_content(resp);
+    ASSERT_NOT_NULL(inner);
+    ASSERT_NOT_NULL(strstr(inner, "search_mode: bm25"));
+    ASSERT_NOT_NULL(strstr(inner, "handle_payment"));
+    ASSERT_NULL(strstr(inner, "unrelated_helper"));
+
+    free(inner);
+    free(resp);
+    cbm_mcp_server_free(srv);
+    PASS();
+}
+
 /* Resource discovery methods this server doesn't populate must return EMPTY
  * lists, not -32601 Method-not-found: clients like Cline probe them on connect
  * and surface the errors as a failed connection (#958). */
@@ -10332,6 +10394,7 @@ SUITE(mcp) {
     RUN_TEST(tool_output_regression_gate);
     RUN_TEST(tool_output_byte_budgets);
     RUN_TEST(tool_search_graph_query_honors_file_pattern_issue552);
+    RUN_TEST(tool_search_graph_query_matches_non_ascii_docstring);
     RUN_TEST(mcp_resource_discovery_methods_return_empty_lists);
     RUN_TEST(tool_query_graph_basic);
     RUN_TEST(tool_index_status_no_project);
-- 
2.54.0

Thank you for separating the non-ASCII query behavior from the missing-docstring search surface and for providing a patch sketch. Current storage uses the Unicode-aware FTS tokenizer, while docstrings are consumed by the semantic pipeline rather than exposed through the same BM25 document path; the observed result therefore spans two concrete indexing/query contracts. I have routed this as a high-priority 0.9.1-rc correctness bug. Any implementation should arrive as a focused, signed PR with independent tests for both contracts so authorship and review remain clean.

@Metatau

Thank you, @Metatau, for splitting this into two contracts. That split was exactly right, because half of it has landed since!

Docstrings are now in the full-text index: nodes_fts gained a body column that carries each node's docstring (#518/#519). The query side is still open. bm25_build_match() still keeps only ASCII identifier bytes, so on current main a Cyrillic query falls back to an unranked listing instead of a BM25 search, while the English control ranks correctly.

Your fix is the right shape for what remains: treat bytes ≥0x80 as token bytes and let unicode61 handle word boundaries. Since you've already written it, would you like to open it as a small signed PR covering only the tokenizer, with a non-ASCII query test? If you'd rather not, just say so and we'll carry it with credit to you.

Thanks for the update, and glad the docstring half landed. Please go ahead and carry the tokenizer fix yourselves. I won't be opening a signed PR, and a mention in the PR or changelog is all the credit I need. One case worth covering in the test: a mixed-script query such as `parse договор`. The ASCII token should still rank, and the Cyrillic one should no longer be dropped. A CJK query is a useful second non-Latin case, since it has no spaces between words and depends entirely on `unicode61` for segmentation. Once it ships in a release, I'll drop my local build and move back to the regular `update`. @Metatau 26 сент. 2026 г., 08:56 +0500, DeusData/codebase-memory-mcp ***@***.***>, писал(а):

…

Thank you, @Metatau, for splitting this into two contracts. That split was exactly right, because half of it has landed since! Docstrings are now in the full-text index: nodes_fts gained a body column that carries each node's docstring (#518/#519). The query side is still open. bm25_build_match() still keeps only ASCII identifier bytes, so on current main a Cyrillic query falls back to an unranked listing instead of a BM25 search, while the English control ranks correctly. Your fix is the right shape for what remains: treat bytes ≥0x80 as token bytes and let unicode61 handle word boundaries. Since you've already written it, would you like to open it as a small signed PR covering only the tokenizer, with a non-ASCII query test? If you'd rather not, just say so and we'll carry it with credit to you.

Thank you for explicitly offering the tokenizer work for maintainer follow-through and identifying mixed-script and CJK cases. No signed PR is needed from you; the remaining tokenizer work is on us. Sorry for the delay; this is waiting on maintainer follow-up, and I’ll follow up soon.

Navigation Menu

Repository navigation

search_graph: non-ASCII queries return noise, and docstrings are never indexed #1401

Summary

Reproduction

Suggested fix

Result after the fix

Environment

What it does

Verified

Known limitation

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

search_graph: non-ASCII queries return noise, and docstrings are never indexed #1401

Description

Summary

Reproduction

Suggested fix

Result after the fix

Environment

Activity

Metatau commented on Aug 1, 2026

What it does

Verified

Known limitation

DeusData commented on Aug 3, 2026

DeusData commented on Sep 26, 2026

Metatau commented on Sep 26, 2026

DeusData commented on Oct 25, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions