You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9ef1c23
Browse filesBrowse the repository at this point in the historyBrowse files
fix(search): index docstrings and accept non-ASCII query bytes
BM25 search was unusable for any project documented in a non-Latin
script, for two independent reasons.
bm25_build_match() treated only [a-zA-Z0-9_] as token bytes, so a
Cyrillic (or Greek, or CJK) query was tokenized to nothing, reached
FTS5 as an empty MATCH and returned unranked noise. Bytes >= 0x80 are
now token bytes: no UTF-8 decoding is needed here, only keeping
multi-byte sequences intact, since the unicode61 tokenizer already
applies the real word-boundary rules on both sides.
nodes_fts also indexed identifiers only. Identifiers are English in
practice even when the team writes prose in another language, so the
one field that could match a native-language query was the one field
never indexed. A docstring column is added and populated from
json_extract(properties, '$.docstring') in both backfill paths.
Databases created before this carry the 4-column table, which
CREATE ... IF NOT EXISTS would leave in place and every backfill would
then fail on, so the new column is probed and the table dropped when
missing — it is contentless, nothing is lost, the next backfill
repopulates it from nodes.
English search improves as a side effect: docstrings carry intent that
identifiers alone do not.
Fixes#1401
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0 commit comments