Skip to content

perf(semantic-search): evaluate TurboVec/TurboQuant as an optional vector index backend #1238

Open

Description

@mlandolfi90

What problem does this solve?

As code graphs and cross-repository indexes grow, the semantic-search path could benefit from a lower-memory vector index without weakening retrieval quality or incremental indexing.

TurboVec implements Google Research's TurboQuant in Rust and advertises online ingestion, 2/4-bit compression, SIMD search, persistence, stable IDs, deletion, and allowlist filtering:

https://github.com/RyanCodrai/turbovec

The allowlist filtering is especially interesting for codebase-memory-mcp: structural/BM25/graph signals could narrow candidates, then dense retrieval could rerank only the allowed symbol IDs.

This is a recommendation to evaluate, not a request to replace the current implementation based on upstream benchmarks alone.

Proposed solution

Create a bounded benchmark or design spike comparing the current semantic-search implementation against TurboVec/TurboQuant on representative code embeddings and repository sizes.

Measure:

  • recall@k and ranking quality on code-symbol queries
  • memory usage and persisted index size
  • indexing and incremental-add latency
  • deletion/update behavior
  • single-query and batch-query latency
  • filtered retrieval using graph-derived candidate IDs
  • portability across Windows, Linux, macOS, x86, and ARM

Preserve these constraints:

  • the pure-C, single-static-binary distribution remains the default
  • AST, SQLite graph storage, BM25, and graph relationships remain unchanged
  • no mandatory Rust runtime or external service
  • no adoption unless project-owned benchmarks show a meaningful end-to-end gain

If results justify further work, possible paths include:

  1. Port the relevant TurboQuant indexing/search algorithm to C.
  2. Keep an optional experimental external backend behind a stable internal interface.
  3. Adopt only the quantization or filtered-search ideas, not TurboVec itself.

Alternatives considered

  • Keep the current semantic-search implementation unchanged; this remains the correct baseline.
  • FAISS adds a much larger native dependency and conflicts with the zero-dependency distribution goal.
  • Integrating TurboVec directly would introduce Rust build and supply-chain complexity.
  • Replacing SQLite or the graph model is unnecessary; this proposal concerns only dense vector retrieval.

Confirmations

  • I searched existing issues and this is not a duplicate.

Activity

  1. added
    enhancementNew feature or request
    stability/performanceServer crashes, OOM, hangs, high CPU/memory
    dependenciesPull requests that update a dependency file
    priority/backlogValuable contribution, lower scheduling urgency; review when maintainer capacity opens.
    and removed
    windowsWindows-specific issues
    on Jul 24, 2026
  2. added this to the 0.9.2-rc milestone on Jul 24, 2026
  3. DeusData commented on Jul 25, 2026

    @DeusData
    Owner

    Thanks for this — and specifically for writing it as an evaluation proposal with explicit non-goals rather than a "please adopt X" request. The constraints you list (pure-C single static binary stays the default, no mandatory Rust runtime or external service, AST/SQLite/BM25/graph untouched, no adoption without project-owned benchmarks) are the projects actual constraints, so this lands as a coherent proposal rather than something transplanted from a different architecture.

    For grounding, the surface you are proposing against does exist today. Semantic embeddings live in src/semantic/ (semantic.c / semantic.h, plus rotsq.c and ast_profile.c), with vector storage in the graph buffer (src/graph_buffer/graph_buffer.h:149, "Vector storage (for semantic embeddings)"). The dimension is 768 to match nomic-embed-code (src/semantic/semantic.h:35), and the whole path is opt-in behind CBM_SEMANTIC_ENABLED=1 (semantic.h:87). It currently produces SEMANTICALLY_RELATED edges as a pre-dump pass (src/pipeline/pipeline_internal.h:603).

    Two observations that matter for how such an evaluation would have to be designed:

    The allowlist-filtering angle you highlight is the genuinely interesting part, and you are right that it fits this project unusually well — narrowing candidates by structural/BM25/graph signals and reranking only allowed symbol IDs matches how the graph is already used. That is worth measuring independently of any quantization result, because it is a retrieval-architecture question rather than a vector-index question.

    The portability line in your measurement list is not a formality here; it is close to decisive. The distribution model is a single static C binary across Windows, Linux and macOS on both x86 and ARM, and a Rust dependency is a build- and release-pipeline change well before it is a performance question. Of your three follow-on paths, that asymmetry means path 1 (port the algorithm to C) and path 2 (optional backend behind a stable internal interface) carry very different costs, and the benchmark would need to be structured to inform that choice rather than just producing recall/latency numbers.

    Whether to spend the bench cycles on this at all is a roadmap call rather than a triage one, so I am recording it rather than scheduling it, and leaving it at priority/backlog where you filed it. Flagging one thing for whoever picks it up: the current implementation is algorithmic-embedding based and opt-in, so a fair comparison needs a baseline measured with CBM_SEMANTIC_ENABLED=1 on representative code embeddings, not against the default build where the path is inert.

    Thanks for the link and for the disciplined framing — recommendations that state their own falsification criteria are rare and much appreciated.

  4. nwn900 commented on Aug 19, 2026

    @nwn900

    A concrete integration direction that seems worth benchmarking is to use TurboVec as a query-time semantic vector index, while leaving the rest of codebase-memory-mcp intact.

    The goal would be enhancement, not replacement: preserve CBM’s tree-sitter/LSP knowledge graph, SQLite graph storage, BM25/FTS, structural queries, call tracing, impact analysis, architecture tooling, and existing semantic signal generation. TurboVec would replace only the current dense-vector candidate-search hot path.

    Suggested architecture

                    +-- SQLite graph / BM25 / structural search
    

    Source -> CBM ------+
    +-- TurboVec IdMapIndex
    768 dimensions
    node_id -> stable u64 id
    4-bit TurboQuant / TQ+

    For semantic_query, the flow could be:

    1. Build the same 768-dimensional query vectors CBM already uses.
    2. Search a TurboVec IdMapIndex for a broader candidate set, for example limit * 5.
    3. Map TurboVec’s stable u64 IDs directly to CBM node IDs.
    4. Preserve CBM’s existing multi-keyword semantics by reranking those candidates with the current per-keyword minimum-cosine logic.
    5. Return results through the existing search_graph MCP surface unchanged.

    This keeps TurboVec behind an internal retrieval abstraction rather than making it the system of record.

    Why this boundary looks promising

    CBM’s query-time semantic vectors are 768-dimensional int8 vectors, and the current search path scans them using a SQLite cosine function before reranking candidates.

    TurboVec supports dimensions that are multiples of 8, stable external IDs through IdMapIndex, O(1) deletion, incremental sync(), and allowlist filtering, so it maps naturally onto the existing node-vector workload.

    At the raw vector-payload level, 768-dimensional storage is approximately:

    • Current int8: 768 B/vector
    • TurboVec 4-bit: ~384 B/vector
    • TurboVec 3-bit: ~288 B/vector
    • TurboVec 2-bit: ~192 B/vector

    TurboVec has additional index metadata, so these are not total-index-size claims.

    I would start evaluation with 4-bit TQ+, not 2-bit, and treat retrieval quality as a hard acceptance criterion.

    Preserve CBM’s strengths

    I would explicitly not use TurboVec to replace:

    • AST/LSP extraction
    • Graph relationships and traversal
    • BM25/FTS search
    • Architecture and impact-analysis tools
    • Semantic signal construction
    • SQLite as the authoritative graph and metadata store

    TurboVec should only accelerate and compress dense candidate retrieval.

    Hybrid retrieval opportunity

    TurboVec’s allowlist search could also enable a stronger hybrid path:

    1. Use BM25, graph structure, module boundaries, or other CBM signals to narrow the candidate set.
    2. Pass those node IDs to TurboVec as an allowlist.
    3. Run dense retrieval only within that subset.
    4. Apply CBM’s existing reranking and graph-aware logic afterward.

    This may be more valuable than simply replacing one full scan with another because CBM’s structural knowledge can directly reduce vector-search work.

    Engineering concerns

    A few areas would need explicit design:

    • C/Rust boundary: CBM is pure C while TurboVec is Rust. A small stable C ABI/FFI layer, or a C port of the relevant indexing/search pieces, would need to preserve CBM’s distribution goals.
    • Persistence consistency: a .tvim index and SQLite cannot participate in one atomic transaction. The vector index should carry a CBM generation/version marker, with a rebuild or resync path after interrupted updates.
    • Incremental updates: add/remove operations should follow the CBM node lifecycle so stale vector IDs cannot survive graph mutation.
    • Calibration: if TQ+ is used, calibration should come from a representative sample and be versioned with the vector index.
    • Fallback: the existing SQLite/int8 path should remain available until project-owned benchmarks show a clear end-to-end improvement.

    Suggested benchmark

    Compare the current CBM semantic-search implementation with TurboVec-backed candidate retrieval on representative repositories and several symbol counts.

    Measure:

    • semantic answer quality and recall@k
    • end-to-end semantic_query latency, not only vector-kernel time
    • peak and steady-state RAM
    • persisted index size
    • initial indexing cost
    • incremental add/remove/update latency
    • cold-load and first-query latency
    • filtered/allowlist retrieval
    • x86_64 and ARM64 behavior

    If the results are favorable, this could make semantic retrieval faster and more memory-efficient without sacrificing any of the structural code-intelligence functionality that makes CBM valuable.

  5. DeusData commented on Oct 19, 2026

    @DeusData
    Owner

    Thank you for the additional query-time design and generation-consistency considerations. A benchmark and any dependency/FFI choice still require a maintainer roadmap decision. No evaluation has been scheduled or backend approved in this triage pass.

  6. DeusData commented on Oct 25, 2026

    @DeusData
    Owner

    Thank you for the query-time proposal and consistency considerations. Benchmark scope and any backend/dependency choice still need a maintainer roadmap decision. Sorry for the delay; this is waiting on maintainer follow-up, and I’ll follow up soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dependenciesPull requests that update a dependency fileenhancementNew feature or requestpriority/backlogValuable contribution, lower scheduling urgency; review when maintainer capacity opens.stability/performanceServer crashes, OOM, hangs, high CPU/memory

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions