feat(context): add hybrid retrieval and typed agent decisions

Cargo.toml

@@ -79,7 +79,7 @@ chrono = { version = "0.4", features = ["serde"] }
 regex = "1"
 tracing = "0.1"
 tracing-subscriber = { version = "0.3", features = ["env-filter"] }
-rmcp = { version = "1.7.0", features = ["server", "macros", "transport-streamable-http-server"] }
+rmcp = { version = "1.7.0", features = ["server", "macros", "transport-io", "transport-streamable-http-server"] }
 tokio-util = "0.7"
 tokio-stream = { version = "0.1", features = ["sync"] }
 async-stream = "0.3"

README-vi.md

@@ -30,13 +30,25 @@ npm install -g vibervn-context-engine@latest
 vibervn-context-engine --port 6699
 ```
 
+Để kết nối Codex, Claude Code, Cursor hoặc MCP client chạy dưới dạng subprocess,
+dùng transport stdio:
+
+```bash
+vibervn-context-engine --mcp-stdio
+```
+
+`ask-context` có thể gọi thêm System One của TypeSafe. Đặt `TYPESAFE_API_KEY` để
+bật API `jev-latest`; có thể ghi đè bằng `TYPESAFE_API_URL`, `TYPESAFE_MODEL` và
+`TYPESAFE_TIMEOUT_MS`. Nếu không có key, quyết định local vẫn hoạt động.
+
 Nền tảng được hỗ trợ: Linux x64/arm64, macOS arm64, Windows x64.
 
 ## Tính năng
 
 | Tính năng | Mô tả |
 |-----------|-------|
 | Semantic code search | Tìm mã theo ý nghĩa thông qua embedding, không khớp văn bản thuần |
+| Hybrid retrieval | Kết hợp vector semantic với lexical matching có giới hạn để bắt exact symbol, path và error string |
 | Multi-language parsing | Extract symbol bằng Tree-sitter cho 22 ngôn ngữ (xem bảng bên dưới) |
 | Call-graph expansion | Resolve caller/callee edge và BFS expand các symbol khớp khi query |
 | Import-path resolution | Trace import tới file thật cho TS/JS, Python, Go, Rust — resolve cross-module call mà name matching bỏ lỡ |
@@ -48,9 +60,10 @@ Nền tảng được hỗ trợ: Linux x64/arm64, macOS arm64, Windows x64.
 | Real-time file watching | `notify` (debounce) tự động trigger re-index khi tệp thay đổi |
 | Voyage AI embedding | HTTP embedding client có disk cache để tránh gọi API thừa |
 | LLM rerank | Sắp xếp lại các candidate chunk bằng LLM (OpenAI / Google); tùy chọn, có thể tắt |
+| Structured Ask decision | `ask-context` trả JSON gồm evidence, trạng thái source, confidence, uncertainty và cảnh báo degradation; TypeSafe bổ sung answer noul/choice/score khi được cấu hình |
 | Embedded SurrealDB | Lưu chunk, symbol và edge; một datastore cho mỗi repo |
 | HTTP API + Web UI | Quản lý cấu hình, index explorer và bảng điều khiển thử query |
-| MCP server | Cung cấp `codebase-retrieval` và `file-retrieval` tool qua streamable HTTP |
+| MCP server | Cung cấp retrieval dạng text và `ask-context` dạng cấu trúc qua streamable HTTP hoặc stdio |
 | SSE progress stream | Truyền sự kiện indexing progress trực tiếp tới UI |
 | Large-repo scaling | Bounded memory và không có đường O(n²) — xây dựng cho codebase quy mô Linux/Chromium |
 
@@ -110,13 +123,15 @@ flowchart TD
     end
 
     Clients --> Q1[Embed query]
-    Q1 --> Q2[Vector search: top-k cosine]
-    Q2 -.reads.-> VecIndex
+    Q1 --> Q2[Hybrid retrieval: vector + lexical]
+    Q2 -.semantic reads.-> VecIndex
+    Q2 -.lexical reads.-> Store
     Q2 --> Q3[Graph expand: BFS caller/callee]
     Q3 --> Q4[Merge + dedup adjacent range]
     Q4 --> Q5[LLM rerank]
     Q5 --> Q6[Format: path#Lstart-end + dòng đánh số]
-    Q6 --> Result([Kết quả])
+    Q6 --> Decision[Typed decision: action + confidence + evidence validity]
+    Decision --> Result([Kết quả])
 ```
 
 ## Đóng góp

README-zh.md

@@ -27,13 +27,24 @@ npm install -g vibervn-context-engine@latest
 vibervn-context-engine --port 6699
 ```
 
+对于 Codex、Claude Code、Cursor 以及其他以子进程启动 MCP 的客户端，可使用 stdio：
+
+```bash
+vibervn-context-engine --mcp-stdio
+```
+
+`ask-context` 可选调用 TypeSafe System One。设置 `TYPESAFE_API_KEY` 后启用
+`jev-latest`；也可通过 `TYPESAFE_API_URL`、`TYPESAFE_MODEL` 和
+`TYPESAFE_TIMEOUT_MS` 覆盖默认值。没有密钥时仍使用本地决策。
+
 支持的平台：Linux x64/arm64、macOS arm64、Windows x64。
 
 ## 功能特性
 
 | 功能 | 描述 |
 |------|------|
 | 语义代码搜索 | 通过嵌入向量按语义查找代码，而非字面文本匹配 |
+| 混合检索 | 将向量检索与有界 lexical matching 结合，保留精确符号、路径和错误字符串 |
 | 多语言解析 | 使用 Tree-sitter 为 22 种语言提取符号（见下表） |
 | 调用图扩展 | 解析调用方/被调方边，并在查询时对匹配符号进行 BFS 扩展 |
 | Import 路径解析 | 为 TS/JS、Python、Go、Rust 追踪 import 到实际文件 —— 解析 name matching 遗漏的跨模块调用 |
@@ -45,9 +56,10 @@ vibervn-context-engine --port 6699
 | 实时文件监听 | `notify`（带去抖）在文件变更时自动触发重新索引 |
 | Voyage AI 嵌入 | 带磁盘缓存的 HTTP 嵌入客户端，避免重复 API 调用 |
 | LLM 重排序 | 使用 LLM（OpenAI / Google）对候选片段重新排序；可选，可禁用 |
+| 结构化 Ask 决策 | `ask-context` 返回 JSON evidence、源码有效性、confidence、uncertainty 和降级警告；配置 TypeSafe 后还会返回 noul/choice/score typed answers |
 | 内嵌 SurrealDB | 存储片段、符号和边；每个仓库一个数据存储 |
 | HTTP API + Web 界面 | 配置管理、索引浏览器和查询测试控制台 |
-| MCP 服务器 | 通过可流式 HTTP 暴露 `codebase-retrieval` 和 `file-retrieval` 工具 |
+| MCP 服务器 | 通过可流式 HTTP 或 stdio 暴露文本检索和结构化 `ask-context` 工具 |
 | SSE 进度流 | 将实时索引进度事件流式传输到界面 |
 | 大型仓库扩展 | 内存有界且无 O(n²) 路径 —— 为 Linux/Chromium 规模的代码库而构建 |
 

README.md

@@ -29,13 +29,26 @@ npm install -g vibervn-context-engine@latest
 vibervn-context-engine --port 6699
 ```
 
+For Codex, Claude Code, Cursor, and other subprocess-launched MCP clients, run
+the stdio transport instead of the HTTP server:
+
+```bash
+vibervn-context-engine --mcp-stdio
+```
+
+`ask-context` can optionally add TypeSafe System One judgments. Set
+`TYPESAFE_API_KEY` to enable the real `jev-latest` API; `TYPESAFE_API_URL`,
+`TYPESAFE_MODEL`, and `TYPESAFE_TIMEOUT_MS` are optional overrides. Without a
+key, the local retrieval decision remains active.
+
 Supported platforms: Linux x64/arm64, macOS arm64, Windows x64.
 
 ## Features
 
 | Feature | Description |
 |---------|-------------|
 | Semantic code search | Finds code by meaning via embeddings, not literal text matching |
+| Hybrid retrieval | Fuses semantic vectors with bounded lexical matching for exact symbols, paths, and error strings |
 | Multi-language parsing | Tree-sitter symbol extraction for 22 languages (see table below) |
 | Call-graph expansion | Resolves caller/callee edges and BFS-expands matched symbols at query time |
 | Import-path resolution | Traces imports to actual files for TS/JS, Python, Go, and Rust — resolves cross-module calls that name matching misses |
@@ -47,9 +60,10 @@ Supported platforms: Linux x64/arm64, macOS arm64, Windows x64.
 | Real-time file watching | `notify` (debounced) triggers re-index automatically on file changes |
 | Voyage AI embeddings | HTTP embedding client with an on-disk cache to avoid redundant API calls |
 | LLM reranking | Reorders candidate chunks with an LLM (OpenAI / Google); optional, can be disabled |
+| Structured Ask decisions | `ask-context` returns JSON evidence, source validity, confidence, uncertainty, and degradation warnings; TypeSafe adds typed noul/choice/score answers when configured |
 | Embedded SurrealDB | Stores chunks, symbols, and edges; one datastore per repo |
 | HTTP API + Web UI | Settings management, index explorer, and a query test console |
-| MCP server | Exposes `codebase-retrieval` and `file-retrieval` tools over streamable HTTP |
+| MCP server | Exposes text retrieval plus structured `ask-context` over streamable HTTP or stdio |
 | SSE progress stream | Streams live indexing progress events to the UI |
 | Large-repo scaling | Bounded memory and no O(n²) paths — built for Linux/Chromium-scale codebases |
 
@@ -111,14 +125,16 @@ flowchart TD
 
     Clients --> QF[Parse field filters: kind:/lang:/path:/name:]
     QF --> Q1[Embed remaining query text]
-    Q1 --> Q2[Vector search: top-k cosine]
-    Q2 -.reads.-> VecIndex
+    Q1 --> Q2[Hybrid retrieval: vector + lexical]
+    Q2 -.semantic reads.-> VecIndex
+    Q2 -.lexical reads.-> Store
     Q2 --> Q3[Apply filters + downrank generated files]
     Q3 --> Q4[Graph expand: BFS callers/callees]
     Q4 --> Q5[Merge + dedup adjacent ranges]
     Q5 --> Q6[LLM rerank]
     Q6 --> Q7[Format: path#Lstart-end + caller/callee names + numbered lines]
-    Q7 --> Result([Results])
+    Q7 --> Decision[Typed decision: action + confidence + evidence validity]
+    Decision --> Result([Results])
 ```
 
 ## Contributing

plans/2026-09-22-ctxe-system-one.md

@@ -0,0 +1,57 @@
+# CtxE and System One Integration
+
+Status: completed
+
+## Outcome
+
+Improve `vibervn-context-engine` with the strongest compatible ideas found in
+CtxE and TypeSafe's System One announcement: hybrid semantic and lexical
+retrieval, a goal-driven structured Ask surface, source-evidence validity, and
+typed local confidence decisions that clients can consume without parsing prose.
+
+## Constraints
+
+- Preserve the existing HTTP UI, streamable HTTP MCP endpoint, indexing model,
+  and existing `codebase-retrieval` response contract.
+- Keep all retrieval bounded and repository-scoped.
+- Do not pretend to call Jev or reproduce proprietary calibration; expose an
+  explicit local heuristic decision with its uncertainty and signals.
+- Keep optional LLM/network work outside database write transactions and degrade
+  to local retrieval when optional enrichment is unavailable.
+
+## Non-goals
+
+- Replacing the existing embedding or LLM providers.
+- Adding a hosted TypeSafe dependency without a documented public API.
+- Rebuilding the entire CtxE records database or daemon/service manager.
+
+## Acceptance criteria
+
+1. Queries combine vector candidates with bounded lexical candidates and keep
+   exact symbol/path matches useful when embeddings miss them.
+2. Query JSON includes a typed decision (`action`, confidence, uncertainty,
+   evidence coverage, and reasons) and evidence records that distinguish
+   unchanged, changed, partial, missing, and unavailable source content.
+3. MCP exposes a default `ask-context` tool returning structured JSON while
+   retaining the existing text retrieval tools.
+4. The binary supports a stdio MCP mode suitable for Codex, Claude Code,
+   Cursor, and other subprocess-launched clients.
+5. Focused unit/integration checks cover lexical scoring, decision outcomes,
+   evidence classification, config migration, and MCP tool wiring; the normal
+   Rust checks pass.
+
+## Sources
+
+- CtxE public contract: `https://ctxe.tlelabs.com/llms.txt` and its current docs
+  pages for hybrid retrieval, Ask, graph tools, evidence validity, and stdio MCP.
+- TypeSafe AI, “Introducing System One Models & Jev”, published 2026-09-15:
+  `https://typesafe.ai/blog/introducing-system-one-models-and-jev`.
+
+## Verification
+
+- `cargo test --lib`: 665 passed, 0 failed, 6 ignored.
+- The `ask-context` schema and native MCP `structuredContent` tests are included in that run.
+- `cargo test --test integration --test router_integration --test mcp_session_restore`: 33 passed.
+- `cargo check --all-targets` and `cargo run -- --help`: passed.
+- A temp-home MCP stdio initialize handshake returned a valid JSON-RPC response.
+- `cargo clippy --all-targets -- -D warnings` still reports five pre-existing findings in `src/embedding/cache.rs`, `src/store/ops.rs`, `src/router/plan.rs`, and `src/server.rs`; no new findings remain in the changed code.

plans/260922-2052-turso-prototype/plan.md

@@ -0,0 +1,26 @@
+# Turso Experimental Backend Prototype
+
+Status: planned
+
+## Outcome
+
+Benchmark a Turso-backed code index against the existing SurrealDB/vector-shard
+path without changing the production default or deleting existing data.
+
+## Boundaries
+
+- Additive feature flag and separate database file only.
+- No automatic migration, destructive schema change, or default backend switch.
+- Reuse the existing parser/chunker/symbol/call-edge output.
+- Keep TypeSafe and MCP behavior independent of the storage choice.
+
+## Acceptance criteria
+
+1. A fixture repository can be imported idempotently into Turso tables for files,
+   chunks, symbols, and edges.
+2. Turso FTS and exact vector queries return a common retrieval record shape.
+3. A benchmark compares both backends on recall@k, p50/p95 latency, indexing
+   time, memory, and recovery after an interrupted update.
+4. Existing SurrealDB tests remain unchanged and green.
+5. Promotion remains a measured decision; the prototype never silently becomes
+   the default backend.

plans/260922-2052-typesafe-api/plan.md

@@ -0,0 +1,36 @@
+# TypeSafe API Integration
+
+Status: completed
+
+## Outcome
+
+Use the public TypeSafe System One HTTP API from the structured `ask-context`
+workflow when `TYPESAFE_API_KEY` is configured, while preserving local retrieval
+and degradation when the key or service is unavailable.
+
+## Scope
+
+- Add a small Rust client for `https://api.typesafe.ai/v1/systemone`.
+- Ask bounded `noul`, `choice`, and `score` questions over retrieved evidence.
+- Return the typed TypeSafe answers, model, and usage in `ask-context` JSON and
+  MCP `structuredContent` without exposing credentials.
+- Keep all TypeSafe calls outside database transactions and cap request state.
+
+## Acceptance criteria
+
+1. `TYPESAFE_API_KEY` enables a real authenticated request using `jev-latest`.
+2. Missing keys and non-success responses degrade to local Ask output with a
+   warning and no secret values in logs or responses.
+3. Valid TypeSafe responses preserve typed answers and usage metadata.
+4. Unit tests cover request shape, response decoding, error redaction, and the
+   no-key path; existing library/integration checks remain green.
+
+## Verification
+
+- `cargo test --lib`: 668 passed, 0 failed, 6 ignored, including the TypeSafe
+  request/response tests.
+- `cargo test --test integration --test router_integration --test mcp_session_restore`:
+  33 passed.
+- `cargo check --all-targets`: passed.
+- TypeSafe failures are logged only as status-level errors and return local Ask
+  output; response bodies and credentials are never copied into errors.

plans/reports/2026-09-22-ctxe-system-one-research.md

@@ -0,0 +1,53 @@
+# CtxE and System One Research
+
+Conducted: 2026-09-22 UTC
+
+## Findings
+
+CtxE's public documentation describes a local, syntax-aware index with a
+two-phase publish pipeline, hybrid semantic plus indexed-text retrieval, typed
+symbol relationships, a goal-driven multi-round Ask flow, source snapshots that
+report evidence validity, and explicit degradation/truncation diagnostics. Its
+MCP integration is stdio-first and exposes separate status, tree, retrieval,
+definition, usage, graph, impact, and record tools.
+
+The current repository already has strong local foundations: Tree-sitter
+parsing, incremental indexing, a persisted call graph, vector shards with lazy
+warming, optional LLM/agentic reranking, freshness gates, and a streamable
+HTTP MCP service. The main gaps are exact lexical recall, machine-readable
+confidence/evidence metadata, a structured Ask entry point, and stdio MCP
+transport.
+
+TypeSafe's article presents System One models as fast decision functions: typed
+outputs selected from a fixed schema, probabilities/confidence on every answer,
+parallel sampling, and a preference for composing many small decisions inside
+ordinary software workflows. The article itself did not include the API
+contract, so the initial implementation used a typed local decision layer;
+follow-up documentation verification now supports the optional API client
+described below.
+
+## Implementation choice
+
+The change keeps vector search as the semantic backbone, adds bounded lexical
+candidate lookup against the already-indexed chunk store, and fuses both scores
+before graph expansion and reranking. Query results gain structured evidence
+and decision metadata. `ask-context` returns that data as JSON, while the
+existing text tool remains unchanged for clients that need concise snippets.
+
+## Follow-up API verification
+
+The current TypeSafe documentation now publishes a stable HTTP contract at
+`POST https://api.typesafe.ai/v1/systemone` with `Bearer` authentication,
+`jev-latest`, and the `noul`, `choice`, and `score` question primitives. The
+follow-up implementation uses `TYPESAFE_API_KEY` and keeps the integration
+optional: a missing key or non-success response falls back to the local typed
+decision without copying credentials or provider response bodies into logs.
+
+## References
+
+- CtxE overview and docs: `https://ctxe.tlelabs.com/`
+- CtxE machine-readable index: `https://ctxe.tlelabs.com/llms.txt`
+- CtxE retrieval/evidence docs: `https://ctxe.tlelabs.com/docs/how-it-works`
+- CtxE MCP contract: `https://ctxe.tlelabs.com/docs/mcp-integration`
+- CtxE tools: `https://ctxe.tlelabs.com/docs/tools-reference`
+- TypeSafe article: `https://typesafe.ai/blog/introducing-system-one-models-and-jev`

plans/reports/turso-260922-2052-assessment.md

@@ -0,0 +1,57 @@
+# Turso Integration Assessment
+
+Conducted: 2026-09-22 UTC
+
+## Decision
+
+Use Turso as an optional experimental storage and benchmark backend, not as an
+immediate replacement for SurrealDB.
+
+The current engine already has large-repository behavior that would be risky to
+discard: per-repository SurrealDB handles, persisted vector shards, mmap warmup,
+quantized resident vectors, LRU eviction, mutation fences, and extensive
+incremental/crash-recovery tests. The Turso path should prove equivalent
+behavior and performance before it becomes the default.
+
+## Evidence
+
+Turso's current public documentation provides a code-indexing schema that maps
+well to this repository: indexed files with content hashes, chunks with names,
+signatures, line ranges and embeddings, FTS indexes over identifiers, and vector
+distance queries. It also supports `BEGIN CONCURRENT` with MVCC for independent
+writes.
+
+The important limits are equally clear:
+
+- FTS is powered by Tantivy and requires the experimental `index_method` feature.
+- Vector similarity queries are exact linear scans; approximate vector indexing
+  remains on the roadmap.
+- FTS and vector syntax/API may change while those features are experimental.
+- Graph edges can be represented in relational tables, but traversal, cleanup,
+  and incremental blast-radius logic remain application-owned.
+
+Sources:
+
+- Turso code indexing guide: `https://docs.turso.tech/guides/code-indexing`
+- Turso vector search: `https://docs.turso.tech/guides/vector-search`
+- Turso FTS reference: `https://docs.turso.tech/sql-reference/functions/fts`
+- Turso Rust reference: `https://docs.turso.tech/sdk/rust/reference`
+- Turso concurrent writes: `https://docs.turso.tech/tursodb/concurrent-writes`
+- Turso repository: `https://github.com/tursodatabase/turso`
+
+## Recommended prototype
+
+1. Add an opt-in `turso-experimental` feature and a separate local database
+   file; do not alter the existing SurrealDB path.
+2. Import one indexed repository into `codebases`, `indexed_files`, `chunks`,
+   `symbols`, and `edges`, using `file_hash`/`chunk_key` for idempotent upserts.
+3. Create weighted FTS over `name`, `signature`, and a bounded snippet field;
+   store vectors as `vector8` or `vector32` and compare exact-search latency to
+   the current resident shard.
+4. Run the same query corpus through SurrealDB and Turso and record indexing
+   time, query p50/p95, recall@k, memory, FTS freshness, and crash recovery.
+5. Promote only if Turso matches the current correctness suite and wins on a
+   measured workload; otherwise retain it as an export/portable-index option.
+
+No database migration has been applied. If the prototype becomes a migration,
+the repository must first create a durable backup and define a rollback path.