Skip to content

Commit d3776bc

Browse files
XiaoConstantineXiaoConstantine
committed
feat(search): add implementation-first ranking mode
Add an opt-in stable partition that places source implementation ahead of documentation, examples, and tests without changing candidate generation or default ordering. Include CLI wiring, cache isolation, documentation, and tests. Preserve the isolated Jev and local-reranker benchmark harness, fixed labels, raw run artifacts, and reports used to evaluate the change. On the frozen 20-query benchmark, implementation-first improves balanced MRR from 0.717 to 0.767 and NDCG@10 from 0.534 to 0.580 while eliminating two prose-promotion errors without measurable reranker latency.
1 parent 7acabf3 commit d3776bc

47 files changed

Lines changed: 125369 additions & 31 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ cd your-repository
4646
sgrep index .
4747

4848
sgrep "database connection error handling" # balanced, the default
49+
sgrep --implementation-first "register and discover tools" # prefer source over prose
4950
sgrep --profile fast "request routing" # lowest latency
5051
sgrep --profile quality "JWT validation middleware" # optional ColBERT reranking
5152
```
@@ -56,6 +57,9 @@ sgrep --profile quality "JWT validation middleware" # optional ColBERT reran
5657
| `balanced` | Semantic + BM25 | Recommended everyday profile; default |
5758
| `quality` | Semantic + BM25 + ColBERT | Optional reranking experiments |
5859

60+
Use `--implementation-first` when you want implementation evidence rather than
61+
documentation, examples, or tests. The default ordering is unchanged.
62+
5963
[Code-search guide →](https://xiaocui.me/sgrep/guides/code-search/)
6064

6165
## Search Agent Conversations

‎bench/jev/ALTERNATIVES_REPORT.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Self-hosted reranker alternatives (exploratory)
2+
3+
## Conclusion
4+
5+
Do not replace the current Nomic balanced default or adopt a local reranker
6+
from this evidence. The strongest code-specialized candidate found,
7+
`lightonai/LateOn-Code`, was also tested: it tied verified live Jina's quality
8+
through the existing shortlist boundary, took 13.0s p50 on CPU, and its direct
9+
ranking was materially worse. The two strongest llama.cpp-compatible
10+
cross-encoders likewise did not beat verified Jina or ColBERT.
11+
12+
All quality numbers are exploratory. The ten-query held-out set retains its
13+
fixed labels, but those labels have not received the requested blind human
14+
audit. No candidate model or Jev was used as a judge.
15+
16+
## Local benchmark
17+
18+
Environment: 8 vCPU, 15 GiB RAM, no GPU; llama.cpp `0.4.0-dev`, build 10809,
19+
commit `5266f24da`. Each arm preserves the existing candidates, filters,
20+
deduplication, paths/ranges, output and fusion boundary. The two added models
21+
ran via `/v1/rerank` on `127.0.0.1`, with `--offline`, 8 threads, 8192 context,
22+
and 2048 logical/physical batch size. Each model arm ran three times.
23+
24+
| Arm | MRR | NDCG@10 | R@10 | Prose promotion errors | p50 / p95 end-to-end |
25+
|---|---:|---:|---:|---:|---:|
26+
| Balanced | 0.667 | 0.499 | 0.438 | 1 | 35.4 / 48.8 ms |
27+
| Balanced + verified live Jina | 0.667 | 0.503 | 0.438 | 1 | 3,641 / 5,504 ms |
28+
| Quality / ColBERT | **0.729** | **0.567** | **0.500** | 1 | 6,645 / 8,638 ms |
29+
| Balanced + LateOn-Code rank fusion | 0.667 | 0.503 | 0.438 | 1 | 12,991 / 18,980 ms |
30+
| LateOn-Code direct top-20 order (diagnostic) | 0.573 | 0.448 | 0.406 | 4 | same inference |
31+
| Balanced + BGE v2-m3 Q4_K_M | 0.667 | 0.503 | 0.438 | 1 | 10,874 / 14,881 ms |
32+
| ColBERT + BGE v2-m3 Q4_K_M | 0.688 | 0.518 | 0.438 | **0** | 17,590 / 23,488 ms |
33+
| Balanced + Qwen3 0.6B Q8 | 0.667 | 0.503 | 0.438 | 1 | 19,079 / 26,954 ms |
34+
| ColBERT + Qwen3 0.6B Q8 | 0.713 | 0.562 | 0.500 | **0** | 25,039 / 36,004 ms |
35+
| Balanced + Jev (prior live run) | 0.708 | 0.553 | 0.500 | 2 | 452 / 521 ms |
36+
| ColBERT + Jev (prior live run) | **0.750** | **0.608** | **0.563** | 1 | 7,179 / 9,102 ms |
37+
38+
The local models produced identical metrics on all three corrected runs.
39+
Qwen's GGUF was 639,153,184 bytes, peaked at 2,886,880 KiB RSS, and processed
40+
367 prompt tokens/s. BGE's Q4_K_M was 438,376,864 bytes, peaked at 1,794,828
41+
KiB RSS, and processed 667 prompt tokens/s. These are single-server CPU
42+
measurements, not cross-hardware throughput claims.
43+
44+
The generic answerability thresholds were fixed before inspecting scores:
45+
Qwen probability >= 0.5 classified 10/10 cases correctly; BGE raw logit >= 0
46+
classified 9/10, rejecting both no-answer controls but also rejecting the
47+
positive self-refine case. This diagnostic scored only each arm's returned top
48+
10 and does not establish calibrated corpus-level rejection.
49+
50+
On six independently authored synthetic implementation/prose pairs, Qwen chose
51+
implementation in 2/6 and BGE in 1/6. A single predeclared code-oriented query
52+
instruction left Qwen at 2/6 and was not evaluated against held-out labels.
53+
54+
LateOn-Code was pinned at revision
55+
`ace431824f35db231178fc602e33296784762a2e` and run locally with PyLate 1.3.4.
56+
It is a 149M ModernBERT late-interaction model trained on CoRNStack, CoIR, and
57+
CodeSearchNet including Go. Its Apache-2.0 model card reports MTEB Code 74.12
58+
and CodeSearchNet MaxSim NDCG@10 0.8942; those are author-reported numbers.
59+
On this frozen transfer set, three production-compatible rank-fusion runs were
60+
deterministic, had no inference failures/fallbacks, peaked at 2,321,672 KiB RSS,
61+
and tied live Jina exactly. A separate direct-order diagnostic was worse on all
62+
three quality metrics and increased prose-promotion errors from one to four.
63+
It returned results for both no-answer controls and has no calibrated
64+
answerability threshold. The model cache was 573 MiB; the isolated Python
65+
environment was 7.0 GiB because PyTorch installed CUDA runtime packages even
66+
though inference used CPU.
67+
68+
## Runtime validity and failure behavior
69+
70+
Initial runs with llama.cpp's default 512-token physical batch were invalid:
71+
longer chunks returned HTTP 500 while `Searcher.rerankResults` intentionally
72+
continued without that reranker signal. Those files are retained with
73+
`invalid-ubatch512` in their names and excluded. Corrected servers used
74+
`--batch-size 2048 --ubatch-size 2048`; post-run full-top-10 protocol checks
75+
returned every score for 10/10 queries for each model.
76+
77+
The benchmark still verifies the scaffold's injected Jev failure gives exact
78+
pre-Jev order. The production search boundary swallows a local reranker error
79+
and continues, but can allocate reranker fusion weight with no scores; that is
80+
not always exact pre-reranker order. No production code was changed here.
81+
82+
## Focused survey
83+
84+
| Candidate | Evidence and fit | Deployment / license | Recommendation |
85+
|---|---|---|---|
86+
| LateOn-Code | Best code-specific hypothesis: 149M multi-vector model trained on CoRNStack/CoIR/CodeSearchNet, author-reported MTEB Code 74.12. Local fused quality only tied verified Jina; direct ranking regressed to 0.573/0.448/0.406 with four prose errors. | 2,048 document / 256 query tokens, 128-dimensional token vectors; PyLate, ONNX and FastPLAID/ColGrep; Apache-2.0. Local CPU p50 13.0s and 2.21 GiB peak RSS. | **Most promising model found, but reject adoption on measured transfer.** Re-test only on a larger blind-human-labeled set or GPU. |
87+
| Qwen3-Reranker-0.6B | Official card claims MTEB-Code 73.42 versus Jina 58.98 and BGE 41.38, but these are Qwen-authored runs over Qwen dense candidates. Local fixed-label quality did not beat Jina; implementation diagnostic 2/6. | 0.6B, 32K, 100+ natural/programming languages; Transformers, vLLM, llama.cpp GGUF; Apache-2.0. | **Experiment on a larger human-labeled set only.** Strongest surveyed llama.cpp cross-encoder candidate, but reject current CPU adoption. |
88+
| BGE-reranker-v2-m3 | Mature multilingual XLM-R cross-encoder. Jina's author-run CodeSearchNet table reports BGE 62.86 versus Jina 71.36. Local quality regressed and diagnostic was 1/6. | 568M; Transformers, FlagEmbedding, TEI, llama.cpp/ONNX ecosystem; Apache-2.0. | **Reject for sgrep's current shortlist.** |
89+
| Existing Jina v2 | Best local quality/latency tradeoff in this experiment and author-reported CodeSearchNet 71.36. Its one prose error remains. | 278M, 1024 tokens plus sliding window; Transformers/ONNX/GGUF; **CC-BY-NC-4.0**, commercial use needs separate terms. | **Keep only where the noncommercial license is acceptable.** License is the main adoption blocker. |
90+
| CodeRankEmbed | Code-specific 137M bi-encoder; author reports CSN MRR 77.9 and CoIR NDCG@10 60.1. Independent 2026 CORE-Bench finds it stronger than BGE-M3 among small off-the-shelf embeddings, but far below in-domain models. It changes first-stage embeddings rather than acting as a drop-in cross-encoder. | 8,192 context, required query prefix; Sentence Transformers; no first-party llama.cpp rerank route; MIT. Requires re-indexing and separate Go/runtime work. | **Highest-priority retrieval experiment**, not a reranker replacement. |
91+
| CodeRankLLM | Code-specific listwise model, but training order was supervised by Qwen2.5-32B and the card supplies no compact runtime benchmark. | 7B/8B-class BF16, community GGUFs, likely several GiB and slow on this CPU; MIT. | **Reject locally; GPU-only research candidate.** |
92+
| answerai-colbert-small-v1 | Very compact 33.4M, 96-dimensional token vectors, author-reported BEIR 53.79. No persuasive code-specific evidence. | 512 positions; Sentence Transformers MultiVector, ONNX, RAGatouille/ColBERT; Apache-2.0. Requires a new late-interaction index/runtime rather than the current llama endpoint. | **Low-cost experiment only** if replacing the current ColBERT implementation is independently desired. |
93+
| mxbai-rerank-base-v2 | Claims code support and 31.73 code-search score, but its card's latency is on A100 and its evidence is model-author supplied. | 0.5B, 109 languages; Transformers/Sentence Transformers/TEI; Apache-2.0. No validated drop-in llama.cpp path in this experiment. | **Defer behind Qwen/CodeRankEmbed.** |
94+
| Compact coder LLM as classifier | llama.cpp-compatible and rubric-capable, but lacks reranker calibration and adds generation/parsing latency. Qwen3 already took 19 s p50 as a purpose-trained 0.6B reranker here. | Many Apache-2.0 Qwen coder sizes; llama.cpp/ONNX varies. | **Reject** for interactive CPU search. |
95+
96+
No credible independent benchmark located compares these rerankers directly on
97+
implementation-versus-prose code search. Model-card tables are therefore marked
98+
as author/vendor evidence. CORE-Bench is independent and code-focused but
99+
benchmarks retrievers/embeddings, not this cross-encoder reranking boundary.
100+
101+
## Privacy and integration
102+
103+
The alternatives are fully offline after model download. Query/source chunks
104+
remain on-device, and the server bound only to loopback with llama.cpp offline
105+
mode. Model files live outside the repository under
106+
`~/.cache/sgrep-rerank-research/`. Artifacts contain queries, paths, line ranges,
107+
scores and metrics, but no source contents or credentials. A privacy scan found
108+
no API-key-shaped values or environment secret values.
109+
110+
Go integration for BGE/Qwen is mechanically simple through the existing
111+
`pkg/rerank.Reranker`, but model-specific score semantics matter: BGE emits raw
112+
logits while Qwen's llama.cpp conversion emits probabilities, and current
113+
fusion applies sigmoid uniformly. A production experiment would first need an
114+
explicit score contract and observable per-query reranker failures. Neither was
115+
added in this isolated benchmark.
116+
117+
## Sources
118+
119+
- Qwen3 model card and author benchmark: <https://huggingface.co/Qwen/Qwen3-Reranker-0.6B>
120+
- Official ggml Qwen Q8 conversion: <https://huggingface.co/ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF>
121+
- BGE model card: <https://huggingface.co/BAAI/bge-reranker-v2-m3>
122+
- BGE GGUF conversion: <https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF>
123+
- Jina v2 model card: <https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual>
124+
- CodeRankEmbed and CodeRankLLM: <https://huggingface.co/nomic-ai/CodeRankEmbed>, <https://huggingface.co/nomic-ai/CodeRankLLM>
125+
- AnswerAI ColBERT: <https://huggingface.co/answerdotai/answerai-colbert-small-v1>
126+
- Mixedbread v2: <https://huggingface.co/mixedbread-ai/mxbai-rerank-base-v2>
127+
- llama.cpp reranking endpoint/flags: <https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md>
128+
- Independent CORE-Bench (retrieval, not reranking): <https://arxiv.org/abs/2606.11864>
129+
- LateOn-Code model card and benchmark: <https://huggingface.co/lightonai/LateOn-Code>
130+
- PyLate runtime: <https://github.com/lightonai/pylate>

‎bench/jev/CODERANK_REPORT.md‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# CodeRankEmbed retrieval experiment
2+
3+
## Decision
4+
5+
**Reject CodeRankEmbed as sgrep's default retriever on this evidence.** It did
6+
not beat the current Nomic balanced retriever, did not beat the previously
7+
reported Jina arm, increased implementation-versus-prose errors, doubled query
8+
latency, and required a 46m47s CPU indexing pass for this 9,172-chunk corpus.
9+
10+
The strongest CodeRank result was CodeRank balanced + a verified live Jina
11+
endpoint. It improved recall over Nomic + verified Jina, but remained below the
12+
previously reported Jina MRR/NDCG, added a prose error, and took 4.68s p50.
13+
That is an experiment result, not an adoption case.
14+
15+
All quality scores are fixed-label exploratory evidence. The frozen labels
16+
have not received the requested blind human audit, and no model was used as a
17+
judge.
18+
19+
## Method
20+
21+
- Corpus: `dspy-go` commit `87cb50fa0611ef8a3905494c8b16fb0aab7666d3`.
22+
- 9,172 existing deterministic sgrep chunks and unchanged SQLite FTS5 index.
23+
- Model: `nomic-ai/CodeRankEmbed`, revision
24+
`3c4b60807d71f79b43f3c4363786d9493691f8b1`, with its required query prefix
25+
`Represent this query for searching relevant code: `.
26+
- Native 8,192-token model context; actual corpus maximum was 1,213 tokens.
27+
- Exact CPU float32 embeddings, normalized cosine retrieval; weighted RRF
28+
remained 0.6 dense / 0.4 BM25.
29+
- Existing filters, significant-overlap deduplication, paths, line ranges,
30+
top-20 Jina boundary, score fusion, and output limit were reconstructed from
31+
production code. The Nomic control exactly reproduced the frozen baseline.
32+
- Three query runs. Ranking and metrics were identical across runs.
33+
34+
## Results
35+
36+
Eight answerable held-out cases contribute to quality metrics; two frozen
37+
no-answer controls contribute only to answerability observations.
38+
39+
| Arm | MRR | NDCG@10 | R@10 | Candidate R@50 | Prose errors | p50 / p95 end-to-end |
40+
|---|---:|---:|---:|---:|---:|---:|
41+
| Nomic balanced control | **0.667** | **0.499** | 0.438 | **0.646** | 1 | **33.2 / 57.9 ms** |
42+
| Verified Nomic + live Jina | 0.667 | 0.503 | 0.438 | **0.646** | 1 | 3,641 / 5,504 ms |
43+
| CodeRank dense | 0.542 | 0.439 | 0.438 | 0.521 | **0** | 74.6 / 87.7 ms |
44+
| CodeRank balanced | 0.577 | 0.423 | 0.469 | 0.615 | 9 | 74.7 / 87.8 ms |
45+
| CodeRank balanced + live Jina | **0.708** | **0.553** | **0.552** | 0.615 | 2 | 4,681 / 5,867 ms |
46+
47+
Against Nomic balanced, CodeRank balanced had one MRR win, two losses, and five
48+
ties by case. It recovered self-refine and more logging recall, but regressed
49+
ReAct, tool registration, and logging rank quality. The hybrid arm's nine prose
50+
promotion errors versus Nomic's one are the clearest rejection signal for the
51+
stated implementation-versus-prose objective. Dense-only CodeRank avoided
52+
promotion errors but had the lowest candidate recall and lower aggregate
53+
quality.
54+
55+
CodeRank + live Jina recovered self-refine and logging results, producing one
56+
MRR win and no losses against verified Nomic + live Jina. It still did not beat
57+
the old reported Jina figures (0.729/0.565/0.500): MRR and NDCG were lower,
58+
recall was higher, and prose errors doubled.
59+
60+
Neither embedding model exposes a calibrated any-answer judgment. Every arm
61+
returned results for both no-answer controls, so interpreting non-empty output
62+
as “answer found” gives 0/2 no-answer rejections. Do not infer answerability
63+
from cosine score without a separately labeled calibration set.
64+
65+
## Jina control finding
66+
67+
The old “balanced + existing Jina” result (0.729/0.565/0.500 at about 36ms)
68+
does **not** measure a successful local Jina request in this orb. Port 8081 is
69+
owned by a `websocat` SSH bridge, not llama.cpp. `Searcher.rerankResults`
70+
silently continues after the resulting reranker error while still applying its
71+
fusion/code-path selection, yielding the old numbers.
72+
73+
The Go benchmark was therefore rerun against the explicit, healthy offline
74+
llama.cpp Jina endpoint on port 8084. It independently matched the Python
75+
experiment: 0.667/0.503/0.438 at 3.68s p50. This is both a benchmark-validity
76+
finding and a production observability concern; no production behavior was
77+
changed.
78+
79+
## Cost and deployment
80+
81+
- Full index build: 46m47s wall time on this 8-vCPU CPU-only orb.
82+
- Peak build RSS observed: 3,718,320 KiB (about 3.55 GiB).
83+
- CodeRank weights: 546,938,168 bytes; Hugging Face cache footprint 523 MiB.
84+
- Sidecar vectors: 28,176,512 bytes (about 26.9 MiB), versus the existing
85+
28 MiB Nomic exact-vector artifact.
86+
- Cached CodeRank query + exact retrieval: 74.7ms p50 / 87.8ms p95, versus
87+
Nomic's local-server query + exact retrieval at 33.2ms / 57.9ms.
88+
- CodeRank's first-party path is Sentence Transformers/PyTorch. This experiment
89+
found no first-party llama.cpp or existing Go-native integration; adopting it
90+
would add a runtime or require an export/runtime validation project.
91+
- Model license: MIT. Local inference is offline after model download.
92+
93+
## Privacy and artifacts
94+
95+
No source or query content was sent to a hosted service. CodeRank, Nomic, and
96+
Jina inference stayed local. Model download is the only network dependency.
97+
Artifacts contain frozen queries, paths, line ranges, rankings, and metrics;
98+
they contain no source bodies or credentials. A post-run secret-pattern and
99+
source-payload scan passed.
100+
101+
- `bench/jev/coderank_benchmark.py`: isolated benchmark scaffold.
102+
- `bench/jev/results-coderankembed.json`: three-run CodeRank/Nomic artifact.
103+
- `bench/jev/results-coderank-jina-control.json`: Go-harness live-Jina control.
104+
- Vector cache (outside repository):
105+
`~/.cache/sgrep-coderank/coderank-3c4b60807d71-9172.npy`.
106+
107+
## Ranked recommendation
108+
109+
1. **Adopt:** keep Nomic balanced as the local default. It has better aggregate
110+
quality than CodeRank balanced, better candidate recall, fewer prose errors,
111+
and lower latency.
112+
2. **Experiment:** first fix reranker success/failure observability, then rerun
113+
Jina and CodeRank+Jina on a larger blind-human-labeled code-search set. The
114+
small fixed set suggests CodeRank+Jina can recover recall, but not at an
115+
interactive CPU latency.
116+
3. **Experiment, hosted opt-in only:** ColBERT + Jev remains the best measured
117+
aggregate arm (0.750/0.608/0.563), but it uploads queries/source, costs money,
118+
and has multi-second latency. That is a separate privacy/product decision.
119+
4. **Reject now:** replacing Nomic with CodeRankEmbed or claiming that the old
120+
0.729 “existing Jina” score proves successful Jina inference.

0 commit comments

Comments
 (0)