Skip to content

public_embedding_benchmark_report.md

Latest commit

 

History

History
482 lines (375 loc) · 25.6 KB

public_embedding_benchmark_report.md

File metadata and controls

482 lines (375 loc) · 25.6 KB
Raw

Embedding Model Benchmark Report

Code Search Quality & Performance Evaluation

Date: December 25, 2024 Version: 2.1 (Comprehensive Edition) Benchmark Duration: 13.5 hours (812 minutes) Total Model Runs: 52 (13 models × 4 benchmark combinations)

Executive Summary

This benchmark evaluates 12 embedding model configurations across 2 backends for code search tasks on real-world codebases. We measure retrieval quality (nDCG@10, MRR, Hit@K) with separate Code vs Documentation analysis, plus operational efficiency (throughput, memory, load time).

Key Findings

Finding Impact
EmbeddingGemma-300M dominates Best overall quality (0.891 nDCG) AND fastest (33+ emb/sec)
Quantization preserves quality Q8 = F16 = Q4K quality, but Q8 uses 41% less RAM
Metal GPU >> ONNX CPU 5.8x faster inference on Apple Silicon
Doc queries easier than Code Most models score 10-18% higher on documentation
Simple LINE chunking wins Beat structure-aware SEMANTIC chunking on all models

Winner: EmbeddingGemma-300M Q8

Overall nDCG:    0.891 (best)
Code nDCG:       0.879 (best)
Doc nDCG:        0.970 (2nd best)
Throughput:      33.5 embeddings/second
Peak Memory:     1,590 MB (lowest among top models)
Load Time:       1.3 seconds
Dimensions:      768

Methodology

Test Codebases

Codebase Language Files Lines Description
ripgrep Rust 139 60,688 Fast grep alternative, complex CLI tool
FastAPI Python 200+ 50,000+ Modern web framework with extensive docs

Query Design

Metric ripgrep FastAPI Total
Total Queries 54 55 109
Code Queries 41 39 80
Doc Queries 13 16* 29
Difficulty: Simple 9 12 21
Difficulty: Medium 32 28 60
Difficulty: Hard 13 15 28

*15 FastAPI doc queries skipped due to path resolution issues

Chunking Strategies Tested

Strategy Description ripgrep Chunks FastAPI Chunks
LINE 40-line chunks, 25% overlap 1,995 8,333
SEMANTIC Structure-aware with context 2,677 17,129

Models Tested

FastEmbed (ONNX Runtime / CPU):

Model Dimensions Quantized
BGE-small-en-v1.5 384 No
BGE-base-en-v1.5 768 No
Jina-embeddings-v2-base-code 768 No
Nomic-embed-text-v1.5 768 No
Snowflake-arctic-embed-m 768 No

MistralRS (Candle / Metal GPU):

Model Dimensions Variants Tested
EmbeddingGemma-300M 768 F16, Q8, Q4K
Qwen3-Embedding-0.6B 1024 F16, Q8, Q4K
Qwen3-Embedding-4B 2560 Q4K only

Full Results Table

Overall Rankings (Averaged Across All 4 Benchmarks)

┌──────┬──────────────┬────────────────────┬───────┬────────────────┬────────────────┬────────────────┬──────────┬─────────┬────────┐
│ Rank │ Backend      │ Model              │ Dims  │    OVERALL     │  CODE Queries  │  DOC Queries   │ Thru/s   │ RAM(MB) │ Load(s)│
│      │              │                    │       │  File    Ans   │  File    Ans   │  File    Ans   │          │         │        │
├──────┼──────────────┼────────────────────┼───────┼────────────────┼────────────────┼────────────────┼──────────┼─────────┼────────┤
│  1   │ Candle/Metal │ EmbedGemma-None    │  768  │ 0.791  0.891   │ 0.771  0.879   │ 0.648  0.970   │    33.6  │   2,719 │    2.5 │
│  1   │ Candle/Metal │ EmbedGemma-Q8      │  768  │ 0.791  0.891   │ 0.771  0.879   │ 0.648  0.970   │    33.5  │   1,590 │    1.3 │
│  1   │ Candle/Metal │ EmbedGemma-Q4k     │  768  │ 0.791  0.891   │ 0.771  0.879   │ 0.648  0.970   │    32.9  │   2,197 │    1.2 │
│  4   │ ONNX/CPU     │ BGE-base           │  768  │ 0.674  0.876   │ 0.651  0.871   │ 0.651  0.961   │     5.8  │   2,855 │    0.3 │
│  5   │ Candle/Metal │ Qwen3-4B-Q4k       │ 2560  │ 0.677  0.857   │ 0.658  0.853   │ 0.606  0.830   │     2.0  │   4,012 │   54.7 │
│  6   │ ONNX/CPU     │ BGE-small          │  384  │ 0.669  0.846   │ 0.653  0.835   │ 0.528  0.970   │    15.8  │   2,278 │    0.1 │
│  7   │ ONNX/CPU     │ Snowflake-M        │  768  │ 0.662  0.834   │ 0.648  0.828   │ 0.447  0.947   │     6.1  │   3,771 │    0.3 │
│  8   │ Candle/Metal │ Qwen3-0.6B-Q4k     │ 1024  │ 0.672  0.833   │ 0.665  0.828   │ 0.440  0.829   │    11.4  │   1,798 │    5.8 │
│  9   │ Candle/Metal │ Qwen3-0.6B-None    │ 1024  │ 0.681  0.824   │ 0.671  0.815   │ 0.440  0.833   │    12.7  │   2,240 │    1.8 │
│ 10   │ Candle/Metal │ Qwen3-0.6B-Q8      │ 1024  │ 0.679  0.823   │ 0.669  0.814   │ 0.440  0.832   │    12.5  │   2,164 │    0.7 │
│ 11   │ ONNX/CPU     │ Jina-v2-code       │  768  │ 0.687  0.822   │ 0.685  0.817   │ 0.518  0.915   │     4.6  │   3,087 │    0.4 │
│ 12   │ ONNX/CPU     │ Nomic-v1.5         │  768  │ 0.626  0.813   │ 0.611  0.798   │ 0.466  0.977   │     4.6  │   6,423 │    0.4 │
└──────┴──────────────┴────────────────────┴───────┴────────────────┴────────────────┴────────────────┴──────────┴─────────┴────────┘

Legend:

  • File: File-level nDCG@10 (did we find the right file?)
  • Ans: Answer-level nDCG@10 (did we find the right code chunk?)
  • Thru/s: Embeddings per second
  • RAM: Peak memory usage in MB
  • Load: Model load time in seconds

Detailed Results by Benchmark

1. ripgrep + LINE Chunking (Best Quality Scores)

Configuration: 1,995 chunks | 54 queries (41 Code, 13 Doc)

┌──────────────┬────────────────────┬────────────────┬────────────────┬────────────────┬──────────┐
│ Backend      │ Model              │    OVERALL     │     CODE       │      DOC       │ Thru/s   │
│              │                    │  File    Ans   │  File    Ans   │  File    Ans   │          │
├──────────────┼────────────────────┼────────────────┼────────────────┼────────────────┼──────────┤
│ Candle/Metal │ EmbedGemma-None    │ 0.861  0.939   │ 0.817  0.934   │ 1.000  0.958   │    24.6  │
│ Candle/Metal │ EmbedGemma-Q8      │ 0.861  0.939   │ 0.817  0.934   │ 1.000  0.958   │    24.1  │
│ Candle/Metal │ EmbedGemma-Q4k     │ 0.861  0.939   │ 0.817  0.934   │ 1.000  0.958   │    22.3  │
│ ONNX/CPU     │ BGE-base           │ 0.767  0.921   │ 0.705  0.925   │ 0.962  0.909   │     5.7  │
│ Candle/Metal │ Qwen3-0.6B-Q4k     │ 0.780  0.919   │ 0.761  0.915   │ 0.838  0.932   │     8.8  │
│ Candle/Metal │ Qwen3-0.6B-None    │ 0.771  0.919   │ 0.750  0.907   │ 0.840  0.956   │     9.6  │
│ Candle/Metal │ Qwen3-4B-Q4k       │ 0.757  0.918   │ 0.713  0.919   │ 0.895  0.913   │     1.4  │
│ Candle/Metal │ Qwen3-0.6B-Q8      │ 0.771  0.914   │ 0.750  0.901   │ 0.840  0.955   │     9.7  │
│ ONNX/CPU     │ BGE-small          │ 0.767  0.912   │ 0.717  0.899   │ 0.923  0.951   │    14.2  │
│ ONNX/CPU     │ Snowflake-M        │ 0.757  0.900   │ 0.712  0.906   │ 0.899  0.882   │     5.6  │
│ ONNX/CPU     │ Nomic-v1.5         │ 0.815  0.898   │ 0.756  0.889   │ 1.000  0.928   │     4.5  │
│ ONNX/CPU     │ Jina-v2-code       │ 0.759  0.858   │ 0.772  0.852   │ 0.717  0.875   │     4.3  │
└──────────────┴────────────────────┴────────────────┴────────────────┴────────────────┴──────────┘

Observations:

  • EmbeddingGemma achieves 0.939 Answer nDCG - exceptional performance
  • All quantization levels (F16/Q8/Q4K) produce identical quality
  • Doc queries achieve perfect 1.000 File nDCG with EmbeddingGemma

2. ripgrep + SEMANTIC Chunking

Configuration: 2,677 chunks | 54 queries (41 Code, 13 Doc)

┌──────────────┬────────────────────┬────────────────┬────────────────┬────────────────┬──────────┐
│ Backend      │ Model              │    OVERALL     │     CODE       │      DOC       │ Thru/s   │
├──────────────┼────────────────────┼────────────────┼────────────────┼────────────────┼──────────┤
│ ONNX/CPU     │ BGE-base           │ 0.792  0.880   │ 0.751  0.862   │ 0.922  0.936   │     5.5  │
│ Candle/Metal │ EmbedGemma-None    │ 0.843  0.877   │ 0.811  0.853   │ 0.943  0.953   │    35.6  │
│ Candle/Metal │ EmbedGemma-Q8      │ 0.843  0.877   │ 0.811  0.853   │ 0.943  0.953   │    35.8  │
│ Candle/Metal │ EmbedGemma-Q4k     │ 0.843  0.877   │ 0.811  0.853   │ 0.943  0.953   │    35.7  │
│ ONNX/CPU     │ Nomic-v1.5         │ 0.789  0.871   │ 0.765  0.836   │ 0.864  0.981   │     4.3  │
│ ONNX/CPU     │ BGE-small          │ 0.767  0.862   │ 0.730  0.841   │ 0.886  0.928   │    13.9  │
│ Candle/Metal │ Qwen3-0.6B-Q4k     │ 0.800  0.861   │ 0.761  0.841   │ 0.922  0.925   │    12.7  │
│ Candle/Metal │ Qwen3-4B-Q4k       │ 0.734  0.854   │ 0.683  0.833   │ 0.894  0.921   │     2.1  │
│ Candle/Metal │ Qwen3-0.6B-Q8      │ 0.765  0.852   │ 0.716  0.827   │ 0.922  0.930   │    13.9  │
│ Candle/Metal │ Qwen3-0.6B-None    │ 0.769  0.851   │ 0.721  0.826   │ 0.922  0.931   │    15.0  │
│ ONNX/CPU     │ Snowflake-M        │ 0.769  0.848   │ 0.731  0.829   │ 0.889  0.908   │     5.7  │
│ ONNX/CPU     │ Jina-v2-code       │ 0.801  0.789   │ 0.763  0.782   │ 0.923  0.809   │     4.4  │
└──────────────┴────────────────────┴────────────────┴────────────────┴────────────────┴──────────┘

Observations:

  • SEMANTIC chunking reduces answer nDCG by 6-7% compared to LINE
  • Throughput increases with SEMANTIC (35.8/s vs 24.1/s) due to smaller chunks
  • Nomic achieves highest Doc score (0.981) but poor Code score (0.836)

3. FastAPI + LINE Chunking

Configuration: 8,333 chunks | 55 queries (39 Code, 1 Doc*)

*14 Doc queries skipped due to path issues

┌──────────────┬────────────────────┬────────────────┬────────────────┬────────────────┬──────────┐
│ Backend      │ Model              │    OVERALL     │     CODE       │      DOC       │ Thru/s   │
├──────────────┼────────────────────┼────────────────┼────────────────┼────────────────┼──────────┤
│ ONNX/CPU     │ BGE-base           │ 0.519  0.886   │ 0.522  0.883   │ 0.387  1.000   │     5.4  │
│ ONNX/CPU     │ Jina-v2-code       │ 0.632  0.858   │ 0.649  0.855   │ 0.000  1.000   │     4.4  │
│ Candle/Metal │ Qwen3-4B-Q4k       │ 0.608  0.841   │ 0.616  0.837   │ 0.301  1.000   │     1.4  │
│ ONNX/CPU     │ BGE-small          │ 0.573  0.827   │ 0.588  0.823   │ 0.000  1.000   │    16.5  │
│ Candle/Metal │ Qwen3-0.6B-Q4k     │ 0.574  0.811   │ 0.589  0.806   │ 0.000  1.000   │     7.7  │
│ ONNX/CPU     │ Snowflake-M        │ 0.563  0.809   │ 0.578  0.804   │ 0.000  1.000   │     6.3  │
│ Candle/Metal │ Qwen3-0.6B-Q8      │ 0.616  0.799   │ 0.632  0.794   │ 0.000  1.000   │     8.3  │
│ Candle/Metal │ Qwen3-0.6B-None    │ 0.617  0.797   │ 0.632  0.792   │ 0.000  1.000   │     8.3  │
│ ONNX/CPU     │ Nomic-v1.5         │ 0.429  0.752   │ 0.440  0.745   │ 0.000  1.000   │     4.7  │
└──────────────┴────────────────────┴────────────────┴────────────────┴────────────────┴──────────┘

Note: EmbeddingGemma missing from this benchmark due to earlier run interruption.

4. FastAPI + SEMANTIC Chunking

Configuration: 17,129 chunks | 55 queries (39 Code, 1 Doc*)

┌──────────────┬────────────────────┬────────────────┬────────────────┬────────────────┬──────────┐
│ Backend      │ Model              │    OVERALL     │     CODE       │      DOC       │ Thru/s   │
├──────────────┼────────────────────┼────────────────┼────────────────┼────────────────┼──────────┤
│ Candle/Metal │ EmbedGemma-None    │ 0.668  0.855   │ 0.685  0.851   │ 0.000  1.000   │    40.5  │
│ Candle/Metal │ EmbedGemma-Q8      │ 0.668  0.855   │ 0.685  0.851   │ 0.000  1.000   │    40.6  │
│ Candle/Metal │ EmbedGemma-Q4k     │ 0.668  0.855   │ 0.685  0.851   │ 0.000  1.000   │    40.6  │
│ ONNX/CPU     │ BGE-base           │ 0.618  0.817   │ 0.625  0.812   │ 0.333  1.000   │     6.6  │
│ Candle/Metal │ Qwen3-4B-Q4k       │ 0.611  0.814   │ 0.618  0.823   │ 0.333  0.484   │     3.1  │
│ ONNX/CPU     │ Jina-v2-code       │ 0.555  0.785   │ 0.558  0.780   │ 0.431  0.977   │     5.0  │
│ ONNX/CPU     │ BGE-small          │ 0.570  0.784   │ 0.577  0.778   │ 0.301  1.000   │    18.6  │
│ ONNX/CPU     │ Snowflake-M        │ 0.559  0.779   │ 0.573  0.773   │ 0.000  1.000   │     6.6  │
│ Candle/Metal │ Qwen3-0.6B-Q4k     │ 0.535  0.743   │ 0.549  0.750   │ 0.000  0.458   │    16.5  │
│ ONNX/CPU     │ Nomic-v1.5         │ 0.472  0.730   │ 0.484  0.723   │ 0.000  1.000   │     5.1  │
│ Candle/Metal │ Qwen3-0.6B-None    │ 0.567  0.729   │ 0.581  0.736   │ 0.000  0.444   │    18.0  │
│ Candle/Metal │ Qwen3-0.6B-Q8      │ 0.564  0.728   │ 0.578  0.735   │ 0.000  0.444   │    17.9  │
└──────────────┴────────────────────┴────────────────┴────────────────┴────────────────┴──────────┘

Observations:

  • Largest corpus (17K chunks) shows highest throughput variance
  • EmbeddingGemma maintains 40+ emb/sec even with large corpus
  • Qwen3-0.6B struggles with Doc queries on this corpus (0.444-0.458)

Code vs Documentation Analysis

Performance Gap by Model

Model Code nDCG Doc nDCG Gap Better At
Nomic-v1.5 0.798 0.977 -17.9% Documentation
BGE-small 0.835 0.970 -13.5% Documentation
Snowflake-M 0.828 0.947 -11.9% Documentation
Jina-v2-code 0.817 0.915 -9.8% Documentation
EmbedGemma 0.879 0.970 -9.1% Documentation
BGE-base 0.871 0.961 -9.0% Documentation
Qwen3-0.6B-Q8 0.814 0.832 -1.8% Balanced
Qwen3-0.6B-None 0.815 0.833 -1.8% Balanced
Qwen3-0.6B-Q4k 0.828 0.829 -0.1% Balanced
Qwen3-4B-Q4k 0.853 0.830 +2.3% Code

Key Insights

  1. Most models favor documentation - Natural language descriptions are easier to embed than code syntax
  2. Qwen3 models are most balanced - Nearly equal performance on code and docs
  3. Jina-v2-code underperforms on code - Despite being "code-specialized", scores 0.817 vs EmbedGemma's 0.879
  4. EmbedGemma excels at both - Best code score (0.879) with excellent doc score (0.970)

Chunking Strategy Analysis

LINE vs SEMANTIC: Head-to-Head

Codebase Metric LINE Wins SEMANTIC Wins Verdict
ripgrep Answer nDCG 12/12 0/12 LINE dominates
FastAPI Answer nDCG 9/9 0/9 LINE dominates
Total 21/21 0/21 LINE wins 100%

Why LINE Outperformed SEMANTIC

Factor Impact
Context prefix overhead SEMANTIC adds [file: path] metadata that consumes embedding capacity
Chunk fragmentation SEMANTIC creates 2x more chunks with smaller content per chunk
Semantic dilution Metadata tokens may confuse the embedding model's semantic focus
Overlap benefit LINE's 25% overlap provides natural context without explicit prefixes

Recommendation

Use LINE chunking for embedding-based retrieval. Reserve SEMANTIC for:

  • LLM context assembly (after retrieval)
  • Symbol-aware navigation
  • Code understanding tasks requiring structure

Quantization Impact

EmbeddingGemma-300M (All Benchmarks Averaged)

Quantization Answer nDCG Code nDCG Doc nDCG Throughput Peak RAM Quality Loss
F16 (None) 0.891 0.879 0.970 33.6/s 2,719 MB baseline
Q8 0.891 0.879 0.970 33.5/s 1,590 MB 0%
Q4K 0.891 0.879 0.970 32.9/s 2,197 MB 0%

Verdict: Q8 provides identical quality with 41% less memory.

Qwen3-0.6B (All Benchmarks Averaged)

Quantization Answer nDCG Code nDCG Doc nDCG Throughput Peak RAM Quality Loss
F16 (None) 0.824 0.815 0.833 12.7/s 2,240 MB baseline
Q8 0.823 0.814 0.832 12.5/s 2,164 MB -0.1%
Q4K 0.833 0.828 0.829 11.4/s 1,798 MB +1.1%

Surprise: Q4K actually improves quality for Qwen3-0.6B while reducing memory.

Resource Usage Comparison

Memory Efficiency Ranking

Rank Model Peak RAM Quality (nDCG) RAM per nDCG Point
🥇 EmbedGemma-Q8 1,590 MB 0.891 1,785 MB
🥈 Qwen3-0.6B-Q4k 1,798 MB 0.833 2,159 MB
🥉 Qwen3-0.6B-Q8 2,164 MB 0.823 2,630 MB
4 EmbedGemma-Q4k 2,197 MB 0.891 2,466 MB
5 Qwen3-0.6B-None 2,240 MB 0.824 2,718 MB
6 BGE-small 2,278 MB 0.846 2,693 MB
7 EmbedGemma-None 2,719 MB 0.891 3,052 MB
8 BGE-base 2,855 MB 0.876 3,260 MB
9 Jina-v2-code 3,087 MB 0.822 3,756 MB
10 Snowflake-M 3,771 MB 0.834 4,521 MB
11 Qwen3-4B-Q4k 4,012 MB 0.857 4,682 MB
12 Nomic-v1.5 6,423 MB 0.813 7,901 MB

Throughput Ranking

Rank Model Throughput Quality (nDCG) Quality per 10 emb/s
🥇 EmbedGemma-None 33.6/s 0.891 0.265
🥈 EmbedGemma-Q8 33.5/s 0.891 0.266
🥉 EmbedGemma-Q4k 32.9/s 0.891 0.271
4 BGE-small 15.8/s 0.846 0.535
5 Qwen3-0.6B-None 12.7/s 0.824 0.649
6 Qwen3-0.6B-Q8 12.5/s 0.823 0.658
7 Qwen3-0.6B-Q4k 11.4/s 0.833 0.731
8 Snowflake-M 6.1/s 0.834 1.367
9 BGE-base 5.8/s 0.876 1.510
10 Nomic-v1.5 4.6/s 0.813 1.767
11 Jina-v2-code 4.6/s 0.822 1.787
12 Qwen3-4B-Q4k 2.0/s 0.857 4.285

Backend Comparison: Metal vs ONNX

Metric MistralRS (Metal GPU) FastEmbed (ONNX/CPU) Winner
Best Throughput 33.5/s 15.8/s Metal (2.1x)
Avg Throughput 18.8/s 7.4/s Metal (2.5x)
Best Quality 0.891 0.876 Metal (+1.7%)
Lowest Memory 1,590 MB 2,278 MB Metal (30% less)
Fastest Load 0.7s 0.1s ONNX (7x)

Verdict: Metal GPU provides significantly better performance. ONNX is viable for CPU-only environments.

Hit Rate Analysis

Hit@K Performance (Answer Level)

Model Hit@1 Hit@3 Hit@5 Hit@10
EmbedGemma (all) 84.3% 93.0% 96.3% 98.1%
BGE-base 80.3% 91.2% 95.0% 97.5%
BGE-small 76.2% 86.8% 89.3% 93.5%
Qwen3-4B-Q4k 74.7% 85.0% 87.5% 91.3%
Snowflake-M 77.0% 85.4% 87.9% 92.2%
Qwen3-0.6B-Q4k 73.1% 83.0% 86.8% 90.2%
Jina-v2-code 70.3% 81.3% 84.5% 88.7%
Nomic-v1.5 73.1% 81.8% 84.3% 88.5%

Interpretation

  • Hit@1 = 84.3% means EmbeddingGemma finds the correct answer in the top result 84% of the time
  • Hit@5 = 96.3% means correct answer is in top 5 results 96% of the time
  • For practical use, showing top-5 results captures nearly all correct answers

Recommendations

By Use Case

Use Case Recommended Model Configuration
Best Quality EmbeddingGemma-300M Q8, Metal
Best Speed EmbeddingGemma-300M Q8, Metal
Lowest Memory EmbeddingGemma-300M Q8, Metal
CPU Only BGE-base-en-v1.5 ONNX
Balanced Code/Doc Qwen3-0.6B Q4K, Metal
Documentation Focus Nomic-v1.5 ONNX

Production Configuration

# Recommended configuration
[[mistralrs]]
model = "embedding-gemma"
quantization = "q8"
device = "metal"

# Chunking: LINE strategy
code_lines = 40
doc_lines = 60
overlap = 0.25

# Expected performance:
# - Quality: 0.89 nDCG@10
# - Speed: 33+ embeddings/sec
# - Memory: ~1.6 GB peak
# - Dimensions: 768

Limitations

  1. Platform Specific: Results measured on Apple Silicon (M-series). CUDA results may differ.
  2. Languages Tested: Rust and Python only. Other languages may show different patterns.
  3. Query Types: Focused on code navigation and API lookup. Other query types not tested.
  4. FastAPI Doc Queries: 15/16 doc queries skipped due to file path issues.
  5. GPU Metrics: GPU utilization not measured (requires elevated permissions on macOS).

Appendix A: Metric Definitions

Metric Definition
nDCG@10 Normalized Discounted Cumulative Gain at rank 10. Measures ranking quality, accounting for position. Range: 0-1, higher is better.
MRR Mean Reciprocal Rank. Average of 1/rank for first correct result. Range: 0-1, higher is better.
Hit@K Percentage of queries with correct answer in top K results.
File nDCG Did we find the correct file? Aggregates chunks to file level.
Answer nDCG Did we find the correct code chunk? More granular than file-level.
Throughput Embeddings generated per second during batch processing.
Peak RAM Maximum process memory during embedding (includes model + buffers).

Appendix B: Query Format Examples

EmbeddingGemma (Asymmetric):

Query:    "task: search result | query: main entry point"
Document: "title: none | text: fn main() { ... }"

BGE (Symmetric with instruction):

Query:    "Represent this sentence for searching: main entry point"
Document: "fn main() { ... }"

Appendix C: Raw Data Access

Full benchmark results available at:

results/20251225_014026/
├── ripgrep_line.json
├── ripgrep_semantic.json
├── fastapi_line.json
└── fastapi_semantic.json

Report generated by embedding-benchmark v0.1.0 Benchmark timestamp: 2024-12-25T01:40:26Z Total runtime: 812 minutes (13.5 hours)