Skip to content

[codex] Add AcuRank and BM25 top-20 benchmark - #2

Merged
pko89403 merged 1 commit into
mainfrom
codex/acurank-benchmark-release
May 29, 2026
Merged

[codex] Add AcuRank and BM25 top-20 benchmark#2
pko89403 merged 1 commit into
mainfrom
codex/acurank-benchmark-release

Conversation

@pko89403

Copy link
Copy Markdown
Owner

Summary

  • Add AcuRank as a built-in uncertainty-aware adaptive reranking strategy with sync/async support, TrueSkill-based scoring, score priors, strict validation, and an offline example.
  • Add BM25 top-20 reranking benchmark support, method aliases, Pyserini candidate artifact handling, and committed benchmark summary evidence.
  • Refresh README/README.ko, wiki/spec/reference docs, benchmark documentation, and bump the package version to 0.5.0.

Benchmark Evidence

  • Dataset: AskUbuntuDupQuestions test data
  • First-stage candidates: Pyserini BM25 top-20
  • Evaluation: rerank top-20, evaluate @5, 361 queries
  • Evidence artifact: benchmark-results/live/askubuntu-bm25-top20-default-live.v2.merged.json
  • Candidate run artifact: benchmark-results/pyserini/askubuntu-bm25-top20.trec

Validation

  • ./scripts/verify.sh -> 209 passed, lint/type/build succeeded
  • UV_CACHE_DIR=.uv-cache uv tool run twine check dist/* -> passed for 0.5.0 wheel/sdist
  • Pre-push hook reran verify + twine check successfully

Notes

  • Raw *.query_results.jsonl benchmark rows and retry artifacts are intentionally ignored and not committed.
  • Remaining invalid benchmark rows are reported in the summary artifact instead of being repaired.

@pko89403
pko89403 marked this pull request as ready for review May 29, 2026 02:47
@pko89403
pko89403 merged commit 833bb1b into main May 29, 2026
5 checks passed
@pko89403
pko89403 deleted the codex/acurank-benchmark-release branch May 29, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant