feat: add CBDR strategy, confidence-gain reranking, and LM Studio pipeline - #9
Merged
Conversation
…DR estimators Adds the local LM Studio confidence-generation/training pipeline (provider, answer generator, dataset/report CLIs) and fixes CBDRStrategy reloading its BERT+LightGBM estimators on every benchmark query by caching them per artifact path. Runs the full 361-query AskUbuntu CBDR benchmark and records the results in README with an out-of-domain disclosure. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-reranking ; Conflicts: ; docs/wiki/02_architecture.md ; src/ranksmith/confidence_training/features.py ; src/ranksmith/strategies/__init__.py ; tests/test_confidence_scorer.py
…-resolved The merge commit only staged the files that had textual conflicts. Several files auto-merged cleanly but still referenced pre-rename module paths (ranksmith.confidence._encoder, confidence_training._dataset, strategies _common/_confidence_gain, integrations._answer_generator, etc.) or relied on ModelRequest.response_format/temperature, which origin/main's cleanup had dropped and this branch's LM Studio/answer-generator code still needs. This commit finishes that reconciliation; it was verified locally before the merge commit but not staged with it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CBDRStrategy(confidence-gain reranking:Conf(Q+D) - Conf(Q)via trainedConf(Q)/Conf(Q+C)scorers) and wires it intoscripts/compare_reranking.pyas an opt-in--algorithm cbdr.LMStudioModelProvider,ProviderAnswerGenerator, dataset generation/report CLIs) soConf(Q)/Conf(Q+C)scorer artifacts can be produced without Azure.CBDRStrategyreloading its BERT encoder + LightGBM scorer on every single benchmark query (functools.cache); this was going to take ~4 hours of pure reload overhead for the 361-query AskUbuntu benchmark before the fix.cbdrand records it in the READMEBenchmarkingtable with an explicit out-of-domain disclosure (scorers trained on TriviaQA, evaluated on AskUbuntu).Test plan
uv run pytest -q(640 passed)uv run ruff check .uv run mypy src/--algorithm cbdrrun against the committed AskUbuntu BM25 top-20 candidates (361/361 valid, invalid_rate 0.000); evidence:benchmark-results/askubuntu-bm25-top20-cbdr-live.json🤖 Generated with Claude Code