Repository navigation
fix(vocab): stop consolidation on repeated LLM failures, flag partial failures - #599
Merged
Merged
Conversation
… failures A failed evaluation skipped the max_iterations check, so with a revoked key or a provider outage the live loop called the LLM and re-ran vocabulary analysis once for every remaining candidate pair. Both the live and dry-run loops now stop after three consecutive failed evaluations; a decision in between resets the count. kg vocab consolidate printed a green check and exited 0 unless every evaluation failed. Any failed evaluation now prints a warning naming how many pairs went undecided and exits 1. Fixes #592 Fixes #593
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MAX_CONSECUTIVE_EVAL_FAILURES(3) failed LLM evaluations in a row. Previously a failure skipped themax_iterationscheck, so an outage meant one failing LLM call plus a fullanalyze_vocabulary()per remaining pair. A decision in between resets the count.max_iterationssemantics are unchanged (it still counts toward merges only).kg vocab consolidateprints a yellow warning with the undecided count and exits 1 when any evaluation failed (previously green check, exit 0, unless all failed).Test plan
tests/unit/services/test_vocab_consolidation_failures.py(3 tests). Control: against the old loop (constant added only) the live and dry-run tests fail, the live run making 20 LLM calls through a simulated outage.make test-unit: 887 passedcd cli && npm run build && npm test: 46 passedFixes #592
Fixes #593