fix: reconcile model-buyer results with transcripts, add run provenance - #24
Merged
Merged
Conversation
Three honesty gaps, closed. Model buyer contradicted its own transcripts. It reported "0 wrong orders" while the transcripts showed payments on decline_* tasks that a reroute exemption scored correct. Verified each: every such order was a legitimate network reroute — the named merchant could not fulfil the task, the index found one that serves the pincode and holds the stock, and the gate verified both. Re-ran clean on gpt-4o (40 tasks, none rate-limited): 30 orders split into 14 at the named merchant and 16 via reroute, 9 model walk-aways, 0 orders the gate should have blocked. The results file now lists every reroute by task (from -> to) so the transcripts and the headline agree, and a new consistency test pins the reroute count and the zero-violations claim to the README. Provenance. The gpt-4o run is a cache replay, so a reader could not tell whether the model or the fallback produced it. The report now carries llm_health (backend, total_failures, total_failovers) and RESULTS.md renders a Provenance section: 0 failovers proves gpt-4o answered, with the real hit/miss split shown (this run: 371 live calls, the rest cached) — no more silent zero-miss ambiguity. Consistency drift. Test count synced to 91 in both docs; the cache and latency lines now point to the generated provenance section / give a range instead of quoting transient per-run numbers, so they cannot drift again; the threat model cites the 190-attack corpus instead of calling the probes "equally small"; the demo docs' stale 17/17 and 24/24-checks corrected. Two new consistency tests guard the generated red team and the model-buyer claims. 91 tests, ruff clean, consistency guard green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BLrj9TWybDCNw1mWxZgAks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: reconcile model-buyer results with transcripts, add run provenance
Three honesty gaps, closed.
Model buyer contradicted its own transcripts. It reported "0 wrong
orders" while the transcripts showed payments on decline_* tasks that a
reroute exemption scored correct. Verified each: every such order was a
legitimate network reroute — the named merchant could not fulfil the
task, the index found one that serves the pincode and holds the stock,
and the gate verified both. Re-ran clean on gpt-4o (40 tasks, none
rate-limited): 30 orders split into 14 at the named merchant and 16 via
reroute, 9 model walk-aways, 0 orders the gate should have blocked. The
results file now lists every reroute by task (from -> to) so the
transcripts and the headline agree, and a new consistency test pins the
reroute count and the zero-violations claim to the README.
Provenance. The gpt-4o run is a cache replay, so a reader could not tell
whether the model or the fallback produced it. The report now carries
llm_health (backend, total_failures, total_failovers) and RESULTS.md
renders a Provenance section: 0 failovers proves gpt-4o answered, with
the real hit/miss split shown (this run: 371 live calls, the rest
cached) — no more silent zero-miss ambiguity.
Consistency drift. Test count synced to 91 in both docs; the cache and
latency lines now point to the generated provenance section / give a
range instead of quoting transient per-run numbers, so they cannot drift
again; the threat model cites the 190-attack corpus instead of calling
the probes "equally small"; the demo docs' stale 17/17 and 24/24-checks
corrected. Two new consistency tests guard the generated red team and
the model-buyer claims.
91 tests, ruff clean, consistency guard green.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BLrj9TWybDCNw1mWxZgAks