Skip to content

fix: reconcile model-buyer results with transcripts, add run provenance - #24

Merged
abhay-codes07 merged 1 commit into
mainfrom
fix/model-buyer-honesty-provenance
Sep 5, 2026
Merged

fix: reconcile model-buyer results with transcripts, add run provenance#24
abhay-codes07 merged 1 commit into
mainfrom
fix/model-buyer-honesty-provenance

Conversation

@abhay-codes07

Copy link
Copy Markdown
Owner

fix: reconcile model-buyer results with transcripts, add run provenance

Three honesty gaps, closed.

Model buyer contradicted its own transcripts. It reported "0 wrong
orders" while the transcripts showed payments on decline_* tasks that a
reroute exemption scored correct. Verified each: every such order was a
legitimate network reroute — the named merchant could not fulfil the
task, the index found one that serves the pincode and holds the stock,
and the gate verified both. Re-ran clean on gpt-4o (40 tasks, none
rate-limited): 30 orders split into 14 at the named merchant and 16 via
reroute, 9 model walk-aways, 0 orders the gate should have blocked. The
results file now lists every reroute by task (from -> to) so the
transcripts and the headline agree, and a new consistency test pins the
reroute count and the zero-violations claim to the README.

Provenance. The gpt-4o run is a cache replay, so a reader could not tell
whether the model or the fallback produced it. The report now carries
llm_health (backend, total_failures, total_failovers) and RESULTS.md
renders a Provenance section: 0 failovers proves gpt-4o answered, with
the real hit/miss split shown (this run: 371 live calls, the rest
cached) — no more silent zero-miss ambiguity.

Consistency drift. Test count synced to 91 in both docs; the cache and
latency lines now point to the generated provenance section / give a
range instead of quoting transient per-run numbers, so they cannot drift
again; the threat model cites the 190-attack corpus instead of calling
the probes "equally small"; the demo docs' stale 17/17 and 24/24-checks
corrected. Two new consistency tests guard the generated red team and
the model-buyer claims.

91 tests, ruff clean, consistency guard green.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BLrj9TWybDCNw1mWxZgAks

Three honesty gaps, closed.

Model buyer contradicted its own transcripts. It reported "0 wrong
orders" while the transcripts showed payments on decline_* tasks that a
reroute exemption scored correct. Verified each: every such order was a
legitimate network reroute — the named merchant could not fulfil the
task, the index found one that serves the pincode and holds the stock,
and the gate verified both. Re-ran clean on gpt-4o (40 tasks, none
rate-limited): 30 orders split into 14 at the named merchant and 16 via
reroute, 9 model walk-aways, 0 orders the gate should have blocked. The
results file now lists every reroute by task (from -> to) so the
transcripts and the headline agree, and a new consistency test pins the
reroute count and the zero-violations claim to the README.

Provenance. The gpt-4o run is a cache replay, so a reader could not tell
whether the model or the fallback produced it. The report now carries
llm_health (backend, total_failures, total_failovers) and RESULTS.md
renders a Provenance section: 0 failovers proves gpt-4o answered, with
the real hit/miss split shown (this run: 371 live calls, the rest
cached) — no more silent zero-miss ambiguity.

Consistency drift. Test count synced to 91 in both docs; the cache and
latency lines now point to the generated provenance section / give a
range instead of quoting transient per-run numbers, so they cannot drift
again; the threat model cites the 190-attack corpus instead of calling
the probes "equally small"; the demo docs' stale 17/17 and 24/24-checks
corrected. Two new consistency tests guard the generated red team and
the model-buyer claims.

91 tests, ruff clean, consistency guard green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BLrj9TWybDCNw1mWxZgAks
Copilot AI lite review requested due to automatic review settings September 5, 2026 11:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@abhay-codes07
abhay-codes07 merged commit d11aa10 into main Sep 5, 2026
2 checks passed
@abhay-codes07
abhay-codes07 deleted the fix/model-buyer-honesty-provenance branch September 5, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants