Skip to content

docs(efficacy): round 4's report and verdicts (#1875) (#1876) (#1949) - #1954

Merged
braboj merged 1 commit into
mainfrom
docs/round-4-verdicts
Oct 17, 2026
Merged

braboj merged 1 commit into
mainfrom
docs/round-4-verdicts

Conversation

@braboj

@braboj braboj commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator

Closes #1875, closes #1876, closes #1949. Replaces #1953.

What

  • docs/audits/2026-10-17-efficacy-thin.md and docs/audits/2026-10-17-efficacy-weaker.md: report.py output for each round-4 root, unedited. report.py names both 2026-10-17-efficacy.md, so each is renamed with a run suffix.
  • docs/audits/2026-10-17-round-4.md: the verdicts on design §12.3 and §12.4. It has a growth table for every registered row, and thin hybrid's registered scores beside the key-supplied re-run (Settle whether the application must start without a secret key #1949).

Verdicts

Hypothesis Verdict Rows that count
12.3, thin spec (#1876) Supported hybrid, judged security +1.45; full +27 pp and hand +32 pp, security probes
12.4, weaker model (#1875) Supported full, judged security +0.56; hybrid, judged data protection +1.0; hand +18 pp, security probes
  • Task success didn't grow under haiku. Full, short and hybrid scored below none.
  • Both reports owe an escalation to K = 5, which hasn't been run.
  • Under the thin spec, hybrid's 42% build suite and its probe rows measure the missing TARIFF_SECRET_KEY, not its code. With the key supplied, its suite scores are 87.1%, 94.6% and 84.8%.

How it was produced

  • Both roots were judged three times per trial by claude-opus-5-5, with no cross-check (§12.2). Thin hybrid-2 was judged after fix(efficacy): keep a virtualenv that lost its pyvenv.cfg out of the judge's bundle (#1950) #1951.
  • The archives were verified from a fresh clone of braboj/tariff-runs (rounds/2026-10-16-thin, rounds/2026-10-15-weaker, rounds/2026-10-16-thin-sensitivity).
  • report.py ran on main at 8f09318.
  • Growth is round 4's mean paired difference against none minus round 3's, as 2026-09-27-efficacy.md prints it. A row counts where that reaches the registered bar and round 4's verdict is "better".

No CHANGELOG entry: these are benchmark reports, not a template change. run_smoke.py 33/33; run_conformance.py 0 failed.

🤖 Generated with Claude Code

Both registered hypotheses are supported on a few rows at K = 3:
the thin spec (12.3) on hybrid's judged security and full's and hand's
security probes, the weaker model (12.4) on full's judged security,
hybrid's judged data protection and hand's security probes. Task
success did not grow under the weaker model. Thin hybrid's registered
scores stand beside the key-supplied re-run.

Closes #1875, closes #1876, closes #1949.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@braboj
braboj merged commit 07e68db into main Oct 17, 2026
1 check passed
@braboj
braboj deleted the docs/round-4-verdicts branch October 17, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant