benchmark-integrity-appendix.md

Benchmark integrity appendix

This appendix records benchmark cases where a higher score would not demonstrate a better coding harness. It is a living companion to the exact evidence in the failure journal. Final aggregates will be added only after the final-candidate campaign finishes; the completed frozen diagnostic aggregate is retained below.

The rule

Peritus keeps the score produced by the unchanged benchmark. It does not read hidden verifiers or reference solutions while solving a task, hard-code task names or private vocabulary, alter tasks, fixtures, resources, deadlines, or scoring, or add behavior supported only by a benchmark-specific win.

A low score is not automatically a benchmark gotcha. Real Peritus defects, ordinary model misses, provider failures, and infrastructure failures remain failures. General corrections stay in the product only when they improve ordinary coding work without relying on private benchmark facts. Installing ordinary prerequisites such as Python, Make, R, compilers, or declared packages inside an authorized disposable task environment, and keeping those tools discoverable on PATH, is normal task execution rather than benchmark cooking. It becomes cooking only when behavior depends on a task or verifier identity, hidden reference data, or altered fixtures, resources, deadlines, or scoring.

Each entry below links to its published contract, hidden or contradictory expectation, exact retained result, and evidence. The text here names the score-only shortcut Peritus refused.

Frozen diagnostic aggregate

The completed serialized baseline contains all 445 trials: 239 reward-1, 151 reward-0, and 55 unscored. Accuracy is 0.6128205128 across 390 scored trials and 0.5370786517 across all completed trials. The no-overwrite normalized report is retained outside Git at /home/doll/.local/state/peritus/benchmarks/terminalbench/reports/frozen-baseline-445.final.json with SHA-256 d7feff820c7d38d204744f75ef9214cb7b91949cac2c8c3b5625f10c39321bc0. It preserves the official results; the categories below explain evidence without adjusting scores. Because general fixes landed during this diagnostic campaign, it is not the final-candidate result.

Broken or internally inconsistent evaluators

These cases cannot be satisfied reliably by following the published contract. Raising the score would require changing the evaluator, relocating hidden data, exploiting its string processing, or changing correct product behavior to accommodate an evaluator bug.

• HBI-002: the scoring formula cannot reach its own pass threshold. Refused: patching the formula or reporting an invented passing score.
• HBI-003: the oracle looks outside the task for ground truth. Refused: copying hidden data to the mistaken path.
• HBI-007: normalization makes the required filename token unmatchable. Refused: unnatural text crafted to exploit the bug.
• HBI-013: a denial is scored as a forbidden promise. Refused: removing a clear customer-facing safety boundary.
• HBI-022: the direct oracle loads an ordinary dataclass module incorrectly. Refused: weakening normal Python code to fit the loader defect.
• HBI-027: the runner moves a sandbox without rewriting earlier evidence paths. Refused: rewriting immutable prior reports or patching the pinned runner.
• HBI-035: a forbidden-claim check ignores negation. Refused: deleting explicit language that says the forbidden promise will not be made.
• HBI-039: the oracle expects an input script absent from the fixture. Refused: fabricating the missing source or silently altering the fixture.
• TBI-008: the clean-input check requires parser reserialization despite an exact formatting-preservation contract, while browser startup failures count as safe. Refused: violating the stated byte contract, copying the hidden reference behavior, or treating unexecuted checks as proof.
• TBI-009: the public interface defines output_path as a folder, while the verifier and hidden reference require that exact argument to be the CSV file. Refused: reading the hidden interface, overriding explicit prose from a .csv suffix, or adding task-specific dual behavior.

Ground truth that contradicts the supplied inputs

These cases publish enough evidence to derive a valid answer, then grade against a hidden answer that violates that evidence or the stated boundary. Winning would require knowingly returning a wrong, unsafe, or contract-breaking result.

• HBI-006: every hidden meeting slot conflicts with supplied calendars. Refused: scheduling known conflicts.
• HBI-012: hidden truth requires an identifier the appendix retires. Refused: reusing the prohibited identifier.
• HBI-014: direct negative evidence is graded as insufficient evidence. Refused: discarding a sourced fact.
• HBI-028: hidden stock membership violates the published threshold. Refused: adding an ineligible item.
• HBI-036: hidden shortlist expectations contradict the must-have threshold. Refused: weakening the stated hiring rule.
• HBI-040: hidden grading contradicts the published evidence rubric. Refused: changing a rubric-faithful assessment to match the private grade.
• HBI-041: hidden confidence categories overlap contrary to the stated calibration. Refused: replacing a reasoned classification with the private label.
• TBI-011: repeated bases permit equivalent circular-plasmid junction decompositions, but the verifier grades only its private split. Refused: reading the verifier and replacing a valid one-pair primer design solely to match that hidden boundary.

Unpublished vocabulary, schemas, and serialization

These tasks request meaning or a general shape but grade exact private labels, tokens, aliases, field spellings, row conventions, or prose. Consistently winning them would require reading the hidden verifier or reference solution, or maintaining task-specific guesses. Peritus retains the ordinary, evidence-grounded result instead.

• HBI-005: private error labels and row numbering.
• HBI-008: private risk and renewal wording.
• HBI-009: private to-do and approval phrases.
• HBI-010: private expense issue labels.
• HBI-011: private no-change row convention.
• HBI-015: private evidence tokens and nonessential details.
• HBI-016: private priority verb set.
• HBI-017: private key convention plus double-counted rename.
• HBI-018: private quotation and authority conventions.
• HBI-019: private path and row-citation shape.
• HBI-020: private architecture substrings.
• HBI-021: private schema-version spelling.
• HBI-023: private glob and prose string rules.
• HBI-024: private documentation tokens.
• HBI-025: private test-source tokens.
• HBI-026: private release-note substrings.
• HBI-029: private state and log serialization.
• HBI-030: private conflict-location word.
• HBI-031: private severity taxonomy and synonym.
• HBI-032: private incident identifier.
• HBI-033: private blocker severity mapping.
• HBI-034: private normal-status enum.
• HBI-037: private reply keys and primary clause.
• HBI-038: private compensation token.
• HBI-042: private adjacent-word sequence.
• HBI-043: private archive-chain serialization.
• HBI-044: private normalized source schema.
• HBI-045: private conflict-source alias.
• HBI-047: private orphan-reason phrase.
• HBI-053: private severity, priority, and summary shape.
• HBI-054: private cross-session carryover and duplicate routing.
• HBI-063: private refusal synonym set.
• HBI-066: private partial-snapshot and ledger shapes.
• HBI-067: private repeated alias convention.
• TBI-010: private MTEB task and query/passage prompt convention that changes the requested embedding rank.

Hidden runtime and effect conventions

These cases require an unpublished mechanism rather than a published artifact contract. The score-only shortcut would be private-verifier inspection or adding an otherwise unjustified effect.

• HBI-046: the exact local DOM is supplied, but scoring demands a redundant root request. Refused: extra network traffic whose only purpose is the hidden trace check.
• TBI-004: the verifier requires a private socket path absent from the task. Refused: reading the verifier or hard-coding its path.
• TBI-005: the checkpoint does not reveal the hidden two-argument model signature. Refused: reading the verifier or reference implementation and hard-coding its private interface.

Cases deliberately excluded from this appendix

Entries labeled HBF or TBF are product or adapter defects and are fixed generally when the evidence supports a correction. HBM and TBM entries are ordinary model limitations. Clean control tasks remain unchanged. Infrastructure failures such as unavailable dependencies, provider terminals, container failures, and verifier timeouts are reported separately and never converted into an inferred score.

The final delivery report will add the final-candidate comparison and link each demonstrated general correction to its unchanged rerun. It will not delete, reclassify, or rewrite a retained failure merely because a later candidate performs better.