Skip to content

fix: align parity evidence with BlindBench ingest - #17

Merged
nmogil merged 1 commit into
mainfrom
fix-parity-blindbench-ingest
Jul 12, 2026
Merged

fix: align parity evidence with BlindBench ingest#17
nmogil merged 1 commit into
mainfrom
fix-parity-blindbench-ingest

Conversation

@nmogil

@nmogil nmogil commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Align multi-attempt provider-parity evidence with BlindBench's strict full-span ingest contract, and allow the completed paid matrix to be re-exported offline without rerunning providers.

  • Give every strict evidence record a deterministic, attempt-specific run.id while retaining the real attempt ID and private logical grouping metadata.
  • Exclude generated Python cache files from future trusted workspace evidence.
  • Narrowly sanitize the serialized letter + colon + newline sequence that BlindBench's Windows-path detector treats as an absolute path.
  • Add rollback-safe aggregate evidence re-export from retained checksummed bundles.
  • Require independently unique run IDs and attempts in producer/parity validation.

Activation finding

The completed 18-run matrix was valid at the execution layer but could not be imported as one BlindBench batch because:

  1. Mogil repeated stable run IDs across attempts while BlindBench requires independent uniqueness.
  2. Three inventory projections triggered BlindBench's serialized absolute-path detector on source strings such as except ValueError as e:\n.

This change regenerates only reviewer-safe aggregate evidence. It does not alter retained bundle bytes or rerun any model/provider.

Verification

  • pytest -q tests/test_harbor_bundle.py -k 'reexport or aggregate_pair or python_cache' — 5 passed
  • pytest -q -m 'not daytona_smoke' — 120 passed, 1 paid Daytona test deselected
  • ruff check . — passed
  • mypy — passed
  • git diff --check — passed
  • Real retained run re-export — 18 valid JSON + 18 valid JSONL records
  • BlindBench parseHarborEvidenceV1 — 18/18 parsed, 18 unique run IDs, 18 unique attempts
  • All 18 retained bundle checksums remain valid
  • Reviewer projection provenance/credential/canary/path scan — clean

Replacement safety

Both aggregate files are staged and validated before replacement. Each same-filesystem rename is atomic. An observed replacement failure triggers rollback of both prior destination states; this is deliberately not described as a crash-atomic multi-file transaction.

Safety boundaries

  • No provider calls or paid execution
  • No BlindBench upload
  • No bundle mutation
  • No training approval or export

@nmogil
nmogil merged commit 27e1601 into main Jul 12, 2026
2 checks passed
@nmogil
nmogil deleted the fix-parity-blindbench-ingest branch July 12, 2026 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant