Add the methods note and an executable blinding harness for the H2 holdout

[thabatarj8-stack]

What this adds

Two pieces of the same publication step: the method written down, and the method made enforceable.

Methods note v0.1, bilingual, linked from the reading path, the roadmap, and the discovery checklist. It records the invalidated first holdout as a preserved failure rather than omitting it.

experiments/002/harness/, which turns the blinding protocol into artifacts instead of operator discipline. The first #2 holdout was burned because ground truth reached the evaluator context before predictions existed, so the ordering is now structural rather than procedural:

Module	Context	May read the key
blinding.py	shared	n/a
preflight.py	curator	yes
holdout_run.py	evaluator	no code path exists
holdout_reveal.py	curator	only after verifying the prediction seal
hub.py	curator	yes

• preflight.py freezes a holdout only after checking schema conformance, ID convention, frozen composition, evidence mix, and both structural and lexical answer leaks.
• holdout_run.py projects each trace to the input boundary its frozen prompt permits: the content channel sees trace_id and final_output, the trajectory channel sees trace_id and ordered events. Withholding the output from the trajectory channel is what keeps the OR aggregation a measure of incremental signal.
• holdout_reveal.py verifies the prediction_freeze seal before opening the key, and exits without reading it if verification fails. It never rewrites the sealed manifest; reveal-time facts go to run-manifest.final.json.
• hub.py keeps traces and key in separate private dataset repositories, so the commit carrying the predictions is timestamped independently of the commit that publishes the key. A local seal proves integrity but cannot prove ordering to a third party.

Also adds ground-truth.schema.json, an open-weight evaluator backend that records exact_weights_available, and evaluator-failure recording that refuses to report a partial run as complete.

Notes for review

• Two incompatible trace conventions exist in the repo. reproducibility/ uses lowercase trajectory_id; experiments/002/schemas/ uses NIR-002-*. The harness follows the Build a bilingual research reading path and article series #2 schemas, which are authoritative for H2, and does not reuse the older validator.
• is_semantic_leak implements a lexical policy over curator-authored trajectory prose. final_output is exempt, because that text is the object under judgement rather than a description of it. The policy is blunt by design; a rejection is a prompt to rewrite the case, not to loosen LABEL_VOCABULARY.
• Preflight now rejects a holdout with no unsafe final output. Without one, the final-only condition scores zero recall by construction and the comparison is degenerate.
• 26 tests, wired into quality.yml. Each guard has a test asserting it fails closed.

Evidence status unchanged

This makes the H2 holdout executable, not executed. The roadmap items for a fresh blind pilot stay open. No confirmatory claim is made or implied here.

🤖 Generated with Claude Code