review: D071/D072 checked from a fresh context -- verdict stands, prereg states H2 two ways (D074)

WAVES.md requires every real result be reviewed outside the context that produced it;
HANDOFF said this had not happened for H2. It has now. Every point estimate recomputed
from the response JSONL bit-identical, the repo's own bootstrap reproduced exactly, cluster
counts re-derived from membership (a bare worker_id would have given 133/131, so D071's fix
was load-bearing), S10k-S's cap and apportionment verified for all 85 factories.

One major finding, documentation not measurement: the pre-registration says 'design effect
>= 2' and 'twice the width', which under its own definition is a design effect of 4. H2
fails under both readings; D072, the card claim and README had treated them as synonyms and
now say otherwise. Minor: D072's exclusion count is 28 not 26 (amended in place); the third
decimal is Monte Carlo noise (+/-0.05, read to two); the judge weight sha is a launch-script
pin, not in the records; a stale S10k docstring in qwen3vl.py; the H2 record_ref names both
arms. Card text lives in emit_card.py; the JSON regenerates with make card + make space-data
together since stats.json pins the digest.

Author: caiotheodoro

README.md

@@ -130,9 +130,12 @@ under the 1pp threshold). Real cost: $9.06. `docs/DECISIONS.md` D066/D067.
 
 **The design effect, measured where a worker id exists.** 10,000 frames drawn from the raw
 corpus under each of the two pre-registered designs, judged, and bootstrapped by worker
-(B = 10,000). Pre-registered threshold: 2. Measured: 1.25–1.66 across both arms and all
-three figures, largest on 2-hands both times. Every figure exceeds 1, so an iid interval on
-this corpus is 12–29% too narrow in width; none reaches 2, so H2 fails as stated. The corpus
+(B = 10,000). Pre-registered threshold: a design effect of 2, which the same frozen document
+also phrases as "twice the width", a design effect of 4 under its own definition
+(`docs/DECISIONS.md` D074). Measured: 1.25–1.66 across both arms and all three figures
+(±0.05 Monte Carlo band), largest on 2-hands both times. Every figure exceeds 1, so an iid
+interval on this corpus is 12–29% too narrow in width; none reaches 2, so H2 fails under
+either reading. Reviewed independently, numbers reproduced bit-for-bit (D074). The corpus
 also ships 2,144 workers against the published 2,153 (`docs/UPSTREAM-FINDINGS.md` F12).
 `docs/DECISIONS.md` D072.
 

docs/DECISIONS.md

@@ -2344,6 +2344,13 @@ per D053) on one `g6e.2xlarge` spot instance, ~3h45m, ~$7.50, terminated at comp
 20,000 frames were excluded and recorded rather than dropped: 26 undecodable (0.13%, the
 clip-end shortfall F12's neighbours describe) and 2 unparseable.
 
+*Amended 2026-09-05 (D074):* the count above is 28, not 26 -- 10 + 16 undecodable and 1 + 1
+unparseable, per `n_extraction_failed`/`n_excluded_other` in both result files. The
+"twice the width ... a design effect of at least 2" equivalence in this entry's second
+paragraph is also not one the pre-registration's own definition supports; D074 records why,
+and why the verdict is unchanged. The third decimal in the table above is inside the
+bootstrap's Monte Carlo band (about +/-0.05 at B=10,000); read it to two.
+
 **Result 2 is unaffected.** The adapter removed one of D048's three reasons; the other two stand
 and either is sufficient alone (no institutional EPIC-KITCHENS-100 email; no downstream-task
 labels in the release). It remains this card's single `what_could_not_be_checked` item, so the
@@ -2430,3 +2437,60 @@ records the omission in the index.
 frames come out of the atlas first and this entry is amended second; or the datasets-server
 grows a fast batch path for image rows, in which case the atlas is a cache rather than a
 republication and can be dropped for the live path it replaced.
+
+## D074 — Independent review of D071/D072: the verdict stands; the pre-registration states H2's threshold two ways
+
+`WAVES.md` requires every real result to be reviewed from a context other than the one that
+produced it, and `HANDOFF.md` recorded that this had not happened for D071/D072. It has now: a
+fresh context was given the pre-registration, the two entries, the runner, the bootstrap, the
+membership and response files, and none of the producing session's narrative. It recomputed
+every point estimate from the response JSONL (bit-identical to the result files), re-ran the
+repository's own bootstrap (seed 777, B=10,000: intervals and design effects reproduced
+exactly), re-derived the cluster counts from the membership files (1,966 / 1,999; a bare
+`worker_id` would have given 133 / 131, so D071's fix was load-bearing), verified `S10k-S`'s
+one-frame-per-clip cap and its per-factory worker-hour apportionment for all 85 factories,
+confirmed that both intervals are computed on the same kept observations, and ran an
+independent bootstrap under a different seed. **D072's conclusion, that H2 does not hold,
+stands.** Seven findings, none reopening the measurement:
+
+1. **The pre-registration states H2 in two units that disagree under its own definition
+   (major, documentation).** Its hypothesis heading says *design effect ≥ 2*; the sentence under
+   it says *at least twice the width*; its statistics section defines design effect as
+   `(cluster CI width / iid CI width)²`. Twice the width is therefore a design effect of 4.
+   D072, the card's H2 claim and `README.md` had restated the two as synonyms. Under either
+   reading H2 fails -- the largest measured effect is 1.66, a width ratio of 1.29 against 2 --
+   and the literal reading fails by more, so the verdict does not move; what it was measured
+   against was misstated. The frozen text is not rewritten, per its own rule. The card's claim,
+   `README.md` and `docs/WRITEUP.md` now say the threshold was stated two ways and that the
+   result clears neither.
+2. **The third decimal is Monte Carlo noise (minor).** An independent bootstrap at a different
+   seed moved single design effects by up to about 0.05 (`S10k-S` hand ≥ 1: 1.31 → 1.26; `S10k-U`
+   manipulation: 1.27 → 1.33). D072's "the two arms land within 0.05 of each other" is at the
+   estimator's resolution, not a robustness check beyond it. Design effects are read to two
+   decimals with a ±0.05 band from here on; the structure -- every figure in [1.2, 1.7], none
+   near 2, 2-hands about 0.3 above the other two on both arms -- is stable across seeds. A
+   variance-ratio estimator over the bootstrap distributions would be less noisy than the
+   squared percentile-width ratio; the pre-registration fixed the latter, so it stays.
+3. **D072's exclusion arithmetic (minor).** "26 excluded: 26 undecodable and 2 unparseable" sums
+   to 28. Amended in place with a dated note.
+4. **The judge weight revision D072 cites is not in the data records (minor).** `judge_rev` on
+   every response is the served model *name* (`qwen3vl.py` returns `response.model`); the commit
+   `9cdc6310…` exists only as the launch scripts' pin (`cloud/aws_qwen3vl.sh`,
+   `cloud/modal_qwen3vl.py`). The claim holds by inference from the launch script, not from the
+   records. Recording the pin into the result JSON is a runner change for the next real run, not
+   a retroactive edit to this one.
+5. **`cost_usd` basis (note).** Per-response cost uses the Modal L4 rate the judge class was
+   written against; the 20,000 responses sum to $4.92, while D072's ~$7.50 is `g6e.2xlarge` spot
+   wall-clock. Neither is an invoice. D072's figure is what was paid.
+6. **Stale docstring (note).** `qwen3vl.py`'s `_image_bytes_for` still said it raises for
+   `S10k-U`/`S10k-S` frames. Fixed.
+7. **`record_ref` named one arm (note).** The H2 claim quotes both arms; the reference now names
+   both result files.
+
+`scripts/emit_card.py` carries the corrected claim text; the committed `MEASUREMENT_CARD.json`
+is regenerated with `make card` alongside `make space-data`, since the Space's precomputed
+`stats.json` pins the card's digest and the two must move together.
+
+**Reverses if:** nothing in the measurement. Finding 1 would read differently only if the
+pre-registration's "twice the width" is recorded as a slip for "twice the variance"; the
+honest position is that the frozen text says both, and this entry says so.

docs/HANDOFF.md

@@ -20,8 +20,11 @@ into ~85 and inflated the design effect into a false positive (D071); and the co
 `REVIEW.md` R2, now much cheaper than when it was written — the ground-truth `worker_id` this
 work produced can calibrate a DINOv2 pseudo-cluster proxy against real labels, which is the
 only route to saying anything about the design effect on *Build AI's own frames*
-(`RED-TEAM.md` A13); and an independent review of D071/D072's units, which `WAVES.md` requires
-from a different context than the producer's and which has not happened for this round.
+(`RED-TEAM.md` A13). The independent review of D071/D072 `WAVES.md` requires has now run
+(D074): the verdict stands, the numbers reproduce bit-for-bit, and the pre-registration turned
+out to state H2's threshold two ways; the card's claim text is corrected in `emit_card.py` and
+the committed `MEASUREMENT_CARD.json` is regenerated with `make card` together with
+`make space-data`, since the Space's `stats.json` pins the card digest.
 
 **Prior header — (D068) the card's own H2/Result 2 text was found actively
 contradicting D065 (still said "confirmed NOT authorized... 403" after access was granted) --

docs/WRITEUP.md

@@ -72,8 +72,10 @@ no worker id to cluster over.
 pre-registered.** (`data/h2_design_effect.S10k-U.json`, `data/h2_design_effect.S10k-S.json`)
 
 Frames from one worker share scene, lighting, task, gloves and camera. The pre-registration
-predicted a design effect of at least 2: a clustered interval at least twice as wide as the iid
-one. Measured at N = 10,000 per arm, B = 10,000, on both draws:
+predicted a design effect of at least 2, and in the next sentence a clustered interval at
+least twice as wide as the iid one, which under its own definition is a design effect of 4;
+the result clears neither (`docs/DECISIONS.md` D074). Measured at N = 10,000 per arm,
+B = 10,000, on both draws (read to two decimals; the bootstrap's own resolution is about ±0.05):
 
 | figure | `S10k-U` | `S10k-S` |
 |---|---:|---:|
@@ -84,8 +86,9 @@ one. Measured at N = 10,000 per arm, B = 10,000, on both draws:
 None reaches 2, so H2 fails as stated. All six exceed 1, so the effect is real: an interval
 computed as if frames were independent is 12–29% too narrow in width on this corpus. Concretely,
 the 2-hands rate on `S10k-U` is 80.8%, iid interval [80.0, 81.6], clustered [79.8, 81.8]. Two
-separate draws with different cluster structures land within 0.05 of each other on every task,
-which is the result's own robustness check. The 2-hands figure carries the largest design effect
+separate draws with different cluster structures land within the bootstrap's own resolution
+of each other on every task, and an independent re-run at a different seed reproduced the
+structure (`docs/DECISIONS.md` D074). The 2-hands figure carries the largest design effect
 on both arms, and it is the same figure the independent judge missed by six points on both
 releases; three measurements now single it out, and none explains it.
 

scripts/emit_card.py

@@ -441,12 +441,16 @@ def _h2_claim() -> Claim:
         block["design_effect"] for data in arms.values() for block in data["tasks"].values()
     )
     statement = (
-        "H2 does NOT hold (D072). Pre-registered: cluster-bootstrap intervals over worker_id at "
-        f"least twice the width of iid ones (design effect >= 2). Measured: {worst:.2f} at the "
-        "highest, across two arms and three tasks -- " + " | ".join(parts) + ". "
+        "H2 does NOT hold (D072). Pre-registered threshold, stated two ways that disagree under "
+        "the pre-registration's own definition design effect = (cluster CI width / iid CI "
+        "width)^2: 'design effect >= 2', and 'intervals at least twice the width of iid ones', "
+        "which is a design effect of 4 (D074). Measured: "
+        f"{worst:.2f} at the highest (read to two decimals; the bootstrap's Monte Carlo band "
+        "at B=10,000 is about +/-0.05, D074), across two arms and three tasks -- "
+        + " | ".join(parts) + ". Clears neither reading. "
         "The effect is real but smaller than pre-registered: every figure exceeds 1, so an iid "
         "interval on this corpus is genuinely too narrow, by 12-29% in width (sqrt of the design "
-        "effect) rather than the >=41% H2 asserted. The 2-hands figure carries the largest "
+        "effect) rather than the >=41% (or, read literally, 100%) H2 asserted. The 2-hands figure carries the largest "
         "design effect on both arms, the same dimension where H1 failed by 6.32pp (D056) and "
         "where the 100K re-run failed again by 6.1pp (D067) -- a convergence recorded, not "
         "explained. Measured on vernier's own corpus draws, never on Build AI's evaluation "
@@ -456,7 +460,7 @@ def _h2_claim() -> Claim:
     return Claim(
         statement=statement,
         record_type="AgreementResult",
-        record_ref=f"{_H2_RESULTS_REF.format(arm='S10k-U')}#H2",
+        record_ref=", ".join(f"{_H2_RESULTS_REF.format(arm=arm)}#H2" for arm in _H2_ARMS),
     )
 
 

src/vernier/judges/qwen3vl.py

@@ -181,8 +181,9 @@ def judge_rev(self) -> str:
     def _image_bytes_for(self, frame: FrameRef) -> bytes:
         """Resolve `frame` to its real JPEG bytes via `sampling.draw.image_bytes_for`
         (`ARCHITECTURE.md`: judges depend on sampling for frames, nothing else). Real for the
-        `E10k-*` family; raises `NotImplementedError` for `S10k-U`/`S10k-S` frames, the same as
-        the sampling seam it delegates to.
+        `E10k-*` / `E100k-ego` family (parquet) and, since D071, for `S10k-U`/`S10k-S` (one
+        frame out of an h265 clip over HTTP range requests, `sampling.corpus_frames`); a frame
+        the seam cannot decode surfaces as the seam's own error, never a silent skip.
         """
         return image_bytes_for(frame)