|
3 | 3 | The resume point. A fresh session should be able to continue from this file without |
4 | 4 | re-deriving anything. |
5 | 5 |
|
6 | | -**Last updated: 2026-09-04 — (D071/D072) H2 is measured and closed.** The raw-corpus adapter |
| 6 | +**Last updated: 2026-09-05 — delivery is current on every surface; what is left is |
| 7 | +publication.** Main is green (a `constraints.txt` lock pins CI's install; the Space's stats |
| 8 | +snapshot no longer fails on a test count or a float ulp). `MEASUREMENT_CARD.json` |
| 9 | +carries 18 claims and one unmet item (Result 2), verdict `NOT_VERIFIED`; the Hub dataset, |
| 10 | +the Hub model and the Space all serve that exact card (digest `3aacbe46…`), and the dataset's |
| 11 | +`results/` now ships both `h2_design_effect` files the H2 claim cites. D074 is the independent |
| 12 | +review of D071/D072 `WAVES.md` required: the verdict stands, every number reproduced from the |
| 13 | +response records, and the pre-registration turned out to state H2's threshold two ways (design |
| 14 | +effect ≥ 2, and "twice the width", which is 4 under its own definition; the result clears |
| 15 | +neither). D075 opened a third, salted, blind labelling pass over the 18 rater/judge |
| 16 | +disagreements plus an equal control arm; it is a separate `review` pass and **feeds no published |
| 17 | +number** — folding it needs its own entry with before and after. Two sessions wrote a D074 |
| 18 | +within minutes of each other on 2026-09-05; the later one is now D075. That will recur while two |
| 19 | +sessions share one append-only file: check `grep -c '^## D0' docs/DECISIONS.md` against the |
| 20 | +last number before appending. |
| 21 | + |
| 22 | +**Still open, in order:** publication (the section below); the D075 fold decision, once the |
| 23 | +review pass is complete and `scripts/review_labels_cli.py report` says whether the controls |
| 24 | +moved; a second rater on `G200-ego` (`RED-TEAM.md` A1, still the first weakness on the front |
| 25 | +page, ~35 minutes of someone else's time); `judge_weights_rev` recorded into the result JSON on |
| 26 | +the next real judge run (D074 finding 4); `REVIEW.md` R2, the pseudo-cluster proxy on Build AI's |
| 27 | +own frames, now calibratable against real `worker_id`s (`RED-TEAM.md` A13); Result 2, blocked |
| 28 | +on two reasons the adapter did not touch. |
| 29 | + |
| 30 | +## Next session: publication |
| 31 | + |
| 32 | +Three steps. One is the critical path, one follows it, one runs beside it. `docs/private/` |
| 33 | +holds the outreach sequence and the email draft; it is gitignored and is never quoted here or in |
| 34 | +anything public (`AGENTS.md` rule 5). |
| 35 | + |
| 36 | +### Gate, before anything goes public |
| 37 | + |
| 38 | +Run all of it; publish only if every line is clean. |
| 39 | + |
| 40 | +``` |
| 41 | +git status --short && git rev-list --left-right --count origin/main...main # clean, 0 0 |
| 42 | +gh run list --limit 1 # ✓ on main |
| 43 | +make validate # tests, mypy, lint |
| 44 | +make card && git diff --exit-code MEASUREMENT_CARD.json # committed card == fresh |
| 45 | +python3 -c "import json;print(json.load(open('MEASUREMENT_CARD.json'))['content_digest'])" |
| 46 | +python3 -c "import json;print(json.load(open('space/public/data/stats.json'))['generated_from']['card_digest'])" |
| 47 | +curl -sL https://caiotheodoro-vernier.static.hf.space/data/stats.json | python3 -c "import sys,json;print(json.load(sys.stdin)['generated_from']['card_digest'])" |
| 48 | +curl -sL https://huggingface.co/datasets/caiotheodoro/vernier/raw/main/MEASUREMENT_CARD.json | python3 -c "import sys,json;print(json.load(sys.stdin)['content_digest'])" |
| 49 | +``` |
| 50 | + |
| 51 | +The four digests must be identical. The digest does not see UI-only changes, so also compare |
| 52 | +`git log -1 --format='%h %ci' -- space/` with the Space repo's `lastModified` |
| 53 | +(`curl -s https://huggingface.co/api/spaces/caiotheodoro/vernier`); as of this header the last |
| 54 | +Space commit (f5ef78a, 23:31 local) is four minutes newer than the last upload. If either is |
| 55 | +behind, `make space-data && make space` |
| 56 | +then `hf upload caiotheodoro/vernier space/dist . --type space`; if the dataset's is behind, |
| 57 | +`make hf-dataset` then `hf upload caiotheodoro/vernier hf/dataset . --type dataset` (both need |
| 58 | +`HF_TOKEN` from `.env`; `hf` is the 1.30 CLI at `~/.local/bin/hf`). If the other session has |
| 59 | +folded D075 into the primary labels in the meantime, every number downstream moves: run |
| 60 | +`make agreement`, `make distil`, `make card`, `make space-data`, republish, and only then |
| 61 | +continue — a writeup that cites numbers a later commit restates is the exact failure this |
| 62 | +project audits in its subject. |
| 63 | + |
| 64 | +Then the writeup's own rule: every figure in `docs/WRITEUP.md` is copied from the card or a |
| 65 | +`data/*.json` file, and the card wins on disagreement. Check the ones that have moved most |
| 66 | +recently by hand: the H2 table (1.25 / 1.62 / 1.27 and 1.31 / 1.66 / 1.29), the 2,144-vs-2,153 |
| 67 | +worker count, the decision range (D001–D075), and the two cost figures ($17.63 from the two E2 |
| 68 | +result files; ~$7.50 from D072, the one number in the writeup that cites a decision rather than a |
| 69 | +data file, because no result file carries it). |
| 70 | + |
| 71 | +### 1. The writeup, as a Hugging Face community article — critical path |
| 72 | + |
| 73 | +There is no CLI for this; it is the web editor at https://huggingface.co/new-blog, published |
| 74 | +under `caiotheodoro`. Body is `docs/WRITEUP.md` with two edits: drop the italic *"Draft for the |
| 75 | +Hugging Face community blog"* paragraph at the top (keep the sentence about where the numbers |
| 76 | +come from, reworded as a statement, not a draft note), and turn the bare repository paths |
| 77 | +(`docs/PRE-REGISTRATION.md`, `data/e2_full_n10000.json#H1`, …) into links to |
| 78 | +https://github.com/caiotheodoro/vernier at `main`, since on the Hub they resolve to nothing. The |
| 79 | +H1 title is the article title. Tags: `egocentric`, `dataset-audit`, `llm-as-judge`, |
| 80 | +`evaluation`, `build-ai`. Link the collection |
| 81 | +(https://huggingface.co/collections/caiotheodoro/vernier-same-judge-same-6pp-on-2-hands-both-releases-6a9b1bc9fac182a7a2d9c997) |
| 82 | +in the "Where everything is" list; if the editor offers to attach datasets and models, attach |
| 83 | +`caiotheodoro/vernier`, `caiotheodoro/vernier-rung1-probe`, and the two `builddotai/*-Evaluation` |
| 84 | +datasets so the article surfaces on their pages, which is the whole point of publishing on the |
| 85 | +Hub rather than anywhere else. |
| 86 | + |
| 87 | +After it is live: a `DECISIONS.md` entry (D076 unless the other session has taken it) recording |
| 88 | +the URL and the date; the URL added to `README.md`'s map, `llms.txt`, the dataset card |
| 89 | +(`scripts/export_hf_dataset.py`, then `make hf-dataset` and re-upload), and the collection. |
| 90 | +Nothing in the article is edited after that without a dated note in the same entry — it is a |
| 91 | +published result now, and the project's own standard for Build AI's card applies to it. |
| 92 | + |
| 93 | +### 3. The email — after 1, never before |
| 94 | + |
| 95 | +`docs/private/OUTREACH.md` is the sequence and the reasons; `docs/private/EMAIL-DRAFT.md` is the |
| 96 | +text. Five lines, link first. What a session can do: check every link in the draft resolves, |
| 97 | +check every figure in it against the card, confirm the article is live, and stop. Sending is the |
| 98 | +user's act, from the user's account, and is not automated. Two things the draft must reflect as |
| 99 | +of now: the article URL (it is the first link), and `docs/ETHICS.md` §4 as rewritten in D073 — |
| 100 | +24 downscaled stills are republished in the Space and come down on request; that is the one |
| 101 | +sentence in the email that is a courtesy to them rather than a result. D009's exception (private |
| 102 | +disclosure first, only for a live privacy or security exposure) has not been triggered; nothing |
| 103 | +found in this project is one. Record the send as a one-line `DECISIONS.md` entry with the date |
| 104 | +and nothing else — the content stays private. |
| 105 | + |
| 106 | +### 2. arXiv — beside the other two, gating nothing |
| 107 | + |
| 108 | +cs.CV, single author, no affiliation ("Independent"). An unaffiliated first-time submitter needs |
| 109 | +an endorser; ask with the article already public, since the ask is then "read this" rather than |
| 110 | +"trust me" (`docs/private/OUTREACH.md` says why in more words). Do not name a candidate in any |
| 111 | +public file. Build the submission from the writeup, not from a rewrite: |
| 112 | + |
| 113 | +``` |
| 114 | +pandoc -s docs/WRITEUP.md -o arxiv/vernier.tex # TeX source, which arXiv requires |
| 115 | + # for anything TeX-produced |
| 116 | +``` |
| 117 | + |
| 118 | +Then a title, an abstract (the first two paragraphs of the writeup, tightened), CC BY 4.0, and |
| 119 | +cross-list `stat.AP` if it is offered. The arXiv version cites the article URL and the Git |
| 120 | +commit it was built from. Submit after 1 is live so v1 links to it; submit before 3 only if the |
| 121 | +endorsement arrives first — the email should carry the arXiv link if one exists, and does not |
| 122 | +wait for it. |
| 123 | + |
| 124 | + |
| 125 | +**Prior header — (D071/D072) H2 is measured and closed.** The raw-corpus adapter |
7 | 126 | D065 declined to size turned out to be small: ffmpeg's `subfile` protocol extracts one frame |
8 | 127 | from an h265 mp4 inside a tar over HTTP range requests, so no new dependency was needed, and a |
9 | 128 | tar header is 512 bytes at a computable offset, so `scripts/build_corpus_manifest.py` indexed |
@@ -252,7 +371,7 @@ A live E2 or E5 run now also leaves per-frame records behind, next to its checkp |
252 | 371 | per-file negation. Forward-only -- the committed aggregates predate this and have no jsonl. |
253 | 372 | `docs/DECISIONS.md` D069. |
254 | 373 |
|
255 | | -## The next action |
| 374 | +## The next action, as it stood in Wave 2 (historical; the current one is "Next session: publication" above) |
256 | 375 |
|
257 | 376 | **The Qwen3-VL judge is deployed and fully smoke-tested live end to end, image + logprobs |
258 | 377 | included — real correct output.** `cloud/modal_qwen3vl.py` is live on Modal |
|
0 commit comments