Skip to content

Commit dbe1ee8

Browse files
Merge pull request #8 from sofiapetrusenko/phase-3-gate2-artefacts
Phase 3 Gate 2: real-blot artefacts and acquisition tools
2 parents f27775e + 930bf4c commit dbe1ee8

49 files changed

Lines changed: 1827 additions & 1 deletion

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎DEBT.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,14 @@ at commit `52200a5` (Phases 0–2 merged, plus the README and this register). **
2828
at each phase gate**: confirm the evidence still reproduces, close what the phase closed, and add
2929
what it introduced. A register without a snapshot goes stale invisibly.
3030

31+
**Phase 3 Gate 2 artefacts landed on 2026-08-18** (branch `phase-3-gate2-artefacts`): `data/real/`
32+
now holds the frozen pre-registration, 19 crops, 13 of 21 source figures, and the provenance and
33+
manifest tables. **No entry below has been re-checked against them, and no real blot has been
34+
measured**, so S1 — the generator has never been validated against a real blot — is unchanged: the
35+
data to close it now exists in the tree, and the measurement that would close it does not. S6, S15
36+
and E6 likewise still stand. The one thing that did change mechanically: `tools/check_claims.py`
37+
verifies every crop and committed parent against its recorded digest on each CI run.
38+
3139
**Phase 4a is in progress** on branch `phase-4a-api`, under the human ruling of 2026-08-17 that
3240
runs Phase 4 before the remainder of Phase 3 (P2 entry 7). **Four entries have been updated for it,
3341
and only four:** P2 gains entries (7) and (8); **P1 is widened from numeric claims to claims

‎NOTES.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2911,6 +2911,76 @@ property carefully: it is *no item requiring a code change*, not *nothing behavi
29112911
saw something behavioural and reported it as a claim defect, which is the hedge the paragraph above
29122912
sets out.
29132913

2914+
## Phase 3 — Gate 2 artefacts
2915+
2916+
Gate 2 approved the real-image list on 2026-08-18. This section records where the artefacts
2917+
landed and what was decided about them; **no real blot has been measured yet**, so nothing here
2918+
changes a figure.
2919+
2920+
### The pre-registration is imported byte-identical, not restated
2921+
2922+
`data/real/DECISION_unit_of_analysis.md` — 11 sections plus one amendment — was written outside
2923+
this repository and frozen before any measurement, and its digest is cited:
2924+
2925+
```
2926+
sha256 = 994acc30daf82011acfe957adff2866f824b05aa3b39bade118e539f4cfd88d0
2927+
```
2928+
2929+
**So it is committed unchanged rather than folded into this file.** NOTES.md records decisions in
2930+
its own voice; rewriting a pre-registration into that voice would destroy the single property
2931+
that makes it one, and any edit — including tidying — invalidates a hash that is quoted in three
2932+
places. This file points at it and deliberately does not paraphrase it. `tools/check_claims.py`
2933+
pins the digest, so an edit fails CI rather than being noticed later.
2934+
2935+
One consequence worth surfacing here, because a reader of §11 alone would get it wrong: §11a
2936+
re-cropped `PMC13135388_Figure4__E-Vinculin` after §11 was written, so §11's stated
2937+
colour-fraction maximum of 0.0158 describes a **superseded** crop. The current set maximum is
2938+
0.0068, and `crop_log.csv` holds the current crop's digest.
2939+
2940+
### A source figure is committed if and only if a measured artefact derives from it
2941+
2942+
19 crops are committed without argument: they are the measured artefacts, and §9 requires ImageJ
2943+
to run on the same crop file byte-identical, so a comparison whose crops are not fixed in the
2944+
tree is not reproducible at all.
2945+
2946+
Of the 21 source figures, **13 are committed and 8 are not** — the 13 being exactly those a crop
2947+
derives from. The rule is a column (`committed`) in `data/real/sources.csv` rather than a
2948+
convention someone has to remember.
2949+
2950+
**The brief's premise was that sources are "recoverable from the recorded URLs and sha256s", and
2951+
as recorded they were not.** Checking it before relying on it: `provenance.md` carries
2952+
**truncated** 16-hex digests and **no URL at all**, and the shortlist's `graphic_href` is a bare
2953+
filename, not a link. PMC serves figures from a CDN path containing an opaque per-blob hash that
2954+
cannot be derived from anything recorded, so a URL has to be re-read from the live article page
2955+
every time. Recovery therefore depended on data that existed nowhere in the tree.
2956+
2957+
That is now fixed in the direction that makes the premise true: `sources.csv` carries the **full**
2958+
64-hex digest, the DOI, the licence, the article URL and the CDN graphic name for all 21, and
2959+
`tools/gate2/refetch_sources.py` performs the recovery — it re-resolves each figure's CDN URL
2960+
from its article page, downloads it, and verifies against the recorded digest, failing loudly if
2961+
upstream has been re-rendered. Runnable, not described.
2962+
2963+
**Committing the 13 anyway is the deliberate belt-and-braces**, on the same argument that commits
2964+
`data/images/`: recovery depends on someone else's website continuing to serve byte-identical
2965+
bytes, and a measured claim has to stay auditable when it does not. 2.5 MB against that is cheap.
2966+
2967+
**Recovery was subsequently verified live: 21 of 21** — see the closed open item below. The first
2968+
run failed on all 8 for a TLS-interception reason unrelated to PMC, which is why
2969+
`refetch_sources.py` now builds its TLS context from `certifi`.
2970+
2971+
### The digest chain is now checked mechanically, because it already caught two real failures
2972+
2973+
Cropping was done by hand in Preview, whose Cmd+S writes back to the *open* file rather than
2974+
exporting, silently re-encoding the JPEG. **Twice during Gate 2 a parent figure's digest stopped
2975+
matching what `fetch_v2.py` recorded at download**, which is the only reason either overwrite was
2976+
noticed: the pixels look identical and nothing else would have flagged it.
2977+
2978+
`tools/check_claims.py` now re-verifies both halves of that chain on every CI run — every crop
2979+
against `crop_sha256`, and every committed parent against `parent_sha256` — so a re-saved
2980+
artefact fails the build rather than being measured. Each failure mode was falsified before being
2981+
trusted: a re-saved crop, a missing crop, a re-encoded parent and an edited pre-registration all
2982+
produce the expected failure.
2983+
29142984
## Open items
29152985

29162986
Unresolved questions carried out of a phase. Not decisions — each one names the phase
@@ -2922,6 +2992,47 @@ meaning of the numbers, the usability of the tool, or the trustworthiness of the
29222992
carries the evidence for each. This section stays the per-phase narrative; DEBT.md is the
29232993
index.
29242994

2995+
### ~~Recovery of the 8 uncommitted Gate 2 sources is untested end to end~~ — CLOSED, 21 of 21 verified live
2996+
2997+
**Closed 2026-08-18. The recovery works, and the first run found a defect in the tool rather than
2998+
in the record.**
2999+
3000+
The first networked run failed on all 8 with `CERTIFICATE_VERIFY_FAILED: self signed certificate
3001+
in certificate chain` on every article URL. That is **TLS interception on the operator's network**
3002+
— a middlebox re-signing traffic with a CA the Python default store does not carry — and not a PMC
3003+
failure: the same URLs return 200 when the context is built from certifi's bundle. The tool now
3004+
builds its `SSLContext` from certifi when importable and falls back to the system store with a
3005+
logged note. **Verification is not disabled and will not be**: every fetched byte is checked
3006+
against a recorded sha256, and hashing bytes from a connection nobody authenticated would prove
3007+
only that they match what *someone* served, which is the question the digest exists to answer.
3008+
3009+
With that fixed, the live result:
3010+
3011+
- **8 of 8 uncommitted sources recovered**, each matching its recorded sha256.
3012+
- **13 of 13 committed sources re-verified against upstream**, and each byte-identical to the
3013+
copy in this tree.
3014+
3015+
So all 21 are recoverable today, and the 13 committed parents are confirmed to be what PMC still
3016+
serves. **This does not retire the argument for committing them** — it confirms the premise held
3017+
on one day, against one set of publishers, and the reason for committing was always that the
3018+
premise can stop holding without notice.
3019+
3020+
**The run also exposed a defect in the tool, which mattered more than the failure.** It reported
3021+
`0 of 8 verified` and the exit status was read as 0. The status was in fact 1 — the 0 came from
3022+
`$?` after a shell pipe, which reports the last command in the pipeline, not the tool. But the
3023+
episode is why the exit contract is now pinned by `tests/test_refetch_sources.py` rather than
3024+
assumed: the count printed and the status returned are derived from one counter, so they cannot
3025+
disagree, and four tests fail if that is reverted. One genuine defect of the same family was found
3026+
and fixed while looking: a source whose upstream matched the record but whose *committed copy*
3027+
differed was printed as `ok <name>: … COMMITTED COPY DIFFERS` and counted toward the total. A line
3028+
beginning `ok` is what a reader skims past — the same defect as a success exit code, in smaller
3029+
type. It is now a `FAIL` line and excluded from the count.
3030+
3031+
**Two archived scripts have the same shape and were deliberately not fixed**, since editing them
3032+
would make them differ from what produced `data/real/`: `fetch_v2.py` exits 0 after reporting
3033+
failed downloads, and `log_crops.py` exits 0 after skipping a crop whose parent it could not
3034+
match. Both are recorded in `tools/gate2/README.md` with the controls that now cover them.
3035+
29253036
### Whether band ROIs become caller-supplyable — Phase 4b to settle
29263037

29273038
Phase 4a made **lane** rectangles supplyable and left band ROIs detector-only, so the band object
Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# Phase 3 — unit of analysis, minimum N, QC-excluded ratios
2+
3+
Decided: 2026-08-18, before any real image was measured.
4+
Decided by: Sofia. Status: pre-registered; frozen once the first image is
5+
downloaded. Amendments after that point require a dated entry below stating
6+
what changed and why, and are themselves part of the record.
7+
8+
## 1. Unit of analysis
9+
10+
DECISION: ratio (one normalized ratio = one point in the ImageJ comparison).
11+
12+
Rationale: blot-level N (8–10 reachable blots) is below Spearman's
13+
discriminating power — the Gate 1 deferred question "can Spearman discriminate
14+
at expected N" answers itself as "no" at that N. Ratio-level N≈30 is reachable
15+
within the time budget and is testable against the pre-registered thresholds
16+
(>=0.9 / 0.7–0.9 / <0.7).
17+
18+
## 2. Target N, source, and ratio definition
19+
20+
DECISION: 30 ratios from 8–10 blots with >=3 usable lanes each.
21+
22+
Ratio definition (fixed now, before any measurement):
23+
- Per blot, the reference lane/band is designated at image-selection time,
24+
from the figure caption alone, before any measurement is run.
25+
- Each remaining usable lane contributes exactly one ratio to that reference:
26+
a blot with L usable lanes yields L−1 ratios. No other pairings enter N.
27+
- Normalization mode per blot is likewise fixed at selection time from the
28+
caption: a stated loading-control band -> housekeeping; stated stain-based
29+
loading (Ponceau, total protein stain) -> total_protein.
30+
- A blot whose caption states neither is REJECTED at selection, not decided
31+
at measurement time. Guessing the loading control from the data is
32+
forbidden (consistent with Phase 2 Ruling 2).
33+
34+
Arithmetic check: 8–10 blots × (L−1) >= 2 ratios each gives 16–30+; the
35+
stopping rule below caps collection, not the definition.
36+
37+
## 3. Non-independence of ratios within a blot
38+
39+
DECISION: disclosed, not corrected. Ratios within one blot share a background
40+
estimate and an exposure; effective N is below nominal N; any reported
41+
confidence interval is optimistic. This becomes a DEBT entry when merged into
42+
the repo. The reported r itself is unaffected; only its uncertainty is
43+
understated, and that is stated wherever r is stated.
44+
45+
Agreed: Sofia, 2026-08-18.
46+
47+
## 4. QC-excluded ratios in the ImageJ comparison
48+
49+
DECISION: excluded from the correlation; reported as a separate count
50+
("n excluded by QC", with the flags that excluded them).
51+
52+
Rationale: the correlation measures agreement on numbers the tool certifies.
53+
Including numbers the tool itself refuses to certify would mix two populations
54+
and answer neither question. A secondary correlation on flagged ratios was
55+
considered and rejected: at the expected flagged n (likely <10) it would be
56+
uninterpretable noise, and reporting an uninterpretable number invites
57+
misreading it.
58+
59+
Failure mode I accept: QC may leave fewer than 15 surviving ratios. I will
60+
report that count as a finding in its own right ("QC excluded X of Y ratios
61+
from N real blots") and will not relax any QC threshold to save N. QC
62+
thresholds are frozen as shipped (Phase 3 Gate 1); nothing here reopens them.
63+
64+
## 5. Stopping rule
65+
66+
DECISION: collection stops at 30 ratios or 10 blots, whichever comes first.
67+
68+
If fewer than 15 ratios survive QC, the ImageJ correlation is reported with
69+
the explicit caveat that N is below the pre-registered minimum and the
70+
agreement claim is downgraded to descriptive (no threshold verdict). The
71+
survival count itself is reported as a primary finding. Criteria are not
72+
relaxed to reach 15.
73+
74+
Agreed: Sofia, 2026-08-18.
75+
## 6. Source image format (recorded before download, 2026-08-18)
76+
77+
Every candidate figure in the Gate 2 shortlist is distributed as JPEG. The
78+
`lossy_format` QC flag is therefore expected to fire on the entire real-blot
79+
set. This is reported as a finding about the OA literature — published figures
80+
do not preserve the acquisition format the measurement should be made on — and
81+
not treated as grounds to suppress or exempt the flag.
82+
83+
## 7. Pseudocoloured blots (recorded before download, 2026-08-18)
84+
85+
DECISION: a blot panel whose pixels are not single-channel is REJECTED at
86+
selection; it is not converted to greyscale.
87+
88+
Scope: the rule applies to the CROPPED blot panel, not to the source figure.
89+
Published figures are montages — coloured bar charts and MW ladders share a
90+
file with greyscale blot panels — so a whole-figure colour test would reject
91+
nearly every candidate while saying nothing about the measured region. The
92+
download step records a whole-figure colour fraction as a screening aid only.
93+
94+
Rationale: choosing an RGB→grey conversion would introduce a measurement
95+
parameter selected without data to select it. Phase 3's only reference is the
96+
ImageJ comparison, and ImageJ applies its own conversion; part of any observed
97+
disagreement would then be attributable to the conversion rather than to the
98+
method, with no third reference available to separate the two contributions.
99+
100+
Failure mode I accept: fluorescence-detection blots are common, so this may
101+
remove a substantial share of candidates and push N down. That count is
102+
reported ("X of Y candidate panels rejected as pseudocoloured") rather than
103+
recovered by converting.
104+
105+
Carried to backlog, not Phase 3: blotquant should detect non-single-channel
106+
input and BLOCK with an explanatory message directing the user to the
107+
scanner's single-channel export, consistent with the three-state QC model
108+
(pass | flagged | blocked). Silently converting and returning a number is the
109+
failure mode this project exists to refuse. -> DEBT entry.
110+
111+
## 8. Gate 2 image review — ratified 2026-08-18, after download, before any crop
112+
113+
(a) FINDING: every total-protein stain panel in the downloaded set (Ponceau,
114+
2 blots) is reproduced in colour. Under §7 these reference panels are not
115+
single-channel and are excluded; no per-channel extraction is applied. The
116+
real-blot set is therefore HOUSEKEEPING-ONLY, and the README will state that
117+
total_protein agreement was not measurable on published figures for this
118+
reason. Feeds the non_grayscale_source blocking-flag DEBT entry.
119+
120+
(b) Removed at image review: PMC13135388 Figure 1 (reference panel
121+
pseudocoloured), PMC13019689 Figures 1 and 3 (Ponceau reference in colour),
122+
PMC13138064 Figure 4 (2 lanes, below the >=3 minimum — visible only in the
123+
image, not the caption). Retained with restriction: PMC13135388 Figure 4,
124+
panels TIGAR + vinculin only (adjacent Parkin panel is pseudocoloured; §7
125+
applies to the cropped panel). Retained: PMC12956003 Figure 2 (Pon S panel is
126+
coloured but is not the designated reference; α-tubulin is, and is greyscale).
127+
128+
(c) Borderline-resolution rule: no figure is excluded by eye for band size.
129+
All remaining figures are cropped; the crop's measured pixel height decides
130+
against the pre-registered >=15 px band-height minimum. Selection by
131+
measurement, not by impression.
132+
133+
## 9. Crop rule — fixed before the first crop, 2026-08-18
134+
135+
- One crop = one blot panel: the rectangle bounds the panel's lanes, full lane
136+
height, plus >=10 px background margin on every side where the montage
137+
allows it.
138+
- The crop must include the panel's designated reference band row if the
139+
reference shares the panel; if the reference is a separate strip (e.g.
140+
loading-control row below), it is cropped as part of the same rectangle
141+
whenever they are vertically contiguous in the figure, else as a second
142+
rectangle recorded against the same blot_id.
143+
- Annotation overlays (arrows, kDa markers, labels) may remain if they sit
144+
outside lane boundaries; a crop where annotation touches a measured lane is
145+
rejected.
146+
- No rotation, no rescaling, no level/contrast adjustment, no resampling.
147+
Export as PNG (lossless container; the source pixels are already JPEG and
148+
are not re-encoded into further loss).
149+
- Every crop records: parent file, parent sha256, crop rectangle is NOT
150+
recorded manually — the crop file's own sha256 and pixel size are recorded
151+
at creation by the logging step, and the parent linkage makes the crop
152+
auditable against its source.
153+
- ImageJ runs on the SAME crop file, byte-identical, never on its own crop.
154+
155+
## 10. Gate 2 CLOSED — 2026-08-18
156+
157+
Real-image list approved by Sofia. The approved set is exactly the rows of
158+
crops/crop_log.csv at this timestamp; provenance (DOI, licence, sha256 of crop
159+
and parent) recorded per image in images/provenance.md and crops/crop_log.csv.
160+
Housekeeping-only per §8(a). Borderline crops enter measurement; the >=15 px
161+
band-height minimum and lane-width reality decide there, per §8(c).
162+
163+
## 11. §7 verified mechanically on crops — 2026-08-18
164+
165+
Colour fraction measured on all 19 crops after Gate 2 closure: 13 crops at
166+
0.0000, 5 between 0.0034 and 0.0068 (labels and kDa markers, not bands),
167+
maximum 0.0158 (PMC13135388_Figure4__E-Vinculin, adjacent to the
168+
pseudocoloured Parkin panel). All below the 0.02 screening threshold. The §7
169+
single-channel rule is therefore confirmed by measurement, not by eye.
170+
171+
### 11a. Amendment — 2026-08-18, after §11
172+
173+
PMC13135388_Figure4__E-Vinculin was re-cropped: the original rectangle
174+
included the lower edge of the adjacent pseudocoloured Parkin panel
175+
(colour fraction 0.0158). The band region was unaffected, but that edge sits
176+
in the background region the pipeline uses for background estimation. The
177+
re-crop excludes it; measured colour fraction is now 0.0000 and the crop
178+
sha256 in crop_log.csv is updated accordingly. §11's stated maximum of 0.0158
179+
refers to the superseded crop; the current set maximum is 0.0068.

0 commit comments

Comments
 (0)