Skip to content

Commit 89dfe19

Browse files
Merge pull request #11 from sofiapetrusenko/phase-3b1-collapse
Phase 3b-1: ruled channel collapse, two confirmed tables, detection on 12 real crops — and no N
2 parents c77fae3 + fe22253 commit 89dfe19

44 files changed

Lines changed: 7629 additions & 76 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,10 @@ runs/
1818
# Session working file: the human's task specification for this session, not project
1919
# content. It describes what to build; the tree records what was built.
2020
kickoff_3b0.md
21+
22+
# Session working file: the human's task specification for Phase 3b-1, on the same
23+
# footing as kickoff_3b0.md above. It describes what to build; the tree records what
24+
# was built. Ignored rather than committed so that a task description can never be
25+
# mistaken for a project decision — the decisions this phase acts on live in
26+
# data/real/AMENDMENT_2026-08-19_*.md, which are ratified and digest-pinned.
27+
kickoff_3b1.md

‎DEBT.md‎

Lines changed: 142 additions & 28 deletions
Large diffs are not rendered by default.

‎NOTES.md‎

Lines changed: 405 additions & 1 deletion
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
QC-first western blot densitometry with a measured evaluation harness.
44

5-
blotquant quantifies protein bands in single-channel gel-doc images and attaches a quality-control verdict, an explicit parameter set, and full provenance to every number it reports. It is for molecular biologists who need densitometry numbers they can defend in review, and for reviewers who need to see what a number was measured from. It is a command-line tool at present, aimed at people willing to run `python -m pipeline run` on an exported image.
5+
blotquant quantifies protein bands in single-channel gel-doc images (and, under a narrow ratified rule, in a colour scan whose three channels carry the same signal — see [Limitations](#limitations)) and attaches a quality-control verdict, an explicit parameter set, and full provenance to every number it reports. It is for molecular biologists who need densitometry numbers they can defend in review, and for reviewers who need to see what a number was measured from. It is a command-line tool at present, aimed at people willing to run `python -m pipeline run` on an exported image.
66

77
- **Status: Phase 2 of 5 merged; Phase 4a in review** — CLI pipeline, QC and normalization are merged. An HTTP API (`POST /analyze`, `GET /results/{id}`) and caller-supplied lane ROIs are built and staged on `phase-4a-api`, not yet merged. No UI, no deploy, no export, and no real-blot validation.
88
- **Band detection F1 0.851**; intensity recovery **7.05% mean / 4.60% median** on clean bands — synthetic dev split only.
@@ -98,7 +98,7 @@ as absent.
9898
| 0 — synthetic gold set + metrics | complete |
9999
| 1 — core pipeline (CLI) + eval loop | complete |
100100
| 2 — QC + normalization + provenance | complete |
101-
| 3 — full evals + real-blot cross-validation | **not started** |
101+
| 3 — full evals + real-blot cross-validation | **in progress** — Gate 2 image set approved and pre-registered; the loader now reads the crops. No ImageJ comparison, no real-blot number yet |
102102
| 4 — API + UI | **4a built, unmerged** (`phase-4a-api`); 4b (UI, deploy) not started |
103103
| 5 — export + polish + deploy | **not started** |
104104

@@ -107,7 +107,7 @@ as absent.
107107
- **No HTTP API on `main`.** `POST /analyze` and `GET /results/{id}` are built and staged on the unmerged `phase-4a-api` branch; nothing described in this section is available from a clone of `main`.
108108
- **No user interface.** There is no `web/` module, no upload page, no ROI editing, no overlay. The tool is CLI-only, so this README has no screenshots.
109109
- **No export.** No CSV, no XLSX, no charts, no generated methods paragraph.
110-
- **No real-blot validation.** No ImageJ cross-comparison has been run. `data/real/` does not exist. **Every number in this README comes from synthetic images.**
110+
- **No real-blot validation.** No ImageJ cross-comparison has been run, and **every number in this README comes from synthetic images.** `data/real/` does now exist — 19 approved crops, a frozen pre-registration and two ratified amendments — and since Phase 3b-1 the pipeline can load 12 of them. The reference-band designations and the blot identities have been confirmed by a human against the figure captions and the images. What does not exist is the comparison: **no ImageJ measurement has been made, so no agreement statistic — no correlation, no Bland–Altman — has been computed from a real blot, and none will be until the ImageJ side exists.**
111111
- **No published accuracy claim.** The figures above are dev-split figures from one generator; they are not a general accuracy claim and are not a claim about real blots.
112112
- **No eval history log.** `evals/history.md`, which [CLAUDE.md](CLAUDE.md) and [PLAN.md](PLAN.md) both reference as the per-iteration record and the place a generator break marker would go, does not exist yet. It arrives with Phase 3.
113113

@@ -158,7 +158,7 @@ This is the anti-circularity discipline costing something measurable, and both a
158158

159159
Honest ones, not a formality. [DEBT.md](DEBT.md) is the full register — every known weakness with its evidence, a phase or trigger that should close it, and its status; the list below draws from it — the part a user needs before trusting a number.
160160

161-
- **Single-channel grayscale chemiluminescence only.** 8/16-bit TIFF, PNG, JPEG. Multichannel fluorescence, dot blots and 2D gels are out of scope. Unsupported bit depths raise rather than being squashed to 8-bit.
161+
- **Single-channel grayscale chemiluminescence only.** 8/16-bit TIFF, PNG, JPEG. Multichannel fluorescence, dot blots and 2D gels are out of scope. Unsupported bit depths raise rather than being squashed to 8-bit. **One narrow exception, and it is a ruling rather than a feature:** a 3-channel *8-bit* image whose maximum per-pixel channel divergence is **at most 2 DN** — that is, a colour file carrying the same greyscale signal in all three planes up to codec noise — is collapsed to its green channel on the way in, and the operation is recorded in `source.channel_collapse` with the divergence that was measured. Above 2 DN it is refused, and the refusal names the divergence it measured, because at that point which plane you pick can change the number reported. The bound and the method are fixed by [`data/real/AMENDMENT_2026-08-19_channel_collapse.md`](data/real/AMENDMENT_2026-08-19_channel_collapse.md) and are not configurable: the bound was named before any image was measured against it, and it is not widened to admit an image. Everything else multi-channel — RGBA, 16-bit colour, anything above the bound — still raises. This does **not** put fluorescence in scope; it lets a greyscale blot that was saved as a colour file be measured as what it is.
162162
- **Chemiluminescence is not linear in protein amount** over an arbitrary range. blotquant measures the image it is given; it cannot correct for an exposure taken outside the detector's linear range, only warn that the dynamic range looks wrong.
163163
- **Unresolved doublets report one band.** Where two bands sit too close to resolve, one ROI spans both and over-reads. The tool warns via `unresolved_shoulder` rather than inventing a second band.
164164
- **The geometric `overlapping` flag scores 0.000 by construction, and what little it does fire on is mostly spurious.** Because unresolved doublets yield one ROI, there is usually nothing to overlap: at the shipped threshold of 0.05, only 3 detected same-lane band pairs across the whole dev split overlap enough to flag, putting the flag on 6 detected bands — against 104 truth `overlapping` labels, none of which it matches. Reconciling that 6 with the QC table's single false positive: **only 1 of the 6 is a matched detection; the other 5 are detections that match no truth band at all.** So the flag fires predominantly on detection false positives — two spurious peaks in the same lane overlapping each other — rather than on real bands. That is worth knowing: as it stands the flag is closer to a weak detector-artifact signal than a band-overlap warning. (The generator labels overlap at a different threshold, 0.15, where the pipeline finds 2 such pairs; the shipped 0.05 is chosen from an independent criterion, not copied.) **Phase 3 Gate 1 ruled that the flag is kept as a recorded negative result, not as functionality** — retiring it would delete the evidence that a plausible-sounding QC flag can fail completely. Do not read it as a working band-overlap warning; `unresolved_shoulder` is the flag that answers that question.

‎api/app.py‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -67,8 +67,11 @@
6767
6868
Processing is synchronous: the pipeline runs inside this request and the finished result
6969
comes back in the response. The `result_id` is content-addressed over the image bytes, the
70-
config digest, the reference band ids and the supplied lane ROIs, so re-posting the
71-
identical request returns the same id.
70+
config digest, the reference band ids, the supplied lane ROIs and the reference-designation
71+
source, so re-posting the identical request returns the same id — and changing `lane_roi` or
72+
`reference_band_id` changes it, because both are inputs to it. The designation source is the
73+
one input this API does not expose: it is always absent here, so it is constant across every
74+
request and never the reason two ids differ.
7275
7376
**What is reproducible under that id, and what is not.** Everything the analysis measured:
7477
every ROI, every intensity, every ratio, every QC flag, and `source.sha256`. Two fields are
@@ -113,8 +116,9 @@
113116
114117
**Which analysis you get back.** The stored entry is the most recent analysis written under
115118
that id, not necessarily the first. `result_id` is content-addressed over the image bytes,
116-
the config digest, the reference band ids and the supplied lane ROIs -- but not over
117-
`source.path`, which names the per-request temporary file the upload was read from. So two
119+
the config digest, the reference band ids, the supplied lane ROIs and the
120+
reference-designation source -- but not over `source.path`, which names the per-request
121+
temporary file the upload was read from. So two
118122
posts of identical bytes under different filenames share an id, produce documents that
119123
differ in `source.path` and `provenance.created_at`, and the later one replaces the earlier.
120124
Everything the analysis measured is identical across them; see `POST /analyze` for the full
@@ -312,7 +316,17 @@ def pipeline_error_handler(request: Request, exc: Exception) -> JSONResponse:
312316

313317
@app.post("/analyze", description=ANALYZE_DESCRIPTION, summary="Analyse one image")
314318
def analyze(
315-
image: Annotated[UploadFile, File(description="8/16-bit grayscale TIFF, PNG or JPEG")],
319+
image: Annotated[
320+
UploadFile,
321+
File(
322+
description=(
323+
"8/16-bit grayscale TIFF, PNG or JPEG. An 8-bit 3-channel image whose "
324+
"channels diverge by at most 2 DN is collapsed to its green channel and "
325+
"the collapse is recorded in the result's source block; anything else "
326+
"multi-channel is refused with 415"
327+
)
328+
),
329+
],
316330
config: Annotated[
317331
str, Form(description="name of a parameter set in configs/, e.g. 'default'")
318332
],

‎api/display.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,8 @@ def render_display(pixels: np.ndarray, max_value: int) -> DisplayDerivative:
145145
if pixels.ndim != 2:
146146
raise DisplayError(
147147
f"the display derivative needs a 2D single-channel image, got shape "
148-
f"{pixels.shape}; the pipeline only measures single-channel images, so this is "
148+
f"{pixels.shape}; the loader delivers 2D pixels for every image it accepts -- a "
149+
f"multi-channel input is either collapsed to one plane or refused -- so this is "
149150
f"an internal inconsistency rather than an input problem"
150151
)
151152
if pixels.size == 0:

‎api/errors.py‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -87,10 +87,15 @@ class ResultSchemaError(PipelineError):
8787
# NormalizationError, which is a 500 below, because status_for takes the first match.
8888
(ReferenceBandError, 400),
8989
# The upload is an image this service cannot quantify: a container outside TIFF/PNG/JPEG,
90-
# a pixel type outside uint8/uint16, a multi-channel image, or a payload that does not
91-
# decode. 415 is exactly "the entity's media type is unsupported", and the verbatim
92-
# message distinguishes the four cases. The pipeline refuses to squash or rescale any of
93-
# them, so there is no representation of these inputs that a 200 could have carried.
90+
# a pixel type outside uint8/uint16, a multi-channel image *the ruled channel collapse does
91+
# not admit*, or a payload that does not decode. The qualifier is load-bearing since the
92+
# 2026-08-19 §7 amendment: an 8-bit 3-channel image at or below the ruled divergence bound
93+
# is collapsed to one plane and quantified, so multi-channel is no longer unquantifiable per
94+
# se. 415 is exactly "the entity's media type is unsupported", and the verbatim message
95+
# distinguishes the cases -- including naming the divergence it measured, so a caller can
96+
# tell a colour image from one that missed the bound by a digit. The pipeline refuses to
97+
# squash or rescale any of them, so there is no representation of these inputs that a 200
98+
# could have carried.
9499
(UnsupportedFormatError, 415),
95100
(UnsupportedBitDepthError, 415),
96101
(UnsupportedImageError, 415),

‎data/real/blot_identity.csv‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# blotquant real-blot identities. One row per measurable crop.
2+
#
3+
# RULED FROM THE IMAGES AT THE HUMAN GATE, 2026-08-20 (ruling G2). Each blot_id was
4+
# PROPOSED from the parent figure stem and the panel token and from nothing else, and a
5+
# panel letter is not evidence of physical identity -- TWO PANELS OF ONE FIGURE MAY BE TWO
6+
# DIFFERENT PHYSICAL BLOTS, and one physical blot may be split across two panels. The
7+
# proposals did not decide that; the human did, by looking at the images.
8+
#
9+
# What the images decided, where it mattered: PMC13135410_Figure4 panels A/B/C are THREE
10+
# SEPARATE PHYSICAL BLOTS -- three distinct experiments with distinct animal cohorts
11+
# (4-day vs 2-week denervation; drug vs genotype), 12 lanes per panel each with its own MW
12+
# annotation, and categorically different background texture and exposure per panel with
13+
# no continuity across panel boundaries. None of that is visible in a filename. All other
14+
# ids were ruled distinct.
15+
#
16+
# identity_source 'image_confirmed_human' means a human ruled from the images.
17+
# 'proposed_pending_human' means a filename proposed it and nobody has looked. Tools that
18+
# read this file refuse every value they do not know and never read one as a confirmation.
19+
#
20+
# NOT every blot here contributes a ratio: PMC13135388_Figure4__E-Vinculin is the §9
21+
# separate reference strip for the E-TIGAR target, which §7 refuses at 255 DN. See
22+
# designations.csv. The ratio-contributing blot count is therefore one lower than the
23+
# number of rows in this file.
24+
#
25+
# Why this is a sibling file and not a column in crop_log.csv: §9 of
26+
# DECISION_unit_of_analysis.md provides for the column in the crop record --
27+
# "if the reference is a separate strip (e.g. loading-control row below), it is cropped as part of the same rectangle whenever they are vertically contiguous in the figure, else as a second rectangle recorded against the same blot_id"
28+
# -- but crop_log.csv is the frozen Gate 2 record and is byte-identical to what the gate
29+
# approved. That conflict is reported for the human to rule on, not resolved by editing
30+
# the approved artefact. See DEBT draft D5.
31+
#
32+
# Lines beginning with # are this note. The CSV header is the first line that is not one.
33+
crop_filename,parent_figure,panel,blot_id,identity_source,notes
34+
PMC12686555_FIGURE1__A-p16p21-ACTIN.png,PMC12686555_FIGURE1,A,PMC12686555_FIGURE1_A,image_confirmed_human,"ruled distinct from the images (G2, 2026-08-20)"
35+
PMC12686555_FIGURE2__C-HSPA12B-ACTIN.png,PMC12686555_FIGURE2,C,PMC12686555_FIGURE2_C,image_confirmed_human,"ruled distinct from the images (G2, 2026-08-20)"
36+
PMC12708318_Figure5__A1-GSDME-actin.png,PMC12708318_Figure5,A1,PMC12708318_Figure5_A1,image_confirmed_human,"ruled distinct from the images (G2, 2026-08-20)"
37+
PMC12895598_Fig3__A-EMT-GAPDH.png,PMC12895598_Fig3,A,PMC12895598_Fig3_A,image_confirmed_human,"ruled distinct from the images (G2, 2026-08-20)"
38+
PMC12956003_Figure2__A-Htt-tub.png,PMC12956003_Figure2,A,PMC12956003_Figure2_A,image_confirmed_human,"ruled distinct from the images (G2, 2026-08-20)"
39+
PMC13017922_Figure8__D-ABCB1-Actin.png,PMC13017922_Figure8,D,PMC13017922_Figure8_D,image_confirmed_human,"ruled distinct from the images (G2, 2026-08-20)"
40+
PMC13025488_Figure5__C-pSMAD-GAPDH.png,PMC13025488_Figure5,C,PMC13025488_Figure5_C,image_confirmed_human,"ruled distinct from the images (G2, 2026-08-20)"
41+
PMC13135410_Figure3__B-Fib-CCN2-GAPDH.png,PMC13135410_Figure3,B,PMC13135410_Figure3_B,image_confirmed_human,"ruled distinct from the images (G2, 2026-08-20)"
42+
PMC13135410_Figure4__A-PDGFRa-GAPDH.png,PMC13135410_Figure4,A,PMC13135410_Figure4_A,image_confirmed_human,"ruled from the images (G2, 2026-08-20): panels A/B/C of this figure are THREE SEPARATE PHYSICAL BLOTS -- three distinct experiments with distinct animal cohorts (4-day vs 2-week denervation; drug vs genotype), 12 lanes per panel each with its own MW annotation, and categorically different background texture and exposure per panel with no continuity across panel boundaries"
43+
PMC13135410_Figure4__B-PDGFRa-GAPDH.png,PMC13135410_Figure4,B,PMC13135410_Figure4_B,image_confirmed_human,"ruled from the images (G2, 2026-08-20): panels A/B/C of this figure are THREE SEPARATE PHYSICAL BLOTS -- three distinct experiments with distinct animal cohorts (4-day vs 2-week denervation; drug vs genotype), 12 lanes per panel each with its own MW annotation, and categorically different background texture and exposure per panel with no continuity across panel boundaries"
44+
PMC13135410_Figure4__C-PDGFRa-GAPDH.png,PMC13135410_Figure4,C,PMC13135410_Figure4_C,image_confirmed_human,"ruled from the images (G2, 2026-08-20): panels A/B/C of this figure are THREE SEPARATE PHYSICAL BLOTS -- three distinct experiments with distinct animal cohorts (4-day vs 2-week denervation; drug vs genotype), 12 lanes per panel each with its own MW annotation, and categorically different background texture and exposure per panel with no continuity across panel boundaries"
45+
PMC13135388_Figure4__E-Vinculin.png,PMC13135388_Figure4,E,PMC13135388_Figure4_E,image_confirmed_human,"ruled from the images (G2, 2026-08-20). This is the §9 case the missing blot_id column was meant to record: a reference cropped as a second rectangle 'recorded against the same blot_id' as its target. The target is PMC13135388_Figure4__E-TIGAR.png, same figure and same panel E, which §7 refuses at 255 DN -- confirmed at the gate as publisher annotation (red arrowheads), not blot content. So this blot_id is a real and distinct blot, and it CONTRIBUTES NO RATIO because the target it references does not load. See designations.csv"

0 commit comments

Comments
 (0)