Skip to content

feat: add kronos_exclude_markers, and fix split-series channel reads - #42

Merged
mikemcka merged 1 commit into
WEHI-SODA-Hub:devfrom
mikemcka:feat/kronos-exclude-markers
Aug 6, 2026
Merged

mikemcka merged 1 commit into
WEHI-SODA-Hub:devfrom
mikemcka:feat/kronos-exclude-markers

Conversation

@mikemcka

@mikemcka mikemcka commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two changes, one of which is a silent correctness bug worth reading first.

1. kronos_exclude_markers — withhold non-marker channels from KRONOS2

A COMET panel's Autofluorescence channel is not a marker and is not in KRONOS2's
288-marker vocabulary, so the run fails:

ERROR: 1 of 8 markers are outside KRONOS2's vocabulary and will use DEFAULT
normalisation stats: ['Autofluorescence']

There was no way to say "don't embed this channel". --marker-mapping only renames,
and --allow-novel-defaults still feeds the channel to the model, normalised with
default statistics — the opposite of what is wanted for a channel carrying no biology.

--kronos_exclude_markers 'Autofluorescence'

Scoped to the embedding step; nothing is deleted. A withheld channel is simply
never read, so it stays in the image, remains available to segmentation and every
other process, and keeps whatever intensity measurements CELLMEASUREMENT wrote for it
in the GeoJSON. This is deliberately unlike remove_markers, which does strip
channels out of the background-subtracted image — the parameter shape is borrowed from
it, the blast radius is not, and the log line, marker report and docs all say "not
shown to KRONOS2" rather than "removed" so the distinction is not mistakable.

Names are comma separated and case sensitive, matched against the image's own channel
names before any mapping. A name matching nothing fails the run and lists the
channels that are present; the nuclear channel cannot be withheld because it is the
model's preferred_dapi.

2. ⚠️ KRONOS2 embedded one channel under every marker's name

Found while making channel indices load-bearing for the above.

On OME-TIFFs that store each channel as its own 2D series, ChannelReader sized itself
from series[0] alone, so an 8-series image read as 1 channel — while
CellPatchDataset still sized the patch buffer from the 8-name marker list. NumPy then
broadcast that single channel across all eight slots.

The run completed, logged Found 8 channels, and produced 768-d vectors computed from
eight copies of DAPI. No error, no warning. The tell in the log is Found 8 channels
on one line and image 1x27454x26980 a few lines later.

Confirmed on the COMET _fixed images (inForm unmixing export): every one of the 8
"markers" came back byte-identical to DAPI. Embeddings produced from such images by
earlier versions are invalid and need regenerating.

Now: the layout is detected once by channel_layout() and shared between the name list
and the pixel reader; each channel is read from its own series via its own zarr store;
read_window refuses to return fewer channels than it was asked for, so this class of
bug cannot recur silently; and the run fails outright if the channel-name count and the
image's channel count disagree.

Also fixed in passing: a channel whose OME Channel element carried neither Name nor
ID was dropped from the name list but not from the image, sliding every later marker
onto the wrong channel. Names are now built positionally, one per channel always.

Marker report

Gains the embedded/not-shown split and names withheld channels in place. It is now
written before the novel-marker gate can exit — a failing run is exactly when you
need it, since it lists the channel names to map or withhold.

Channels: 8 (7 embedded, 1 not shown)

Channel -> name given to KRONOS2
  DAPI -> DAPI
  TCR -> TCR_B  (mapped)
  ...
  Autofluorescence -> (not shown to KRONOS2)

Withheld from KRONOS2 via --exclude-marker (still present in the image):
  Autofluorescence

Testing

tests/python/test_kronos2_common.py is the first module under the tests/python/
scaffold, which had a conftest.py and no tests since the cellmeasurement extraction.
22 tests covering channel naming, selection, the reader and the report. Pure python —
no torch, no GPU, no gated weights — so a pytest job has been added to the linting
workflow and it actually runs in CI.

The split-layout test is a genuine regression guard: against the pre-fix code its
fixture returns eight identical channels, against the fix it returns eight distinct
ones.

Also adds a stub nf-test for KRONOS2EMBEDDINGS, which had none — the KRONOS1 test
was deleted in the v2 migration and never replaced.

Verified locally beyond the unit tests:

  • Reader and channel selection against the real 8-channel COMET image — patches are now
    (7, 64, 64) with per-channel values matching the raw file exactly.
  • All four failure paths exit 1 with actionable messages (unknown name, wrong case,
    nuclear channel, everything withheld).
  • The Groovy ext.args expansion quotes names containing spaces ('Collagen 4'),
    trims whitespace and drops empty entries from a trailing comma.
  • pre-commit run --all-files passes.

Not run: the model forward itself, which needs the gated snapshot and a GPU. The
preprocess contract was verified by reading the model source — it builds its stats
list from marker_names and broadcasts, so it is channel-count agnostic.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs — n/a, no new tool
  • Make sure your code lints (nf-core pipelines lint) — nf-core not available locally; pre-commit passes
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>) — not run locally (no container engine on the login node)
  • Check for unexpected warnings in debug mode
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated — n/a, it delegates the parameter list to docs/usage.md

🤖 Generated with Claude Code

Adds a way to withhold channels that are not markers -- autofluorescence,
blanks, empty cycles -- from KRONOS2, so they contribute nothing to the
embedding. Until now the only escape for an out-of-vocabulary channel was
--allow-novel-defaults, which still *feeds* it to the model normalised with
default statistics: the opposite of what is wanted for a channel that
carries no biology.

This is scoped to the embedding step and deletes nothing. A withheld
channel is simply never read, so it stays in the image, remains available
to segmentation and every other process, and keeps whatever intensity
measurements CELLMEASUREMENT wrote for it in the GeoJSON. Contrast
remove_markers, which does strip channels out of the background-subtracted
image. The log line, the marker report and the docs all say "not shown to
KRONOS2" rather than "removed", so the scope is not mistakable.

Names are comma separated and case sensitive, matched against the image's
own channel names before any kronos_marker_mapping rename. A name matching
nothing fails the run and lists the channels that are present; the nuclear
channel cannot be withheld because it is the model's preferred_dapi.

While wiring this up, channel indices became load-bearing and exposed a
silent correctness bug on OME-TIFFs that give each channel its own 2D
series. ChannelReader sized itself from series[0] alone, so an 8-series
image read as 1 channel while the patch buffer was still sized from the
8-name marker list -- and numpy broadcast that one channel across all
eight. The run completed, reported 8 channels, and produced 768-d vectors
computed from eight copies of DAPI. Verified on the COMET _fixed images.

The layout is now detected once and shared by the name list and the pixel
reader, each channel is read from its own series, read_window refuses to
return fewer channels than were asked of it, and the run fails outright if
the name count and the image's channel count disagree. Channel names are
also built positionally, so an OME Channel carrying neither Name nor ID no
longer shifts every later marker onto the wrong channel.

The marker report gains the embedded/not-shown split and names withheld
channels in place, and is now written before the novel-marker gate can
exit -- a failing run is when it is most useful.

Adds tests/python/test_kronos2_common.py, the first module under the
tests/python scaffold, plus a pytest job in the linting workflow so it
runs, and a stub nf-test for the module, which had none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mikemcka
mikemcka merged commit c697c00 into WEHI-SODA-Hub:dev Aug 6, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant