Skip to content

Commit 08e972c

Browse files
mikemckaclaude
andcommitted
feat: global cellpose normalisation to remove residual tiling grid
Even with sopa's CLAHE and gaussian disabled, cellpose still rescales every patch to that patch's own 1-99 percentile, so an identical cell lands on a different scale in a dense versus a sparse patch -- the residual tile grid at patch seams. Measured on one identical synthetic cell across patches of differing content, its post-normalisation peak varied by 284% under per-patch percentile and 1.8% under a shared global scale. Add opt-in `cellpose_normalize_global`: compute one [low, high] pair per channel from the whole image (at `cellpose_normalize_percentiles`, default 1,99) and pass it to cellpose as `--method-kwargs '{"normalize": {"lowhigh": ...}}'`, so every patch shares one scale. Bounds are ordered to match the channels cellpose receives (membrane then nucleus for whole-cell). The cellpose normalize dict is now single-sourced in the SOPA_SEGMENTATIONCELLPOSE module rather than conf/modules.config, because the global bounds are computed per sample at run time and sopa would choke on --method-kwargs appearing twice. Precedence: global > tile-local (cellpose_tile_norm_blocksize) > cellpose's stock whole-patch percentile. Off by default (preserves cellpose's stock contract); assumes cellpose_clip_limit=0, since bounds are computed on raw intensities. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 8ec03af commit 08e972c

8 files changed

Lines changed: 359 additions & 14 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
189189

190190
### Added
191191

192+
- **`cellpose_normalize_global`, which normalises intensity globally across the
193+
whole image instead of per patch, removing the residual tiling grid.** Even
194+
with sopa's CLAHE and gaussian disabled, Cellpose still rescales every patch to
195+
that patch's own 1–99 percentile, so an identical cell lands on a different
196+
scale in a dense versus a sparse patch and a faint grid persists at patch seams.
197+
Measured on one identical synthetic cell across patches of differing content,
198+
its post-normalisation peak varied by 284% under per-patch percentile and 1.8%
199+
under a shared global scale. When enabled, the pipeline computes one
200+
`[low, high]` pair per channel from the whole image (at the new
201+
`cellpose_normalize_percentiles`, default `1,99`) and passes it to Cellpose as
202+
`--method-kwargs '{"normalize": {"lowhigh": ...}}'`, ordered to match the
203+
channels Cellpose receives. Off by default; assumes `cellpose_clip_limit = 0`
204+
and supersedes `cellpose_tile_norm_blocksize`. The Cellpose normalize dict is
205+
now single-sourced in the `SOPA_SEGMENTATIONCELLPOSE` module rather than
206+
`conf/modules.config`, since the global bounds are computed per sample at run
207+
time.
208+
192209
- **`kronos_exclude_markers`, which withholds channels from KRONOS2 so they
193210
contribute nothing to the embedding.** A COMET panel's `Autofluorescence`
194211
channel is not a marker and is not in the 288-marker vocabulary, so the run

‎bin/compute_cellpose_norm.py‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
#!/usr/bin/env python
2+
'''
3+
Module : compute_cellpose_norm
4+
Description : Compute per-channel global intensity bounds (low, high) for a
5+
SpatialData .zarr image, to be handed to Cellpose as its
6+
`normalize={"lowhigh": ...}` argument.
7+
8+
Cellpose normalises every patch to that patch's own 1-99
9+
percentile. Identical cells therefore land on different scales in
10+
a dense versus a sparse patch, which is the source of the residual
11+
tiling grid once sopa's CLAHE/gaussian stages are disabled.
12+
Computing one (low, high) pair per channel from the WHOLE image
13+
and applying it to every patch removes that patch dependence.
14+
15+
Bounds are emitted in the SAME channel order they are passed on
16+
the command line, which must match the order in which the caller
17+
passes `--channels` to `sopa segmentation cellpose` (membrane then
18+
nucleus for whole-cell, nucleus alone for nuclear-only). The
19+
output is a JSON/Python-literal list of [low, high] pairs, ready to
20+
splice into `--method-kwargs`.
21+
Copyright : (c) WEHI SODA Hub, 2025
22+
License : MIT
23+
Maintainer : Marek Cmero (@mcmero)
24+
Portability : POSIX
25+
'''
26+
import json
27+
from pathlib import Path
28+
from typing import Annotated, List, Optional
29+
30+
import numpy as np
31+
import typer
32+
33+
app = typer.Typer()
34+
35+
36+
def _get_image(zarr_path: str):
37+
"""Return the single image element of a SpatialData store as a DataArray or
38+
DataTree (multiscale). Uses sopa's helper when available, else falls back to
39+
the sole image in the store."""
40+
import spatialdata as sd
41+
42+
sdata = sd.read_zarr(zarr_path)
43+
try:
44+
from sopa.utils import get_spatial_image
45+
46+
return get_spatial_image(sdata)
47+
except Exception:
48+
keys = list(sdata.images.keys())
49+
if len(keys) != 1:
50+
raise ValueError(
51+
f"Expected exactly one image in {zarr_path}, found {keys}"
52+
)
53+
return sdata.images[keys[0]]
54+
55+
56+
def _pick_level(image, max_pixels: int):
57+
"""From a (possibly multiscale) image, return a single DataArray to compute
58+
percentiles on. For a multiscale pyramid, pick the finest level whose Y*X is
59+
at or below `max_pixels` (falling back to the coarsest if even that is
60+
larger). p1/p99 are robust to this downsampling, and it bounds memory on
61+
large slides."""
62+
# Multiscale images are DataTree-like: they expose scale groups.
63+
scales = None
64+
try:
65+
scales = [k for k in image.keys() if str(k).startswith("scale")]
66+
except Exception:
67+
scales = None
68+
69+
if scales:
70+
# scale0 is full resolution; scales are ordered coarser as the index grows.
71+
def _yx(scale):
72+
arr = image[scale]["image"]
73+
return int(arr.sizes["y"]) * int(arr.sizes["x"])
74+
75+
scales_sorted = sorted(scales, key=lambda s: _yx(s), reverse=True)
76+
chosen = scales_sorted[-1] # coarsest as a safe default
77+
for s in scales_sorted: # finest first
78+
if _yx(s) <= max_pixels:
79+
chosen = s
80+
break
81+
return image[chosen]["image"]
82+
83+
# Single-scale DataArray.
84+
return image
85+
86+
87+
def _percentile_bounds(values: np.ndarray, plow: float, phigh: float,
88+
max_pixels: int) -> List[float]:
89+
"""Compute [low, high] percentile bounds from a 2D intensity array.
90+
91+
Pure numpy so it is unit-testable without xarray/spatialdata. Stride-
92+
subsamples a single-scale array far above the cap to bound memory, and
93+
guards a flat channel: Cellpose divides by (high - low), so equal bounds
94+
would give inf/NaN."""
95+
values = np.asarray(values)
96+
n = values.size
97+
if n > max_pixels:
98+
stride = int(np.ceil(np.sqrt(n / max_pixels)))
99+
values = values[::stride, ::stride]
100+
101+
lo, hi = np.percentile(values.ravel(), [plow, phigh])
102+
lo, hi = float(lo), float(hi)
103+
if hi <= lo:
104+
hi = lo + 1.0
105+
return [lo, hi]
106+
107+
108+
def _bounds_for_channel(arr, channel: str, plow: float, phigh: float,
109+
max_pixels: int) -> List[float]:
110+
"""Compute [low, high] percentile bounds for one named channel."""
111+
a = arr.sel(c=channel).data # dask or numpy, shape (Y, X)
112+
return _percentile_bounds(np.asarray(a), plow, phigh, max_pixels)
113+
114+
115+
@app.command()
116+
def main(
117+
zarr: Annotated[str, typer.Argument(help="Path to the SpatialData .zarr store")],
118+
channel: Annotated[
119+
List[str],
120+
typer.Option(
121+
"--channel",
122+
help="Channel name to compute bounds for. Repeat in the SAME order "
123+
"the channels are passed to `sopa segmentation cellpose` "
124+
"(membrane then nucleus for whole-cell).",
125+
),
126+
],
127+
percentiles: Annotated[
128+
str,
129+
typer.Option(help="Low,high percentiles for the bounds."),
130+
] = "1,99",
131+
max_pixels: Annotated[
132+
int,
133+
typer.Option(
134+
help="Compute on the finest multiscale level with at most this many "
135+
"pixels (single-scale images above it are stride-subsampled).",
136+
),
137+
] = 8_000_000,
138+
as_method_kwargs: Annotated[
139+
bool,
140+
typer.Option(
141+
"--as-method-kwargs",
142+
help="Emit the full Cellpose method-kwargs dict "
143+
'`{"normalize": {"lowhigh": ...}}` instead of the bare list of '
144+
"pairs, ready to pass to `sopa segmentation cellpose --method-kwargs`.",
145+
),
146+
] = False,
147+
output: Annotated[
148+
Optional[str],
149+
typer.Option("--output", "-o", help="Write JSON here instead of stdout."),
150+
] = None,
151+
):
152+
"""Emit per-channel [low, high] bounds as a JSON list of pairs, ordered to
153+
match the given `--channel` order."""
154+
plow, phigh = (float(x) for x in percentiles.split(","))
155+
156+
image = _get_image(zarr)
157+
arr = _pick_level(image, max_pixels)
158+
159+
available = {str(c) for c in arr.coords["c"].values}
160+
missing = [c for c in channel if c not in available]
161+
if missing:
162+
raise ValueError(
163+
f"Channel(s) {missing} not found in image. Available: {sorted(available)}"
164+
)
165+
166+
lowhigh = [
167+
_bounds_for_channel(arr, c, plow, phigh, max_pixels) for c in channel
168+
]
169+
170+
payload = {"normalize": {"lowhigh": lowhigh}} if as_method_kwargs else lowhigh
171+
text = json.dumps(payload)
172+
if output:
173+
Path(output).write_text(text)
174+
else:
175+
typer.echo(text)
176+
177+
178+
if __name__ == "__main__":
179+
app()

‎conf/modules.config‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -92,15 +92,14 @@ process {
9292
// to 'cpsam'. Model choice goes through --pretrained-model.
9393
params.containsKey('cellpose_pretrained_model') && params.cellpose_pretrained_model != null ?
9494
"--pretrained-model ${params.cellpose_pretrained_model}" : null,
95-
// Patch-local normalisation. sopa forwards --method-kwargs into
96-
// cellpose_patch's **cellpose_eval_kwargs, which land on
97-
// model.eval, so this is the only route to cellpose's own
98-
// normalize dict. Passed only when non-zero: cellpose's default
99-
// is tile_norm_blocksize=0 (whole-patch percentile), and
100-
// sending the dict regardless would pin percentile even when
101-
// tiling is off.
102-
params.containsKey('cellpose_tile_norm_blocksize') && params.cellpose_tile_norm_blocksize ?
103-
"--method-kwargs '{\"normalize\": {\"tile_norm_blocksize\": ${params.cellpose_tile_norm_blocksize}, \"percentile\": [1.0, 99.0]}}'" : null,
95+
// NB: --method-kwargs (cellpose's normalize dict) is NOT set
96+
// here. It is single-sourced in the module itself, because
97+
// global normalisation (cellpose_normalize_global) needs
98+
// per-sample bounds computed at run time, and sopa would choke
99+
// on the flag appearing twice. The module writes exactly one
100+
// method_kwargs.json from cellpose_normalize_global /
101+
// cellpose_tile_norm_blocksize and always passes it.
102+
//
104103
// sopa's CLAHE, applied before cellpose. Left at sopa's
105104
// defaults unless asked for.
106105
params.containsKey('cellpose_clip_limit') && params.cellpose_clip_limit != null ?

‎docs/usage.md‎

Lines changed: 43 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -308,14 +308,53 @@ These four parameters form the **Cellpose preprocessing options** group:
308308

309309
| Parameter Name | Default | Description |
310310
| ---------------------------- | ------- | -------------------------------------------------------------------------------------- |
311-
| cellpose_gaussian_sigma | `0` | sopa's gaussian smoothing sigma. `0` disables it; `null` restores sopa's `1`. |
312-
| cellpose_clip_limit | `0` | sopa's CLAHE clip limit. `0` disables CLAHE; `null` restores sopa's `0.2`. |
313-
| cellpose_clahe_kernel_size | `null` | CLAHE kernel in pixels. `null` means `patch.shape // 8`. No effect while CLAHE is off. |
314-
| cellpose_tile_norm_blocksize | `0` | Cellpose tile-local normalisation block. `0` is Cellpose's whole-patch default. |
311+
| cellpose_gaussian_sigma | `0` | sopa's gaussian smoothing sigma. `0` disables it; `null` restores sopa's `1`. |
312+
| cellpose_clip_limit | `0` | sopa's CLAHE clip limit. `0` disables CLAHE; `null` restores sopa's `0.2`. |
313+
| cellpose_clahe_kernel_size | `null` | CLAHE kernel in pixels. `null` means `patch.shape // 8`. No effect while CLAHE is off. |
314+
| cellpose_tile_norm_blocksize | `0` | Cellpose tile-local normalisation block. `0` is Cellpose's whole-patch default. |
315+
| cellpose_normalize_global | `false` | Normalise intensity globally across the whole image instead of per patch. See below. |
316+
| cellpose_normalize_percentiles | `1,99` | Low,high percentiles for the global bounds. `1,99` matches Cellpose's per-patch default.|
315317

316318
To restore sopa's behaviour, set `--cellpose_clip_limit 0.2` and
317319
`--cellpose_gaussian_sigma 1`.
318320

321+
##### Global normalisation (the residual tiling grid)
322+
323+
Disabling sopa's CLAHE and gaussian removes the largest source of patch
324+
dependence, but not all of it. Cellpose's **own** normalisation — the percentile
325+
(1, 99) rescale it always applies — is still computed on each patch
326+
independently, because sopa hands Cellpose one patch at a time. An identical cell
327+
therefore lands on a different scale in a dense versus a sparse patch, and where
328+
those patches meet you can still see a faint grid, most visibly as a step in cell
329+
yield across a patch seam.
330+
331+
Measured directly: placing one identical synthetic cell in patches of differing
332+
content, its post-normalisation peak intensity varied by **284%** across patches
333+
under Cellpose's per-patch percentile. Sharing one global scale dropped that to
334+
**1.8%**.
335+
336+
`cellpose_normalize_global = true` removes it. The pipeline computes one
337+
`[low, high]` pair per channel from the **whole image** (at the
338+
`cellpose_normalize_percentiles` percentiles), then passes it to Cellpose as
339+
`--method-kwargs '{"normalize": {"lowhigh": [[lo, hi], ...]}}'`, so every patch is
340+
rescaled against the same bounds. Bounds are computed and ordered to match the
341+
channels Cellpose receives — membrane then nucleus for whole-cell, nucleus alone
342+
for nuclear-only.
343+
344+
Two caveats:
345+
346+
- It **assumes `cellpose_clip_limit = 0`** (the default). The bounds are computed
347+
on raw intensities, which is what Cellpose sees only while CLAHE is off — CLAHE
348+
rescales each patch to 0–1, so raw-intensity bounds would not correspond to the
349+
array Cellpose is normalising.
350+
- It **supersedes `cellpose_tile_norm_blocksize`**: with global bounds set,
351+
Cellpose uses `lowhigh` and ignores tile-local percentiles. Both configure the
352+
same `normalize` dict, so only one is passed at a time
353+
(`cellpose_normalize_global` wins).
354+
355+
It is off by default so the stock Cellpose contract (per-patch percentile) is
356+
preserved unless you opt in.
357+
319358
##### Why CLAHE is off
320359

321360
Segmentation always runs on patches, and all of this normalisation is computed

‎modules/local/sopa/segmentationcellpose/main.nf‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,31 @@ process SOPA_SEGMENTATIONCELLPOSE {
2424

2525
script:
2626
def args = task.ext.args ?: ''
27-
def membrane_channel_arg = (membrane_channel && membrane_channel != "[]") ? "--channels \"${membrane_channel}\"" : ""
27+
def has_membrane = (membrane_channel && membrane_channel != "[]")
28+
def membrane_channel_arg = has_membrane ? "--channels \"${membrane_channel}\"" : ""
29+
30+
// Cellpose's normalize dict is single-sourced here (rather than in
31+
// modules.config) because global normalisation needs the per-sample bounds
32+
// that only exist at run time. sopa forwards --method-kwargs straight into
33+
// model.eval, and passing it twice would conflict, so exactly one
34+
// method_kwargs.json is written below and always passed.
35+
//
36+
// Precedence: global normalisation > tile-local percentile > cellpose's
37+
// stock whole-patch percentile ({} == no override).
38+
def tile_norm = (params.containsKey('cellpose_tile_norm_blocksize') && params.cellpose_tile_norm_blocksize) ? params.cellpose_tile_norm_blocksize : 0
39+
def global_norm = params.containsKey('cellpose_normalize_global') && params.cellpose_normalize_global
40+
def percentiles = (params.containsKey('cellpose_normalize_percentiles') && params.cellpose_normalize_percentiles) ? params.cellpose_normalize_percentiles : '1,99'
41+
// Channels in the SAME order sopa builds the image below (membrane then
42+
// nucleus), so the per-channel bounds line up with the array cellpose sees.
43+
def norm_channel_args = (has_membrane ? "--channel \"${membrane_channel}\" " : "") + "--channel \"${nuclear_channel}\""
44+
def norm_setup = ''
45+
if (global_norm) {
46+
norm_setup = "compute_cellpose_norm.py ${zarr} ${norm_channel_args} --percentiles ${percentiles} --as-method-kwargs -o method_kwargs.json"
47+
} else if (tile_norm) {
48+
norm_setup = "printf '%s' '{\"normalize\": {\"tile_norm_blocksize\": ${tile_norm}, \"percentile\": [1.0, 99.0]}}' > method_kwargs.json"
49+
} else {
50+
norm_setup = "printf '%s' '{}' > method_kwargs.json"
51+
}
2852
"""
2953
export NUMBA_CACHE_DIR=\$PWD/.numba_cache
3054
@@ -43,13 +67,17 @@ process SOPA_SEGMENTATIONCELLPOSE {
4367
done
4468
export CELLPOSE_LOCAL_MODELS_PATH=\$PWD/.cellpose_models
4569
70+
# Write exactly one cellpose normalize dict (see script block above).
71+
${norm_setup}
72+
4673
sopa segmentation cellpose \\
4774
${args} \\
4875
--patch-index ${index} \\
4976
${membrane_channel_arg} \\
5077
--channels "${nuclear_channel}" \\
5178
--diameter ${params.cellpose_diameter} \\
5279
--min-area ${params.cellpose_min_area} \\
80+
--method-kwargs "\$(cat method_kwargs.json)" \\
5381
${zarr}
5482
5583
cat <<-END_VERSIONS > versions.yml

‎nextflow.config‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,22 @@ params {
8585
// cellpose's contract, not about stability. null restores sopa's default
8686
// of 1.
8787
cellpose_gaussian_sigma = 0
88+
// Global (cross-patch) intensity normalisation. Even with CLAHE and gaussian
89+
// off, cellpose still normalises each patch to ITS OWN 1-99 percentile, so an
90+
// identical cell lands on a different scale in a dense versus a sparse patch
91+
// -- the residual tile grid once sopa's stages are disabled. When true, one
92+
// [low, high] pair per channel is computed from the WHOLE image and applied
93+
// to every patch (cellpose's normalize={"lowhigh": ...}), removing the patch
94+
// dependence. Off by default to preserve cellpose's stock per-patch contract.
95+
//
96+
// Assumes cellpose_clip_limit = 0: the bounds are computed on raw image
97+
// intensities, which is what cellpose sees only while CLAHE is off (CLAHE
98+
// rescales each patch to 0-1, so raw-intensity bounds would be meaningless).
99+
// It supersedes cellpose_tile_norm_blocksize when both are set.
100+
cellpose_normalize_global = false
101+
// Low,high percentiles for the global bounds. (1, 99) matches cellpose's own
102+
// per-patch default; the only change is computing them once, globally.
103+
cellpose_normalize_percentiles = '1,99'
88104

89105
// mesmer options
90106
mesmer_segmentation_level = null

0 commit comments

Comments
 (0)