denoiq-core

Denoising under a data-processing ceiling: observer-dependent benefits, fidelity–task divergence, and an information floor.

The code and results behind the study of that name: a simulation study of when denoising helps a detection task, when it hurts one, and how both depend on who is looking.

Post-processing cannot create information. That is not an opinion about denoising algorithms; it is the data-processing inequality, and it has a consequence that can be measured: once the input image is fixed, the task information available to any downstream observer has a ceiling, and no filter, no network and no reader can exceed it. Separately, a study can prespecify how much detectability a task requires; the acquisition contour where the input's ideal observer falls below that requirement is the operational floor. It is a task requirement, not a zero-information boundary — measurable information remains below it — but no post-processing can bring the requirement back into reach. What a denoiser can still do there is produce a cleaner-looking image, which is why visual plausibility is not evidence that the required information survived.

This repository formalises both limits, implements them, and demonstrates them on purely synthetic data:

	what it is	where
Ceiling	detectability after any processing ≤ analytic ideal-observer detectability of the unprocessed input	denoiq_core/bound.py
Floor	the kV–mAs contour where the input's ideal d' crosses a prespecified requirement (default: Rose, d' = 5)	denoiq_core/redlamp.py
Red lamp	green / amber / red with a written reason, from the floor, the task estimate, excess lesion-like responses and contrast recovery	denoiq_core/redlamp.py

Everything is synthetic and seeded. There is no patient data, no DICOM, no scanner calibration, no executable, and no proprietary model in this repository, and none is required to reproduce any number in it.

The question, and what the study answers

Under a fixed data-processing ceiling, when does denoising improve task performance, when does it degrade it despite improving fidelity, and how do these effects depend on observer efficiency and on the detectability available in the input?

Measured over a matrix of dose, noise correlation length and lesion configuration, three deterministic denoisers, three observers of differing efficiency, and ten independent realisations, with cross-fitted estimation and cluster-bootstrap intervals. The headline numbers are in results/statistics.json; the paper is paper/manuscript.md.

Install

# the observer / task engine (not on PyPI: install the archived release by tag)
pip install "taskiq-core @ git+https://github.com/Institute-of-One/taskiq-core@v0.4.0"

pip install -e ".[dev]"          # everything except deep learning
pip install -e ".[dev,dl]"       # adds the optional CNN (torch, CPU)
pip install -e ".[dev,pdf]"      # adds the review-PDF build (reportlab)

Python 3.10–3.12. The deep-learning extra is genuinely optional: the classical denoisers, all three observers, the ceiling, the red lamp, every sweep and every figure run without it.

Reproduce

python -c "from denoiq_core.experiment import run_primary; run_primary()"  # the 10-realisation matrix
python -c "from denoiq_core.experiment import run_all; run_all()"      # representative-realisation artefacts
python paper/make_figures.py                                        # writes paper/figures/
python paper/build_manuscript.py                                    # resolves the numbers in both documents
python paper/build_pdf.py                                           # JMI-style PDF (needs the [pdf] extra)
python paper/build_pdf.py --document supplementary                  # the supplement
pytest                                                              # ~2 minutes, all synthetic

Two example scripts show the two headline results on their own:

python examples/run_denoise_taskbench.py            # raw vs classical (add --cnn <ckpt> for the network)
python examples/run_redlamp_atlas.py                # the kV-mAs atlas and Figure 6

Optional, needs [dl]:

python -m denoiq_core.train --epochs 40             # deterministic; records the checkpoint sha256
python examples/run_denoise_taskbench.py --cnn checkpoints/cnn.pt

How the measurement works

The task. Signal-known-exactly / background-known-exactly detection of a low-contrast disk, generated by taskiq-core together with the analytic noise power spectrum of the images it just made. Observers (ideal_linear, cho, npwe), trial generation, ROC/d'/AUC and the physical metrics are imported from that package and not reimplemented here.

The ceiling. On the unprocessed input the ideal linear (prewhitening) observer is available in closed form, so the ceiling carries no sampling error. On processed images neither the noise spectrum nor the effective signal is known analytically — a non-linear denoiser has no transfer function — so both are estimated on a training split and the resulting prewhitening template is scored on a disjoint test split. That estimator is a held-out prewhitening linear observer, not the likelihood-ratio ideal observer of the processed data: it is measurable and leakage-controlled, and it need not attain processed-data ideal performance. Held-out scoring is what makes the comparison honest — a template fitted and scored on the same images is biased high, and against an upper bound a biased-high estimator manufactures violations of the inequality under test.

The acquisition model. physics.py maps (kV, mAs) to contrast and noise through documented proportionalities — photons ∝ mAs·kV², noise ∝ 1/√photons, contrast falling with kV — all relative to a reference setting. It is a model, not a calibration; see the scope note below.

Excess false structure, and erasure. In signal-absent images the underlying object is spatially uniform, but the acquired image is not: noise alone produces lesion-like matched-filter responses. false_structure_rate counts how often one reaches a given fraction of a real lesion's amplitude, and is only interpretable against the same measurement on the unprocessed input — what it detects is an excess, not structure that processing alone invented. contrast_recovery measures how much of a real lesion survives.

What the tests guarantee

test	what would break it
test_bound.py	any denoiser beating the raw ideal observer; a processor that peeks at the truth (asserted to be caught); an estimator so weak the bound is trivially satisfied
test_closed_form.py	the ideal observer disagreeing with d' = ‖s‖/σ in white noise by more than 1 % (it agrees to ~1e-9)
test_physics.py	the model drifting from σ ∝ 1/√(mAs·kV²), dose ∝ mAs, or monotone contrast
test_redlamp.py	the floor contour missing the threshold; the alert logic firing wrongly; a planted lesion going uncounted
test_determinism.py	any number, figure or trained checkpoint that changes between runs at a fixed seed
test_manuscript_consistency.py	a number in the manuscript that is not re-derivable from results/; a typed number in the prose; a stale built manuscript; overstated wording ("processed ideal observer", "chance-level", a floor described as zero information) in the text or in a figure label

Scope

Included: synthetic phantom generation, the relative kV/mAs degradation model, classical and (optional) CNN denoisers, model-observer evaluation, the data-processing ceiling, the red-lamp method, condition sweeps, figures, tests, and results/.

Not included, by design: patient or clinical data of any kind, DICOM, .exe or MATLAB runtimes, commercial GUIs, third-party proprietary data or weights, and — the important one — measured dose-to-noise calibration of any particular scanner and the absolute kV/mAs threshold tables that would follow from it. The red lamp here is a method plus a synthetic demonstration. Applying it to a specific device requires that device's calibration, which is not part of this open core.

Citing

Cite the archived release (Zenodo concept DOI resolves to the latest version) — see CITATION.cff. This work uses taskiq-core as its observer/task engine; please cite that release too (concept DOI 10.5281/zenodo.21422924).

Licence and provenance

MIT — see LICENSE. Copyright (c) 2026 Institute of One, LISIT Co., Ltd., Tokyo, Japan. Contact: yamamoto@lisit.jp · ORCID 0000-0001-9211-1071.