Skip to content

Commit 7fb4813

Browse files
committed
Add multimodal early-fusion tutorial (radiograph + clinical tabular)
Adds a self-contained tutorial fusing a 2D image stream (MONAI DenseNet121) with a low-dimensional clinical tabular stream via dictionary-based transforms and torch.cat, using a fully synthetic, locally generated dataset modeled after a Nakaseke Hospital hypertension-screening schema. No downloads or real patient data. Signed-off-by: Paul Sentongo <134306188+sentongo-web@users.noreply.github.com>
1 parent b3c6c99 commit 7fb4813

3 files changed

Lines changed: 596 additions & 0 deletions

File tree

README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,10 @@ This tutorial demonstrates how to construct a training workflow of [HoVerNet](ht
235235
##### [Nuclei Classification](./pathology/nuclick#nuclei-classification-model)
236236
The notebook demonstrates examples of training and inference pipelines with interactive annotation for pathology, NuClick is used for delineating nuclei, cells and a squiggle for outlining glands.
237237

238+
#### <ins>**Multimodal**</ins>
239+
##### [Multimodal Early-Fusion Network](./multimodal/nakaseke_multimodal_early_fusion)
240+
An end-to-end tutorial fusing a 2D radiograph stream (MONAI `DenseNet121`) with a low-dimensional clinical tabular stream via dictionary-based transforms and `torch.cat`, using a fully synthetic, locally generated dataset.
241+
238242
#### <ins>**Acceleration**</ins>
239243
##### [fast_model_training_guide](./acceleration/fast_model_training_guide.md)
240244
The document introduces details of how to profile the training pipeline, how to analyze the dataset and select suitable algorithms, and how to optimize GPU utilization in single GPU, multi-GPUs or even multi-nodes.
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Multimodal Early-Fusion Network: Radiographs + Clinical Tabular Data
2+
3+
This tutorial demonstrates an **early-fusion** architecture that combines a 2D medical image stream
4+
with a low-dimensional clinical tabular stream in a single MONAI dictionary-based pipeline, using
5+
[`multimodal_early_fusion_tutorial.ipynb`](./multimodal_early_fusion_tutorial.ipynb).
6+
7+
## Motivation
8+
9+
Most MONAI tutorials focus on a single imaging modality. In real deployments, especially in
10+
resource-constrained clinics, an image is rarely read in isolation -- a clinician also has vitals,
11+
labs, or a short structured history on hand, and a model that ignores that context is throwing away
12+
signal it doesn't have to. This tutorial shows the minimum end-to-end MONAI/PyTorch pattern for
13+
combining the two without hand-rolling a custom `Dataset`.
14+
15+
The clinical feature schema (age, BMI, salivary pH, systolic blood pressure) and the binary screening
16+
task are modeled after a hypertension-screening workflow explored at Nakaseke Hospital, Uganda. The
17+
underlying patient data is confidential and is **not** included in or downloaded by this tutorial.
18+
19+
## Dataset
20+
21+
**Fully synthetic, generated locally, no download required.** The notebook's
22+
`simulate_nakaseke_multimodal_dataset()` function creates, on the fly:
23+
24+
- one synthetic 2D radiograph per patient, saved as a NIfTI (`.nii.gz`) file with `nibabel`
25+
- a 4-dimensional tabular vector `[age, bmi, salivary_ph, systolic_bp]`
26+
- a binary label
27+
28+
Both modalities are generated from a shared hidden "risk factor" per patient, so neither the image
29+
nor the tabular vector is fully predictive of the label on its own -- this is what motivates fusing
30+
them. There is no claim that the notebook's results reflect real-world diagnostic performance; the
31+
synthetic cohort exists only to exercise the pipeline end-to-end without any real or downloadable
32+
data.
33+
34+
## What the notebook covers
35+
36+
1. Reproducible setup with `monai.utils.set_determinism`.
37+
2. Synthetic multimodal data generation and a MONAI-style data manifest
38+
(`{"image": ..., "nakaseke_tabular": ..., "label": ...}`).
39+
3. A dictionary-based `Compose` pipeline where image-only transforms (`LoadImaged`,
40+
`EnsureChannelFirstd`, `ScaleIntensityRanged`) touch only the `"image"` key, while the tabular and
41+
label keys are cast to tensors with `EnsureTyped` and otherwise left untouched.
42+
4. `ResilientMultimodalClassifier`: a MONAI `DenseNet121` visual stream (512-d embedding) fused via
43+
`torch.cat` with a small tabular projection stream (16-d embedding) into a 528-d representation,
44+
followed by a dropout-regularized classification head.
45+
5. A short training loop (`max_epochs`, `val_interval`) showing validation accuracy improve as the
46+
model learns to use both streams.
47+
48+
## Requirements
49+
50+
Everything needed is installed by the notebook's own `Setup environment` cell
51+
(`monai-weekly[nibabel, tqdm]`, `matplotlib`). No GPU is required -- the default image size (64x64)
52+
and cohort size (200 patients) are chosen to train in well under a minute on CPU.

0 commit comments

Comments
 (0)