|
| 1 | +# Multimodal Early-Fusion Network: Radiographs + Clinical Tabular Data |
| 2 | + |
| 3 | +This tutorial demonstrates an **early-fusion** architecture that combines a 2D medical image stream |
| 4 | +with a low-dimensional clinical tabular stream in a single MONAI dictionary-based pipeline, using |
| 5 | +[`multimodal_early_fusion_tutorial.ipynb`](./multimodal_early_fusion_tutorial.ipynb). |
| 6 | + |
| 7 | +## Motivation |
| 8 | + |
| 9 | +Most MONAI tutorials focus on a single imaging modality. In real deployments, especially in |
| 10 | +resource-constrained clinics, an image is rarely read in isolation -- a clinician also has vitals, |
| 11 | +labs, or a short structured history on hand, and a model that ignores that context is throwing away |
| 12 | +signal it doesn't have to. This tutorial shows the minimum end-to-end MONAI/PyTorch pattern for |
| 13 | +combining the two without hand-rolling a custom `Dataset`. |
| 14 | + |
| 15 | +The clinical feature schema (age, BMI, salivary pH, systolic blood pressure) and the binary screening |
| 16 | +task are modeled after a hypertension-screening workflow explored at Nakaseke Hospital, Uganda. The |
| 17 | +underlying patient data is confidential and is **not** included in or downloaded by this tutorial. |
| 18 | + |
| 19 | +## Dataset |
| 20 | + |
| 21 | +**Fully synthetic, generated locally, no download required.** The notebook's |
| 22 | +`simulate_nakaseke_multimodal_dataset()` function creates, on the fly: |
| 23 | + |
| 24 | +- one synthetic 2D radiograph per patient, saved as a NIfTI (`.nii.gz`) file with `nibabel` |
| 25 | +- a 4-dimensional tabular vector `[age, bmi, salivary_ph, systolic_bp]` |
| 26 | +- a binary label |
| 27 | + |
| 28 | +Both modalities are generated from a shared hidden "risk factor" per patient, so neither the image |
| 29 | +nor the tabular vector is fully predictive of the label on its own -- this is what motivates fusing |
| 30 | +them. There is no claim that the notebook's results reflect real-world diagnostic performance; the |
| 31 | +synthetic cohort exists only to exercise the pipeline end-to-end without any real or downloadable |
| 32 | +data. |
| 33 | + |
| 34 | +## What the notebook covers |
| 35 | + |
| 36 | +1. Reproducible setup with `monai.utils.set_determinism`. |
| 37 | +2. Synthetic multimodal data generation and a MONAI-style data manifest |
| 38 | + (`{"image": ..., "nakaseke_tabular": ..., "label": ...}`). |
| 39 | +3. A dictionary-based `Compose` pipeline where image-only transforms (`LoadImaged`, |
| 40 | + `EnsureChannelFirstd`, `ScaleIntensityRanged`) touch only the `"image"` key, while the tabular and |
| 41 | + label keys are cast to tensors with `EnsureTyped` and otherwise left untouched. |
| 42 | +4. `ResilientMultimodalClassifier`: a MONAI `DenseNet121` visual stream (512-d embedding) fused via |
| 43 | + `torch.cat` with a small tabular projection stream (16-d embedding) into a 528-d representation, |
| 44 | + followed by a dropout-regularized classification head. |
| 45 | +5. A short training loop (`max_epochs`, `val_interval`) showing validation accuracy improve as the |
| 46 | + model learns to use both streams. |
| 47 | + |
| 48 | +## Requirements |
| 49 | + |
| 50 | +Everything needed is installed by the notebook's own `Setup environment` cell |
| 51 | +(`monai-weekly[nibabel, tqdm]`, `matplotlib`). No GPU is required -- the default image size (64x64) |
| 52 | +and cohort size (200 patients) are chosen to train in well under a minute on CPU. |
0 commit comments