ZoomLDM: Latent Diffusion Model for multi-scale image generation

CVPR 2025

         

Setup

git clone https://github.com/cvlab-stonybrook/ZoomLDM/
conda activate zoomldm
pip install -r requirements.txt

The model weights are hosted on huggingface. The inference scripts we provide below download the model weights using huggingface hub.

Inference

Patch level generation

We demonstrate patch-level generation at any scale in sample_patches_brca.ipynb and sample_patches_naip.ipynb.

Large image generation

For large image generation, we use the proposed joint multi-scale sampling algorithm. For a more in-depth analysis of the inference algorithm refer to Fast constrained sampling in pre-trained diffusion models (NeurIPS 2025)

An implementation of the algorithm for generating very large images using ZoomLDM is given in joint_multiscale.ipynb.

For more examples of large images, see our website.

Super-resolution

Super-resolution uses the condition inversion algorithm proposed in the paper and with the joint multi-scale sampling to enforce the low-resolution constraint.

We prove an implementation in superres.ipynb.

Training

To train the model, you need to prepare a multi-scale dataset of {images, conditioning}.

Patch extraction

We use the codebase of DS-MIL to extract regions from the WSIs, first at the base 20x magnification. The patches range from 256x256 to 32768x32768 pixels. You might want to use a lower tissue threshold for larger images.

The following command will extract 1024x1024 patches at 20x:

python deepzoom_tiler.py -m 0 -b 20 -s 1024

Refer to this issue for satellite image patch extraction.

Feature extraction

Histopathology

We pre-extract UNI embeddings (conditioning) from the full resolution images in a patch-based manner. A 2048x2048 image would result in 64x256x256 patches -> 64x1024 UNI embedding.

We then resize images to 256x256, extract VAE features, and save them together with the UNI embeddings.

For NAIP, we use the pre-trained DINO-v2 ViT-Large (dinov2_vitl14_reg) checkpoint to extract embeddings.

Please take a look at the demo datasets: brca/naip or our dataloader scripts: brca/naip for more details.

Training command

Create a config file similar to this, which specifies the dataset, model, and training parameters.

Then, run the training script:

python main.py -t --gpus 0,1,2 --base configs/zoomldm_brca.yaml

Conditioning diffusion model (CDM)

Our image synthesis pipeline requires a set of SSL embeddings and a magnification level, which involves extracting embeddings from a reference image. This becomes impractical when access to the reference image is unavailable. To address this, we introduce the Conditioning Diffusion Model (CDM) that generates SSL embeddings conditioned on the specified magnification level.

We provide the training and inference scripts for CDM in cdm_dit/.

Pretrained model weights for CDM are available here.

Bibtex

@InProceedings{Yellapragada_2025_CVPR,
  author = {Yellapragada, Srikar and Graikos, Alexandros and Triaridis, Kostas and Prasanna, Prateek and Gupta, Rajarsi and Saltz, Joel and Samaras, Dimitris},
  title = {ZoomLDM: Latent Diffusion Model for Multi-scale Image Generation},
  booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
  month = {June},
  year = {2025},
  pages = {23453-23463}
}