Multi-Scale Features and Parallel Transformers Based Image Quality Assessment

Code for our paper "Multi-Scale Features and Parallel Transformers Based Image Quality Assessment" Abhisek Keshari, Komal, Sadbhawna, Badri Subudhi (IIT Jammu) arXiv:2204.09779 · Submission to the NTIRE 2022 Perceptual Image Quality Assessment Challenge (CVPR 2022 Workshops)

Overview

Full-reference image quality assessment (FR-IQA) predicts how a human would rate a distorted image, given its pristine reference. The PIPAL dataset makes this hard by including outputs of GAN-based restoration models, whose distortions don't match traditional quality metrics.

Transformer-based IQA (IQT, Cheon et al., NTIRE 2021) works well on PIPAL, but it sees each image at a single scale. Classical IQA has long benefited from multi-scale analysis. This work combines the two approaches.

Method

1. Feature extraction: The reference and distorted images go through an ImageNet-pretrained InceptionResNetV2. Feature maps from six intermediate blocks (Mixed_5b and the Block35 layers) are concatenated, giving 1920 channels.
2. Transformer regression: The difference features (reference − distorted) go to a transformer encoder, and the reference features go to the decoder. A learnable [CLS] token is passed through an MLP head to predict the quality score.
3. Multi-scale, parallel transformers: The input images are rescaled by factors of ×2 (upscale), ×1, ×½ and ×⅓. Feature maps are resized to a common 21×21 grid. A separate transformer is trained at each scale.
4. Fusion: The final quality score is the average of the four scale-specific predictions.

Training uses L1 loss, Adam with cosine-annealed learning rate, random crops/flips/rotations, and an 8:2 scene-level train/validation split of PIPAL.

Repository structure

IQT-multiscale submission/
├── train.py            # training + validation (all options in the `config` dict)
├── test.py             # inference on a folder of distorted images
├── trainer.py          # train / eval loops (SROCC, PLCC)
├── model/
│   ├── backbone.py     # InceptionResNetV2 feature extractor
│   └── model_main.py   # transformer encoder–decoder regressor
├── data/               # PIPAL and LIVE dataset loaders (with rescaling)
├── IQA_list/           # image lists with MOS labels
└── utils/util.py       # augmentations and data split

Setup

Tested with Python 3.6–3.9, PyTorch 1.7.1 (CUDA 11.0), einops 0.3.0, OpenCV 4.2, SciPy 1.4, tqdm.

1. Download the ImageNet-pretrained InceptionResNetV2 weights, rename the file to inceptionresnetv2.pth and put it in model/.
2. Download PIPAL (train/valid) from the official repository.
3. In train.py / test.py, set db_path (dataset root) and txt_file_name (path to IQA_list/PIPAL.txt).

Training

The scale field in the config dict selects the scale branch:

scale	Input transform
0.5	upscale ×2
1	original resolution
2	downscale ×2
3	downscale ×3

cd "IQT-multiscale submission"
python train.py          # repeat once per scale value

Inference

Set scale, weight_file and result_file in test.py, then run it once per scale:

python test.py

This gives four score files, which are averaged into the final prediction. The averaging script (averaging.py) and trained weights are in the official code release.

Citation

@article{keshari2022multiscale,
  title   = {Multi-Scale Features and Parallel Transformers Based Image Quality Assessment},
  author  = {Keshari, Abhisek and Komal and Sadbhawna and Subudhi, Badri},
  journal = {arXiv preprint arXiv:2204.09779},
  year    = {2022}
}

Acknowledgements

The single-scale baseline builds on an open-source PyTorch implementation of IQT. Experiments were run on GPU servers at IIT Jammu.