Semantic Segmentation with DeepLabV3-ResNet50

A PyTorch project for semantic segmentation on the Pascal VOC 2012 dataset using a pretrained DeepLabV3-ResNet50 model.

The project explores different fine-tuning strategies and shows how training configuration, particularly Batch Normalization with small batch sizes, can strongly affect segmentation performance.

The best configuration improved the validation mean Intersection over Union (mIoU) from 71.31% to 75.78%.

Overview

Semantic segmentation assigns a class label to every pixel in an image.

This project uses DeepLabV3 with a ResNet-50 backbone to segment images into the 21 Pascal VOC classes:

background
aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
cow
dining table
dog
horse
motorbike
person
potted plant
sheep
sofa
train
TV/monitor

The workflow includes:

loading and preprocessing Pascal VOC 2012
image augmentation
transfer learning with pretrained weights
different fine-tuning strategies
mIoU evaluation
early stopping
model checkpointing
training-curve visualization
qualitative comparison of predicted and ground-truth segmentation masks

Model

The model is based on:

DeepLabV3 + ResNet-50

implemented using torchvision.models.segmentation.

The pretrained model is initialized with:

DeepLabV3_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1

This provides a strong pretrained baseline for the Pascal VOC semantic segmentation classes.

DeepLabV3 uses atrous convolution and Atrous Spatial Pyramid Pooling to capture contextual information at multiple spatial scales.

Dataset

The project uses the Pascal VOC 2012 Semantic Segmentation Dataset.

Dataset split used in this project:

Training images:   1464
Validation images: 1449

The images and segmentation masks are resized to:

256 × 256 pixels

The dataset itself is not included in this repository.

Place the VOC dataset in a local data directory before running the training script.

Example structure:

data/
└── VOCdevkit/
    └── VOC2012/
        ├── JPEGImages/
        ├── SegmentationClass/
        ├── ImageSets/
        └── ...

Data Augmentation

The training pipeline applies lightweight image augmentation while keeping the segmentation mask geometrically aligned with the image.

The augmentations include:

random horizontal flipping
brightness adjustment
contrast adjustment
saturation adjustment

Images are normalized using ImageNet statistics:

mean = [0.485, 0.456, 0.406]
std  = [0.229, 0.224, 0.225]

Experiments

Several transfer-learning strategies were evaluated.

1. Pretrained Baseline

The pretrained DeepLabV3-ResNet50 model was evaluated directly on the Pascal VOC 2012 validation set without additional training.

Validation Loss: 0.2182
Validation mIoU: 0.7131

This corresponds to a baseline validation mIoU of:

71.31%

2. Full Fine-Tuning

The complete pretrained network was fine-tuned using a single learning rate.

The model increasingly fitted the training set, but the validation performance did not improve beyond the pretrained baseline.

Best result:

Best Validation mIoU: 0.7049

At the final epoch:

Train mIoU: 0.8177
Validation mIoU: 0.6982

The increasing gap between training and validation performance indicated overfitting.

3. Differential Learning Rates

A second fine-tuning strategy used separate learning rates for the pretrained backbone and segmentation classifier:

Backbone learning rate:   1e-6
Classifier learning rate: 1e-5

Because the available GPU memory required a small batch size of 2, the model showed a substantial decrease in validation performance when Batch Normalization layers remained in training mode.

Best observed validation mIoU:

64.32%

This experiment motivated a modified strategy in which Batch Normalization statistics were kept fixed during fine-tuning.

4. Differential Learning Rates + Frozen BatchNorm

The final training strategy combined:

pretrained DeepLabV3-ResNet50 weights
a low learning rate for the ResNet backbone
a higher learning rate for the segmentation classifier
frozen Batch Normalization layers
batch size of 2
data augmentation
early stopping

The BatchNorm layers were kept in evaluation mode during fine-tuning so that their pretrained running statistics were not updated using very small mini-batches.

The best model achieved:

Train mIoU:      0.8463
Validation mIoU: 0.7578
Validation Loss: 0.1672

Final validation performance:

75.78% mIoU

This represents an absolute improvement of:

+4.47 percentage points

over the pretrained baseline.

Results

Experiment	Validation mIoU
Pretrained baseline	71.31%
Full fine-tuning	70.49%
Differential LR with unfrozen BatchNorm	64.32%
Differential LR with frozen BatchNorm	75.78%

The experiments demonstrate that simply fine-tuning all parameters does not necessarily improve a strong pretrained segmentation model.

For the small batch size used in this project, preserving pretrained Batch Normalization statistics was important for stable fine-tuning.

Training Curves

mIoU

Loss

Example Predictions

Each example shows:

Input Image → Ground Truth → Model Prediction

Example 1

Example 2

Example 3

Example 4

Example 5

Example 6

Real-World Inference

The fine-tuned model can also be applied to custom images outside the Pascal VOC validation set using the separate inference script in src/predict.py.

For real-world images, the inference pipeline preserves the image aspect ratio by resizing with padding before prediction and then restores the segmentation mask to the original image dimensions.

Example command:

python src/predict.py \
  --input examples/street_scene.jpg \
  --checkpoint checkpoints_finetune3/best.pt \
  --output results/real_world_inference.png \
  --size 512

Using a larger inference resolution such as 512 can help preserve smaller objects in real-world street scenes.

Example real-world prediction

Project Structure

Python-proj/
│
├── src/
│   ├── train.py
│   └── predict.py
│
├── examples/
│   └── street_scene.jpg
│
├── results/
│   ├── loss_curve.png
│   ├── miou_curve.png
│   ├── real_world_inference.png
│   │
│   └── predictions/
│       ├── sample_0_1.png
│       ├── sample_0_2.png
│       ├── sample_1_1.png
│       ├── sample_1_2.png
│       ├── sample_2_1.png
│       └── sample_2_2.png
│
├── README.md
├── requirements.txt
└── .gitignore

Large files such as datasets, model checkpoints, log files, and virtual environments are excluded from the repository.

Installation

Clone the repository:

git clone <repository-url>
cd <repository-name>

Create a Python virtual environment:

python3 -m venv venv

Activate it on Linux/macOS:

source venv/bin/activate

Then install the required packages:

pip install -r requirements.txt

GPU Acceleration

Training with CUDA is strongly recommended.

The final experiments in this project were executed with an NVIDIA GPU using CUDA-enabled PyTorch.

You can verify whether PyTorch detects your GPU with:

python -c "import torch; print(torch.cuda.is_available())"

and display the GPU name with:

python -c "import torch; print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'No CUDA GPU')"

The training script automatically selects CUDA when it is available:

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

Training

Run the training script with:

python src/train.py

The default configuration uses:

Epochs:                 5
Batch size:             2
Image size:             256 × 256
Backbone learning rate: 1e-6
Classifier learning rate: 1e-5
Weight decay:           1e-4

Batch Normalization layers are frozen during fine-tuning.

Inference

Run semantic segmentation on a custom image with:

python src/predict.py \
  --input path/to/image.jpg \
  --checkpoint path/to/best.pt \
  --output prediction.png \
  --size 512

The script automatically uses CUDA when available and prints the detected Pascal VOC foreground classes.

Running Training in the Background

On a remote Linux workstation, training can be run with nohup:

nohup python -u src/train.py > training.log 2>&1 &

Training progress can then be monitored with:

tail -f training.log

Evaluation Metric

The main evaluation metric is mean Intersection over Union (mIoU).

IoU measures the overlap between the predicted segmentation region and the corresponding ground-truth region for each class.

The IoU values are averaged across the valid classes to obtain mIoU.

Higher mIoU indicates better segmentation performance.

Key Observation

One of the main findings of this project was the importance of Batch Normalization behavior during transfer learning with small mini-batches.

When BatchNorm statistics were updated with a batch size of only 2, validation performance deteriorated substantially despite the use of conservative learning rates.

Freezing the BatchNorm layers while continuing to optimize the backbone and segmentation classifier increased validation mIoU from the 71.31% pretrained baseline to 75.78%.

This highlights the importance of training configuration, rather than simply increasing the number of fine-tuning epochs, when adapting pretrained deep-learning models to a new training setup.

Technologies

Python
PyTorch
Torchvision
NumPy
Matplotlib
CUDA
DeepLabV3
ResNet-50
Pascal VOC 2012

Name		Name	Last commit message	Last commit date
examples		examples
results		results
src		src
.gitignore		.gitignore
README.md		README.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation