Extracting building footprints from high-resolution aerial imagery of Dubai — by running self-attention in the frequency domain instead of over pixels.
An implementation and evaluation of FsaNet (Zhang, Panahi & Gao, IEEE Transactions on Image Processing, 2023) integrated into a Mask R-CNN with a ResNet-50 FPN backbone, benchmarked against a standard spatial-attention baseline.
Headline result: attending over 256 frequency coefficients instead of 65,536 pixels produced a model that was both cheaper and more accurate — final training loss fell from 0.8489 to 0.7666, and the over-segmentation artefacts that plagued the baseline disappeared.
- Why building segmentation
- The problem with spatial self-attention
- The frequency-domain insight
- Implementation
- Results
- Why the smaller model won
- Repository structure
- Getting started
- Limitations
- References
Precise extraction of building footprints from aerial maps underpins urban planning, infrastructure assessment, and site selection — including identifying viable warehouse locations for automated dark stores in dense urban areas like Dubai.
Good segmentation needs two things at once: internal consistency, so a building is recognised as one coherent region, and boundary precision, so its edges are sharp. These pull in opposite directions, which is where the architecture gets interesting.
Convolutional networks capture local texture well but struggle with long-range dependencies, because their receptive fields are inherently local. Self-attention fixes that by letting every pixel attend to every other pixel:
O = V · Softmax(Kᵀ Q)
Two problems follow.
Cost. Computing pairwise interactions across all pixels is O(N²) in the number of spatial tokens. At high resolution this becomes prohibitive — a hard blocker for deployment on constrained hardware.
Indiscriminate processing. Standard self-attention applies identical learnable parameters to every frequency component. But the inner body of an object corresponds to low-frequency content and its edges to high-frequency content. Treating both the same way means promoting regional consistency and preserving sharp boundaries pull against each other, and one degrades as the other improves.
Transform the feature map into the frequency domain with the Discrete Cosine Transform, discard the high-frequency components, and run attention only over what remains:
N = H × W → k² where k ≪ H, W
With k = 16, attention operates over a 16×16 block of low-frequency coefficients — 256 values instead of 65,536 — before the network reconstructs spatial features from the processed coefficients via inverse DCT.
The attention computation is reformulated over the low-frequency representation f′:
O′ = (Wᵥf′)(W_k f′)ᵀ (W_q f′) / (HW)
The counterintuitive part is that throwing away that much data improved accuracy. See Why the smaller model won.
The dataset is high-resolution aerial imagery of Dubai (Humans in the Loop, Kaggle). Standard architectures can't ingest full satellite tiles, so an automated OpenCV pipeline handles preparation:
- Recursive directory traversal across the raw tile set
- Slicing into uniform 256×256 patches
- Binary mask generation per patch, isolating the building class by BGR colour threshold on the dark-purple building annotation (
RGB 152, 16, 60) with ±15 tolerance to absorb JPEG compression artefacts
Mask R-CNN with a ResNet-50 Feature Pyramid Network backbone in PyTorch, initialised from pre-trained weights, trained for 5 epochs with Adam. This establishes the control for the ablation — both segmentation accuracy and GPU processing time.
The frequency self-attention module is injected into Layer 4, the final block of the ResNet-50 backbone. A sequence of reshape, 2D-DCT and 2D-IDCT operations is expressed as a linear map so the transform composes cleanly with the rest of the network, replacing the computationally intensive spatial attention with a lightweight spectral alternative.
| Model | Epoch 1 loss | Final loss |
|---|---|---|
| Mask R-CNN baseline (spatial attention) | 1.2783 | 0.8489 (epoch 4) |
| Mask R-CNN + FsaNet (spectral attention) | 1.2646 | 0.7666 (epoch 5) |
The spectral model converged faster and further despite a severe reduction in input dimensionality.
On a test patch containing exactly two rectangular structures:
- Spatial baseline — severe over-segmentation. Misclassified surrounding shadows and ground texture as building infrastructure, predicting four structures and bleeding well beyond true architectural boundaries.
- FsaNet — identified exactly two buildings, with footprints aligned to the ground-truth geometry and the deceptive background textures ignored entirely.
This is the part worth understanding, and it isn't obvious.
Spatial attention forces the network to compute gradients over everything, including high-frequency background noise — road texture, sand, shadow edges. In desert aerial imagery that noise is abundant and actively misleading, which is exactly what produced the baseline's phantom buildings.
Stripping the high frequencies didn't just make the model cheaper. It removed the signal that was confusing it. The network received a cleaner input and learned the structural information that actually matters for building footprints more efficiently than the baseline did with full information.
Less data, better result — because the discarded data was mostly noise.
Digital_Image_Processing_FSAnet/
├── fsanet.py # Frequency self-attention module: DCT, low-frequency selection, IDCT
├── model.py # Mask R-CNN + ResNet-50 FPN assembly, FsaNet injection into Layer 4
├── preprocessing.py # OpenCV tile slicing and binary mask generation
├── train.py # Training loop, Adam optimiser, loss tracking
├── predict.py # Inference and visualisation of predicted masks
├── dip-training.ipynb # Training notebook with epoch-by-epoch loss output
├── dataset/ # Dubai semantic segmentation dataset
└── images/ # Baseline vs FsaNet comparison figures
- Python 3.9+
- PyTorch with CUDA (training on CPU is impractical)
git clone https://github.com/mayaalkhzaee/Digital_Image_Processing_FSAnet.git
cd Digital_Image_Processing_FSAnet
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txtDownload the aerial imagery dataset into dataset/, then slice it into patches:
python preprocessing.pypython train.py --model baseline # spatial attention control
python train.py --model fsanet # spectral attention, k=16python predict.py --checkpoint checkpoints/fsanet.pt --image path/to/patch.pngStated plainly, because they bound what the results show:
- Training loss, not validation IoU. The comparison tracks training loss over 5 epochs. A full evaluation would report mean IoU and mask AP on a held-out set; these numbers demonstrate convergence behaviour, not generalisation.
- 5 epochs. Short by segmentation standards. The trend is consistent and the qualitative gap is visible, but neither model was trained to saturation.
- Single-class. Only the building class is segmented; the source dataset carries additional semantic classes that are discarded during mask generation.
- k = 16 is unablated. The low-frequency cutoff was fixed rather than swept, so the accuracy-versus-compression curve is uncharacterised.
- Computational gains are argued, not benchmarked. The token reduction from 65,536 to 256 at the attention stage is structural, but wall-clock and memory profiling would be needed to quantify the end-to-end saving.
[1] Gonzalez, R. C. & Woods, R. E. (2018). Digital Image Processing, 4th ed. Pearson. [2] Long, J., Shelhamer, E. & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. CVPR. [3] Ronneberger, O., Fischer, P. & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI. [4] Wang, X., Girshick, R., Gupta, A. & He, K. (2018). Non-local Neural Networks. CVPR. [5] Rao, K. R. & Yip, P. (1990). Discrete Cosine Transform: Algorithms, Advantages, Applications. Academic Press. [6] Zhang, F., Panahi, A. & Gao, G. (2023). FsaNet: Frequency Self-attention for Semantic Segmentation. IEEE TIP. arXiv:2211.15595 [7] Humans in the Loop (2018). Semantic segmentation of aerial imagery. Kaggle.
Attribution: The frequency self-attention mechanism is the work of Zhang, Panahi & Gao (2023). This repository implements their method, integrates it into a Mask R-CNN pipeline, and independently evaluates their claim that low-frequency self-attention can match or exceed full-frequency spatial attention.