FDY-CNN and Masked-Attention Transformer for Sound Event Detection

Sound event detection with FDY-CNN and Masked-Attention Transformer.

This repository extends the original Transformer4SED codebase with:

Support for Urban-SED dataset (Salamon et al., 2017) and DataSED dataset (Fredianelli et al., 2025) in addition to the original DESED dataset
Reproducible multi-seed training (seeds 21, 42, 123) via SLURM job scripts
Hyperparameter optimization via Optuna TPE sampler (CNN-SED: 20 trials, MAT-SED: 10 trials)
Statistical comparison (paired t-test) between baseline and tuned configurations
Evaluation with PSDS (Scenario 1 and 2), event-based F1, and segment-based F1

This repository accompanies the master's thesis: "Comparative Analysis of Deep Learning Architectures for Audio Tagging and Sound Event Detection" Yu Chia Kuo, McGill University, 2026.

Datasets

Dataset	Classes	Split	Notes
Urban-SED	10	train / val / test (official)	Synthesised urban soundscapes (Salamon et al., 2017)
DataSED	21	60 / 20 / 20 (seed=42)	Real-world environmental noise (Fredianelli et al., 2025); recording-level stratified split

Model Architectures

Model	Encoder	Context Network	Pretrain	Training
CNN-SED	Frequency Dynamic CNN (FDY-CNN, 7 layers, 128-dim)	TransformerXL (3 layers)	None	finetune1 (15 ep) + finetune2 (30 ep)
MAT-SED	PaSST (12 blocks, 768-dim)	TransformerXL (3 layers)	MLM reconstruction	pretrain (15 ep) + finetune1 (15 ep) + finetune2 (30 ep)

Both models use the mean-teacher framework with consistency regularization. The student model is reported (DCASE convention).

Repository Structure

Transformer4SED/
├── config/
│   ├── cnn-sed/{urban_sed,datased}/    # YAML configs per seed and stage
│   └── mat-sed/{urban_sed,datased}/    # YAML configs per seed and stage
├── recipes/desed/
│   ├── finetune/cnn_trans/main.py      # CNN-SED entry point
│   ├── finetune/passt/main.py          # MAT-SED entry point
│   ├── finetune/train.py              # Shared Trainer
│   └── mlm/mlm_passt/main.py          # MAT-SED pretrain entry point
├── src/
│   ├── models/passt/                   # PaSST encoder
│   ├── models/cnn/                     # FDY-CNN encoder
│   ├── preprocess/                     # Dataset classes, data loading
│   └── utils/                          # Logger, metrics
├── meta/
│   ├── urban_sed/                      # Label dict, TSV metadata
│   └── datased/                        # Label dict, TSV metadata
├── slurm/                              # SLURM submission scripts
├── optuna/                             # Optuna search scripts and results
└── requirements.txt

Requirements

torch>=1.13.0
torchaudio>=0.13.0
pytorch-lightning>=2.0.0
sed_scores_eval>=0.0.1
psds_eval>=0.5.0
tensorboard
optuna>=3.0.0
pyyaml
numpy
scipy
pandas
soundfile
librosa
h5py
tqdm

Setup

Create and activate a Python 3.11 virtual environment.
Install dependencies:

   pip install -r requirements.txt

Copy and edit the local environment file:

   cp env.sh.example env.sh
   # Edit PROJECT_ROOT, VENV_PATH, DATASET_DIR
   source env.sh

Prepare datasets at data/:

   data/
   ├── URBAN-SED_v2.0.0/audio_32k/{train,validate,test}/
   └── DataSED/audio_32k/{train,validate,test}/

Usage

Training

source env.sh

# CNN-SED Urban-SED (seed 21)
sbatch slurm/slurm_cnnsed_urbansed.sh

# MAT-SED DataSED (pretrain -> ft1 -> ft2 dependency chain)
PRE=$(sbatch --parsable slurm/slurm_matsed_datased_pretrain.sh)
FT1=$(sbatch --parsable --dependency=afterok:$PRE slurm/slurm_matsed_datased_ft1.sh)
sbatch --dependency=afterok:$FT1 slurm/slurm_matsed_datased_ft2.sh

Hyperparameter Search (Optuna)

# CNN-SED: 20 trials, ft1+ft2, Urban-SED val PSDS1
sbatch optuna/run_optuna_cnn.sh

# MAT-SED: 10 trials, ft2-only from ft1 checkpoint, Urban-SED val PSDS1
sbatch optuna/run_optuna_mat.sh

Results

All results are student model, test set, reported as mean ± std across 3 seeds (21, 42, 123). CNN-SED reports Tuned (adopted via VAL paired t-test). MAT-SED reports Baseline (Optuna showed no improvement).

Urban-SED — CNN-SED Baseline (Student, Test Set)

Metric	Seed 21	Seed 42	Seed 123	Mean ± Std
PSDS1	0.265	0.279	0.277	0.274 ± 0.008
PSDS2	0.395	0.412	0.399	0.402 ± 0.009
Eb F1	0.306	0.302	0.298	0.302 ± 0.004
Sb F1	0.689	0.685	0.695	0.690 ± 0.005

Urban-SED — CNN-SED Tuned (Student, Test Set)

Metric	Seed 21	Seed 42	Seed 123	Mean ± Std
PSDS1	0.279	0.292	0.280	0.284 ± 0.007
PSDS2	0.405	0.396	0.398	0.400 ± 0.005
Eb F1	0.303	0.308	0.314	0.308 ± 0.006
Sb F1	0.694	0.699	0.698	0.697 ± 0.003

Decision: ADOPT OPTUNA. VAL paired t-test p = 0.125 (n.s.), Δ PSDS1 = +0.016 ≥ 0.01, Cohen's d = 1.475.

Urban-SED — MAT-SED Baseline (Student, Test Set)

Metric	Seed 21	Seed 42	Seed 123	Mean ± Std
PSDS1	0.384	0.379	0.376	0.380 ± 0.004
PSDS2	0.503	0.510	0.512	0.508 ± 0.005
Eb F1	0.397	0.395	0.389	0.394 ± 0.004
Sb F1	0.746	0.742	0.745	0.744 ± 0.002

Urban-SED — MAT-SED Tuned (Student, Test Set)

Metric	Seed 21	Seed 42	Seed 123	Mean ± Std
PSDS1	0.381	0.376	0.379	0.379 ± 0.003
PSDS2	0.507	0.520	0.522	0.516 ± 0.008
Eb F1	0.396	0.382	0.390	0.389 ± 0.007
Sb F1	0.746	0.740	0.745	0.744 ± 0.003

Decision: KEEP BASELINE. Urban-SED VAL t-test p = 0.101 (n.s.), DataSED VAL t-test p = 0.944 (n.s.). No improvement on either dataset.

DataSED — CNN-SED Baseline (Student, Test Set)

Metric	Seed 21	Seed 42	Seed 123	Mean ± Std
PSDS1	0.311	0.326	0.301	0.313 ± 0.013
PSDS2	0.506	0.500	0.479	0.495 ± 0.014
Eb F1	0.115	0.123	0.126	0.121 ± 0.006
Sb F1	0.622	0.623	0.626	0.624 ± 0.002

DataSED — CNN-SED Tuned (Student, Test Set)

Metric	Seed 21	Seed 42	Seed 123	Mean ± Std
PSDS1	0.327	0.350	0.343	0.340 ± 0.012
PSDS2	0.501	0.531	0.505	0.512 ± 0.016
Eb F1	0.122	0.125	0.123	0.123 ± 0.002
Sb F1	0.617	0.639	0.635	0.630 ± 0.012

Decision: ADOPT OPTUNA. VAL paired t-test p = 0.053, Δ = 0.010 ≥ 0.01, Cohen's d = 2.412.

DataSED — MAT-SED Baseline (Student, Test Set)

Metric	Seed 21	Seed 42	Seed 123	Mean ± Std
PSDS1	0.454	0.449	0.461	0.455 ± 0.006
PSDS2	0.674	0.668	0.707	0.683 ± 0.021
Eb F1	0.169	0.162	0.168	0.166 ± 0.004
Sb F1	0.751	0.767	0.777	0.765 ± 0.013

DataSED — MAT-SED Tuned (Student, Test Set)

Metric	Seed 21	Seed 42	Seed 123	Mean ± Std
PSDS1	0.451	0.467	0.452	0.457 ± 0.009
PSDS2	0.685	0.682	0.669	0.679 ± 0.009
Eb F1	0.165	0.166	0.175	0.169 ± 0.006
Sb F1	0.761	0.762	0.741	0.755 ± 0.012

Decision: KEEP BASELINE. VAL paired t-test p = 0.944, Cohen's d = −0.046.

Summary

Model	Config	Urban-SED PSDS1	DataSED PSDS1
CNN-SED	Baseline	0.274 ± 0.008	0.313 ± 0.013
CNN-SED	Tuned	0.284 ± 0.007	0.340 ± 0.012
MAT-SED	Baseline	0.380 ± 0.004	0.455 ± 0.006
MAT-SED	Tuned	0.379 ± 0.003	0.457 ± 0.009

Optimized Hyperparameters

CNN-SED adopts Tuned for both datasets. MAT-SED retains Baseline (Optuna parameters ≈ baseline defaults).

CNN-SED Optuna Search Space and Best Parameters

Parameter	Range	Scale	Baseline	Tuned
enc_lr	[5e-5, 5e-4]	log-uniform	1e-4	3.40e-4
ctx_lr	[5e-5, 5e-4]	log-uniform	1e-4	8.15e-5
w_cons_max	[2, 40]	log-uniform	20	3.45

Search: 20 trials on Urban-SED validation PSDS1 (ft1 + ft2 per trial). No pruner. TPE sampler.

MAT-SED Optuna Search Space and Best Parameters

Parameter	Range	Scale	Baseline	Tuned
enc_lr	[5e-7, 1e-5]	log-uniform	5e-6	4.17e-6
ctx_lr	[5e-5, 5e-4]	log-uniform	1e-4	5.24e-5
w_cons_max	[2, 40]	log-uniform	40	36.55

Search: 10 trials on Urban-SED validation PSDS1 (ft2-only from ft1 checkpoint). No pruner. TPE sampler.

Evaluation Metrics

Metric	Description
PSDS1	Polyphonic Sound Detection Score, Scenario 1 (dtc=0.7, gtc=0.7, cttc=0.3, α_ct=0, α_st=1)
PSDS2	Polyphonic Sound Detection Score, Scenario 2 (dtc=0.1, gtc=0.1, cttc=0.3, α_ct=0.5, α_st=1)
Eb F1	Event-based macro F1 (200 ms collar, 0.2 intersection ratio)
Sb F1	Segment-based macro F1 (1-second resolution)

Based On

Lu, Pengfei, Yiming Yan, Xuenan Zou, Jun Du, and Chin-Hui Lee. 2024. "MAT-SED: A Masked Audio Transformer with Masked-Reconstruction Based Pre-training for Sound Event Detection." In Interspeech 2024.

Original repository: cai525/Transformer4SED

References

Fredianelli, Luca, et al. 2025. "DataSED: A Real-World Environmental Sound Event Detection Dataset." Zenodo. https://zenodo.org/records/15346092.

Lu, Pengfei, Yiming Yan, Xuenan Zou, Jun Du, and Chin-Hui Lee. 2024. "MAT-SED: A Masked Audio Transformer with Masked-Reconstruction Based Pre-training for Sound Event Detection." In Interspeech 2024.

Salamon, Justin, Duncan MacConnell, Mark Cartwright, Peter Li, and Juan Pablo Bello. 2017. "Scaper: A Library for Soundscape Synthesis and Augmentation." In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA).

Citation

@inproceedings{lu2024matsed,
  title={MAT-SED: A Masked Audio Transformer with Masked-Reconstruction Based Pre-training for Sound Event Detection},
  author={Lu, Pengfei and Yan, Yiming and Zou, Xuenan and Du, Jun and Lee, Chin-Hui},
  booktitle={Interspeech},
  year={2024}
}

License

This repository is based on cai525/Transformer4SED, released under the MIT License.

Navigation Menu

Repository files navigation

FDY-CNN and Masked-Attention Transformer for Sound Event Detection

Datasets

Model Architectures

Repository Structure

Requirements

Setup

Usage

Training

Hyperparameter Search (Optuna)

Results

Urban-SED — CNN-SED Baseline (Student, Test Set)

Urban-SED — CNN-SED Tuned (Student, Test Set)

Urban-SED — MAT-SED Baseline (Student, Test Set)

Urban-SED — MAT-SED Tuned (Student, Test Set)

DataSED — CNN-SED Baseline (Student, Test Set)

DataSED — CNN-SED Tuned (Student, Test Set)

DataSED — MAT-SED Baseline (Student, Test Set)

DataSED — MAT-SED Tuned (Student, Test Set)

Summary

Optimized Hyperparameters

CNN-SED Optuna Search Space and Best Parameters

MAT-SED Optuna Search Space and Best Parameters

Evaluation Metrics

Based On

References

Citation

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
config		config
docs		docs
meta		meta
recipes		recipes
scripts		scripts
slurm		slurm
src		src
.gitignore		.gitignore
README.md		README.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

FDY-CNN and Masked-Attention Transformer for Sound Event Detection

Datasets

Model Architectures

Repository Structure

Requirements

Setup

Usage

Training

Hyperparameter Search (Optuna)

Results

Urban-SED — CNN-SED Baseline (Student, Test Set)

Urban-SED — CNN-SED Tuned (Student, Test Set)

Urban-SED — MAT-SED Baseline (Student, Test Set)

Urban-SED — MAT-SED Tuned (Student, Test Set)

DataSED — CNN-SED Baseline (Student, Test Set)

DataSED — CNN-SED Tuned (Student, Test Set)

DataSED — MAT-SED Baseline (Student, Test Set)

DataSED — MAT-SED Tuned (Student, Test Set)

Summary

Optimized Hyperparameters

CNN-SED Optuna Search Space and Best Parameters

MAT-SED Optuna Search Space and Best Parameters

Evaluation Metrics

Based On

References

Citation

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages