Experimental project for DT2119 Speech and Speaker Recognition at KTH Royal Institute of Technology. Graded A.
Authors: Matteo Camellini, Shadman Hassan, Grace Xu.
Full write-up: report/Evaluating_Classical_Preprocessing_for_Neural_Speaker_Diarization.pdf
Modern neural diarizers such as pyannote/speaker-diarization-3.1 chain a segmentation network, a speaker-embedding network and agglomerative clustering. They work well on clean audio and degrade sharply under two conditions: overlapping speech, which contaminates the frames embeddings are extracted from, and background noise, which reduces embedding discriminability.
The intuitive fix is to clean the audio first. This project tests that intuition directly: take two classical signal-processing front-ends, put them in front of an unmodified neural baseline, and measure whether diarization improves.
The answer is largely no. The artefacts a classical front-end introduces can hurt a modern neural pipeline more than the acoustic corruption they remove.
| Condition | Baseline DER | With pre-processing | Change |
|---|---|---|---|
| Set A clean control | 20.4 ± 9.0 % | (no pre-processing) | reference floor |
| Set B 10 % overlap | 25.5 ± 15.0 % | 53.7 ± 13.5 % | +28.2 pp |
| Set B 20 % overlap | 25.0 ± 11.1 % | 48.7 ± 12.7 % | +23.7 pp |
| Set B 30 % overlap | 23.1 ± 9.1 % | 41.9 ± 10.4 % | +18.7 pp |
| Set C 5 dB SNR | 33.2 ± 12.5 % | 29.2 ± 10.7 % | −4.0 pp |
| Set C 10 dB SNR | 31.2 ± 13.2 % | 37.2 ± 15.0 % | +6.0 pp |
| Set C 15 dB SNR | 35.8 ± 16.2 % | 30.3 ± 12.5 % | −5.5 pp |
Across all thirteen condition-by-pipeline cells, no pre-processed condition reaches the 20.4 % clean-speech floor.
DER changes are small and non-monotonic in SNR: better at 5 dB and 15 dB, worse at 10 dB. The component breakdown identifies the mechanism. At 5 and 15 dB, Speaker Confusion falls sharply after cleaning (18.0 to 11.4 %, and 19.5 to 13.4 %), consistent with denoising restoring spectral detail the embeddings depend on. At 10 dB, Speaker Confusion rises from 16.0 to 21.1 % while False Alarm stays near zero.
That last detail is the interesting one. The failure is not musical-noise artefacts triggering false speech, which would show up as False Alarm. It is spectral distortion pushing embeddings further apart in the embedding space than the original noise did. The likely reason is a training-distribution mismatch: WeSpeaker embeddings are trained with heavy augmentation using realistic additive noise (MUSAN, RIR), which makes them robust to noise but leaves them untested against spectral-subtraction artefacts. The cleaned signal lands outside the training distribution in a way the noisy original does not.
PESQ corroborates this: 1.21 to 1.67 on a 0 to 4.5 scale. The cleaned audio has better broadband SNR yet remains perceptually degraded, and that residual degradation is what the diarizer responds to.
The separation pipeline degrades DER at every overlap level, and the breakdown localises the blame precisely. False Alarm rises from ~0 % in the baseline to 20 to 27 %, dwarfing every other component and accounting for roughly half the total inflation.
Since this pipeline has no embedding stage, the diarization decision is taken entirely by an energy-threshold VAD on the two separated tracks. So False Alarm has exactly one interpretation: residual cross-track energy survives suppression and triggers spurious activations.
Meanwhile NMF separation quality is flat at ~3.6 dB SNDR across all overlap levels. The tracks contain recognisable speech. The separation works; the downstream energy-based VAD cannot distinguish target speech from residual leakage.
Two structural problems compound it. The frame-level mutual-exclusion stage encodes a hard one-speaker-per-frame assumption, which is fundamentally incompatible with the overlap it is meant to handle: simultaneous speech is forcibly assigned to whoever is louder, and the resulting Missed Speech duly climbs from 5.6 % at 10 % overlap to 11.9 % at 30 %. And that stage is memoryless, evaluating each frame independently, so temporal continuity is left entirely to post-hoc gap-bridging. The report proposes a two-state HMM over the per-frame energy ratio, which would absorb residual leakage as transient observation noise rather than reading it as a turn boundary.
Partitioning every condition by speaker-pair gender, same-gender DER exceeds cross-gender DER in 11 of 13 conditions, averaging 9.1 pp, and reaching +20.8 pp on Set B 10 % baseline. The gap appears even on the clean Set A control (+9.5 pp), which indicates an embedding-level property rather than something induced by the stress conditions: same-gender embeddings sit closer together, so clustering struggles to separate them. Neither pre-processing pipeline reduces it.
70 two-speaker conversations synthesised from LibriSpeech test-clean, with stationary noise from MS-SNSD. Synthesis rather than real conversation is a deliberate choice: it yields frame-exact ground truth, which is what makes strict scoring meaningful.
| Set | Variable | Levels | Files |
|---|---|---|---|
| A | clean control | none | 10 |
| B | overlap percentage | 10 %, 20 %, 30 % | 30 |
| C | signal-to-noise ratio | 5, 10, 15 dB | 30 |
Each file has 4 to 6 turns of 3 to 8 s separated by 0.2 to 0.8 s of silence, 16 kHz mono, with pairs stratified roughly half cross-gender and half same-gender. Set C is prefixed by a 0.5 s pure-noise lead-in to seed the noise estimator. Noise types are restricted to genuinely stationary categories (air conditioner, copy machine, vacuum, washer, low hum) so that Spectral Subtraction's time-invariance assumption is not violated by construction. Each file ships with an RTTM ground truth, with overlapping segments written as temporally co-active entries, and a UEM defining the scored region. The generator is deterministic given one seed.
Spectral Subtraction (Set C) estimates a noise profile and subtracts it from the magnitude spectrogram with a floor to limit musical noise:
|Ŝ(f,t)| = max( |X(f,t)| − α·N̂(f) , β·|X(f,t)| ) α = 1.5, β = 0.02
The noise profile is averaged over the 10 % quietest frames ranked by dB power, an adaptive estimator that replaces the classical leading-silence scheme, which breaks whenever speech starts during the lead-in. Original phase is recombined and the signal inverted by ISTFT. The cleaned waveform then goes to the unmodified baseline.
NMF + VAD (Set B) factors the magnitude spectrogram into a rank-2 product, reconstructs each component by Wiener-style soft masking and ISTFT, applies frame-level mutual exclusion to suppress crosstalk, then runs an independent RMS-threshold VAD per track (3x the 15th percentile, adaptive to absolute amplitude) with 200 ms gap-bridging and a 300 ms minimum activation. Unlike Spectral Subtraction, this pipeline replaces the neural baseline rather than feeding it.
Rank 2 is a deliberate restriction. Standard NMF speech separation uses 20 to 100 components with downstream clustering; rank 2 keeps the pipeline fully blind and in the spirit of the classical methods under test. The report argues a higher rank would not have rescued it, since the dominant failure is the VAD, not the factorisation.
DER and its decomposition into False Alarm, Missed Speech and Speaker Confusion via pyannote.metrics, with Hungarian-algorithm label mapping. Two choices make the numbers harsher and more informative:
- Collar = 0 s. No forgiveness window. Justified because the ground-truth boundaries are exact by construction, and any collar would mask precisely the small errors the experiment exists to surface.
skip_overlap = False. Overlap regions are scored fully, since overlap handling is the explicit object of study.
PESQ (wideband) measures Set C cleaned audio against the clean reference; SNDR measures the higher-energy NMF track against the original mixture. Together with the DER components these trace a causal chain from raw audio through pre-processing to the diarization decision.
All hyperparameters were fixed before the final run, with no per-condition tuning.
generate_diarization_datasets.py synthesises Sets A/B/C with RTTM and UEM ground truth
run_baseline_diarizer.py unmodified pyannote 3.1 over every set
noise_pipeline.py Spectral Subtraction front-end (Set C)
nmf_pipeline.py rank-2 NMF, Wiener masking, mutual exclusion, VAD (Set B)
evaluate_der.py DER, component breakdown, PESQ, SNDR
analyze_speaker_pairs.py cross-gender against same-gender partition
generate_graphs.py the ten report figures
run_full_experiment.py orchestrator: runs the whole pipeline end to end
all_csvs/ per-condition DER tables with per-file rows plus MEAN/STD
baseline_predictions/ RTTM output, unmodified baseline
nmf_predictions/ RTTM output, NMF + VAD
spectral_predictions/ RTTM output, Spectral Subtraction then baseline
graphs/ the ten figures, plus speaker_pair_analysis.csv
report/ the paper
Every DER table carries per-file rows with MEAN and STD appended, so any number quoted above can be traced back to the ten files that produced it.
pip install pyannote.audio==3.3.2 torch torchaudio librosa scikit-learn scipy \
soundfile numpy pandas matplotlib pesq rich "huggingface_hub<1.0"
python generate_diarization_datasets.py # needs LibriSpeech test-clean and MS-SNSD
python run_full_experiment.py --hf_token <YOUR_TOKEN>The orchestrator runs NMF + VAD on Set B, Spectral Subtraction on Set C, re-diarises the cleaned audio, evaluates every condition, and emits the figures. Each step runs as a subprocess so a failed stage can be re-run in isolation from the logged command. The full run takes roughly an hour on a CPU-only workstation.
Not included in this repository: the LibriSpeech and MS-SNSD source corpora, and the generated audio for the synthetic sets and the two pre-processing front-ends. These are large and reproducible from the scripts. The corpora are publicly available from OpenSLR and MS-SNSD.
A HuggingFace token with access to the gated pyannote/speaker-diarization-3.1 weights is required. Pass it on the command line or set HF_TOKEN; never commit it.
The Python files have been stripped of all comments and docstrings. The method, the hyperparameters and the reasoning behind each design choice are documented in this README and, in full, in the report.
Stated in the report and worth repeating. Ten files per condition limits statistical power, and several reported differences have standard deviations comparable to the effect size, so these are indicative rather than significant. The synthesised LibriSpeech audio is read speech without natural conversational dynamics, so absolute DER values are not comparable to AMI or DIHARD. Spectral Subtraction was tested only on stationary noise, the regime where its assumptions hold.