WHAM! Extensions: Enhancing Noisy Speech Separation

• WHAM! extensions are datasets and methods that expand upon the original corpus by incorporating authentic ambient noise and room reverberation to simulate real-world conditions.
• They employ advanced techniques such as image-source modeled reverberation and permutation-invariant training, ensuring robust benchmarks for single-channel speech separation.
• These extensions drive innovation in deep speech separation, significantly reducing overfitting while improving model generalization and evaluation across diverse acoustic scenarios.

The term “WHAM! extension” refers to datasets, methods, and analytic expansions that build upon the original WSJ0 Hipster Ambient Mixtures (WHAM!) single-channel speech separation corpus, as well as other domain-specific extensions of the acronym WHAM. The most prominent usage pertains to speech separation datasets, in which WHAM! augments WSJ0-2mix mixtures with real ambient noise to more realistically simulate everyday noisy environments. Subsequent extensions, such as WHAMR!, LibriMix, and analytic methods in other domains, broaden WHAM!’s scope by enhancing acoustic realism, generalization, or algorithmic methodology.

1. WHAM!: Noisy Speech Separation Dataset

WHAM! (“WSJ0 Hipster Ambient Mixtures”) is a dataset designed to benchmark single-channel speech separation algorithms under realistic noisy conditions by adding real-world ambient environmental noise to the WSJ0-2mix corpus (Wichern et al., 2019). Ambient noise samples—collected in cafés, bars, and similar venues—are combined with two-speaker WSJ0-2mix utterances at SNRs spanning [–6, +3] dB, across four SPL bins (20–70 dB(A)). Preprocessing removes foreground speech and calibrates loudness. Table 1 summarizes data splits:

Split	#Mixtures	Speech (h)	Noise (h)
Train	20,000	30	≈30
Val	5,000	10	≈10
Test	3,000	5	≈5

Mixtures are provided in both “min” (truncated) and “max” (unaltered) versions at 8 kHz and 16 kHz. WHAM! retains the WSJ0-2mix inter-speaker SNRs and ensures no speaker overlap between train/val/test.

2. Motivation for Extensions Beyond WHAM!

While WHAM! advanced realistic speech separation by introducing authentic noise, it exhibits key limitations (Cosentino et al., 2020):

• Restricted speech pool (~30 hours, ~100 speakers, single recording setting), leading to poor generalization to new voices and conditions.
• Fully overlapping mixtures (~100% overlap) are not representative of conversational speech (with typical overlaps ≤20%).
• Speaker identities are repeated across splits, increasing overfitting risk.

These limitations motivated the development of “WHAM! extensions,” aiming to improve generalization, realism, and methodological rigor.

3. Principal Dataset Extensions: WHAMR! and LibriMix

WHAMR!: Incorporating Room Reverberation

The WHAMR! dataset extends WHAM! by synthesizing room reverberation for each speaker using image-source modeled RIRs (room impulse responses) with T₆₀ distributed between 0.1 and 1.0 seconds, over realistic room geometries and source-microphone distances (Maciejewski et al., 2019). The construction procedure is:

• Reverberate each speaker utterance with its own RIR.
• Add noise from WHAM! at realistic SNRs (–6 to +3 dB).
• Provide ground-truth clean, anechoic, and reverberant references.
• Dataset statistics: 20,000 train, 5,000 val, 3,000 test mixtures, both at 8 kHz and 16 kHz, identical splits as WHAM!.

Baseline evaluations using Conv-TasNet and TasNet-BLSTM demonstrate that reverberation causes significantly greater SI-SDR performance degradation than additive noise, and that multi-stage cascaded denoise/separate/dereverb architectures yield up to ~1 dB additional SI-SDR improvement over single-stage models.

LibriMix: Expanding Speaker and Acoustic Diversity

LibriMix broadens the speech pool by sourcing utterances from LibriSpeech—including >1,200 speakers and ~470 hours—while retaining WHAM! noise (Cosentino et al., 2020). LibriMix supports two- and three-speaker mixtures, both clean and noisy, plus sparse-overlap (20%, 40%, …, 100%) and VCTK-based cross-dataset evaluation. Mixtures are normalized in loudness (speech: –33 to –25 LUFS; noise: –38 to –30 LUFS). Notably, training data is constructed to avoid utterance repetition, addressing overfit risks.

Empirically, models trained on LibriMix generalize markedly better to unseen speakers, conditions, and the VCTK-2mix set, compared to those trained on WHAM!. The cross-dataset performance gap is reduced by about 3–4 dB SI-SDR, and larger training sets (LibriMix-train-360 vs. train-100) correlate with further improved generalization.

4. Evaluation Protocols and Loss Functions

WHAM! extensions standardize the evaluation procedure:

• Permutation-invariant, scale-invariant signal-to-distortion ratio (SI-SDR) is the principal metric,

$$\mathrm{SI\!-\!SDR}(s,\hat s) =10\log_{10}\frac{\|\alpha\,s\|^2}{\|\alpha\,s-\hat s\|^2},\quad\alpha = \frac{\langle \hat s, s\rangle}{\|s\|^2}$$

augmented by SDR and perceptual (PESQ) scores as required.

• Models (e.g., Chimera++, TasNet-BLSTM, Conv-TasNet) are benchmarked in “min” and “max” trimming regimes at 8 kHz and 16 kHz.
• Loss functions include tPSA, deep clustering objectives (classic and whitened), and PIT (Permutation Invariant Training) to handle source assignment ambiguity.

Performance degrades when moving from clean to noisy or reverberant conditions, with learned-basis architectures outperforming STFT-based models especially for dereverberation tasks.

5. Impact on Deep Speech Separation Research

The progression from WSJ0-2mix to WHAM!, WHAMR!, and LibriMix has established a more rigorous, ecologically valid set of testbeds for monaural speech separation systems. The inclusion of authentic ambient noise, physically modeled reverberation, greater speaker diversity, and variable overlap scenarios directly challenges separation models to address real-world variability. Cross-dataset benchmarks and sparse-overlap tests expose generalization error, highlighting overfitting and dataset-induced biases.

These extensions have also driven architectural innovation, including mask-based and learned-basis separation networks, cascaded and joint enhancement-separation pipelines, and refined objective functions measuring separation and enhancement quality in difficult “cocktail party” scenarios.

6. Open-Source Accessibility and Future Directions

WHAM! extensions are accompanied by open-source code, reproducible data-preparation scripts, and detailed usage documentation (see e.g., https://github.com/JorisCos/LibriMix; http://wham.whisper.ai) (Cosentino et al., 2020, Maciejewski et al., 2019). This fosters standardization across the community and accelerates methodological advancement.

Future avenues include:

• Exploiting multichannel (stereo/binaural) noise recordings in WHAM! for spatial separation.
• Systematic investigation of architectures tailored for sparse overlap and reverberant-noisy mixtures.
• End-to-end learning with upstream enhancement modules and domain adaptation.
• Extending the analytic methodology to more complex soundscapes and incorporating additional metadata (speaker demography, spatial layout).

7. Summary Table: Major WHAM! Extensions in Speech Separation

Name	Main Additions	Purpose/Outcome	Reference
WHAM!	Real ambient noise	Noisy augmentation of WSJ0-2mix, baseline	(Wichern et al., 2019)
WHAMR!	Reverberation + noise	Models far-field, non-anechoic scenarios	(Maciejewski et al., 2019)
LibriMix	LibriSpeech, overlap	Increased realism & generalization; sparse	(Cosentino et al., 2020)

Collectively, these WHAM! extensions have transformed evaluation standards in monaural speech separation, guiding research toward robust models that better approximate real-world acoustic variability.