WSJ0-2mix Dataset for Speech Separation

• WSJ0-2mix is a benchmark dataset for single-channel two-speaker speech separation, created by algorithmically mixing WSJ0 corpus utterances.
• The dataset is divided into training, validation, and test splits, and employs evaluation metrics like SI-SDR, SDR, and PESQ to gauge performance.
• Variants such as WSJ0-2mix-extr and Ambisonic WSJ0-2mix expand its utility, supporting research in spatialization and robust deep learning separation models.

The WSJ0-2mix dataset is the de facto standard benchmark for single-channel, two-speaker speech separation. It is constructed by algorithmically mixing utterances from the Wall Street Journal corpus (WSJ0), yielding highly controlled but challenging fully-overlapping speech mixtures for assessing supervised speech separation algorithms. The dataset’s widespread adoption underpins major methodological and architectural advances in deep learning-based source separation and is foundational for reproducible comparison of algorithms via metrics such as SI-SDR, SDR, and PESQ.

1. Dataset Construction and Protocol

WSJ0-2mix is synthesized from the single-speaker, read-speech WSJ0 corpus (LDC93S6A), wherein each audio file contains an individual reading a newspaper sentence. The dataset is split into training (20,000 mixtures, ≈30 hours), validation (5,000 mixtures, ≈10 hours), and test (3,000 mixtures, ≈5 hours), with no overlap of speakers across these splits (Wichern et al., 2019, Scheibler et al., 2022, Subakan et al., 2020, Libera et al., 2022, Xu et al., 2022, Zeng et al., 2022, Herzog et al., 2022, Jepsen et al., 20 Aug 2025).

Each mixture is a sum of exactly two utterances, selected at random from different speakers. The relative level or signal-to-noise ratio (SNR) between the sources is sampled uniformly—most commonly in [0, 5] dB (Wichern et al., 2019, Subakan et al., 2020, Xu et al., 2022), but some recipes, including (Scheibler et al., 2022), use SNR $\in [-5, 5]$ dB. The equation for mixture generation is

$x(t) = s_1(t) + s_2(t),$

where $s_1, s_2$ are rescaled utterances. Fully-overlapped mixing (“min” version) is canonical, achieved by truncating the longer utterance to the duration of the shorter, enforcing near-total synthentic overlap. The signals are downsampled to 8 kHz and quantized at 16 bits per sample. No external noise or reverberation is added (Wichern et al., 2019, Xu et al., 2022); the only non-anechoic content in the references stems from the original WSJ0 studio recordings—which include minor air-conditioning, microphone noise, and low-level ambient sound (Jepsen et al., 20 Aug 2025).

2. Dataset Variants, Spatialization, and Augmentation

Several closely related variants expand the utility of WSJ0-2mix:

• WSJ0-2mix-extr: Here, both source signals for every mixture are retained, and the SNR sampling and pair matching follow the WSJ0-2mix protocol. Sampling rate is 8 kHz (Zeng et al., 2022).
• Ambisonic WSJ0-2mix: Clean two-speaker WSJ0-2mix mixtures are convolved with simulated first-order Ambisonic room impulse responses to yield spatialized versions for evaluation of multichannel and reverberant speech separation algorithms (Herzog et al., 2022).
• Noisy/augmented WSJ0-2mix: The WHAM! and Enhanced+WHAM! extensions introduce real ambient noise or denoised references to study robustness and breakdown the coupling between SI-SDR and reference noise memorization (Wichern et al., 2019, Jepsen et al., 20 Aug 2025).

3. Data Preprocessing and Feature Representation

The dataset is provided as raw time-domain mixtures and clean reference signals. Separation systems either operate directly in the waveform domain (e.g., via convolutional or TasNet-style encoders (Subakan et al., 2020, Libera et al., 2022, Xu et al., 2022)) or apply learnable time-frequency transformations (STFT-based or hybrids as in DiffSep (Scheibler et al., 2022)). No global normalization or data augmentation is performed in classic recipes, though more recent work uses dynamic mixing—that is, randomizing mixture generation on-the-fly during training—to expose models to a broader diversity of mixtures (Libera et al., 2022, Subakan et al., 2020).

In spatialized or reverberant variants, each WSJ0-2mix mixture is convolved separately for each channel with computed or measured room impulse responses, and the resulting signal is recast via Ambisonic encoding for use in multichannel neural architectures (Herzog et al., 2022).

4. Evaluation Metrics and Training Objectives

WSJ0-2mix evaluations typically employ scale-invariant SDR (SI-SDR) improvement (SI-SDRi), defined as the increment in SI-SDR over the unprocessed mixture, along with BSS_EVAL SDR improvement (SDRi), Perceptual Evaluation of Speech Quality (PESQ), ESTOI (Extended Short-Time Objective Intelligibility), and non-intrusive metrics such as DNSMOS or NISQA.v2 (Scheibler et al., 2022, Xu et al., 2022, Jepsen et al., 20 Aug 2025). For two-source mixtures, models are evaluated under permutation invariant training/inference (PIT/uPIT) to account for label ambiguity (Subakan et al., 2020, Xu et al., 2022, Scheibler et al., 2022).

Utterance-level SI-SDR is computed as:

$$\text{SI-SDR}(\hat{s}, s) = 10 \log_{10} \frac{\| \alpha s \|^2}{\| \hat{s} - \alpha s \|^2}, \quad \alpha = \frac{\langle \hat{s}, s \rangle}{\|s\|^2}$$

For noisy ground truth references ($s = s_t + n$), it is analytically established that SI-SDR is upper-bounded by the SNR of the reference signal (Jepsen et al., 20 Aug 2025). This can bias model training towards reproducing the reference noise. Non-intrusive metrics are increasingly used to evaluate perceptual quality independent of reference artifacts (Jepsen et al., 20 Aug 2025, Scheibler et al., 2022).

5. Model Training and Performance Benchmarks

Multiple architectures have been benchmarked on WSJ0-2mix, enabling direct comparison across methodological generations.

Model	SI-SDRi (dB)	SDRi (dB)	PESQ	Params (M)	Notes
Conv-TasNet ’19	15.3–16.0	—	3.29	—	Early time-domain
DPRNN	18.8	—	—	—	Dual-path RNN
DPTNet	20.2	—	—	—	Dual-path Transformer
SepFormer	22.3	22.4	—	25.7	SOTA Transformer
RE-SepFormer	18.6	18.9	—	8.0	Resource-efficient
DiffSep [DNSMOS]	—	—	3.29	~10s	Generative SDE
MSFFT-2P	21.0	—	—	—	Multi-scale fusion
MTSS-Softmax (extr)	19.5/17.0	19.9/17.4	3.62	—	Multi-target extract.
AmbiSep Triple-Path	—	—	—	—	17.7 dB SI-SDRi, spatialized

Performance varies with training strategy (e.g., dynamic mixing, speed perturbation), architectural depth, and choice of objective (SI-SDR vs. perceptual MOS). Dynamic augmentation and fusion-based Transformers yield SOTA SI-SDRi scores, while generative diffusion/separation models provide competitive perceptual quality (Scheibler et al., 2022, Jepsen et al., 20 Aug 2025, Xu et al., 2022).

6. Dataset Limitations and Reference Noise Artifacts

WSJ0-2mix is often assumed "clean." However, analysis shows that residual noise and mild reverberation are present in the WSJ0 recordings (Jepsen et al., 20 Aug 2025). This has two key implications:

• SI-SDR and related metrics are capped by the reference’s inherent SNR. Models trained using noisy references may learn to memorize and replicate the background noise in the target, reducing generalizability.
• Comparing SI-SDR scores across papers may be unreliable if reference enhancement/denoising is inconsistently applied, or if noise augmentation (e.g., WHAM!) is not controlled.

Reference enhancement (e.g., with MetricGAN+) can reduce noise memorization but may add artifacts (coloration, discontinuity) and does not guarantee an overall improvement in perceived quality (Jepsen et al., 20 Aug 2025). There is a documented negative correlation between SI-SDR and subjective noisiness MOS metrics, highlighting a limitation of using reference-based, intrusive metrics on this benchmark (Jepsen et al., 20 Aug 2025).

7. Impact and Evolving Practice

WSJ0-2mix has standardized the evaluation and facilitated the rapid development of deep learning models for speech separation. It has also exposed challenges in metric selection (SI-SDR limitations), driven the adoption of data augmentation and domain adaptation, and prompted exploration of non-intrusive perceptual metrics. Extensions such as Ambisonic convolution (Herzog et al., 2022), noisy ambient overlays (WHAM!) (Wichern et al., 2019, Jepsen et al., 20 Aug 2025), and multi-reference versions (Zeng et al., 2022), along with Transformer-based model innovations (Subakan et al., 2020, Xu et al., 2022, Libera et al., 2022), continue to refine its role as a rigorous benchmark.

This suggests that as model quality approaches the upper bounds set by reference SNR, further progress may require development of new corpora with genuinely anechoic references or redefinition of target objectives—particularly in research aiming to decouple perceptual quality from intrusive metric ceilings (Jepsen et al., 20 Aug 2025).