WSJ0-2mix Dataset for Speech Separation

• WSJ0-2mix dataset is a benchmark resource for single-channel speech separation derived from the WSJ0 corpus, enabling controlled evaluation of overlapping utterances.
• The dataset employs speaker-disjoint splits with 'min' and 'max' mixing strategies to ensure reproducibility in tasks like TS-ASR and deep clustering.
• Extensions, including multi-speaker variants and noise/reverberation simulations, have advanced research in source separation and codec-based applications.

The WSJ0-2mix dataset is the prevailing benchmark for single-channel speech separation and related tasks such as target-speaker automatic speech recognition (TS-ASR). Constructed from the Wall Street Journal 0 (WSJ0) read-speech corpus, it enables reproducible evaluation of algorithms that aim to disentangle overlapping utterances from two distinct speakers under strictly controlled acoustic conditions. Over time, WSJ0-2mix has been extended to create more challenging variants (e.g., with additive noise or multiple interferers), and repurposed for downstream ASR, perceptual quality, and codec-based separation research. Its protocol, split structure, and reference ambiguities have progressively shaped methodologies and evaluation standards for speech separation.

1. Corpus Construction and Mixture Generation

WSJ0-2mix is created by drawing utterance pairs from the WSJ0 corpus, recorded with close-talk microphones in acoustically controlled settings. For each mixture, two distinct speakers are randomly selected, and a single utterance from each is chosen. The mixture is generated in the time domain as:

$x[n] = s_1[n] + \alpha\,s_2[n]$

where $s_1[n]$ and $s_2[n]$ are the clean reference waveforms, and the gain factor $\alpha$ is computed to achieve a target signal-to-noise ratio (SNR), randomly sampled from a uniform distribution (commonly $\mathcal{U}(0,5)$ dB). The precise formula for $\alpha$ is:

$$\alpha = \frac{\|s_1\|_2}{\|s_2\|_2} \cdot 10^{-\mathrm{SNR}/20}$$

depending on the chosen SNR and the source energies (Zhang et al., 2023, Wichern et al., 2019, Menne et al., 2019).

Two main mixing strategies are provided:

-Min variant: The output mixture is truncated to the length of the shorter utterance, ensuring near-complete temporal overlap (overlap ratio ≃ 100%). -Max variant: The shorter utterance is padded with silence to match the length of the longer, aligned at onset, leading to possible non-overlapping trailing segments (Wichern et al., 2019).

Recent repurposings (e.g., WSJ0-2mix-extr (Zhang et al., 2023), see section 3) require additional logic such as auxiliary utterance selection or silence padding for temporal alignment.

2. Dataset Splits and Extensions

WSJ0-2mix adopts strictly speaker-disjoint splits. The canonical configuration is:

Split	#Mixtures	Duration (h)	Sources
Train	20,000	~30	si_tr_s
Validation	5,000	~10	si_dt_05
Test	3,000	~5	si_et_05

Speakers do not overlap between splits. Each mixture is an independent random pairing; utterance durations range from approximately 1–8 seconds. The “extr” extension (WSJ0-2mix-extr) appends a third, auxiliary utterance from the target speaker, supporting explicit target-speaker modeling in TS-ASR regimes (Zhang et al., 2023).

Multi-speaker expansions (e.g., WSJ0-3mix-extr) follow an analogous procedure using three utterances from three speakers, creating mixtures with two interferers for robustness assessment (Zhang et al., 2023, Gu et al., 2020).

3. Preprocessing and Feature Extraction

The dataset is distributed at both 8 kHz and 16 kHz sample rates (Wichern et al., 2019, Yip et al., 2024). All recordings are free from artificially added noise or reverberation (“clean”), although subsequent studies frequently simulate reverberant conditions via convolution with room impulse responses (RIRs) (Gu et al., 2020).

Feature extraction protocols vary by downstream task:

• ASR and source separation: 80-dimensional log-Mel filterbanks (25 ms window, 10 ms hop) are standard; SpecAugment and volume/speed perturbation are applied for data augmentation (Zhang et al., 2023).
• Codec-based approaches: If model frontends require a higher sample rate, mixtures and references are resampled appropriately (e.g., to 16 kHz for neural audio codecs) (Yip et al., 2024).
• “Far-field” variants: Clean waveforms are convolved with randomly synthesized RIRs, yielding multi-channel, reverberant mixtures (Gu et al., 2020).
• Sparsely overlapping variants: Silence is inserted between utterances to modulate overlap ratio, with per-gap energy normalization to preserve perceptual continuity (Menne et al., 2019).

Explicit voice-activity detection is generally not performed in the standard pipeline.

4. Evaluation Protocols, Metrics, and Benchmarks

WSJ0-2mix is evaluated primarily in two regimes: generic source separation and target-speaker ASR.

Standard Metrics

• Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) and SDR (BSS_EVAL): Intrusive measures used to quantify separation fidelity. SI-SDR has become preferred due to invariance to scaling (Gu et al., 2020, Jepsen et al., 20 Aug 2025).
• Word Error Rate (WER): For ASR tasks, WER is computed after decoding separated streams, using standard n-gram LLMs (Zhang et al., 2023, Menne et al., 2019).
• Perceptual metrics: DNSMOS, STOI, and PESQ are employed to measure speech intelligibility and quality, especially for codec or end-user evaluation scenarios (Yip et al., 2024).

Results and Comparative Baselines

Recent results on WSJ0-2mix(-extr) exhibit rapid advances:

Model	Loss	2-mix WER	Year	Reference
Conformer-CTC (1spk)	CTC	36.7%	2023	(Zhang et al., 2023)
SpeakerBeam	x-ent	30.6%	2019	(Zhang et al., 2023)
Exformer+Conformer-CTC	Si-SiNR+CTC	13.2%	2023	(Zhang et al., 2023)
CONF-TSASR (CTC+Spec)	CTC+Spec	4.2% ★	2023	(Zhang et al., 2023)
DPCL+DNN-HMM		16.5%	2019	(Menne et al., 2019)

★ Denotes new state of the art as of the respective publication.

Codec-based models evaluated on WSJ0-2mix achieve DNSMOS-OVRL in the range 1.73–1.89 and STOI values 0.74–0.83, varying by loss type and codec (Yip et al., 2024).

5. Variants and Related Datasets: WHAM! and Noisy/Enhanced References

The WHAM! extension overlays real ambient noise recordings onto WSJ0-2mix mixtures, sampled from San Francisco cafés, parks, and offices (Wichern et al., 2019). Noise segments are precisely selected to avoid foreground speech and matched in duration to the underlying mixture. The noise is added at an SNR sampled from $\mathcal{U}(-6, 3)$ dB, substantially increasing separation difficulty. Both “min” and “max” overlap configurations are available at 8 kHz and 16 kHz. SI-SDR performance drops by 1–2 dB relative to clean WSJ0-2mix, illustrating the increased challenge (Wichern et al., 2019, Jepsen et al., 20 Aug 2025).

A salient finding is that WSJ0-2mix's “clean” references sometimes contain low-level noise and recording artifacts. This latent noise introduces a ceiling effect in SI-SDR—no algorithm can surpass the actual SNR of the reference, and models are incentivized to pass through noise if SI-SDR is used for training (Jepsen et al., 20 Aug 2025). Approaches that enhance references (e.g., via MetricGAN+) and reconstruct mixtures can yield perceptually cleaner output but may introduce artifacts, notably in Colouration and Discontinuity MOS. Negative correlation between SI-SDR and perceived noisiness is observed across models (Jepsen et al., 20 Aug 2025).

6. Applications, Generalizations, and Research Impact

WSJ0-2mix serves as a canonical testbed for deep clustering, mask-inference, time-domain separation (e.g., Conv-TasNet, SepFormer), TS-ASR, and codec-based separation. Extensions to three-speaker mixtures (WSJ0-3mix), far-field/multi-channel arrangements (Gu et al., 2020), and overlapping ratio control (Menne et al., 2019) facilitate controlled exploration of separation in realistic and adversarial scenarios. The dataset's rigorously defined splits and construction protocol have standardized comparative evaluation, shaped the development of permutation-invariant training objectives, and exposed critical methodology gaps (e.g., reference noisiness and SI-SDR limitations).

A plausible implication is that future research should prioritize development of cleaner reference corpora and non-intrusive evaluation metrics, given the demonstrated limitations of SI-SDR as both loss and metric in the presence of reference noise (Jepsen et al., 20 Aug 2025).

7. Common Misconceptions and Methodological Caveats

A prominent misconception is that WSJ0-2mix is acoustically pristine and thus suitable for arbitrary separation/denoising objectives. However, intrinsic recording noise in the WSJ0 corpus constrains maximum achievable SI-SDR, biases model outputs toward reference noise when SI-SDR is used for optimization, and complicates fair benchmarking (Jepsen et al., 20 Aug 2025). Another misconception concerns overlap: the standard WSJ0-2mix mixtures are fully overlapped except for minor silence frames. Realistic conversational or meeting scenarios (partial overlap, far-field, multi-speaker) are not directly represented and require simulation or alternative benchmarks (Menne et al., 2019, Gu et al., 2020).

In summary, WSJ0-2mix is a foundational speech separation resource with meticulously defined protocols and a diverse ecosystem of extensions. Its ongoing usage continues to inform both fundamental and applied research in source separation and robust speech recognition.