LibriMix Dataset Overview

• LibriMix is an open-source dataset that provides both clean and noisy speech mixtures, enabling robust monaural separation and diarization research.
• It uses a principled loudness-based mixing protocol and multiple overlap strategies to simulate realistic conversational and environmental conditions.
• The dataset includes rich metadata and is compatible with frameworks like Asteroid, facilitating reproducible experiments and advanced model training.

LibriMix is an open-source family of single-channel speech separation datasets constructed to enable generalizable training and evaluation of monaural source separation and speaker diarization systems. Built atop LibriSpeech clean data and WHAM! ambient noise, LibriMix offers two- and three-speaker mixtures in both clean and noisy conditions, synthetic mixtures for cross-corpus testing, and sparsely overlapping mixtures to more closely match conversational scenarios. Its principled, loudness-based mixing protocol, large speaker pool, and metadata-rich format make it central in benchmarking speech separation models under realistic conditions and evaluating cross-dataset generalization.

1. Origins and Motivation

LibriMix was developed in response to the limited speaker variability and cross-dataset generalization challenges observed with the wsj0-2mix/WHAM! benchmarks, which had become de facto standards for single-channel speech separation. Empirical findings indicate that models trained on wsj0-2mix exhibit substantial performance drops (>3 dB SI-SDRi) when tested on other corpora, largely due to the narrow recording conditions and limited number of speakers. To mitigate these issues, LibriMix expands the diversity and scale of both sources and mixing paradigms, using publicly available read English speech and realistic environmental noise captured in public spaces. The entire pipeline, including generation scripts and data loaders, is freely available for reproducibility and extensibility (Cosentino et al., 2020).

2. Dataset Construction and Variants

LibriMix is constructed by algorithmically mixing utterances drawn from the LibriSpeech “train-clean-100” and “train-clean-360” subsets (as well as development and test sets), yielding a pool of ≈470 hours of clean speech from 1,252 speakers. Ambient noise is sourced from WHAM! (≈58 h train, 14.7 h dev, and 9 h test), which contains background environmental sounds (bars, restaurants, coffee shops) with no speech overlap.

Mixing processes differ by scenario:

• Fully Overlapping Mixtures: Two- (Libri2Mix) and three-speaker (Libri3Mix) mixtures are synthesized by summing C distinct utterances ($u_1, ..., u_C$) and, if applicable, one noise segment. Each signal is first normalized to a target loudness in LUFS (ITU-R BS.1770–4). Speech source loudness is assigned $L(s_i) \sim \mathrm{Uniform}(-33, -25~\text{dB})$ LUFS; noise $L(w) \sim \mathrm{Uniform}(-38, -30~\text{dB})$. Scaling factors $a_i$ and $b$ ensure correspondence with these targets. The resulting mixture waveform:

$y[n] = \sum_{i=1}^C a_i s_i[n] + b w[n]$

In clean conditions ($b = 0$), $$SNR_{\rm clean} \sim \mathcal{N}(0\,\mathrm{dB},\,4.1\,\mathrm{dB})$$; with noise, $$SNR_{\rm noisy} \sim \mathcal{N}(-2\,\mathrm{dB},\,3.6\,\mathrm{dB})$$. Noisy mixtures are amplitude-clipped at $0.9$.

• Sparse Overlap and Cross-Corpus Evaluation: SparseLibri2Mix/3Mix test sets are produced from “test-clean” by concatenating voice-activity aligned sub-utterances (≤15 s each) and controlling the overlap ratio from 0% to 100%. The VCTK-2Mix test set adapts the same mixing protocol to 3,000 mixtures from 108 VCTK speakers with WHAM! noise.
• Min and Max Mixing Modes: “Min” mode truncates all sources to the shortest utterance in the mixture; "max" mode zero-pads to the longest. This distinction is often referenced in recent evaluations (Mun et al., 2023).

A summary of sizes (16 kHz, “max” mode): | Dataset | Split | Mixtures | Duration (h) | |-------------------|-------------|----------|--------------| | Libri2Mix | train-360 | 50,800 | 212 | | | train-100 | 13,900 | 58 | | | dev | 3,000 | 11 | | | test | 3,000 | 11 | | Libri3Mix | train-360 | 33,900 | 146 | | | train-100 | 9,300 | 40 | | | dev | 3,000 | 11 | | | test | 3,000 | 11 | | SparseLibri2Mix | test | 3,000 | 6 | | SparseLibri3Mix | test | 3,000 | 6 | | VCTK-2Mix | test | 3,000 | 9 |

(Cosentino et al., 2020)

3. Data Formats, Organization, and Metadata

LibriMix adopts a directory structure compatible with existing wsj0-2mix/WHAM! conventions, facilitating integration with standard data loaders. Each split contains:

• Mixture WAV files (mix_clean, mix_noisy) and per-source files (s1, s2, s3 if applicable), all 16-bit PCM WAV, at 16 kHz or 8 kHz.
• JSON metadata files mapping mixture IDs to source utterance IDs, speakers, noise clips, and scaling factors.

This structure and detailed metadata support reproducible experiments and easy adoption in PyTorch and related frameworks. The Asteroid toolkit provides ready-to-use data loaders and recipes tailored for LibriMix, while the public repositories supply full code for data generation and forced alignment (Cosentino et al., 2020).

4. Preprocessing, Features, and Model Input

Preprocessing conventions depend on task and architecture:

• In direct speech separation, raw waveforms $x(t)$ are typically used. For example, in Transformer-based models (Rijal et al., 2023), the only explicit preprocessing is a 1-D convolutional encoder:

$e = \mathrm{ReLU}\bigl(\mathrm{Conv1d}(x)\bigr)$

where Conv1d implements 256 filters (kernel size 3, stride 1, no padding). No STFT, frequency-domain features, or per-utterance normalization is applied beyond this learned front end.

• For diarization and speech activity tasks (Mun et al., 2023), 80-dimensional log-Mel filterbanks are extracted (window: 25 ms, frame shift: 10 ms), and these are fed directly into neural architectures, typically without cepstral mean-variance normalization (CMVN) or global normalization.

No additional data augmentation, such as speed-perturbation or room simulation, is described in standard LibriMix experiments (Cosentino et al., 2020, Mun et al., 2023).

5. Evaluation Protocols and Benchmark Results

LibriMix defines standard train/validation/test splits. For Libri2Mix in (Rijal et al., 2023), the split is 69:21:10 for train/cross-validation/test; for Libri3Mix and VCTK-2Mix (Cosentino et al., 2020): fixed mixture counts and durations as indicated above.

Speech separation systems are commonly evaluated via SI-SDR improvement (SI-SDRi), comparing model outputs to oracle Ideal Ratio/Ideal Binary Mask (IRM/IBM) results. For LibriMix (16 kHz, “max” mode), baseline Conv-TasNet achieves: | Task | SI-SDRi (dB) | |----------|--------------| | 2spk-C | 16.0 | | 2spk-N | 13.5 | | 3spk-C | 13.0 | | 3spk-N | 10.9 | Oracle IRM/IBM reach 13.4–14.9 dB depending on condition, indicating that Conv-TasNet surpasses frequency-domain masks on 2-speaker mixtures but still trails by 1–4 dB on the harder 3-speaker tasks (Cosentino et al., 2020).

Diarization is evaluated via Diarization Error Rate (DER):

$$\mathrm{DER} = \frac{T_{\rm FA} + T_{\rm Miss} + T_{\rm Conf}}{T_{\rm Total}} \times 100\%$$

with FA = False Alarm, Miss = Missed detection, Conf = Speaker Confusion, and zero collar tolerance. State-of-the-art EEND-DEMUX achieves DER as low as 3.79% (Libri2Mix, min mode) (Mun et al., 2023).

Permutation-invariant training (PIT) is standard both for separation (minimizing negative SI-SDR) and for diarization (minimizing binary cross-entropy losses over speaker hypotheses and existence heads) (Rijal et al., 2023, Mun et al., 2023).

6. Generalization and Impact

LibriMix's diverse speaker pool (∼1,000 speakers vs. ∼100 in prior datasets), wide vocabulary, and realistic noise contribute to models with enhanced cross-corpus robustness. For example, Conv-TasNet trained on LibriMix exhibits <1 dB SI-SDRi drop on the WHAM! test set, compared to a ∼4 dB drop for WHAM!-trained models on LibriMix. On unseen VCTK-2Mix mixtures, LibriMix-trained models outperform WHAM!-based models by 3–4 dB SI-SDRi, demonstrating improved speaker independence and domain generalization (Cosentino et al., 2020). This suggests LibriMix is better suited for the development of speaker- and condition-agnostic separation models.

The availability of cross-corpus and sparsely overlapping test sets enables more realistic evaluation for conversational and meeting-style audio signals—scenarios not well-captured in legacy datasets.

7. Availability and Adoption

LibriMix is fully open-source, with all audio, metadata, and recipe code publicly accessible. Official repositories provide data generation tools, forced alignment scripts for sparse mixing, and pre-defined data splits. The integration with frameworks like Asteroid streamlines adoption for large-scale training and benchmarking, supporting reproducible research across the separation and diarization communities (Cosentino et al., 2020).

A plausible implication is that the dataset’s design principles and accompanying infrastructure have helped establish LibriMix as the reference benchmark for modern monaural speech separation and multi-speaker diarization research.

---

References:

• (Cosentino et al., 2020) "LibriMix: An Open-Source Dataset for Generalizable Speech Separation"
• (Rijal et al., 2023) "Monaural Multi-Speaker Speech Separation Using Efficient Transformer Model"
• (Mun et al., 2023) "EEND-DEMUX: End-to-End Neural Speaker Diarization via Demultiplexed Speaker Embeddings"