VCTK-2Mix Speech Separation Benchmark

• VCTK-2Mix is an open-source, two-speaker mixture dataset derived from the VCTK corpus, designed to assess speech separation under out-of-domain, unseen speaker and accent conditions.
• It follows two established mixing protocols—one mimicking WSJ0-2mix and another replicating LibriMix—to standardize mixture generation using strict SNR and LUFS normalization techniques.
• The dataset facilitates zero-shot evaluation by serving as a held-out test set, revealing performance drops in models and emphasizing the need for training on more diverse audio conditions.

VCTK-2Mix is a fully open-source, two-speaker mixture test set constructed from the multi-accent British English CSTR VCTK corpus and designed for the rigorous evaluation of single-channel speech separation systems, with particular emphasis on generalization to out-of-domain, speaker- and corpus-unseen conditions. It establishes a standardized, reproducible benchmark for testing separation models trained on alternative corpora such as WSJ0-2mix or LibriMix, mirroring their mixing recipes but using a distinct, phonetically diverse speaker population.

1. Dataset Construction and Statistics

VCTK-2Mix leverages the full VCTK corpus, which contains approximately 110 speakers, each with multiple utterances. In one protocol, 4,000 unordered utterance pairs are randomly sampled without enforcing further speaker balancing or utterance length constraints; in the other, 3,000 mixtures are created by selecting pairs of different speakers, with long silences stripped using a 20 dB threshold energy-based VAD, leading to a post-VAD pool of 108 speakers (Kadioglu et al., 2020, Cosentino et al., 2020).

Each mixture contains two utterances combined to produce time-domain mixtures, resulting in variable-length audio (1–12 s) dependent on the longer source utterance, with the shorter utterance zero-padded at summing. The total test set duration is approximately 9 hours of speech. No training or validation splits are provided; all VCTK-2Mix mixtures are reserved exclusively for evaluation.

2. Mixing Procedure and Signal Processing

VCTK-2Mix strictly follows established mixture generation schemes. For the 4,000-pair version (Kadioglu et al., 2020), the SNR for each pair is randomly and uniformly sampled within $[-5, +5]$ dB. For source waveforms $s_1$ and $s_2$, the mixture $x = s_1 + \alpha s_2$, where $\alpha$ is chosen to satisfy

$$\text{SNR} = 10 \log_{10} \frac{\|s_1\|^2}{\|\alpha s_2\|^2}$$

with the target SNR. No further speaker or utterance balancing is performed beyond this random pairing.

For the 3,000-mixture protocol (Cosentino et al., 2020), the mixing recipe faithfully replicates LibriMix. Each source utterance undergoes LUFS loudness normalization to a target $L_i \sim \mathrm{Uniform}[-33, -25]$ dB LUFS (ITU-R BS.1770-4). The loudness-normalized utterances, $s_i'(t) = \alpha_i s_i(t)$, are summed:

$x_C(t) = s_1'(t) + s_2'(t)$

For the noisy variant, noise $n(t)$ from WHAM! undergoes separate normalization to $s_1$0 dB LUFS; the result is clipped pointwise to $s_1$1. The speech-only SNRs are roughly Gaussian with mean $s_1$2 dB and $s_1$3 dB.

All waveforms are output at 16 kHz sample rate (downsampling to 8 kHz is supported), using 16-bit PCM WAV format. Preprocessing beyond downsampling and VAD (for the 3,000-mixture protocol) is not performed.

3. Benchmark Role and Zero-Shot Evaluation

VCTK-2Mix is purpose-built as a test-only, cross-domain evaluation set. No training or validation mixtures from VCTK are included in model development; thus, all VCTK speakers and linguistic content are entirely held out during training. This protocol enables assessment of a model’s ability to generalize to unseen speaker populations, accents, and recording conditions.

Evaluation paradigms are strictly zero-shot: models trained on WSJ0-2mix, LibriMix, or WHAM!, including their noisy variants, are deployed directly on VCTK-2Mix. This test suite consequently provides a robust estimate of a model’s out-of-domain generalization capability (Kadioglu et al., 2020, Cosentino et al., 2020).

4. Evaluation Protocols and Metrics

Evaluation on VCTK-2Mix follows the same objective criteria as used for prior mixtures. The principal quantitative metric is the scale-invariant signal-to-noise ratio improvement (SI-SNRi), computed as the difference between SI-SNR of separated sources and the corresponding mixture. Some studies additionally report improvement in signal-to-distortion ratio (SDRi), and scale-invariant SDR (SI-SDR).

Permutation-invariant training and evaluation (PIT) matching is applied to resolve speaker–output assignment ambiguity. For model comparison, the drop in SI-SNRi when transferring from in-domain to VCTK-2Mix (e.g., from WSJ0-2mix or LibriMix test sets) serves as a proxy for generalization performance. For instance, models trained on WSJ0-2mix lose approximately 6 dB SI-SNRi, and those trained on LibriTTS-2Mix lose approximately 7 dB when evaluated on VCTK (Kadioglu et al., 2020).

For the LibriMix-aligned protocol, a Conv-TasNet trained on LibriMix “train-360” achieves SI-SDR $s_1$4 dB on VCTK-2Mix, compared to $s_1$5 dB for a model trained on WHAM! or $s_1$6 dB for LibriMix “train-100”. No fine-tuning on VCTK-2Mix is performed (Cosentino et al., 2020).

5. Implementation and Public Accessibility

VCTK-2Mix mixing, preprocessing, and corpus assembly are reproducible and fully open-source. Generation scripts—including YAML configurations, energy-VAD routines using librosa, LUFS gain matching, and file I/O—are available at https://github.com/JorisCos/VCTK-2Mix (Cosentino et al., 2020). Both 16 kHz and 8 kHz waveforms are available; all mixtures have corresponding ground truth source files.

Table 1 summarizes key aspects by source paper:

Protocol	Speakers	Mixtures	Mixing Recipe	Sampling Rate	Preprocessing	SNR/Loudness Range
(Kadioglu et al., 2020)	~110	4,000	WSJ0-2mix	8 kHz	None	SNR $s_1$7 dB
(Cosentino et al., 2020)	108	3,000	LibriMix	16/8 kHz	VAD, LUFS matching	$s_1$8 LUFS

Mixing parameters, speaker selection, and all further details are defined unambiguously by the respective generation scripts and protocols.

6. Significance and Applications in Speech Separation Research

VCTK-2Mix is critically positioned as a standardized evaluation set for measuring the cross-corpus generalization of source separation systems, especially Conv-TasNet and its variants. Both (Kadioglu et al., 2020) and (Cosentino et al., 2020) demonstrate that models trained on comparatively homogeneous corpora (e.g., WSJ0-2mix) exhibit markedly degraded performance when tested on VCTK-2Mix. Models trained on corpora with greater speaker and phonetic variability, such as LibriMix, demonstrate diminished but still significant drop-off—highlighting the pronounced domain-mismatch in current deep separation models.

A plausible implication is that speech separation architectures require broader speaker and signal diversity during training to maintain robust generalization. The construction and open-source availability of VCTK-2Mix enable systematic comparison across methods and reinforce its role as a reference testbed for new architectures and training paradigms targeting improved out-of-domain robustness (Kadioglu et al., 2020, Cosentino et al., 2020).