LibriCSS: Benchmark for Continuous Speech Separation

• LibriCSS is a benchmark derived from LibriSpeech that simulates continuous, partially overlapped speech, making it ideal for meeting transcription evaluation.
• It uses far-field multi-channel recordings in meeting rooms to test separation, diarization, and ASR performance across varying overlap conditions.
• Numerous studies leverage LibriCSS to advance CSS methods, beamforming, and integrated pipelines that jointly address speech separation and speaker attribution.

Searching arXiv for recent and foundational LibriCSS papers to ground the article. LibriCSS is a real-recording benchmark for continuous speech separation, meeting transcription, diarization, and multi-talker automatic speech recognition, derived from LibriSpeech and designed to evaluate systems under conditions closer to conversational speech than pre-segmented, fully overlapped synthetic mixtures. It was introduced together with continuous speech separation (CSS) as the task of generating non-overlapped speech streams from a continuous audio input containing multiple utterances with partial and intermittent overlap, and it has since become a standard evaluation corpus for both modular and end-to-end meeting transcription systems (Chen et al., 2020). Subsequent work has used LibriCSS to study single-channel and multi-channel separation, speaker-attributed ASR, diarization, neural beamforming, and error analysis, making it a central benchmark for long-form overlapped speech in meeting-like acoustics (Raj et al., 2020, Kanda et al., 2022, Vieting et al., 12 Sep 2025).

1. Origin, construction, and task definition

LibriCSS was created to address two limitations of earlier speech separation benchmarks: speech in practical scenarios is continuous rather than pre-segmented, and overlap is usually partial rather than fully overlapped throughout an utterance (Chen et al., 2020). The benchmark is derived from LibriSpeech test-clean utterances, which are concatenated to simulate conversation-like sessions and replayed through loudspeakers in a real meeting room before being captured by a 7-channel circular microphone array (Chen et al., 2020). The resulting corpus contains 10 hours of audio in 10 sessions, each approximately 1 hour long and subdivided into six 10-minute mini-sessions (Chen et al., 2020).

Each session uses 8 speakers selected from a pool of 40 speakers, and the corpus is organized by overlap condition. The defined conditions are 0S, 0L, 10%, 20%, 30%, and 40% overlap, where the overlap ratio is

$$\mathrm{OVR} = \frac{L_{\mathrm{ovl}}}{L_{\mathrm{all}}}.$$

For the non-overlap case, 0S denotes short silence uniformly sampled between 0.1 and 0.5 s, whereas 0L denotes long silence of 2.9 to 3.0 s (Chen et al., 2020). These two conditions were included to test whether a system degrades when adjacent speaker turns are separated only by brief gaps, even without actual overlap (Chen et al., 2020).

The recordings were made in a regular meeting room using far-field microphones, with loudspeaker-to-microphone distances ranging from 33 cm to 409 cm (Chen et al., 2020). Later work characterizes the same array as a 6-microphone circular array plus a center microphone, with the center mic used as the reference channel in some multi-channel systems (Taherian et al., 2023). This suggests that LibriCSS occupies an intermediate position between tightly controlled synthetic mixtures and unconstrained in-the-wild meeting corpora: it preserves exact transcript provenance from LibriSpeech while imposing real room acoustics and long-form conversational structure.

2. Evaluation protocols and scoring methodology

A major contribution of the original LibriCSS work was to define evaluation protocols aligned with downstream ASR rather than only signal-level criteria (Chen et al., 2020). Two principal modes are used.

Utterance-wise evaluation uses ground-truth segmentation to cut each utterance from the far-field recording, separate it independently, and feed the outputs to ASR. For each input utterance, two transcriptions and WERs are generated, and the lower-WER output is chosen (Chen et al., 2020). This mode isolates separation quality but assumes oracle boundaries.

Continuous input evaluation is the more practically relevant CSS setting. The system processes long recordings as streams; because open-source streaming ASR was limited in the original setup, each mini-session is split into about 60–120 second segments during silence, each containing about 8–10 utterances, and the outputs are scored using Asclite from SCTK to produce speaker-agnostic WERs (Chen et al., 2020). Many later papers follow this continuous-input protocol, often on the first microphone channel for the monaural setting (Kanda et al., 2022, Vieting et al., 12 Sep 2025).

For speaker-attributed meeting transcription, later work adopts concatenated minimum-permutation word error rate (cpWER). In the modular system description paper, cpWER is defined by concatenating all utterances by speaker in both reference and hypothesis, computing WER for all speaker permutations, and choosing the minimum (Raj et al., 2020). In diarization-aware transcription work, cpWER is complemented by DIcpWER, which preserves segment boundaries but chooses the speaker assignment that minimizes the error, thereby isolating the contribution of diarization errors (Boeddeker et al., 2023).

Other task-specific measures have been used as LibriCSS became a broader benchmark. These include DER for diarization (Taherian et al., 2023), SAgWER for streaming multi-talker ASR without explicit speaker attribution (Kanda et al., 2022), ORCWER for diarization-free modular ASR pipelines (Vieting et al., 2023), and ORC WER in later segmentation studies (Vieting et al., 12 Sep 2025). The diversity of metrics reflects the benchmark’s expansion from a CSS dataset into a general meeting-transcription testbed.

3. Baseline continuous speech separation and early modular systems

The original LibriCSS paper evaluated a speaker-independent CSS method that estimates two speech masks and one noise mask, then outputs two separated speech channels, which was considered sufficient because three-way overlap is rare in meeting-style speech (Chen et al., 2020). The system uses sliding-window block processing with past, current, and future context; reported chunk configurations include $1.2\!-\!0.8\!-\!0.4$ s, $1.6\!-\!0.8\!-\!0.0$ s, and $0.8\!-\!0.4\!-\!0.4$ s (Chen et al., 2020). Output speech can be generated either by spectral masking or by mask-based adaptive MVDR beamforming (Chen et al., 2020).

That work established several recurrent findings. Overlap sharply hurts ASR, even at 10% overlap. Single-channel separation can help in overlap regions but can also harm the non-overlap conditions. More microphones improve performance substantially, and 7-channel MVDR outperforms time-frequency masking alone. Future context improves CSS, and rapid turn-taking in 0S is harmful without channel routing by CSS (Chen et al., 2020).

The first major modular meeting-transcription pipeline for LibriCSS combined Separation $\rightarrow$ Diarization $\rightarrow$ ASR and argued that explicit separation should precede the other modules because separation does not depend on diarization quality, while diarization and ASR benefit from reduced overlap (Raj et al., 2020). In that study, LibriCSS is described as 10 sessions of about 1 hour each, split into six 10-minute mini-sessions, recorded in a real meeting room with a 7-channel circular microphone array, with session 0 as development and the other 9 sessions for evaluation (Raj et al., 2020).

That paper compared no separation, mask-based MVDR CSS, and sequential multi-frame separation. The reported summary is shown below.

Method	SDR / DER	cpWER
No separation	- / 18.3	31.0
Mask-based MVDR	5.8† / 13.9	22.8
Sequential multi-frame	14.1 / 14.1	19.3

The same work reports a final best pipeline of Sequential separation + SC diarization + Transformer ASR, achieving 12.7% cpWER, versus 23.9% cpWER for a system without separation using TS-VAD + Transformer (Raj et al., 2020). The conclusion was explicit: separation is essential, but SDR and downstream recognition are not perfectly aligned, and diarization models trained for mixed speech may not transfer optimally to separated streams (Raj et al., 2020).

4. Separation front ends and multichannel modeling on LibriCSS

LibriCSS quickly became the principal benchmark for architectural advances in CSS. One major line of work replaced recurrent separators with Transformer- and Conformer-based models. The Conformer CSS paper treats LibriCSS as a 10-hour, seven-channel dataset of mixed and concatenated LibriSpeech utterances recorded in a meeting room, and distinguishes between utterance-wise and continuous evaluation (Chen et al., 2020). Its separator estimates time-frequency masks from STFT and inter-channel phase difference features, reconstructing separated signals via mask-based MVDR beamforming in the multichannel case (Chen et al., 2020). With Conformer-large, the reported seven-channel continuous evaluation WERs were 11.0 / 5.2 for 0S, 8.7 / 4.0 for 0L, 12.6 / 5.8 for 10%, 13.5 / 6.8 for 20%, 17.6 / 9.0 for 30%, and 19.6 / 10.0 for 40%, where the table reports hybrid ASR / Transformer ASR WERs (Chen et al., 2020).

Alternative separation formulations were then explored. RSAN-CSS replaced fixed-channel PIT separation with a recurrent selective attention network that estimates one source at a time and can generate a variable number of outputs (Zhang et al., 2021). Evaluated on the 10-hour LibriCSS dataset using the first microphone channel, it reported the following SA-WER values:

Method	0S	0L	10	20	30	40
uPIT	7.0	7.1	9.2	12.2	16.1	17.2
RSAN	6.6	6.3	9.0	12.4	14.8	16.4
RSAN + Dependency	6.3	6.0	9.1	11.8	14.1	15.9

The reported interpretation was that variable-output separation reduces leakage and better handles “hot spot” regions where the number of active speakers exceeds a fixed output count (Zhang et al., 2021).

In the multichannel setting, multi-microphone complex spectral mapping (MC-CSM) trained on matched fixed-geometry simulations showed that a learned non-linear beamformer can generalize well to LibriCSS (Wang et al., 2020). The method predicts the real and imaginary STFT components of target speech at a reference microphone from the real and imaginary components of multiple microphones, and combines the resulting estimates with MVDR beamforming, speaker-wise post-filtering, and frame-level speaker counting for CSS (Wang et al., 2020). A related line, multi-resolution location-based training (LBT), exploited the fixed array geometry to remove permutation ambiguity and supervise decoder outputs consistently according to speaker location (Taherian et al., 2023). That work reports that on continuous evaluation, the best reported example is a 2.4% absolute WER reduction at 40% overlap compared with PIT (Taherian et al., 2023).

Neural beamforming was also studied directly. The all-neural ADL-MVDR system adapted frame-wise neural beamforming to the CSS setting and used LibriCSS as its main benchmark (Zhang et al., 2021). The best reported configuration improved average WER from 11.1 / 11.7 for a baseline magnitude-mask system to 10.1 / 10.3, with larger gains at higher overlap ratios, and the paper attributes much of the improvement to VAD, steering-vector normalization, and a log-mel loss (Zhang et al., 2021).

A different multichannel direction used unsupervised spatial supervision. The spatial loss paper trained IVA- and MVDR-based neural separators without clean source labels, using MUSIC-derived DOA structure as supervision (Saijo et al., 2022). On LibriCSS, the best unsupervised DNN-IVA DOA$_2$ model achieved 10.7% average WER, improving over 12.3% for Gaussian IVA and 11.2% for a Conformer baseline, while remaining above a prior 6.8% state of the art (Saijo et al., 2022). This indicates that LibriCSS is informative not only for supervised source separation but also for geometry-aware unsupervised training.

5. End-to-end multi-talker and speaker-attributed ASR

LibriCSS has also become a benchmark for end-to-end recognition systems that avoid or reduce explicit separation. One early line of work investigated end-to-end speaker-attributed ASR (SA-ASR) on the monaural first channel of LibriCSS (Kanda et al., 2020). That paper showed that the original profile-dependent SA-ASR model performs poorly when no relevant speaker profiles are given, but that clustering the model’s internal speaker query vectors allows the system to function without prior speaker knowledge (Kanda et al., 2020). With oracle segmentation and 100 irrelevant profiles, the proposed method achieved 16.7% cpWER with oracle speaker count and 17.9% with NME counting, close to the profile-aware model (Kanda et al., 2020). The same paper also reported that changing the serialized label construction from speaker-based FIFO to utterance-based FIFO reduces average cpWER from 26.7% to 17.8% on LibriCSS (Kanda et al., 2020).

Because long-form continuous recordings remain difficult for short-segment SA-ASR models, a later study proposed a hypothesis stitcher for LibriCSS (Chang et al., 2021). On monaural LibriCSS with oracle VAD segmentation and provided speaker profiles, the serialized stitcher (WCO/E) reduced total SA-WER from 17.8 to 15.7, with particularly large gains at high overlap: 28.4 $\rightarrow$ 23.2 at 30% and 30.3 $\rightarrow$ 25.4 at 40% (Chang et al., 2021). This suggests that long-form hypothesis fusion is itself a major challenge on LibriCSS, independent of separation quality.

A more architectural update replaced LSTM-based SA-ASR with a Transformer-based joint model performing speaker counting, recognition, and speaker identification (Kanda et al., 2021). On LibriCSS, which that paper describes as 10 hours of real meeting-room recordings from LibriSpeech test_clean, with 8 speakers per recording and only the first channel used, the proposed Transformer SA-ASR achieved 11.9% cpWER with speaker profiles and 16.3% cpWER in the clustering-based setting without the same use of target profiles, both stated as state of the art for the monaural condition reported there (Kanda et al., 2021).

Another major end-to-end line is token-level serialized output training (t-SOT) for streaming multi-talker ASR (Kanda et al., 2022). That paper evaluated on the monaural first channel of LibriCSS in the official continuous-input setting with 8-speaker recordings, 10 hours total, and sessions 1–9 used for evaluation (Kanda et al., 2022). The reported average SAgWER values are:

Model	Latency	Avg WER
SURT w/ DP-LSTM	350 ms	20.1
SURT w/ DP-Transformer	350 ms	22.9
Single-talker TT-18	160 ms	18.6
Single-talker TT-36	160 ms	18.0
t-SOT TT-18	160 ms	10.1
t-SOT TT-36	160 ms	9.0
t-SOT TT-36	2560 ms	7.6

For t-SOT TT-36 at 160 ms, the condition-wise WERs are

$$\text{0L}=6.7,\; \text{0S}=6.1,\; 10=7.5,\; 20=9.3,\; 30=11.6,\; 40=12.9,\; \text{Avg}=9.0,$$

and for 2560 ms latency they are

$1.2\!-\!0.8\!-\!0.4$0

(Kanda et al., 2022). The paper also reports that t-SOT is on par with or slightly better than a single-talker model on the non-overlap conditions, indicating that one model can serve both single- and multi-talker use cases (Kanda et al., 2022).

6. Joint diarization-separation systems and state-of-the-art meeting transcription

A parallel development trajectory on LibriCSS integrated diarization and separation more tightly. TS-SEP reformulated target-speaker voice activity detection so that the network predicts time-frequency masks for each estimated speaker rather than only frame-level activity (Boeddeker et al., 2023). The paper follows the common LibriCSS setting of 10 one-hour sessions with 8 speakers per session and evaluates both recognition and diarization-sensitive error (Boeddeker et al., 2023). Its strongest reported result, TS-SEP $1.2\!-\!0.8\!-\!0.4$1 GSS $1.2\!-\!0.8\!-\!0.4$2 WavLM, achieved 5.06% cpWER and 3.26% DIcpWER, improving over previously reported 11.6% single-channel and 11.2% multi-channel cpWER baselines cited there (Boeddeker et al., 2023). The distinction between cpWER and DIcpWER is particularly influential: it isolates the fraction of total transcription error attributable to incorrect speaker assignment rather than recognition quality.

An even tighter integration was proposed in speaker separation via neural diarization (SSND), which uses multi-channel end-to-end diarization to produce speaker activity and embeddings, then assigns speakers to separation streams over long recordings (Taherian et al., 2023). In that work, LibriCSS is described as ten one-hour sessions, each subdivided into six ten-minute mini-sessions, recorded at 16 kHz by a 6-microphone circular array plus a center microphone (Taherian et al., 2023). The proposed MC-EEND with LBT achieved 4.68% DER, improving on 8.05% for MC-EEND (PIT) and outperforming prior baselines such as TS-VAD at 7.6% and TS-SEP at 6.49% (Taherian et al., 2023). For speaker-attributed ASR, the best estimated-boundary result was 3.22% cpWER with SSND (SpatialNet) + MC-EEND $1.2\!-\!0.8\!-\!0.4$3 + E2E-SSL, which the paper presents as a new state of the art on LibriCSS (Taherian et al., 2023). In the speaker-agnostic continuous-input setting, SSND (SpatialNet) achieved 7.33% WER, slightly better than 7.43% for MIMO-BF-MISO (Taherian et al., 2023).

The same period also saw major gains in monaural modular systems. A system combining TF-GridNet with a mixture encoder for continuous speech separation and hybrid HMM-DNN ASR reported a new state of the art on LibriCSS among systems trained only on LibriSpeech data (Vieting et al., 2023). Evaluated on the first microphone channel only, with Session0 as development and the remaining sessions as test, the best system reached 5.8 ORCWER with a Transformer LM, compared with 7.6 for t-SOT, while the oracle condition was 2.1 ORCWER (Vieting et al., 2023). The paper notes that the mixture encoder helps clearly for a weak BLSTM separator but gives no further gain with strong TF-GridNet, and interprets this as evidence that TF-GridNet already closes much of the gap to oracle separation (Vieting et al., 2023).

A later error-analysis study of a modular CSS + VAD segmentation + ASR + diarization + ASR pipeline, termed CSSAD, used LibriCSS as the main benchmark to investigate where remaining errors arise (Vieting et al., 12 Sep 2025). It evaluates on the first microphone only, using Session0 as development and the remaining sessions for test (Vieting et al., 12 Sep 2025). The paper reports ORC WER under different segmentations:

Segmentation	4-gram LM	Transformer LM
Oracle	5.8	4.3
Energy-based VAD	7.0	5.6
CSSAD (Whisper diarization)	6.7	5.4
CSSAD (ours)	6.5	5.2

The stated conclusion is that advanced diarization closes about one-third of the gap between energy-based VAD and oracle segmentation (Vieting et al., 12 Sep 2025). The same paper reports a cpWER comparison in which CSSAD (Whisper) and CSSAD (ours) both achieve 5.8, compared with 4.3 for oracle and 4.4 for DCF-DS + WavLM, while noting that DCF-DS uses much larger external pretraining data (Vieting et al., 12 Sep 2025). The paper claims state-of-the-art single-microphone performance on LibriCSS among systems that train the ASR module only on LibriSpeech data (Vieting et al., 12 Sep 2025).

7. Error patterns, benchmark extensions, and research significance

LibriCSS has not only served to rank systems but also to reveal failure modes. The 2025 modular error-analysis paper extends prior leakage analysis by measuring both cross-channel and primary-to-cross leakage with improved temporal locality sensitivity (Vieting et al., 12 Sep 2025). It defines WCR and GCR as framewise word coincidence measures, noting that GCR is optimistic because it checks whether any lattice arc matches at each frame, even if those arcs do not form a valid path (Vieting et al., 12 Sep 2025). The reported coincidence table shows that conventional cross-channel leakage is small, but primary-to-cross leakage can be substantial, especially in regions where only one speaker is active (Vieting et al., 12 Sep 2025). Yet the same study finds that leakage is not the main bottleneck because VAD mostly ignores leaked regions.

Instead, the decisive bottleneck is segmentation. The paper categorizes segmentation errors into Leakage, Missing, Merges, and Boundaries, and reports that fixing missing segments yields the largest improvement. For the authors’ diarization transcript, the baseline ORC WER is 6.5 / 5.2; fixing leaks gives 6.4 / 5.2, fixing missing segments gives 6.0 / 4.7, fixing merges gives 6.5 / 5.2, fixing boundaries gives 6.5 / 5.1, and fixing all error types gives 6.0 / 4.5 (Vieting et al., 12 Sep 2025). The paper concludes that missing segments account for more than half of the remaining gap to oracle (Vieting et al., 12 Sep 2025). A plausible implication is that further progress on LibriCSS for modular systems will depend less on stronger leakage suppression than on more sensitive segmentation and diarization.

LibriCSS has also inspired successor benchmarks. LibriWASN was explicitly designed to “follow closely the LibriCSS design” while replacing the single synchronous array with 9 different devices and 29 total channels whose clocks are not synchronized (Schmalenstroeer et al., 2023). It uses the same LibriSpeech source material, the same eight loudspeakers arranged around a meeting table, the same overlap categories (0L, 0S, OV10, OV20, OV30, OV40), and the same directory structure as LibriCSS (Schmalenstroeer et al., 2023). The crucial difference is the presence of sampling rate offset (SRO) and time-varying inter-device drift, making LibriWASN a synchronization benchmark for methods initially developed on LibriCSS (Schmalenstroeer et al., 2023). This suggests that LibriCSS now functions as a reference design pattern for meeting transcription datasets.

Across the literature, several recurring interpretations have become established. First, LibriCSS is valued because it couples real room acoustics with exact, reproducible transcript construction from LibriSpeech (Chen et al., 2020). Second, it exposes errors that are invisible in fully overlapped, utterance-level benchmarks, especially degradation in non-overlap regions, speaker turn handling in 0S, and the mismatch between signal metrics such as SDR and recognition outcomes (Chen et al., 2020, Raj et al., 2020). Third, the benchmark has gradually broadened from a CSS corpus into a shared evaluation platform for separation, diarization, speaker attribution, streaming recognition, and modular pipeline analysis (Kanda et al., 2022, Taherian et al., 2023, Vieting et al., 12 Sep 2025).

Taken together, the body of work on LibriCSS shows that it is not merely a dataset of rerecorded LibriSpeech mixtures. It is a structured benchmark for the full “who spoke what and when” problem under long-form, partially overlapped, meeting-like conditions, and it has played a formative role in aligning source separation research with downstream transcription performance (Chen et al., 2020, Raj et al., 2020).