Continuous Speech Separation (CSS)

Updated 23 April 2026

CSS is a framework that extracts discrete, overlap-free single-speaker streams from continuous multi-speaker audio using windowed segmentation and spectral masking.
The method employs neural architectures like Conformers and dual-path transformers with permutation alignment to ensure temporal speaker consistency.
Benchmarks such as LibriCSS demonstrate that CSS significantly reduces word error rates and improves performance in transcription and diarization tasks.

Continuous Speech Separation (CSS) is a framework designed to extract discrete, single-speaker speech streams from continuous multi-speaker audio, such as meetings or conversational recordings, in which utterances may partially overlap. CSS is motivated by the limitations of utterance-level speech separation algorithms, which are ill-suited for unsegmented audio with arbitrary overlap patterns. The defining constraint of CSS is to produce a small, fixed number (typically two) of output audio streams, each being overlap-free and suitable for downstream tasks such as automatic speech recognition (ASR) or diarization, even as the number of speakers and overlap patterns within the audio may vary arbitrarily (Chen et al., 2020).

1. Formal Definition, Operational Principles, and Datasets

CSS is formally defined as the task of taking a continuous audio stream $x(t)$ , containing multiple speakers with possibly overlapping utterances, and generating $N$ output streams $\{\hat s_k(t)\}_{k=1}^N$ . These are required to satisfy the constraint that, at any time $t$ , at most one speaker is active per output channel, i.e., the output streams are overlap-free (Chen et al., 2020):

$x(t) = \sum_{i=1}^I s_i(t) + n(t), \quad \hat s_k(t) = \text{CSS}(x(t)), \quad \text{at most one active speaker per } \hat s_k(t)$

In the STFT domain, mask-based approaches are common:

$\hat S_k(t,f) = M^k(t,f) X(t,f), \qquad \hat s_k(t) = \mathrm{iSTFT}\{\hat S_k(t,f)\}$

Permutation-invariant training (PIT) or its generalizations are used to align network outputs to ground-truth sources due to identity ambiguity.

The most representative evaluation benchmark for CSS is the LibriCSS dataset (Chen et al., 2020), which offers 10 h of realistic multi-channel conversational mixtures with various overlap ratios, recorded by replaying concatenated LibriSpeech utterances in real meeting rooms. This allows robust assessment of CSS quality in scenarios matching downstream ASR and diarization workloads.

2. Core CSS Processing Pipeline

A canonical CSS pipeline processes the long audio stream by:

Windowed segmentation: The input is split into overlapping sliding windows (e.g., 2.4–4 s, 50% overlap), typically using multi-channel STFT features for each window (Wang et al., 2021, Morrone et al., 2022).
Separation network: For each window, a neural network (e.g., BLSTM (Chen et al., 2020), Conformer (Chen et al., 2020), dual-path Transformer (Li et al., 2021), etc.) estimates $N$ speech masks (one per potential speaker) and optionally a noise mask, followed by spectral masking or (in multi-channel cases) beamforming operations.
Permutation alignment ("stitching"): Across adjacent windows, output streams are reordered using overlap-region similarity to maintain temporal speaker consistency (Wang et al., 2021).
Overlap-add: Outputs are merged using weighted overlap-add, producing continuous overlap-free output streams.
Post-processing: Additional blocks may include channel selection, speaker counting and merging, or voice activity detection (VAD) (Wang et al., 2021, Wang et al., 2020, Morrone et al., 2023).

The pipeline design and latency are determined by window size and stride; smaller window size reduces algorithmic latency but may degrade separation, especially in high-overlap segments (Morrone et al., 2022). A 4 s window and 2 s shift is a common trade-off (Wang et al., 2021).

3. Neural Network Architectures and Algorithmic Variants

CSS networks are built from several architectural paradigms:

Recurrent/Conformer encoders: BLSTM-based (Chen et al., 2020, Wang et al., 2020), Transformer (Li et al., 2021, Chen et al., 2020), and especially Conformer (Chen et al., 2020, Wang et al., 2021) models dominate. The Conformer interleaves self-attention and convolutional modules, with strong empirical separation and ASR gains.
Dual-Path Models: Dual-Path RNN/Transformer architectures alternate local (within-window) and global (cross-window) context blocks, explicitly modeling both short- and long-range patterns (Li et al., 2021). This yields significant WER reductions compared to pure local or global models.
Graph-PIT and Generalized Assignment: Classical PIT is limited to a fixed number of output streams, but in multi-speaker meetings the number of speakers may exceed the output capacity locally. Graph-PIT (Neumann et al., 2021, Li et al., 2022) generalizes PIT into a graph-coloring problem, enabling truly continuous, arbitrarily long CSS with only the requirement that at most $N$ speakers are simultaneously active at any time.
Recursive and Adaptive Separation: Recurrent Selective Attention Networks (RSAN) recursively emit masks with a learned stop flag, producing a variable number of outputs per block based on active speaker counting (Zhang et al., 2021). Blockwise dependency schemes can propagate separation context forward, further reducing leakage and instability.
Low-latency and Early-exit Variants: For real-time applications, architectures such as Skipping Memory LSTM (SkiM) decouple local and global recurrence, allowing millisecond-latency CSS (Li et al., 2022), while Early Exit Transformers adapt depth dynamically to content complexity (Chen et al., 2020).

Multi-channel CSS systems incorporate geometry-agnostic spatial modules (e.g., VarArray (Yoshioka et al., 2021), cross-channel attention (Wang et al., 2021), TAC blocks (Yoshioka et al., 2021, Wang et al., 2022)), or neural MVDR beamformers (including all-neural ADL-MVDR (Zhang et al., 2021)) to leverage spatial diversity regardless of microphone placement.

4. Data Augmentation, Robustness, and Duplication Mitigation

Realistic deployment with ad hoc or variable microphone configurations requires robustness to array geometry, channel asynchrony, and device mismatch. Techniques include:

Device distortion simulation: Band-pass filtering, amplitude clipping, and delay perturbation are applied probabilistically to simulated training signals to close the gap between synthetic training data and real-world device artifacts (Wang et al., 2021).
Speaker counting and merging: A speaker-counting network (either producing active speaker VAD streams or a scalar count) is used to merge output streams in single-talker regions, eliminating speech duplication (Wang et al., 2021, Wang et al., 2020).
Permutation alignment and clustering: Overlap-region similarity (usually Euclidean) is used to align output streams; additional diarization or direction-of-arrival (DOA) clustering across windows achieves robust speaker consistency and grouping (Wang et al., 2021).

These methods substantially reduce insertion errors (speech leakage or duplication) in single-talker or low-overlap regions after CSS.

5. Evaluation Protocols and Performance Metrics

CSS research emphasizes ASR-based metrics over signal-level SDR due to weak correlation between waveform fidelity and downstream recognition (Chen et al., 2020). Standard evaluation strategies include:

Speaker-agnostic/attributed WER: Use asclite or equivalent algorithms to align hypothesis streams to reference utterances, giving either agnostic or speaker-attributed word error rates. "Concatenated minimum-Permutation WER (cpWER)" is common for diarization-integrated scenarios (Wang et al., 2021).
DER (Diarization Error Rate): In diarization-enabled CSS, DER is used to assess the quality of blockwise or global speaker labeling (Wang et al., 2021, Taherian et al., 2023).
Latency, SI-SDR, Real-time Factor: Latency is measured as the window stride or block shift; real-time factor (RTF) and memory footprint are critical in streaming setups (Morrone et al., 2022, Li et al., 2022).
LibriCSS Benchmarks: Overlap-sensitive partitioning enables controlled evaluation from 0% to 40% overlap (Chen et al., 2020).

Top-performing CSS pipelines (e.g., Conformer-based, dual-path transformer, or VarArray with MVDR) generally halve WER on LibriCSS under high-overlap conditions, reaching WERs in the low tens or single digits for 0–30% overlap with arrays of 5–7 microphones (Chen et al., 2020, Yoshioka et al., 2021, Li et al., 2021, Wang et al., 2021).

6. Strengths, Limitations, and Future Directions

Strengths:

Robust Overlap Removal: CSS consistently improves ASR in mixed and overlapping speech by producing overlap-free streams, independent of the number and order of speakers (Chen et al., 2020).
Scalability to Long Audio: Sliding-window and dual-path, graph-PIT, or recurrent attention designs enable processing of entire meetings or multi-hour dialogues (Li et al., 2021, Neumann et al., 2021, Li et al., 2022).
Microphone and Hardware Flexibility: Modern models are array-geometry-agnostic, generalizing across arbitrary device configurations and supporting ad hoc, asynchronous input (Yoshioka et al., 2021, Wang et al., 2021).
Low-latency and Online Processing: Models with small strides, efficient recurrence, and adaptive depth achieve sub-millisecond inference latencies, suitable for streaming captioning or transcription (Li et al., 2022, Chen et al., 2020).

Limitations:

Speaker Occupancy Constraint: Most systems assume at most two concurrent speakers per block; scaling to dynamic and larger concurrency remains challenging (Wang et al., 2021, Yoshioka et al., 2021).
Duplication and Permutation Sensitivity: Fixed-output architectures risk speech leakage or identity swapping; deep merging, robust speaker counting, or recursive separation partially address but do not eliminate the issue (Zhang et al., 2021).
Performance–Latency Trade-Offs: Large windows or full-meeting context improve separation but increase algorithmic latency and memory usage; ultra-short windows can degrade quality at high overlap (Morrone et al., 2022).
Manual Hyperparameter Tuning: Thresholds for speaker counting or duplication mitigation are often hand-tuned.

Future Directions:

Truly adaptive output models: Recursive or attention-based architectures that discover active speaker count dynamically and allocate output streams on the fly (Zhang et al., 2021).
End-to-end CSS-ASR Training: Fully integrating separation and recognition (and even diarization) to directly optimize sequence-level WER and DER, closing the gap between signal-level and task-level metrics (Wang et al., 2022).
Advanced spatial modeling: Learning array geometry embeddings or extending beamforming front-ends to arbitrary, time-varying topologies (Zhang et al., 2021).
Unified clustering and separation: Joint inventory formation, profile selection, and separation in single networks for better speaker tracking across long meetings (Han et al., 2020).
Minimal-label or semi-supervised adaptation: Leveraging large amounts of real, unlabeled conversational data via teacher-student, semi-supervised, or self-supervised learning (Wang et al., 2022).
Ultra-low latency and edge deployment: Further reduction in latency and footprint for real-world, on-device, or streaming solutions (Li et al., 2022).

CSS thus represents a unifying solution to practical multi-speaker separation challenges in conversation and meeting transcription, with an ecosystem of architectures and evaluation protocols tailored to demanding, unconstrained real-world audio (Chen et al., 2020, Wang et al., 2021, Morrone et al., 2022, Yoshioka et al., 2021, Li et al., 2022, Zhang et al., 2021).