Turn-to-Diarize: Turn-Based Speaker Diarization

• Turn-to-Diarize is a diarization paradigm that treats speaker turns as primary inference units, simplifying the clustering process in multi-speaker audio.
• It leverages ASR-aligned speaker change detection and constrained clustering to efficiently segment and label speakers in both streaming and end-to-end systems.
• Recent advancements integrate turn-level token detection with role-aware models and unified transcription, yielding improvements in DER, CER, and overall computational cost.

Turn-to-Diarize is a diarization paradigm in which speaker turns are treated as the primary units of inference rather than as by-products of dense framewise segmentation or fixed-window clustering. In the original formulation, a streaming Transformer Transducer emits special speaker-turn tokens and diarization is performed over turn-wise embeddings with constrained clustering (Xia et al., 2021). Later work suggests a broader usage: MOSS Transcribe Diarize is described as essentially a “turn-to-diarize” model because it directly emits a sequence of time-stamped speaker turns from raw multi-speaker audio in a single autoregressive decoding stream (Yu et al., 4 Jan 2026).

1. Conceptual scope and historical development

The original “Turn-to-Diarize” proposal was motivated by the mismatch between conversational structure and conventional online diarization pipelines. In fixed-window clustering systems, a 1-hour call segmented into 400 ms windows yields roughly 9,000 segments, making clustering expensive and making segment length itself a compromise: long windows cross speaker boundaries, while short windows contain too little speech for stable speaker modeling (Xia et al., 2021). Turn-based diarization addresses this by using sparse speaker turns as the objects to embed and cluster.

The later literature broadens the idea in two directions. One direction keeps explicit turn boundaries as the central representation, but derives them at token level from ASR-aligned speaker change detection, as in SCDiar (Zheng et al., 28 Jan 2025). The other direction absorbs turn structure into a fully generative sequence model: in MOSS Transcribe Diarize, speaker labels, timestamps, and lexical content are all output tokens, so diarization is realized directly in the decoding stream rather than by a separate clustering stage (Yu et al., 4 Jan 2026). This suggests that “turn-to-diarize” now refers not only to turn-wise clustering, but more generally to diarization systems in which turn boundaries, speaker identity, and often transcription are modeled jointly.

Formulation	Turn unit	Characteristic mechanism
Original streaming Turn-to-Diarize	<st>-delimited speaker turns	Turn-wise 256-D d-vectors plus constrained spectral clustering
SCDiar	Token-level SCD segments	Token–segment similarity matrix and representative segment selection
MOSS Transcribe Diarize	Time-stamped textual turns	Single autoregressive stream with speaker tags and timestamps

2. The original streaming formulation

“Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection” defines a pipeline in which a Transformer Transducer performs ASR and emits a special <st> token for speaker turns, a separate LSTM-based d-vector model extracts one embedding per detected turn, and constrained spectral clustering assigns turn-wise speaker labels (Xia et al., 2021). The Transformer Transducer uses 15 Transformer blocks with 32 frames of left context and no right context, making it streamable, and is trained on transcripts in which <st> is inserted between speakers rather than on time-stamped speaker labels. The speaker encoder has 3 LSTM layers of 768 units with 256-dimensional projection, and each turn is represented by the embedding at roughly 75% of the turn; turns longer than 6 seconds are further split and treated as must-link in clustering.

A central technical feature is that turn boundaries induce pairwise constraints. Adjacent segments on opposite sides of a high-confidence <st> become cannot-link pairs, while segments without an intervening turn marker become must-link pairs. These constraints are propagated with Exhaustive and Efficient Constraint Propagation, after which the refined affinity matrix is clustered spectrally with optional auto-tuning of the row-wise soft-thresholding percentile. The result is an online system that is sparse in the number of segments and therefore much cheaper than dense window-based clustering.

The reported gains are both computational and diarization-specific. On inbound call-center audio, the best Turn-to-Diarize configuration (“Turn + E2CP + Auto-tune”) achieves 15.22 DER versus 18.24 for dense d-vectors with auto-tuning; on outbound audio it achieves 8.78 versus 14.88; and on Callhome it achieves 5.33 versus 7.79 (Xia et al., 2021). Its 1-hour computational cost is reported as 2.81 GFLOP/s with auto-tune and E2CP, compared with 361.37 GFLOP/s for dense d-vectors with auto-tune. The annotation burden is also reduced, because the system needs only transcripts augmented with <st> tokens rather than time-stamped speaker labels.

3. End-to-end turn-aware neural diarization

A parallel line of work brings turn structure into end-to-end diarization even when the output remains frame-level. DIVE formulates diarization as iterative speaker discovery from raw waveform: at each iteration the model searches for frames containing a single novel speaker, anchors a speaker embedding to such a frame, and then predicts that speaker’s activity over all frames without permutation-invariant training (Zeghidour et al., 2021). The paper explicitly characterizes this as a turn-aware strategy, because speaker identity is built from clean single-speaker regions before being propagated across the recording. On CALLHOME, DIVE reports 6.7% DER with overlap and 5.9% DER without overlap, compared with 7.8% DER for the best EEND-based baseline (Zeghidour et al., 2021).

Subsequent EEND research sharpened the role of turn-taking statistics in training data. “End-to-end Neural Diarization: From Transformer to Conformer” shows that Conformer-based EEND benefits from local turn-pattern modeling but is sensitive to mismatch between simulated and real conversational behavior; mixing simulated and real data yields a 24% error reduction over SA-EEND and a 10% improvement over the best augmented Transformer-based system on two-speaker CALLHOME (Liu et al., 2021). “Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization” makes this more explicit by replacing naive mixture generation with four transition types—turn-hold, turn-switch, interruption, and backchannel—whose parameters are estimated from real conversation, producing simulated data that is statistically closer to real CALLHOME in silence and overlap distributions and improving EEND performance on CALLHOME and CSJ (Yamashita et al., 2022).

ASR-derived turn cues also improve frame-level EEND. “ASR-Aware End-to-end Neural Diarization” incorporates phones, position-in-word, word boundaries, and lexical speaker-change features into a Conformer-based EEND through concatenation, contextualized self-attention, and multi-task learning. The strongest result comes from multi-task learning with position-in-word information, which reduces DER by 20% relative to the acoustic-only baseline (Khare et al., 2022). This indicates that turn structure can be encoded not only as explicit segments, but also as auxiliary phonetic and lexical boundary information that shapes the diarization encoder.

4. Token-level segmentation and role-aware variants

SCDiar pushes turn granularity from utterance level to token level. It combines a CIF-based ASR model, a token-level Speaker Change Detection module, and a diarization module that learns a token–segment similarity matrix $\mathbf{A} \in \mathbb{R}^{N \times M}$, where $N$ is the number of tokens and $M$ the number of SCD-derived segments (Zheng et al., 28 Jan 2025). Segment representatives are then selected by solving a bounded least-squares problem,

$$\min_x \|\mathbf{A}x-\mathbf{1}\|_2^2 \quad \text{s.t.} \quad 0 \le x_j \le 1,$$

so that each token is covered once while the number of selected segments implicitly estimates the number of speakers. In streaming mode, these representative segments are merged with a speaker cache and updated by token-count-weighted averaging.

The system is explicitly designed for long meetings, where online diarization often drifts in speaker count and speaker identity. On AISHELL-4, SCDiar reports $\Delta$cpWER 3.42 and WDER 3.56, compared with 10.09 and 7.84 for an online Core sample+VBx baseline. On an in-house corpus of real meetings with more than ten participants, it reports $\Delta$cpWER 10.66 and WDER 15.36, narrowing the gap between online and offline performance (Zheng et al., 28 Jan 2025). The paper attributes much of this gain to representative segment selection: removing that component raises AISHELL-4 $\Delta$cpWER from 3.42 to 11.96.

A related extension replaces anonymous speaker IDs with role labels. “ASR-Guided Speaker-Role Diarization and Diarization-Guided ASR Decoding” uses a frozen ASR transducer plus an auxiliary role diarization transducer trained from forced alignments with cross-entropy rather than RNNT loss (Ghosh et al., 14 Jul 2025). The study shows that word prediction and role prediction require different predictor contexts: a CNN-2 predictor is suitable for ASR, while an RNN predictor improves role diarization. It also uses role posterior activity to suppress ASR blanks for frequent deletion-prone words such as “yeah” and “okay,” slightly reducing deletion errors. This preserves the central turn-to-diarize principle—speaker or role attribution aligned to the decoded token stream—while making the labels semantically functional rather than anonymous.

5. Generative speaker-attributed transcription

MOSS Transcribe Diarize recasts turn-to-diarize as unified Speaker-Attributed, Time-Stamped Transcription. The model is a multimodal LLM with an audio encoder, a learned projection into the text-LLM embedding space, and a 128k token context window supporting up to 90-minute inputs (Yu et al., 4 Jan 2026). Instead of producing framewise activity matrices or clustered turn embeddings, it emits text tokens that interleave timestamps, speaker tags, and lexical content. For AISHELL-4 and Podcast, the output format is enforced as $N$1 and diarization is performed purely through this token stream. The training objective is standard autoregressive cross-entropy over the mixed sequence; no dedicated diarization loss is introduced.

This formulation is explicitly turn/segment level rather than frame level. Each output segment has a speaker label, a contiguous text span, and start and end times. The model learns turn boundaries jointly from full-recording supervision rather than from pre-segmented clips, and long-range speaker memory is handled implicitly by the 128k-context LLM over a single sequence. The paper contrasts this with chunked systems that must manage cross-chunk speaker caches and re-enrollment (Yu et al., 4 Jan 2026).

The quantitative results are reported in CER, cpCER, and $\Delta$cp. On AISHELL-4, MOSS reports CER 15.43, cpCER 20.04, and $\Delta$cp 4.61, compared with Doubao at 18.18, 27.86, and 9.68 and ElevenLabs at 19.58, 37.95, and 18.36. On Podcast, MOSS reports 4.46, 6.97, and 2.50. On Movies, it reports 7.50, 13.36, and 5.86, outperforming Doubao, ElevenLabs, GPT-4o, Gemini 2.5 Pro, and Gemini 3 Pro on cpCER (Yu et al., 4 Jan 2026). Because cpCER is computed from concatenated per-speaker transcripts under optimal speaker permutation, these results indicate improvements in joint “who-said-what” accuracy rather than in ASR alone.

6. Evaluation regimes, limitations, and open problems

Turn-to-Diarize systems are evaluated under several incompatible regimes. The original streaming formulation and most end-to-end diarization papers use DER with collars and explicit overlap choices; DIVE is evaluated on CALLHOME with a 250 ms collar and overlap-inclusive or overlap-exclusive conditions (Zeghidour et al., 2021). MOSS instead evaluates SATS through CER, cpCER, and $\Delta$cp, and explicitly does not report DER, JER, or overlap-specific metrics (Yu et al., 4 Jan 2026). SCDiar reports WDER and $N$0cpWER, because the output is speaker-attributed ASR rather than diarization alone (Zheng et al., 28 Jan 2025). These metrics emphasize different failure modes: boundary placement, speaker confusion, lexical deletions, and permutation stability are weighted differently across them.

A broader benchmark reinforces the remaining difficulty of the underlying task. “Benchmarking Diarization Models” evaluates five state-of-the-art systems on 196.6 hours of multilingual audio and finds that PyannoteAI achieves the best overall performance at 11.2% DER, while DiariZen reaches 13.3% DER as a competitive open-source alternative; the primary cause of diarization errors is missed speech segments followed by speaker confusion, especially in high-speaker-count settings (Lanzendörfer et al., 30 Sep 2025). This finding is consistent with the limitations reported across turn-to-diarize systems. The original Turn-to-Diarize paper leaves overlap largely to segmentation and clustering heuristics (Xia et al., 2021). DIVE is validated only on the two-speaker task (Zeghidour et al., 2021). SCDiar does not model overlap explicitly and excludes segments with fewer than 10 tokens from representative selection and cache updates (Zheng et al., 28 Jan 2025). MOSS models overlaps in training and via tags such as <ovl> on Movies, but remains turn-based and does not output framewise multi-speaker activity (Yu et al., 4 Jan 2026).

Several research directions therefore remain open. One is scalability in speaker cardinality, especially for streaming systems that must avoid speaker duplication over hours-long meetings. Another is overlap representation: turn-level outputs are natural for transcription and dialogue analysis, but they compress simultaneous speech unless augmented with richer tags or multi-label timing models. A third is tighter ASR–diarization coupling. The later literature suggests that token-aligned change detection, role-aware predictors, and fully generative SATS decoding are converging toward a common objective: a system that emits lexical content, timing, and speaker or role structure in a single coherent sequence, while remaining robust in long-form, high-speaker-count, and multilingual conditions.