--- title: 'Turn-to-Diarize: Turn-Based Speaker Diarization' url: https://www.emergentmind.com/topics/turn-to-diarize type: topic --- # Turn-to-Diarize: Turn-Based Speaker Diarization Turn-to-Diarize is a diarization paradigm in which speaker turns are treated as the primary units of inference rather than as by-products of dense framewise segmentation or fixed-window clustering. In the original formulation, a streaming Transformer Transducer emits special speaker-turn tokens and diarization is performed over turn-wise embeddings with constrained clustering [2109.11641]. Later work suggests a broader usage: MOSS Transcribe Diarize is described as essentially a “turn-to-diarize” model because it directly emits a sequence of time-stamped speaker turns from raw multi-speaker audio in a single autoregressive decoding stream [2601.01554]. ## 1. Conceptual scope and historical development The original “Turn-to-Diarize” proposal was motivated by the mismatch between conversational structure and conventional online diarization pipelines. In fixed-window clustering systems, a 1-hour call segmented into 400 ms windows yields roughly 9,000 segments, making clustering expensive and making segment length itself a compromise: long windows cross speaker boundaries, while short windows contain too little speech for stable speaker modeling [2109.11641]. Turn-based diarization addresses this by using sparse speaker turns as the objects to embed and cluster. The later literature broadens the idea in two directions. One direction keeps explicit turn boundaries as the central representation, but derives them at token level from ASR-aligned speaker change detection, as in SCDiar [2501.16641]. The other direction absorbs turn structure into a fully generative sequence model: in MOSS Transcribe Diarize, speaker labels, timestamps, and lexical content are all output tokens, so diarization is realized directly in the decoding stream rather than by a separate clustering stage [2601.01554]. This suggests that “turn-to-diarize” now refers not only to turn-wise clustering, but more generally to diarization systems in which turn boundaries, speaker identity, and often transcription are modeled jointly. | Formulation | Turn unit | Characteristic mechanism | |---|---|---| | Original streaming Turn-to-Diarize | ``-delimited speaker turns | Turn-wise 256-D d-vectors plus constrained spectral clustering | | SCDiar | Token-level SCD segments | Token–segment similarity matrix and representative segment selection | | MOSS Transcribe Diarize | Time-stamped textual turns | Single autoregressive stream with speaker tags and timestamps | ## 2. The original streaming formulation “Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection” defines a pipeline in which a Transformer Transducer performs ASR and emits a special `` token for speaker turns, a separate LSTM-based d-vector model extracts one embedding per detected turn, and constrained spectral clustering assigns turn-wise speaker labels [2109.11641]. The Transformer Transducer uses 15 Transformer blocks with 32 frames of left context and no right context, making it streamable, and is trained on transcripts in which `` is inserted between speakers rather than on time-stamped speaker labels. The speaker encoder has 3 LSTM layers of 768 units with 256-dimensional projection, and each turn is represented by the embedding at roughly 75% of the turn; turns longer than 6 seconds are further split and treated as must-link in clustering. A central technical feature is that turn boundaries induce pairwise constraints. Adjacent segments on opposite sides of a high-confidence `` become cannot-link pairs, while segments without an intervening turn marker become must-link pairs. These constraints are propagated with Exhaustive and Efficient Constraint Propagation, after which the refined affinity matrix is clustered spectrally with optional auto-tuning of the row-wise soft-thresholding percentile. The result is an online system that is sparse in the number of segments and therefore much cheaper than dense window-based clustering. The reported gains are both computational and diarization-specific. On inbound call-center audio, the best Turn-to-Diarize configuration (“Turn + E2CP + Auto-tune”) achieves 15.22 DER versus 18.24 for dense d-vectors with auto-tuning; on outbound audio it achieves 8.78 versus 14.88; and on Callhome it achieves 5.33 versus 7.79 [2109.11641]. Its 1-hour computational cost is reported as 2.81 GFLOP/s with auto-tune and E2CP, compared with 361.37 GFLOP/s for dense d-vectors with auto-tune. The annotation burden is also reduced, because the system needs only transcripts augmented with `` tokens rather than time-stamped speaker labels. ## 3. End-to-end turn-aware neural diarization A parallel line of work brings turn structure into end-to-end diarization even when the output remains frame-level. DIVE formulates diarization as iterative speaker discovery from raw waveform: at each iteration the model searches for frames containing a single novel speaker, anchors a speaker embedding to such a frame, and then predicts that speaker’s activity over all frames without permutation-invariant training [2105.13802]. The paper explicitly characterizes this as a turn-aware strategy, because speaker identity is built from clean single-speaker regions before being propagated across the recording. On CALLHOME, DIVE reports 6.7% DER with overlap and 5.9% DER without overlap, compared with 7.8% DER for the best EEND-based baseline [2105.13802]. Subsequent EEND research sharpened the role of turn-taking statistics in training data. “End-to-end Neural Diarization: From Transformer to Conformer” shows that Conformer-based EEND benefits from local turn-pattern modeling but is sensitive to mismatch between simulated and real conversational behavior; mixing simulated and real data yields a 24% error reduction over SA-EEND and a 10% improvement over the best augmented Transformer-based system on two-speaker CALLHOME [2106.07167]. “Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization” makes this more explicit by replacing naive mixture generation with four transition types—turn-hold, turn-switch, interruption, and backchannel—whose parameters are estimated from real conversation, producing simulated data that is statistically closer to real CALLHOME in silence and overlap distributions and improving EEND performance on CALLHOME and CSJ [2204.11232]. ASR-derived turn cues also improve frame-level EEND. “ASR-Aware End-to-end Neural Diarization” incorporates phones, position-in-word, word boundaries, and lexical speaker-change features into a Conformer-based EEND through concatenation, contextualized self-attention, and multi-task learning. The strongest result comes from multi-task learning with position-in-word information, which reduces DER by 20% relative to the acoustic-only baseline [2202.01286]. This indicates that turn structure can be encoded not only as explicit segments, but also as auxiliary phonetic and lexical boundary information that shapes the diarization encoder. ## 4. Token-level segmentation and role-aware variants SCDiar pushes turn granularity from utterance level to token level. It combines a CIF-based ASR model, a token-level Speaker Change Detection module, and a diarization module that learns a token–segment similarity matrix \( \mathbf{A} \in \mathbb{R}^{N \times M} \), where \(N\) is the number of tokens and \(M\) the number of SCD-derived segments [2501.16641]. Segment representatives are then selected by solving a bounded least-squares problem, \[ \min_x \|\mathbf{A}x-\mathbf{1}\|_2^2 \quad \text{s.t.} \quad 0 \le x_j \le 1, \] so that each token is covered once while the number of selected segments implicitly estimates the number of speakers. In streaming mode, these representative segments are merged with a speaker cache and updated by token-count-weighted averaging. The system is explicitly designed for long meetings, where online diarization often drifts in speaker count and speaker identity. On AISHELL-4, SCDiar reports \(\Delta\)cpWER 3.42 and WDER 3.56, compared with 10.09 and 7.84 for an online Core sample+VBx baseline. On an in-house corpus of real meetings with more than ten participants, it reports \(\Delta\)cpWER 10.66 and WDER 15.36, narrowing the gap between online and offline performance [2501.16641]. The paper attributes much of this gain to representative segment selection: removing that component raises AISHELL-4 \(\Delta\)cpWER from 3.42 to 11.96. A related extension replaces anonymous speaker IDs with role labels. “ASR-Guided Speaker-Role Diarization and Diarization-Guided ASR Decoding” uses a frozen ASR transducer plus an auxiliary role diarization transducer trained from forced alignments with cross-entropy rather than RNNT loss [2507.17765]. The study shows that word prediction and role prediction require different predictor contexts: a CNN-2 predictor is suitable for ASR, while an RNN predictor improves role diarization. It also uses role posterior activity to suppress ASR blanks for frequent deletion-prone words such as “yeah” and “okay,” slightly reducing deletion errors. This preserves the central turn-to-diarize principle—speaker or role attribution aligned to the decoded token stream—while making the labels semantically functional rather than anonymous. ## 5. Generative speaker-attributed transcription MOSS Transcribe Diarize recasts turn-to-diarize as unified Speaker-Attributed, Time-Stamped Transcription. The model is a multimodal LLM with an audio encoder, a learned projection into the text-LLM embedding space, and a 128k token context window supporting up to 90-minute inputs [2601.01554]. Instead of producing framewise activity matrices or clustered turn embeddings, it emits text tokens that interleave timestamps, speaker tags, and lexical content. For AISHELL-4 and Podcast, the output format is enforced as ```text [start_timestamp] [Sxx] [end_timestamp] ``` and diarization is performed purely through this token stream. The training objective is standard autoregressive cross-entropy over the mixed sequence; no dedicated diarization loss is introduced. This formulation is explicitly turn/segment level rather than frame level. Each output segment has a speaker label, a contiguous text span, and start and end times. The model learns turn boundaries jointly from full-recording supervision rather than from pre-segmented clips, and long-range speaker memory is handled implicitly by the 128k-context LLM over a single sequence. The paper contrasts this with chunked systems that must manage cross-chunk speaker caches and re-enrollment [2601.01554]. The quantitative results are reported in CER, cpCER, and \(\Delta\)cp. On AISHELL-4, MOSS reports CER 15.43, cpCER 20.04, and \(\Delta\)cp 4.61, compared with Doubao at 18.18, 27.86, and 9.68 and ElevenLabs at 19.58, 37.95, and 18.36. On Podcast, MOSS reports 4.46, 6.97, and 2.50. On Movies, it reports 7.50, 13.36, and 5.86, outperforming Doubao, ElevenLabs, GPT-4o, Gemini 2.5 Pro, and Gemini 3 Pro on cpCER [2601.01554]. Because cpCER is computed from concatenated per-speaker transcripts under optimal speaker permutation, these results indicate improvements in joint “who-said-what” accuracy rather than in ASR alone. ## 6. Evaluation regimes, limitations, and open problems Turn-to-Diarize systems are evaluated under several incompatible regimes. The original streaming formulation and most end-to-end diarization papers use DER with collars and explicit overlap choices; DIVE is evaluated on CALLHOME with a 250 ms collar and overlap-inclusive or overlap-exclusive conditions [2105.13802]. MOSS instead evaluates SATS through CER, cpCER, and \(\Delta\)cp, and explicitly does not report DER, JER, or overlap-specific metrics [2601.01554]. SCDiar reports WDER and \(\Delta\)cpWER, because the output is speaker-attributed ASR rather than diarization alone [2501.16641]. These metrics emphasize different failure modes: boundary placement, speaker confusion, lexical deletions, and permutation stability are weighted differently across them. A broader benchmark reinforces the remaining difficulty of the underlying task. “Benchmarking Diarization Models” evaluates five state-of-the-art systems on 196.6 hours of multilingual audio and finds that PyannoteAI achieves the best overall performance at 11.2% DER, while DiariZen reaches 13.3% DER as a competitive open-source alternative; the primary cause of diarization errors is missed speech segments followed by speaker confusion, especially in high-speaker-count settings [2509.26177]. This finding is consistent with the limitations reported across turn-to-diarize systems. The original Turn-to-Diarize paper leaves overlap largely to segmentation and clustering heuristics [2109.11641]. DIVE is validated only on the two-speaker task [2105.13802]. SCDiar does not model overlap explicitly and excludes segments with fewer than 10 tokens from representative selection and cache updates [2501.16641]. MOSS models overlaps in training and via tags such as `` on Movies, but remains turn-based and does not output framewise multi-speaker activity [2601.01554]. Several research directions therefore remain open. One is scalability in speaker cardinality, especially for streaming systems that must avoid speaker duplication over hours-long meetings. Another is overlap representation: turn-level outputs are natural for transcription and dialogue analysis, but they compress simultaneous speech unless augmented with richer tags or multi-label timing models. A third is tighter ASR–diarization coupling. The later literature suggests that token-aligned change detection, role-aware predictors, and fully generative SATS decoding are converging toward a common objective: a system that emits lexical content, timing, and speaker or role structure in a single coherent sequence, while remaining robust in long-form, high-speaker-count, and multilingual conditions. Source: https://www.emergentmind.com/topics/turn-to-diarize