--- title: Diarization-Conditioned Spoken Language Models url: https://www.emergentmind.com/topics/diarization-conditioned-spoken-language-models-slms type: topic --- # Diarization-Conditioned Spoken Language Models Diarization-conditioned spoken language models (SLMs) are spoken language models that consume, predict, or are otherwise modulated by speaker diarization information—speaker labels, turn boundaries, timestamps, overlap structure, or speaker-conditioned masks—so that multi-speaker speech is represented with explicit attribution rather than as a single undifferentiated stream. In recent work, the term spans at least three regimes: post-processing language models that refine speaker-attributed transcripts after ASR and diarization, decoder-only Speech LLMs that directly generate interleaved speaker, timestamp, and text tokens, and frozen-decoder SLMs whose acoustic encoder is conditioned on diarization masks to extract target-speaker representations [2401.03506][2507.02927][2606.18134]. Within the broader SLM taxonomy, diarization conditioning can be injected into pure speech LMs, speech+text LMs, or speech-aware text LMs, and it functions as a mechanism for grounding multi-speaker conversational structure inside a model family otherwise centered on content modeling [2504.08528]. ## 1. Conceptual scope and antecedents Speaker diarization answers the question of “who spoke when,” whereas SLMs address speech understanding and generation. Diarization-conditioned SLMs arise when these two objectives are coupled closely enough that speaker attribution becomes part of the model’s conditioning state, output space, or post-hoc correction logic. The coupling may be weak, as in a text-only second pass over ASR and diarization outputs, or strong, as in end-to-end generation of speaker and timestamp tokens from raw audio [2504.08528][2507.02927]. Important antecedents predate fully unified Speech LLMs. In telephonic interviews, linguistic content can be more discriminative than acoustics for speaker assignment when one role asks a known set of questions and the other provides short, formulaic answers. “Language Modelling for Speaker Diarization in Telephonic Interviews” reports that combining character-level lexical features with acoustic scores yields an 84.29% improvement in word-level DER relative to a HMM/VB baseline, with final ASR-condition WDER dropping from 13.55% to 1.98% in a two-speaker Spanish call-center setting [2501.17893]. This establishes a central principle of the area: speaker attribution can be recoverable from discourse structure and lexical regularities even when acoustic segmentation is unreliable. Subsequent lexical correction work generalizes that principle to modular ASR+diarization pipelines. “Lexical Speaker Error Correction” formulates a discriminative correction problem over lexical context and first-pass speaker hypotheses, achieving consistent 15–30% relative WDER reductions on Fisher, RT03-CTS, and CALLHOME American English after training on Fisher only [2306.09313]. “DiarizationLM” moves the same idea into a prompted LLM framework, representing ASR words and speaker labels as compact text and applying transcript-preserving speaker transfer (TPST) so that speaker labels can change while the original ASR wording is retained; on Fisher and Callhome English, a finetuned PaLM 2-S reduces WDER by 55.5% and 44.9% relative, respectively [2401.03506]. These systems show that diarization-conditioned modeling is not synonymous with end-to-end audio generation; it also includes text-conditioned correction layers operating over already transcribed conversations. ## 2. Main architectural paradigms Recent literature organizes diarization-conditioned SLMs around a small number of recurring architectural patterns. | Paradigm | Representative systems | Core mechanism | |---|---|---| | Post-processing correction | SEC, DiarizationLM | Refine word-level speaker labels from transcript and initial diarization | | Unified speech generation | ST-ShinozakiLab Speech LLM | Generate speaker, timestamp, and text tokens jointly from audio | | Encoder-side conditioning | Dixtral | Condition acoustic encoder on diarization masks; keep decoder frozen | In post-processing systems, the model does not replace ASR or diarization. SEC uses a RoBERTa-base language model to produce contextual embeddings, concatenates them with first-pass speaker label features, and applies a lightweight Transformer encoder to output corrected speaker posteriors for two-speaker windows. DiarizationLM instead serializes the ASR transcript and speaker sequence into a compact prompt such as ` good morning how are you`, asks an LLM to rewrite only the speaker attribution, and then maps the result back onto the original ASR words via TPST and a Hungarian label-permutation step [2306.09313][2401.03506]. Unified speech generation internalizes diarization inside the decoder’s output vocabulary. “A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations” reformulates each local turn as an interleaved sequence of `[speaker token, start timestamp token, text tokens, end timestamp token]`, so that diarization and ASR become a single next-token prediction problem. The model uses a Whisper encoder, a sub-sampling projector, and a decoder-only Llama-3.2 backbone with LoRA, and it is explicitly designed for two-speaker multilingual conversations [2507.02927]. Encoder-side conditioning reverses the design choice that unified generation makes. “Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning” argues that adapting the decoder through Serialized Output Training risks catastrophic forgetting of instruction-following, reasoning, summarization, and QA abilities. Its alternative, Dixtral, conditions a DiCoW acoustic encoder on diarization masks so that the frozen Voxtral decoder always receives a target-speaker representation rather than an interleaved multi-speaker sequence. Speaker attribution is recovered by running the model separately for each target speaker [2606.18134]. The broader SLM survey provides the abstraction under which these designs coexist. In its notation, speech is encoded as $H_{sp} = Enc_{sp}(X_{sp})$, adapted into the sequence-model space, and consumed by a sequence model. Diarization conditioning can therefore enter as prompt tokens, auxiliary embeddings, masks, or adapter-side modulation, which is exactly what these concrete systems instantiate in different ways [2504.08528]. ## 3. Conditioning signals, objectives, and inference procedures The unified multilingual Speech LLM operationalizes diarization-conditioned generation by treating speaker identity and timing as ordinary tokens in the target stream. Let $Y = (y_1, y_2, \dots, y_T)$ denote the mixed sequence of speaker, timestamp, and text tokens, and let $z$ be the projected audio feature sequence from the Whisper encoder and projector. Training uses standard next-token cross-entropy, $$ L = - \sum_{t=1}^{T} \log p(y_t \mid y_{ ` suffix and an `[eod]` termination token, recursively binary-partitions long transcripts to fit a 4,096-token budget, and applies TPST so that speaker labels inferred by the LLM are transferred back onto the original ASR words without changing WER. SEC instead runs on sliding windows of words, usually 30 words, tokenizes them with RoBERTa, concatenates contextual lexical embeddings with initial speaker-label features, and predicts corrected token-level speaker labels; windows with more than two hypothesized speakers are bypassed [2401.03506][2306.09313]. An earlier iterative fusion pattern remains relevant because it clarifies how lexical and acoustic evidence can be alternated. In the telephonic interview system, a text-only LSTM first predicts word-level speaker labels, then per-recording speaker GMMs are estimated from MFCC frames grouped by those labels, acoustic posteriors $P(\mathrm{Cus}\mid w)$ are computed for each word, and a second LSTM consumes lexical embeddings plus the acoustic score. This procedure converges in 2–3 iterations and has an EM-like interpretation: label inference alternates with acoustic model re-estimation [2501.17893]. ## 4. Metrics, benchmarks, and downstream error budgets Diarization-conditioned SLMs inherit evaluation protocols from both diarization and multi-speaker ASR. The diarization benchmark most directly framed for SLM integration evaluates five diarization systems on 196.6 hours of multilingual audio across CALLHOME, VoxConverse, AMI, and AliMeeting, using diarization error rate $$ DER = \frac{FA + MISS + CONF}{TOTAL} $$ with a 0.25 s collar and `skip_overlap = False`, so overlapped speech is explicitly scored and penalized when mislabeled. Under this strict policy, PyannoteAI achieves the best overall performance at 11.2% DER, DiariZen reaches 13.3% as a competitive open-source alternative, DiariZen is strongest on VoxConverse at 5.2% DER, and Sortformer v2-stream is best on AliMeeting at 7.0% DER [2509.26177]. The same benchmark isolates failure modes that are especially consequential for downstream SLMs. Missed speech is the dominant source of error across models and datasets, particularly in meetings with substantial overlap, and speaker confusion rises sharply in high-speaker settings. The average duration of missed segments is approximately 350 ms, while segments shorter than 0.5 s contribute less than 5% of errors. The reported interpretation is that timestamp imprecision on longer segments—speech onset and offset placement—matters more than outright failure to detect very short events. This is directly relevant for SLM conditioning, because turn attribution, summarization, and dialogue understanding depend on accurate boundaries as much as on speaker identity [2509.26177]. Multi-speaker ASR metrics capture the combined effect of transcription and attribution. The MLC-SLM Challenge evaluates joint diarization and recognition with DER and time-constrained permutation WER/CER (tcpWER/tcpCER), where the time constraint and a 5 s collar penalize misattribution and timing misalignment at the conversation level. Dixtral instead reports concatenated minimum-permutation WER (cpWER), which minimizes WER over permutations of hypothesized speaker streams after concatenating each speaker’s transcript. Word-level correction papers focus on WDER, the fraction of words with incorrect speaker labels, which is appropriate when the transcript is fixed and only speaker attribution is refined [2507.02927][2606.18134][2401.03506]. The benchmark paper also makes the downstream dependency explicit through a simple attribution bound. Let $S$ be total speech time and $T$ total time. A lower bound on speaker-attribution accuracy for speech tokens conditioned on diarization is $$ A_{speech} \ge 1 - \frac{MISS + CONF}{S}, $$ and for pipelines in which a fraction $p_{asr}$ of false-alarm time survives ASR, overall token attribution can be approximated as $$ A_{total} \approx \frac{S}{T}\left[1 - \frac{MISS + CONF}{S}\right] - p_{asr}\frac{FA}{T}. $$ The practical conclusion is that minimizing missed speech and speaker confusion is more critical than minimizing false alarms in speech-centric downstream tasks [2509.26177]. ## 5. Empirical performance across deployment regimes Results differ markedly by conversational regime, which is why diarization-conditioned SLMs have developed along several parallel lines. In structured two-speaker telephony, lexical conditioning is especially powerful. The telephonic interview system reports that a text-only model already reduces ASR-condition WDER from 13.55% to 5.34%, and the final fused lexical-acoustic iterative model reaches 1.98% WDER. SEC, trained and tuned on Fisher, delivers relative WDER reductions of roughly 32% on Fisher, 27% on RT03-CTS test, and 20% on CALLHOME American English test. DiarizationLM’s finetuned PaLM 2-S reaches WDER 2.37 and cpWER 16.93 on Fisher, and WDER 4.25 and cpWER 20.22 on Callhome, while preserving the original ASR wording through TPST [2501.17893][2306.09313][2401.03506]. In multilingual two-speaker conversational audio, joint generation becomes competitive when local and global consistency are engineered carefully. The unified Speech LLM submitted by ST-ShinozakiLab to the MLC-SLM Challenge obtains 27.25 tcpWER/tcpCER on the official Task II evaluation set versus a 60.39 baseline, a 54.87% relative improvement, and ranks 8th overall despite using a smaller LLM backbone. On the development set, the Llama-3.2-3B version reports DER 13.36, tcpWER/tcpCER 28.23, and cpWER 24.97, outperforming the corresponding 1B variant except in Thai, where the 1B model slightly wins. The architecture’s gains are attributed not to larger scale but to the interleaved token format and diarization-aware inference procedure [2507.02927]. In far-field multi-speaker audio, encoder-side conditioning currently provides the strongest evidence for preserving general SLM capabilities while improving speaker-attributed transcription. Dixtral reports a macro-average cpWER of 15.4% across NOTSOFAR-1, AMI, LibriSpeechMix, and Mixer6, compared with 44.4% for Gemini 3.0 Flash, 35.2% for VibeVoice, and 31.4% for Voxtral Mini Transcribe V2, corresponding to absolute improvements of 29.0%, 19.8%, and 16.0%. On the NSF-QA benchmark, zero-shot Dixtral after ASR-only training reaches 54.6% content QA, on par with Gemini’s 55.1% on far-field audio, and 24.4 ROUGE-L in summarization, approximately matching Voxtral on close-talk audio at 24.1. Fine-tuning on NSF-QA raises performance to 73.0% content QA, 47.6% emotion QA, 95.5% gender QA, and 41.4 ROUGE-L, surpassing both Gemini and Voxtral operating on close-talk across all reported tasks. At the same time, adding LoRA to the decoder improves some ASR numbers but harms summarization, with ROUGE-L dropping to 15.4, which underscores the trade-off between transcription specialization and preserved instruction-following [2606.18134]. These empirical results indicate that the optimal conditioning strategy depends on the interaction between speaker cardinality, overlap rate, domain structure, and the intended downstream task. This suggests that diarization-conditioned SLMs are better understood as a design space than as a single architecture class. ## 6. Limitations, misconceptions, and open directions A recurring misconception is that diarization-conditioned SLMs must be end-to-end joint audio generators. The literature shows otherwise. Post-hoc systems such as SEC and DiarizationLM can substantially reduce speaker-attribution errors without retraining ASR or diarization front-ends, and in some telephony settings their gains are already large [2306.09313][2401.03506]. Another misconception is that diarization errors are dominated by missed ultra-short fragments; the multilingual diarization benchmark instead points to onset/offset boundary precision on longer segments and to speaker confusion under high speaker count as the main bottlenecks [2509.26177]. The field also remains constrained by strong assumptions. Several influential systems are explicitly two-speaker: the multilingual unified Speech LLM uses only `<|SPK0|>` and `<|SPK1|>` tokens, SEC corrects only windows containing exactly two hypothesized speakers, and the telephonic interview model assumes interviewer versus customer roles [2507.02927][2306.09313][2501.17893]. Far-field models such as Dixtral lift the fixed-two-speaker assumption by iterating over diarizer outputs, but they remain dependent on diarization quality, speaker counting, and overlap detection; the paper explicitly notes sensitivity to mask quality and keeps the overall pipeline offline rather than streaming [2606.18134]. A major architectural controversy concerns whether the language model decoder should be adapted for multi-speaker speech. Dixtral argues that decoder adaptation via Serialized Output Training risks catastrophic forgetting, while its own results show that keeping the decoder frozen preserves zero-shot QA and summarization ability better than adding LoRA solely for ASR. By contrast, unified generation approaches deliberately expand the output space with speaker and timestamp tokens and rely on prompt engineering plus post-alignment to control drift [2606.18134][2507.02927]. The present evidence does not collapse these strategies into a single winner; instead, it delineates a trade-off between direct joint decoding and preservation of pretrained generative behavior. A broader adjacent direction treats diarization as one instance of a more general acoustic-semantic segmentation problem. “Unsupervised Speech Segmentation” uses a TWIST 350M SLM over discrete HuBERT-based acoustic tokens, scores adjacent 0.5 s “acoustic-sentences” with PMI, and finds that an adaptive selector A(10) provides a strong PR-F1/R-Value trade-off for emotion and gender boundary detection. The paper does not implement explicit diarization conditioning, but it states that speaker change is one of the acoustic-semantic style changes the framework can expose. This suggests that diarization cues could be integrated as hard constraints or priors on top of SLM-based boundary scoring rather than only as labels to be predicted [2501.03711]. The broader survey of SLMs places these directions inside a common component view. Diarization conditioning can be attached to the speech encoder, the adapter, or the sequence model; it is compatible with pure speech LMs, speech+text LMs, and speech-aware text LMs; and duplex dialogue mechanisms such as learned `[speak]/[listen]` control tokens provide natural scaffolds for turn-taking and overlap handling [2504.08528]. Across the current literature, the most consistent priorities are boundary precision, overlap-aware modeling, robustness beyond four speakers, multilingual adaptation, uncertainty propagation into downstream reasoning, and online integration of diarization-conditioned front-ends [2509.26177][2606.18134]. Source: https://www.emergentmind.com/topics/diarization-conditioned-spoken-language-models-slms