Diarization-Conditioned Spoken Language Models

• Diarization-conditioned SLMs are models that integrate explicit speaker information—such as labels, turn boundaries, and timestamps—to represent multi-speaker speech accurately.
• They employ various paradigms including post-processing correction, unified speech generation, and encoder-side conditioning to refine speaker-attributed transcripts.
• Empirical results demonstrate significant improvements in word-level diarization error rates and downstream tasks, balancing transcription accuracy with preserved generative capabilities.

Diarization-conditioned spoken LLMs (SLMs) are spoken LLMs that consume, predict, or are otherwise modulated by speaker diarization information—speaker labels, turn boundaries, timestamps, overlap structure, or speaker-conditioned masks—so that multi-speaker speech is represented with explicit attribution rather than as a single undifferentiated stream. In recent work, the term spans at least three regimes: post-processing LLMs that refine speaker-attributed transcripts after ASR and diarization, decoder-only Speech LLMs that directly generate interleaved speaker, timestamp, and text tokens, and frozen-decoder SLMs whose acoustic encoder is conditioned on diarization masks to extract target-speaker representations (Wang et al., 2024, Saengthong et al., 26 Jun 2025, Polok et al., 16 Jun 2026). Within the broader SLM taxonomy, diarization conditioning can be injected into pure speech LMs, speech+text LMs, or speech-aware text LMs, and it functions as a mechanism for grounding multi-speaker conversational structure inside a model family otherwise centered on content modeling (Arora et al., 11 Apr 2025).

1. Conceptual scope and antecedents

Speaker diarization answers the question of “who spoke when,” whereas SLMs address speech understanding and generation. Diarization-conditioned SLMs arise when these two objectives are coupled closely enough that speaker attribution becomes part of the model’s conditioning state, output space, or post-hoc correction logic. The coupling may be weak, as in a text-only second pass over ASR and diarization outputs, or strong, as in end-to-end generation of speaker and timestamp tokens from raw audio (Arora et al., 11 Apr 2025, Saengthong et al., 26 Jun 2025).

Important antecedents predate fully unified Speech LLMs. In telephonic interviews, linguistic content can be more discriminative than acoustics for speaker assignment when one role asks a known set of questions and the other provides short, formulaic answers. “Language Modelling for Speaker Diarization in Telephonic Interviews” reports that combining character-level lexical features with acoustic scores yields an 84.29% improvement in word-level DER relative to a HMM/VB baseline, with final ASR-condition WDER dropping from 13.55% to 1.98% in a two-speaker Spanish call-center setting (India et al., 28 Jan 2025). This establishes a central principle of the area: speaker attribution can be recoverable from discourse structure and lexical regularities even when acoustic segmentation is unreliable.

Subsequent lexical correction work generalizes that principle to modular ASR+diarization pipelines. “Lexical Speaker Error Correction” formulates a discriminative correction problem over lexical context and first-pass speaker hypotheses, achieving consistent 15–30% relative WDER reductions on Fisher, RT03-CTS, and CALLHOME American English after training on Fisher only (Paturi et al., 2023). “DiarizationLM” moves the same idea into a prompted LLM framework, representing ASR words and speaker labels as compact text and applying transcript-preserving speaker transfer (TPST) so that speaker labels can change while the original ASR wording is retained; on Fisher and Callhome English, a finetuned PaLM 2-S reduces WDER by 55.5% and 44.9% relative, respectively (Wang et al., 2024). These systems show that diarization-conditioned modeling is not synonymous with end-to-end audio generation; it also includes text-conditioned correction layers operating over already transcribed conversations.

2. Main architectural paradigms

Recent literature organizes diarization-conditioned SLMs around a small number of recurring architectural patterns.

Paradigm	Representative systems	Core mechanism
Post-processing correction	SEC, DiarizationLM	Refine word-level speaker labels from transcript and initial diarization
Unified speech generation	ST-ShinozakiLab Speech LLM	Generate speaker, timestamp, and text tokens jointly from audio
Encoder-side conditioning	Dixtral	Condition acoustic encoder on diarization masks; keep decoder frozen

In post-processing systems, the model does not replace ASR or diarization. SEC uses a RoBERTa-base LLM to produce contextual embeddings, concatenates them with first-pass speaker label features, and applies a lightweight Transformer encoder to output corrected speaker posteriors for two-speaker windows. DiarizationLM instead serializes the ASR transcript and speaker sequence into a compact prompt such as <spk:1> good morning <spk:2> how are you, asks an LLM to rewrite only the speaker attribution, and then maps the result back onto the original ASR words via TPST and a Hungarian label-permutation step (Paturi et al., 2023, Wang et al., 2024).

Unified speech generation internalizes diarization inside the decoder’s output vocabulary. “A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations” reformulates each local turn as an interleaved sequence of [speaker token, start timestamp token, text tokens, end timestamp token], so that diarization and ASR become a single next-token prediction problem. The model uses a Whisper encoder, a sub-sampling projector, and a decoder-only Llama-3.2 backbone with LoRA, and it is explicitly designed for two-speaker multilingual conversations (Saengthong et al., 26 Jun 2025).

Encoder-side conditioning reverses the design choice that unified generation makes. “Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning” argues that adapting the decoder through Serialized Output Training risks catastrophic forgetting of instruction-following, reasoning, summarization, and QA abilities. Its alternative, Dixtral, conditions a DiCoW acoustic encoder on diarization masks so that the frozen Voxtral decoder always receives a target-speaker representation rather than an interleaved multi-speaker sequence. Speaker attribution is recovered by running the model separately for each target speaker (Polok et al., 16 Jun 2026).

The broader SLM survey provides the abstraction under which these designs coexist. In its notation, speech is encoded as $H_{sp} = Enc_{sp}(X_{sp})$, adapted into the sequence-model space, and consumed by a sequence model. Diarization conditioning can therefore enter as prompt tokens, auxiliary embeddings, masks, or adapter-side modulation, which is exactly what these concrete systems instantiate in different ways (Arora et al., 11 Apr 2025).

3. Conditioning signals, objectives, and inference procedures

The unified multilingual Speech LLM operationalizes diarization-conditioned generation by treating speaker identity and timing as ordinary tokens in the target stream. Let $Y = (y_1, y_2, \dots, y_T)$ denote the mixed sequence of speaker, timestamp, and text tokens, and let $z$ be the projected audio feature sequence from the Whisper encoder and projector. Training uses standard next-token cross-entropy,

$L = - \sum_{t=1}^{T} \log p(y_t \mid y_{<t}, z).$

No explicit CTC head or separate diarization classification loss is used; supervision is induced by the presence and positions of the special diarization and timestamp tokens. Inference is chunked but diarization-aware: audio is processed in 24 s local windows, the next window starts at the end time of the second-to-last predicted turn in the previous chunk, the prompt carries the last four predicted turns plus a next-speaker hint, and a “no valid turn” condition advances the window by 0.3 s to avoid stalling. A global 3D-Speaker module then aligns the locally generated STM segments to RTTM speaker segments across the full recording (Saengthong et al., 26 Jun 2025).

Dixtral uses a different conditioning primitive. A diarization model, specifically DiariZen, produces frame-level speaker activity probabilities $d(s,t)$, from which the model constructs soft STNO probabilities for silence, target speech, non-target speech, and overlap. These probabilities modulate every encoder layer through frame-level diarization-dependent transformations (FDDT):

$$\hat{z}_t^l = \sum_{i \in \{S,T,N,O\}} (W_i^l z_t^l + b_i^l) p_i^t .$$

The conditioned encoder output is projected by a frozen modality adapter and passed to a frozen Ministral 3B decoder. Training optimizes only cross-entropy on the decoder outputs after the prompt, while keeping the decoder and adapter fixed. The stated rationale is that speaker attribution should be imposed on the acoustic representation rather than forced into the LLM’s token vocabulary, thereby preserving pretrained generative capabilities (Polok et al., 16 Jun 2026).

Post-processing systems impose conditioning through serialization and alignment rather than acoustic modulation. DiarizationLM uses a prompt-completion format with a -> suffix and an [eod] termination token, recursively binary-partitions long transcripts to fit a 4,096-token budget, and applies TPST so that speaker labels inferred by the LLM are transferred back onto the original ASR words without changing WER. SEC instead runs on sliding windows of words, usually 30 words, tokenizes them with RoBERTa, concatenates contextual lexical embeddings with initial speaker-label features, and predicts corrected token-level speaker labels; windows with more than two hypothesized speakers are bypassed (Wang et al., 2024, Paturi et al., 2023).

An earlier iterative fusion pattern remains relevant because it clarifies how lexical and acoustic evidence can be alternated. In the telephonic interview system, a text-only LSTM first predicts word-level speaker labels, then per-recording speaker GMMs are estimated from MFCC frames grouped by those labels, acoustic posteriors $P(\mathrm{Cus}\mid w)$ are computed for each word, and a second LSTM consumes lexical embeddings plus the acoustic score. This procedure converges in 2–3 iterations and has an EM-like interpretation: label inference alternates with acoustic model re-estimation (India et al., 28 Jan 2025).

4. Metrics, benchmarks, and downstream error budgets

Diarization-conditioned SLMs inherit evaluation protocols from both diarization and multi-speaker ASR. The diarization benchmark most directly framed for SLM integration evaluates five diarization systems on 196.6 hours of multilingual audio across CALLHOME, VoxConverse, AMI, and AliMeeting, using diarization error rate

$DER = \frac{FA + MISS + CONF}{TOTAL}$

with a 0.25 s collar and skip_overlap = False, so overlapped speech is explicitly scored and penalized when mislabeled. Under this strict policy, PyannoteAI achieves the best overall performance at 11.2% DER, DiariZen reaches 13.3% as a competitive open-source alternative, DiariZen is strongest on VoxConverse at 5.2% DER, and Sortformer v2-stream is best on AliMeeting at 7.0% DER (Lanzendörfer et al., 30 Sep 2025).

The same benchmark isolates failure modes that are especially consequential for downstream SLMs. Missed speech is the dominant source of error across models and datasets, particularly in meetings with substantial overlap, and speaker confusion rises sharply in high-speaker settings. The average duration of missed segments is approximately 350 ms, while segments shorter than 0.5 s contribute less than 5% of errors. The reported interpretation is that timestamp imprecision on longer segments—speech onset and offset placement—matters more than outright failure to detect very short events. This is directly relevant for SLM conditioning, because turn attribution, summarization, and dialogue understanding depend on accurate boundaries as much as on speaker identity (Lanzendörfer et al., 30 Sep 2025).

Multi-speaker ASR metrics capture the combined effect of transcription and attribution. The MLC-SLM Challenge evaluates joint diarization and recognition with DER and time-constrained permutation WER/CER (tcpWER/tcpCER), where the time constraint and a 5 s collar penalize misattribution and timing misalignment at the conversation level. Dixtral instead reports concatenated minimum-permutation WER (cpWER), which minimizes WER over permutations of hypothesized speaker streams after concatenating each speaker’s transcript. Word-level correction papers focus on WDER, the fraction of words with incorrect speaker labels, which is appropriate when the transcript is fixed and only speaker attribution is refined (Saengthong et al., 26 Jun 2025, Polok et al., 16 Jun 2026, Wang et al., 2024).

The benchmark paper also makes the downstream dependency explicit through a simple attribution bound. Let $S$ be total speech time and $T$ total time. A lower bound on speaker-attribution accuracy for speech tokens conditioned on diarization is

$Y = (y_1, y_2, \dots, y_T)$0

and for pipelines in which a fraction $Y = (y_1, y_2, \dots, y_T)$1 of false-alarm time survives ASR, overall token attribution can be approximated as

$Y = (y_1, y_2, \dots, y_T)$2

The practical conclusion is that minimizing missed speech and speaker confusion is more critical than minimizing false alarms in speech-centric downstream tasks (Lanzendörfer et al., 30 Sep 2025).

5. Empirical performance across deployment regimes

Results differ markedly by conversational regime, which is why diarization-conditioned SLMs have developed along several parallel lines. In structured two-speaker telephony, lexical conditioning is especially powerful. The telephonic interview system reports that a text-only model already reduces ASR-condition WDER from 13.55% to 5.34%, and the final fused lexical-acoustic iterative model reaches 1.98% WDER. SEC, trained and tuned on Fisher, delivers relative WDER reductions of roughly 32% on Fisher, 27% on RT03-CTS test, and 20% on CALLHOME American English test. DiarizationLM’s finetuned PaLM 2-S reaches WDER 2.37 and cpWER 16.93 on Fisher, and WDER 4.25 and cpWER 20.22 on Callhome, while preserving the original ASR wording through TPST (India et al., 28 Jan 2025, Paturi et al., 2023, Wang et al., 2024).

In multilingual two-speaker conversational audio, joint generation becomes competitive when local and global consistency are engineered carefully. The unified Speech LLM submitted by ST-ShinozakiLab to the MLC-SLM Challenge obtains 27.25 tcpWER/tcpCER on the official Task II evaluation set versus a 60.39 baseline, a 54.87% relative improvement, and ranks 8th overall despite using a smaller LLM backbone. On the development set, the Llama-3.2-3B version reports DER 13.36, tcpWER/tcpCER 28.23, and cpWER 24.97, outperforming the corresponding 1B variant except in Thai, where the 1B model slightly wins. The architecture’s gains are attributed not to larger scale but to the interleaved token format and diarization-aware inference procedure (Saengthong et al., 26 Jun 2025).

In far-field multi-speaker audio, encoder-side conditioning currently provides the strongest evidence for preserving general SLM capabilities while improving speaker-attributed transcription. Dixtral reports a macro-average cpWER of 15.4% across NOTSOFAR-1, AMI, LibriSpeechMix, and Mixer6, compared with 44.4% for Gemini 3.0 Flash, 35.2% for VibeVoice, and 31.4% for Voxtral Mini Transcribe V2, corresponding to absolute improvements of 29.0%, 19.8%, and 16.0%. On the NSF-QA benchmark, zero-shot Dixtral after ASR-only training reaches 54.6% content QA, on par with Gemini’s 55.1% on far-field audio, and 24.4 ROUGE-L in summarization, approximately matching Voxtral on close-talk audio at 24.1. Fine-tuning on NSF-QA raises performance to 73.0% content QA, 47.6% emotion QA, 95.5% gender QA, and 41.4 ROUGE-L, surpassing both Gemini and Voxtral operating on close-talk across all reported tasks. At the same time, adding LoRA to the decoder improves some ASR numbers but harms summarization, with ROUGE-L dropping to 15.4, which underscores the trade-off between transcription specialization and preserved instruction-following (Polok et al., 16 Jun 2026).

These empirical results indicate that the optimal conditioning strategy depends on the interaction between speaker cardinality, overlap rate, domain structure, and the intended downstream task. This suggests that diarization-conditioned SLMs are better understood as a design space than as a single architecture class.

6. Limitations, misconceptions, and open directions

A recurring misconception is that diarization-conditioned SLMs must be end-to-end joint audio generators. The literature shows otherwise. Post-hoc systems such as SEC and DiarizationLM can substantially reduce speaker-attribution errors without retraining ASR or diarization front-ends, and in some telephony settings their gains are already large (Paturi et al., 2023, Wang et al., 2024). Another misconception is that diarization errors are dominated by missed ultra-short fragments; the multilingual diarization benchmark instead points to onset/offset boundary precision on longer segments and to speaker confusion under high speaker count as the main bottlenecks (Lanzendörfer et al., 30 Sep 2025).

The field also remains constrained by strong assumptions. Several influential systems are explicitly two-speaker: the multilingual unified Speech LLM uses only <|SPK0|> and <|SPK1|> tokens, SEC corrects only windows containing exactly two hypothesized speakers, and the telephonic interview model assumes interviewer versus customer roles (Saengthong et al., 26 Jun 2025, Paturi et al., 2023, India et al., 28 Jan 2025). Far-field models such as Dixtral lift the fixed-two-speaker assumption by iterating over diarizer outputs, but they remain dependent on diarization quality, speaker counting, and overlap detection; the paper explicitly notes sensitivity to mask quality and keeps the overall pipeline offline rather than streaming (Polok et al., 16 Jun 2026).

A major architectural controversy concerns whether the LLM decoder should be adapted for multi-speaker speech. Dixtral argues that decoder adaptation via Serialized Output Training risks catastrophic forgetting, while its own results show that keeping the decoder frozen preserves zero-shot QA and summarization ability better than adding LoRA solely for ASR. By contrast, unified generation approaches deliberately expand the output space with speaker and timestamp tokens and rely on prompt engineering plus post-alignment to control drift (Polok et al., 16 Jun 2026, Saengthong et al., 26 Jun 2025). The present evidence does not collapse these strategies into a single winner; instead, it delineates a trade-off between direct joint decoding and preservation of pretrained generative behavior.

A broader adjacent direction treats diarization as one instance of a more general acoustic-semantic segmentation problem. “Unsupervised Speech Segmentation” uses a TWIST 350M SLM over discrete HuBERT-based acoustic tokens, scores adjacent 0.5 s “acoustic-sentences” with PMI, and finds that an adaptive selector A(10) provides a strong PR-F1/R-Value trade-off for emotion and gender boundary detection. The paper does not implement explicit diarization conditioning, but it states that speaker change is one of the acoustic-semantic style changes the framework can expose. This suggests that diarization cues could be integrated as hard constraints or priors on top of SLM-based boundary scoring rather than only as labels to be predicted (Elmakies et al., 7 Jan 2025).

The broader survey of SLMs places these directions inside a common component view. Diarization conditioning can be attached to the speech encoder, the adapter, or the sequence model; it is compatible with pure speech LMs, speech+text LMs, and speech-aware text LMs; and duplex dialogue mechanisms such as learned [speak]/[listen] control tokens provide natural scaffolds for turn-taking and overlap handling (Arora et al., 11 Apr 2025). Across the current literature, the most consistent priorities are boundary precision, overlap-aware modeling, robustness beyond four speakers, multilingual adaptation, uncertainty propagation into downstream reasoning, and online integration of diarization-conditioned front-ends (Lanzendörfer et al., 30 Sep 2025, Polok et al., 16 Jun 2026).