Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning

Abstract: We propose diarization-conditioned spoken LLMs (SLMs), a strategy for extending SLMs to far-field multi-talker audio. Rather than adapting the decoder via Serialized Output Training, which risks catastrophic forgetting, we condition the acoustic encoder on diarization masks to extract target-speaker representations, keeping the decoder frozen. We instantiate this as Dixtral, integrating a Diarization Conditioned Whisper (DiCoW) encoder into the Voxtral SLM. On AMI, NOTSOFAR-1, LibriSpeechMix, and Mixer6, Dixtral outperforms Gemini 3.0 Flash, VibeVoice, and Voxtral Mini Transcribe V2 on speaker-attributed transcription by 29.0%, 19.8%, and 16.0% absolute cpWER respectively. On a novel long-form multi-speaker QA benchmark, zero-shot Dixtral matches Gemini on far-field content understanding, and when fine-tuned surpasses both Gemini and Voxtral operating on close-talk across all tasks.

• The paper introduces diarization-conditioned SLMs in the Dixtral architecture to improve multi-speaker transcription without retraining LLM decoders.
• It employs frame-level diarization masks and FDDT modules in a modified Whisper encoder to yield robust target-speaker representations.
• Experimental results on NOTSOFAR-1, AMI, and Mixer6 show significant cpWER improvements and enhanced zero-shot reasoning performance.

Diarization Conditioning for Spoken LLMs: The Dixtral Approach to Multi-Speaker Audio

Introduction

The paper "Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning" (2606.18134) addresses the acute challenge of integrating contemporary spoken LLMs (SLMs) into multi-talker far-field environments. It emphasizes the necessity for speaker-attributed transcription and complex audio reasoning—capabilities essential for downstream tasks such as summarization and multi-party question answering. Existing solutions, primarily based on serialized output training (SOT) and expansion of LLM vocabularies with auxiliary speaker tokens, either violate architectural modularity or induce catastrophic forgetting in decoder parameters. This work proposes diarization-conditioned SLMs, operationalized in the Dixtral architecture, to establish target-speaker extraction as a scalable and semantically compatible solution for LLM-based reasoning over multi-speaker audio.

Methodology

Diarization-Conditioned Representation Learning

Dixtral leverages a diarization mask to explicitly condition the acoustic encoder—a modified Whisper backbone—on frame-level speaker activity probabilities. This is achieved using the DiCoW architecture, which integrates learned STNO (Silence, Target, Non-target, Overlap) masks derived from external diarization systems. Frame-level conditioning is not restricted to the encoder input but is adaptively applied at every transformer layer via learnable affine transformations as implemented in the Frame-Level Diarization-Dependent Transformations (FDDT). This design yields robust target-speaker acoustic representations without compelling decoder adaptation or expansion.

Modular Integration and Training Paradigm

The SLM is constructed by freezing the LLM decoder (Ministral 3B-based), thereby fully preserving its pre-trained generative capabilities for open-domain reasoning. Only the acoustic encoder and FDDT conditioning modules are updated during supervised MT-ASR training. Initialization exploits weight transferability between pre-trained DiCoW and Voxtral models due to their architectural affinity. This approach minimizes cross-modal alignment drift, reduces the risk of catastrophic forgetting, and decouples the acoustic modeling task from the higher-level semantic and syntactic distribution learned by the LLM.

Computational and Architectural Efficiency

A critical insight is the computational superiority of speaker-specific pass inference. Instead of a joint multi-speaker SOT that results in $O((S \cdot N)^2)$ decoding cost (for $S$ speakers and $N$ sequence length), Dixtral executes separate decoding passes, with complexity scaling as $O(S \cdot N)$. This efficiency becomes pronounced as LLM context window sizes and acoustic sequence lengths grow.

Experimental Results

Speaker-Attributed ASR

Dixtral demonstrates significant improvements in speaker-attributed cpWER over leading SLMs on evaluation datasets including NOTSOFAR-1, AMI, and Mixer6. On NOTSOFAR-1, it achieves 29.1% cpWER, outpacing VibeVoice (35.8%), Gemini (39.1%), and Voxtral MTv2 (54.4%). Comparable trends are observed on AMI (19.8% vs. DiCoW's 18.6%, VibeVoice's 33.7%) and Mixer6 (14.4%). This performance gain is maintained without adversarial decoder adaptation, eliminating trade-offs between transcription quality and downstream reasoning.

Zero-Shot Reasoning, QA, and Summarization

On a novel multi-speaker QA and summarization benchmark, zero-shot Dixtral operates on far-field mixtures and matches Gemini on content QA and closely follows Voxtral for emotion queries and summarization despite lacking oracle close-talk inputs. When fine-tuned, Dixtral surpasses both Gemini and Voxtral on all reasoning tasks, notably in paralinguistic QA, demonstrating 95.5% accuracy on gender queries compared to Gemini's 74.1%, and achieving higher ROUGE-L scores for summarization.

Ablations and Architectural Analysis

Detailed ablation studies highlight that strict encoder decoder modularity is essential. Swapping and reinitializing only the FDDT blocks delivers better cross-modal stability compared to wholesale encoder replacement. Applying LoRA to the decoder can extract latent speaker-dependent cues and improve paralinguistic QA, but strong decoder adaptation may compromise verbatim ASR quality, underscoring the delicate balance required.

Implications and Future Directions

This research reframes diarization as a conditioning signal for spoken LLMs, demonstrating practical and theoretical advantages over SOT and joint-vocabulary approaches. Practically, it enables scalable, zero-shot generalization to multi-speaker downstream tasks without necessitating decoder retraining. Theoretically, it provides evidence that frozen LLM decoders retain robust reasoning capabilities given appropriately aligned semantic embeddings from the acoustic pipeline, extending the transferability observed in frozen encoder paradigms for vision-LLMs to the audio modality.

Future directions include end-to-end joint training with diarization, multi-lingual and multi-accent extension, and scaling to fully end-to-end architectures with large-context LLMs. Integrative research into more refined modality adapters and diarization-aware self-supervised learning could further improve both cross-modal representation alignment and downstream task performance.

Conclusion

Diarization conditioning presents a minimal-intrusion, highly effective solution for grounding spoken LLMs in far-field multi-talker audio. Dixtral, as a proof of concept, establishes new state-of-the-art performance on speaker-attributed transcription and downstream reasoning, achieving substantial improvements without retraining LLM decoders. These findings promote diarization conditioning as a primary strategy for future research in multi-speaker spoken LLMs, paving the way for practical, generalizable, and efficient architectures for multi-talker conversational understanding.