Dixtral: Diarization-Conditioned Spoken LLM

• The paper introduces Dixtral, a diarization-conditioned spoken LLM that selectively trains its acoustic encoder with FDDT modules while keeping the decoder frozen to maintain robust reasoning and summarization capabilities.
• It achieves competitive performance with a macro-average cpWER of 15.4, substantially outperforming systems like Gemini 3.0 Flash and VibeVoice in speaker-attributed transcription tasks.
• The architecture shifts multi-speaker disambiguation to the encoder using target-speaker conditioning, thereby reducing computational cost and preserving alignment with established close-talk language model behaviors.

Searching arXiv for the Dixtral paper and directly related named systems. Searching arXiv for "Dixtral diarization conditioned spoken LLMs". Searching arXiv for "DiCoW diarization conditioned Whisper" and "Voxtral spoken LLM". Dixtral is a diarization-conditioned spoken LLM (SLM) for multi-speaker, far-field audio. It instantiates a design in which a spoken LLM backbone, Voxtral, is extended to mixed and reverberant recordings by replacing its acoustic encoder with a Diarization Conditioned Whisper (DiCoW) encoder, while keeping the decoder frozen. The stated objective is to preserve the downstream reasoning, summarization, question answering, and instruction-following behavior of the spoken LLM while adding target-speaker grounding for speaker-attributed transcription and long-form meeting understanding (Polok et al., 16 Jun 2026).

1. Definition, problem setting, and design rationale

Dixtral addresses a setting in which speech understanding must operate on distant, reverberant, multi-talker recordings such as meetings. In that regime, a model must resolve both speaker identity over time and semantic content, under overlap, lower SNR, channel effects, and far-field reverberation. The paper positions this as a mismatch with standard spoken LLM assumptions, which are more naturally aligned with single-speaker audio or close-talk input (Polok et al., 16 Jun 2026).

The paper’s central contrast is between Serialized Output Training (SOT) and diarization conditioning. In SOT, a model emits a single interleaved multi-speaker token stream ordered by speech onset, typically with speaker-change or speaker-serialization markers. The paper argues that this is a poor fit for spoken LLMs for two stated reasons: architectural mismatch and distribution mismatch. Architecturally, SOT requires special speaker tokens and therefore decoder-side adaptation. Distributionally, interleaved multi-speaker text differs from the structured, instruction-following, mostly single-speaker text distributions on which LLM decoders are trained. Dixtral therefore moves multi-speaker disambiguation to the encoder, so that the decoder remains in a standard single-speaker generation regime (Polok et al., 16 Jun 2026).

Freezing the decoder is presented as the key mechanism for avoiding catastrophic forgetting. The paper argues that heavy decoder fine-tuning on multi-speaker transcription data, especially with SOT-style targets, risks degrading the reasoning and summarization behavior that motivates the use of a spoken LLM in the first place. In Dixtral, multi-speaker adaptation is concentrated in the acoustic front-end, while the decoder remains unchanged (Polok et al., 16 Jun 2026).

The paper also makes a decoder-complexity argument. For a serialized transcript of $S$ speakers each of length $N$, autoregressive cost is stated as $\mathcal{O}((SN)^2)$. If instead one decodes $S$ separate single-speaker outputs, the cost becomes $\mathcal{O}(S \cdot N^2)$. This suggests that target-speaker decomposition becomes increasingly attractive as decoder cost dominates (Polok et al., 16 Jun 2026).

2. Architecture and target-speaker conditioning

Dixtral combines Voxtral and DiCoW. The Voxtral backbone is described as a pretrained spoken LLM with a Whisper-based encoder, a modality adapter, and a Ministral 3B decoder. Dixtral replaces the original acoustic encoder with a diarization-conditioned Whisper encoder from DiCoW, then retains Voxtral’s existing modality adapter and frozen decoder (Polok et al., 16 Jun 2026).

The end-to-end processing sequence is explicit. The model takes mixed audio, obtains diarization activity probabilities for all speakers, derives a target-speaker-specific STNO mask, passes audio plus STNO through the DiCoW encoder, projects encoder outputs through Voxtral’s modality adapter into the LLM embedding space, concatenates those audio embeddings with a text prompt, and then decodes autoregressively with the frozen LLM. The paper’s figure describes this as external DiariZen diarization conditioning the DiCoW encoder via FDDT, with only the encoder side trained in the main setup (Polok et al., 16 Jun 2026).

The diarization representation is organized into four frame-level probabilities known as STNO for a selected target speaker $s_k$: silence $\mathcal{S}$, target-only speech $\mathcal{T}$, non-target speech $\mathcal{N}$, and overlap involving the target speaker $\mathcal{O}$. These are computed from frame-level diarization posteriors $N$0. The paper defines them as

$N$1

$N$2

$N$3

$N$4

These probabilities are then injected into every Transformer encoder layer using Frame-Level Diarization-Dependent Transformations (FDDT) (Polok et al., 16 Jun 2026).

Target-speaker extraction occurs in hidden-state space rather than by raw waveform masking. At encoder layer $N$5, the hidden vector $N$6 is transformed by four learnable affine transforms associated with $N$7, mixed according to the corresponding STNO probabilities: $N$8 The resulting encoder output,

$N$9

is intended to contain target-speaker acoustic content (Polok et al., 16 Jun 2026).

After encoding, the modality adapter is an MLP with linear projection, GELU, and linear projection: $\mathcal{O}((SN)^2)$0 Prompt embeddings $\mathcal{O}((SN)^2)$1 are concatenated with the audio embeddings,

$\mathcal{O}((SN)^2)$2

and the frozen decoder generates the output sequence autoregressively (Polok et al., 16 Jun 2026).

The trainable–frozen partition is central. In the main setup, the acoustic encoder layers and FDDT modules are trainable, while DiariZen, the modality adapter, and the LLM decoder are frozen. The paper argues that this is feasible because Voxtral learned its projector with a frozen Whisper encoder, while DiCoW learned its target-speaker conditioning with a frozen decoder, so the two systems are treated as architecturally compatible (Polok et al., 16 Jun 2026).

3. Formal task formulation and operational modes

The paper uses a standard autoregressive loss on the generated target sequence: $\mathcal{O}((SN)^2)$3 Loss is computed only over output tokens after the prompt, and in the frozen-decoder setup only $\mathcal{O}((SN)^2)$4 is updated. The paper does not specify additional auxiliary losses for Dixtral itself (Polok et al., 16 Jun 2026).

Dixtral supports both target-speaker and global modes. For target-speaker transcription, question answering, or summarization, inference is run separately for each selected speaker. For global reasoning over the full recording, target selection can be bypassed by forcing every frame to be treated as target: $\mathcal{O}((SN)^2)$5 This keeps the same architecture while removing explicit speaker selection (Polok et al., 16 Jun 2026).

The paper gives only limited segmentation details. Training handles 2-minute utterances under GPU memory constraints. AMI decoding uses approximately 5-minute segments split at diarization-derived pauses. For long-form QA and summarization fine-tuning, H100 GPUs are used because the full long-form audio does not fit on A5000s. Beam size, temperature, sampling policy, and other decoding hyperparameters are not specified (Polok et al., 16 Jun 2026).

For evaluation, speaker-attributed transcription uses concatenated minimum-permutation word error rate (cpWER) from MeetEval. The paper does not provide a formal cpWER equation, but it states that the metric concatenates each speaker’s reference utterances, concatenates each hypothesized speaker stream, finds the speaker permutation minimizing total WER, and reports the resulting error rate. QA is measured by binary accuracy judged by Gemini 2.5 Flash, and summarization by ROUGE-L (Polok et al., 16 Jun 2026).

4. Training protocol, initialization, and benchmark suite

Initialization exploits the shared Whisper-family lineage of Voxtral and DiCoW. The paper considers two transfer strategies: replacing Voxtral’s entire encoder with pretrained DiCoW weights, or injecting only DiCoW’s pretrained FDDT parameters into Voxtral’s encoder. The main experiments use Voxtral Mini 3B, which contains a Ministral 3B decoder and a Whisper large-v3 acoustic encoder (Polok et al., 16 Jun 2026).

The main fine-tuning recipe freezes the LLM decoder and modality adapter, while training only the acoustic encoder layers and FDDT modules. The reported setup is 20k steps, peak learning rate $\mathcal{O}((SN)^2)$6, 5k warmup, cosine decay to zero, bfloat16, gradient checkpointing, 4 gradient accumulation steps, effective global batch size 32, and hardware consisting of $\mathcal{O}((SN)^2)$7 A5000 GPUs. The optimizer is not specified by name (Polok et al., 16 Jun 2026).

The transcription evaluation spans AMI, NOTSOFAR-1 (NSF-1), LibriSpeechMix (LSMix), and Mixer6. AMI is used in SDM far-field form. NSF-1 is used in a distant meeting transcription setting, with the table labeled “NSF-1 Small.” LibriSpeechMix is reported in three overlap conditions, LSMix-1, LSMix-2, and LSMix-3. Mixer6 CH4 serves as an out-of-domain test set and is explicitly not included in training (Polok et al., 16 Jun 2026).

The paper also introduces NSF-QA, a long-form target-speaker QA and summarization benchmark built on NSF-1. The reported counts are:

• Train: Entity 1,111; Topic 1,139; Y/N 1,557; Detail 775; Emotion 1,196; Summarization 390
• Dev: Entity 579; Topic 593; Y/N 807; Detail 402; Emotion 579; Summarization 201
• Eval: Entity 1,092; Topic 1,114; Y/N 1,515; Detail 753; Emotion 1,145; Summarization 379

The task suite includes content QA, paralinguistic QA for emotion and speaker gender, and concise target-speaker summarization under 50 words. A notable design choice is that no question references timestamps or segment boundaries, so the model must localize relevant content over the full meeting context (Polok et al., 16 Jun 2026).

Diarization comes from DiariZen. Main ASR results use predicted DiariZen diarization, while the ablation study uses oracle diarization. For Gemini on far-field QA, direct diarization conditioning is unavailable, so the paper instead supplies positional speaker descriptions such as “the speaker who starts speaking third from the beginning” (Polok et al., 16 Jun 2026).

5. Empirical performance

The headline transcription result is a macro-average cpWER of 15.4 for Dixtral. The comparison systems reported in the paper are Gemini 3.0 Flash at 44.4, VibeVoice at 35.2, Voxtral Mini Transcribe V2 at 31.4, and DiCoW v3.3 at 14.0. The corresponding absolute improvements over Gemini 3.0 Flash, VibeVoice, and Voxtral Mini Transcribe V2 are 29.0, 19.8, and 16.0 cpWER points, respectively (Polok et al., 16 Jun 2026).

Per-dataset cpWER values further characterize the model’s behavior. Dixtral reports 29.1 on NSF-1 Small, 19.8 on AMI SDM, 2.1 on LSMix-1, 3.6 on LSMix-2, 23.5 on LSMix-3, and 14.4 on Mixer6 CH4. The paper emphasizes that Dixtral is substantially stronger than the compared spoken LLMs and Gemini on speaker-attributed transcription, while remaining close to DiCoW, the dedicated target-speaker ASR system. It does not surpass DiCoW on pure transcription, but on AMI the gap is small: 19.8 for Dixtral versus 18.6 for DiCoW (Polok et al., 16 Jun 2026).

The long-form QA and summarization results are split into far-field zero-shot, far-field LoRA zero-shot, close-talk baselines, and far-field fine-tuned settings. In zero-shot far-field evaluation, Dixtral reports Content QA 54.6, Emotion QA 25.4, Gender QA 43.2, and Summarization ROUGE-L 24.4. Gemini 3.0 Flash on far-field audio reports 55.1, 29.3, 74.1, and 23.7, respectively. The paper interprets this as near parity on far-field content QA, with Dixtral also matching close-talk Voxtral on emotion QA and summarization despite operating on the harder far-field mixture input (Polok et al., 16 Jun 2026).

The close-talk baselines are Voxtral Mv1 at Content QA 68.3, Emotion QA 25.5, Gender QA 49.7, Summarization ROUGE-L 24.1, and Gemini 3.0 Flash at 68.1, 34.1, 75.0, and 26.4. Against these stronger-input baselines, zero-shot far-field Dixtral remains competitive on some tasks but is clearly weaker on others, especially gender QA (Polok et al., 16 Jun 2026).

A decoder-LoRA zero-shot variant changes the performance profile. Dixtral $\mathcal{O}((SN)^2)$8 LoRA reports Content QA 56.9, Emotion QA 22.2, Gender QA 73.3, and Summarization ROUGE-L 15.4. The paper interprets this as evidence that LoRA can help surface speaker-level attributes such as gender while harming instruction following and summarization behavior (Polok et al., 16 Jun 2026).

The strongest downstream results come from task-specific fine-tuning on NSF-QA. Far-field fine-tuned Dixtral reports Content QA 73.0, Emotion QA 47.6, Gender QA 95.5, and Summarization ROUGE-L 41.4. The paper states that under this setting far-field Dixtral surpasses both Voxtral and Gemini operating on close-talk audio across all tasks. This is the strongest evidence offered for the claim that encoder-side grounding, with a preserved decoder, can support both transcription and long-form speech understanding (Polok et al., 16 Jun 2026).

6. Ablations, limitations, and terminological disambiguation

The ablation study uses oracle diarization to isolate architectural effects. The reported cpWER values are as follows: “Dixtral w/o swap” gives NSF-1 26.3, AMI 19.8, LSMix-1 1.9, LSMix-2 2.3, LSMix-3 3.6, and MX6 12.6; full encoder swap gives 26.4, 21.0, 1.9, 2.4, 4.6, and 20.4; FDDT swap gives 26.3, 17.1, 1.8, 2.2, 3.5, and 16.2; LoRA gives 21.3, 16.4, 2.0, 2.6, 4.8, and 14.5; QA+Summ fine-tuning gives 24.3, 19.4, 2.2, 3.1, 5.6, and 26.1 (Polok et al., 16 Jun 2026).

These ablations support three conclusions stated in the paper. First, full encoder swap converges faster early in training, with NSF-1 cpWER at 2k steps of 32.0 versus 37.7 for FDDT-only swap and 38.8 for random initialization, but later training favors either random initialization or FDDT-only transfer. The paper’s interpretation is that full replacement induces an embedding shift that harms Voxtral’s established audio–language alignment. Second, FDDT-only transfer is treated as a compromise that imports diarization conditioning while preserving more of Voxtral’s original alignment. Third, LoRA improves pure ASR in some conditions but can damage instruction following, with the model sometimes outputting transcription when asked for summarization (Polok et al., 16 Jun 2026).

The paper also identifies several limitations. Dixtral depends on external diarization masks and is therefore not fully end-to-end. It does not provide multilingual evaluation. Fine-tuning for QA and summarization can harm pure transcription, especially out of domain, with Mixer6 cpWER worsening in the discussed comparison from 14.5 to 26.1. Long-form fine-tuning required H100 GPUs. In NSF-QA construction and evaluation, Gemini 2.5 Flash is used both to generate questions or references and to judge answers, which the paper acknowledges as a possible source of circularity (Polok et al., 16 Jun 2026).

The term “Dixtral” also invites disambiguation. In the speech literature summarized here, it denotes the diarization-conditioned spoken LLM introduced in 2026 (Polok et al., 16 Jun 2026). It should not be conflated with generalized Dix-formula work in anisotropic reflection moveout, where the core objects are NMO ellipses, NMO-velocity surfaces, and matrix-valued Dix-type averaging in geophysical wave propagation rather than spoken language modeling (Grechka et al., 2018, Grechka et al., 2019). A plausible implication is that the similarity in naming is mnemonic rather than conceptual: the speech model’s contribution lies in encoder-conditioned target-speaker grounding, whereas the geophysical literature addresses analytic and matrix generalizations of the Dix equation.