DiCoW: Diarization-Conditioned Whisper ASR

• DiCoW is a diarization-conditioned ASR framework that uses STNO masks to isolate target speaker speech in multi-speaker recordings.
• It employs Frame-Level Diarization-Dependent Transformations (FDDT) in the Whisper encoder to robustly extract target speech even during overlaps.
• Extensions like self-enrollment and multilingual adaptations demonstrate notable reductions in word error rates across diverse acoustic conditions.

DiCoW, short for Diarization-Conditioned Whisper, is a target-speaker automatic speech recognition framework that adapts Whisper to multi-speaker audio by conditioning on speaker diarization outputs rather than speaker embeddings (Polok et al., 2024). It is designed for speaker-attributed ASR in meetings, conversational mixtures, and overlapping speech, where the objective is to transcribe one selected speaker at a time from a shared recording. The method’s central representation is a target-dependent Silence–Target–Non-target–Overlap (STNO) mask, injected into the acoustic encoder through diarization-conditioned transformations. Subsequent work has extended DiCoW with self-enrollment for overlap disambiguation, speaker-agnostic acceleration, serialized joint decoding, multilingual challenge systems, synthetic-data training studies, and integration into spoken LLMs such as Dixtral (Polok et al., 27 Jan 2026, He et al., 4 Oct 2025, Kocour et al., 4 Oct 2025, Polok et al., 16 Jun 2025, Polok et al., 14 May 2026, Polok et al., 16 Jun 2026).

1. Concept and problem setting

DiCoW was introduced to solve a specific failure mode of strong single-speaker ASR systems in multi-speaker recordings: off-the-shelf Whisper tends to transcribe all audible speech rather than isolating a chosen speaker (Polok et al., 2024). In the original formulation, the input is a long multi-speaker recording together with diarization labels; for each diarized speaker $s_k$, the system constructs a target-dependent conditioning signal and decodes that speaker’s transcript independently. The resulting workflow is diarization-assisted rather than fully end-to-end: diarization is external, whereas ASR is internal and target-conditioned.

The original motivation is explicitly contrasted with target-speaker ASR systems based on x-vectors, i-vectors, enrollment speech, or other speaker embeddings. DiCoW’s authors argue that embedding-based conditioning imposes a difficult mapping from speaker-identity space into the ASR model’s internal representation space, increases data requirements, and complicates deployment because enrollment speech or a separate speaker-embedding module is required (Polok et al., 2024). DiCoW instead conditions on when the target speaker is active, using diarization activity patterns as the control signal.

This design places DiCoW between classical modular diarization–ASR pipelines and fully joint speaker-attributed ASR systems. It does not emit speaker tags as native decoder outputs in its base form; speaker attribution is obtained by repeating target-conditioned decoding once per diarized speaker. The same conceptual framing was later retained in multilingual challenge systems, synthetic-data studies, and spoken-language-model integration, where DiCoW remains the encoder-side target-speaker acoustic front end rather than a full multi-output recognizer (Polok et al., 16 Jun 2025, Polok et al., 14 May 2026, Polok et al., 16 Jun 2026).

2. Conditioning mechanisms and architecture

In the original system, the acoustic input is a log-Mel spectrogram $\mathbf{X} \in \mathbb{R}^{F\times T}$, and diarization is represented as $\mathbf{D}\in[0,1]^{S\times T}$, where $d(s,t)$ denotes the activity probability of speaker $s$ at frame $t$ (Polok et al., 2024). For a chosen target speaker $s_k$, DiCoW reduces the diarization output to a fixed four-class STNO representation: $p_{\mathcal{S}^t} = \prod_{s=1}^S (1 - d(s, t)),$

$$p_{\mathcal{T}^t} = d(s_k, t)\prod_{\substack{s=1 \ s \neq s_k}}^S (1 - d(s, t)),$$

$$p_{\mathcal{N}^t} = \left(1 - p_{\mathcal{S}^t}\right) - d(s_k, t),$$

$\mathbf{X} \in \mathbb{R}^{F\times T}$0

These four quantities encode, per frame, silence, target-only speech, non-target-only speech, and target-involving overlap. A key architectural consequence is that the conditioning dimensionality is fixed even when the number of speakers changes.

The original paper studies three conditioning mechanisms. The simplest is input masking, where the waveform is multiplied by $\mathbf{X} \in \mathbb{R}^{F\times T}$1, preserving only target-active or overlap regions. The second is query-key biasing (QKb), which modifies encoder self-attention and decoder cross-attention by subtracting a bias $\mathbf{X} \in \mathbb{R}^{F\times T}$2 from attention logits associated with non-target frames. The third, and best-performing, mechanism is Frame-Level Diarization-Dependent Transformations (FDDT), which inserts STNO-conditioned affine transforms into the encoder between Transformer layers (Polok et al., 2024).

For encoder layer $\mathbf{X} \in \mathbb{R}^{F\times T}$3 and frame $\mathbf{X} \in \mathbb{R}^{F\times T}$4, FDDT applies

$\mathbf{X} \in \mathbb{R}^{F\times T}$5

In the original implementation, the transforms are diagonally parameterized, and the initial state is deliberately suppressive: silence and non-target transforms are initialized to zero, while target and overlap transforms are initialized to identity, with all biases set to zero (Polok et al., 2024). This keeps the initial behavior close to “preserve target, suppress non-target” and stabilizes adaptation.

The same paper also adds an optional CTC head on top of Whisper encoder outputs and uses joint CTC/attention decoding: $\mathbf{X} \in \mathbb{R}^{F\times T}$6 An additional Co-Attention module is introduced for heavy-overlap settings so that independently decoded speaker streams can exchange contextual information and reduce redundant assignment of the same dominant speaker (Polok et al., 2024).

3. Training protocol and empirical behavior of the original system

The original DiCoW experiments use Whisper-large-v3-turbo, whose decoder is reduced from 32 to 4 layers, and train in three phases: CTC preheat, FDDT preheat, and full fine-tuning (Polok et al., 2024). The reported optimization setup uses AdamW, batch size 64, weight decay $\mathbf{X} \in \mathbb{R}^{F\times T}$7, peak learning rate $\mathbf{X} \in \mathbb{R}^{F\times T}$8, 5k warm-up steps, linear decay after warm-up, maximum 50k steps, and CTC loss weight 0.3. The CTC preheat stage uses LibriSpeech 960h; multi-speaker fine-tuning uses AMI, NOTSOFAR-1, and Libri2Mix, with a multi-domain sampling ratio of 4:4:1 for the FDDT MD model (Polok et al., 2024).

The baseline contrast is large. Plain Whisper used as a target-speaker recognizer yields tcpWER 220.0 on AMI-sdm test, tcpWER 260.1 on NOTSOFAR-1 eval-small, and tcpWER 588.2 on LibriCSS test (Polok et al., 2024). Under oracle diarization, the best reported DiCoW results include AMI-sdm cpWER 17.2 and ORC-WER 16.5, NOTSOFAR-1 eval-small tcpWER 19.7 and tcORC-WER 19.1, Libri2Mix test-clean cpWER 4.4, Libri2Mix test-both cpWER 10.9, and LibriCSS cpWER 5.6 (Polok et al., 2024). Under real diarization, performance degrades to AMI-sdm cpWER 23.6 / ORC-WER 18.0, NOTSOFAR-1 tcpWER 33.5 / tcORC-WER 22.6, Libri2Mix test-clean cpWER 6.0, Libri2Mix test-both cpWER 15.0, and LibriCSS cpWER 8.5 / ORC-WER 6.5, confirming strong dependence on diarization quality (Polok et al., 2024).

Among conditioning mechanisms, FDDT is consistently superior to input masking and QKb. The paper also reports that FDDT converges much faster than QKb, reaching tcpWER below 30% after 1000 steps on NOTSOFAR-1 (Polok et al., 2024). The optional CTC head improves both training and decoding behavior: on NOTSOFAR-1 eval, greedy/beam-5 results improve from 22.9 / 22.2 without CTC to 21.7 / 20.9 with hybrid decoding at decoding weight $\mathbf{X} \in \mathbb{R}^{F\times T}$9, whereas pure CTC decoding is much worse at 46.5 / 52.8 (Polok et al., 2024).

Single-speaker robustness is largely preserved rather than destroyed. In greedy decoding, LibriSpeech clean/other WER changes from 2.5 / 4.5 for Whisper to 2.1 / 4.3 for the proposed system with $\mathbf{D}\in[0,1]^{S\times T}$0, while TED-LIUM and VoxPopuli change from 4.3 / 10.9 to 5.3 / 11.2, indicating some cost in domain generalization (Polok et al., 2024). The paper also demonstrates transfer beyond Whisper by applying FDDT to a Branchformer CTC-AED model, where AMI segment-group inference improves from ORC-WER 141.2 to 26.8 (Polok et al., 2024).

4. Overlap ambiguity, DiCoW v3.3, and SE-DiCoW

A central limitation of original DiCoW is that STNO masks can become nearly indistinguishable for different target speakers in heavily overlapped audio (Polok et al., 27 Jan 2026). If two speakers are continuously active together, both targets may be labeled mostly as overlap, so the model receives almost the same mixture and almost the same conditioning although the desired transcripts differ. SE-DiCoW addresses this by adding a second conditioning source: an automatically selected enrollment segment from elsewhere in the same recording where the target speaker is most active.

For each target speaker, SE-DiCoW searches the recording for the segment maximizing target-only activity: $\mathbf{D}\in[0,1]^{S\times T}$1 The selected self-enrollment segment is encoded under its own STNO conditioning and injected into every encoder layer of the main input via cross-attention and an MLP fusion block (Polok et al., 27 Jan 2026). Loss is computed only on the main mixture branch, not on the self-enrollment branch.

The same paper also defines DiCoW v3.3, which incorporates several nontrivial refinements before self-enrollment is added. These include corrected flexible data segmentation for long-form Whisper training, a new pre-positional embedding FDDT inserted immediately after convolutional subsampling, less aggressive initialization in which the diagonal scaling factor for non-target and silence transforms is increased from 0.1 to 0.5, and augmentation procedures such as Gaussian noise on STNO masks with $\mathbf{D}\in[0,1]^{S\times T}$2, dominant-class segment flips, SpecAugment, and MUSAN noise (Polok et al., 27 Jan 2026).

The reported effect is large. On the EMMA MT-ASR benchmark, SE-DiCoW reduces macro-averaged tcpWER by 52.4% relative to the original DiCoW (Polok et al., 27 Jan 2026). Under oracle diarization, examples include NOTSOFAR-1 Small-SDM 19.6 $\mathbf{D}\in[0,1]^{S\times T}$3 16.0 $\mathbf{D}\in[0,1]^{S\times T}$4 15.8, AMI SDM 17.5 $\mathbf{D}\in[0,1]^{S\times T}$5 14.5 $\mathbf{D}\in[0,1]^{S\times T}$6 14.3, Libri3Mix-Both 49.1 $\mathbf{D}\in[0,1]^{S\times T}$7 27.7 $\mathbf{D}\in[0,1]^{S\times T}$8 19.9, and Libri3Mix-Clean 39.5 $\mathbf{D}\in[0,1]^{S\times T}$9 16.0 $d(s,t)$0 9.7 for original DiCoW, DiCoW v3.3, and SE-DiCoW respectively (Polok et al., 27 Jan 2026). Under real diarization, gains remain but are smaller, for example AMI SDM 21.4 $d(s,t)$1 18.6 $d(s,t)$2 18.5 and Libri3Mix-Clean 47.1 $d(s,t)$3 31.6 $d(s,t)$4 29.3, reflecting the fact that diarization quality itself becomes the bottleneck (Polok et al., 27 Jan 2026).

5. Multilingual deployment, synthetic data, and robustness studies

DiCoW has also been studied as part of a multilingual diarization-plus-ASR pipeline in the MLC-SLM Challenge, where the BUT system combines DiCoW with DiariZen, a diarization pipeline built on top of Pyannote (Polok et al., 16 Jun 2025). In that setting, the task is two-speaker single-channel multilingual ASR. On the development set with ground-truth segmentation, baseline Whisper large-v3 scores 16.8 overall, out-of-domain DiCoW scores 22.0, and challenge fine-tuned DiCoW scores 12.9. Under real diarization, the baseline fine-tuned Pyannote + chunked Whisper large-v3 system scores 76.1, while DiCoW with fine-tuned DiariZen scores 28.4 out of domain and 20.8 after challenge-data fine-tuning (Polok et al., 16 Jun 2025). The final system achieves a micro-average tcpWER/CER of 16.75% and ranks second in Task 2 (Polok et al., 16 Jun 2025). The same paper argues that encoder-side conditioning allows DiCoW to retain solid multilingual behavior even though target-speaker ASR pretraining before challenge adaptation used English-only data.

A complementary line of work examines synthetic conversational data generation for DiCoW rather than architecture. In that study, DiCoW is trained with Whisper-large-v3-turbo, evaluated with tcpWER under ground-truth diarization and greedy attention-only decoding, and used as the representative MT-ASR model against Sortformer for diarization (Polok et al., 14 May 2026). The most prominent result is that more overlap is beneficial for DiCoW: with 500 h of synthetic conversations generated from NSF-1 close-talk source, a flat prior turn-taking model yields 24.8 on NSF-1 SC and 29.2 on AMI SDM, whereas CALLHOME (OV boost) yields 22.1 and 25.1 respectively (Polok et al., 14 May 2026). The same paper reports that source diversity outweighs exact domain matching: a Combined synthetic source reaches 20.6 on NSF-1, 16.5 on AMI, and 10.0 macro average, outperforming the real (AMI+NSF) baseline at 10.9 macro average (Polok et al., 14 May 2026). The strongest schedule is Synthetic $d(s,t)$5 real, which reaches 8.7 macro average and nearly matches the reported reference at 8.6 (Polok et al., 14 May 2026). Acoustic augmentation affects DiCoW less strongly than turn-taking and source diversity; for LibriSpeech source, macro average changes from 14.0 with no augmentation to 13.7 with noise, 14.3 with reverberation, and 13.8 with noise + reverberation (Polok et al., 14 May 2026).

Taken together, these studies show that DiCoW is not only an architecture but also a training regime sensitive to diarization annotation style, overlap statistics, and source-domain composition. The multilingual challenge system further shows that better diarization error patterns can matter more than lower DER alone: in the BUT study, VAD-assisted diarization reduces test-like development tcpWER/CER to 17.9 and improves test-set tcpWER/CER from 28.6% to 17.4%, even when DER does not improve monotonically (Polok et al., 16 Jun 2025).

6. Acceleration, serialized decoding, and spoken-language-model integration

Later work reuses DiCoW as a modular component rather than treating it as a closed design. One direction addresses DiCoW’s inference cost, which ordinarily scales with the number of diarized speakers because the recognizer is run once per target speaker. The paper on speaker-agnostic activity streams leaves DiCoW’s internal STNO and FDDT machinery unchanged but replaces per-speaker masks with two speaker-agnostic HEAT streams, so the recognizer runs only twice under the assumption that at most two speakers overlap at a time (He et al., 4 Oct 2025). With predicted Diarizen activity and beam search, AMI changes from 18.34 WER / 2.05 RTFx for speaker-conditioned DiCoW to 18.99 / 4.57 for HEAT-conditioned DiCoW, while ICSI changes from 25.55 / 1.50 to 26.24 / 3.89; the reported relative runtime gains are 123% on AMI and 159% on ICSI (He et al., 4 Oct 2025). The same paper emphasizes that naive first-available merging is unusable because it causes model collapse, whereas speaker-continuity and recency-continuity heuristics preserve enough local conversational structure for Whisper/DiCoW compatibility (He et al., 4 Oct 2025).

A second direction converts DiCoW from per-speaker decoding into a serialized joint decoder. SA-DiCoW runs the DiCoW encoder once per speaker, treats each conditioned output as a speaker-channel embedding, applies a learned speaker-specific affine transform, concatenates channels along time, and uses a shared decoder to emit a serialized stream of words, timestamps, and speaker tags (Kocour et al., 4 Oct 2025). This architecture surpasses DiCoW on fully overlapped synthetic mixtures, with Libri2Mix 4.8 $d(s,t)$6 3.9 and Libri3Mix 32.1 $d(s,t)$7 18.0 in cpWER, further improving to 3.4 and 17.2 with 5× loss weight on speaker-timestamp tokens (Kocour et al., 4 Oct 2025). On real meetings, however, original DiCoW remains stronger: on NOTSOFAR, DiCoW scores 18.0 versus 21.0 for SA-DiCoW and 20.8 for the speaker-loss variant, which the paper attributes largely to speaker-assignment errors such as leakage and omission (Kocour et al., 4 Oct 2025).

A third direction embeds DiCoW inside a spoken LLM. In Dixtral, a Diarization Conditioned Whisper (DiCoW) encoder replaces Voxtral’s standard Whisper encoder while the modality adapter and decoder remain frozen, so multi-speaker adaptation occurs entirely on the encoder side (Polok et al., 16 Jun 2026). The paper motivates this design as an alternative to Serialized Output Training for spoken LLMs, arguing that keeping the decoder frozen reduces catastrophic forgetting and preserves instruction-following, summarization, and QA behavior (Polok et al., 16 Jun 2026). On macro-average cpWER, DiCoW v3.3 as a specialized TS-ASR system achieves 14.0%, Dixtral achieves 15.4%, and the baselines Gemini 3.0 Flash, VibeVoice, and Voxtral Mini Transcribe V2 achieve 44.4%, 35.2%, and 31.4% respectively (Polok et al., 16 Jun 2026). The abstract further reports absolute cpWER gains of 29.0% over Gemini 3.0 Flash, 19.8% over VibeVoice, and 16.0% over Voxtral Mini Transcribe V2 (Polok et al., 16 Jun 2026). On the downstream NSF-QA benchmark, zero-shot Dixtral reaches 54.6% content QA, 25.4% emotion QA, and 24.4 ROUGE-L summarization, while task-specific fine-tuning yields 73.0% content QA, 47.6% emotion QA, 95.5% gender QA, and 41.4 ROUGE-L (Polok et al., 16 Jun 2026).

Across these derivative systems, DiCoW remains identifiable by the same core principle: diarization-conditioned acoustic encoding through STNO-style target/non-target structure, most often realized by FDDT inside a Whisper-compatible encoder. What changes across the literature is the surrounding system objective—constant-cost ASR, serialized speaker-attributed decoding, or spoken-language understanding—rather than the underlying role of DiCoW as a target-speaker representation mechanism.