---
title: 'DiCoW: Diarization-Conditioned Whisper ASR'
url: https://www.emergentmind.com/topics/dicow
type: topic
---

# DiCoW: Diarization-Conditioned Whisper ASR

DiCoW, short for **Diarization-Conditioned Whisper**, is a target-speaker automatic speech recognition framework that adapts Whisper to multi-speaker audio by conditioning on speaker diarization outputs rather than speaker embeddings [2501.00114]. It is designed for speaker-attributed ASR in meetings, conversational mixtures, and overlapping speech, where the objective is to transcribe one selected speaker at a time from a shared recording. The method’s central representation is a target-dependent **Silence–Target–Non-target–Overlap (STNO)** mask, injected into the acoustic encoder through diarization-conditioned transformations. Subsequent work has extended DiCoW with self-enrollment for overlap disambiguation, speaker-agnostic acceleration, serialized joint decoding, multilingual challenge systems, synthetic-data training studies, and integration into spoken language models such as Dixtral [2601.19194] [2510.03630] [2510.03723] [2506.13414] [2605.15442] [2606.18134].

## 1. Concept and problem setting

DiCoW was introduced to solve a specific failure mode of strong single-speaker ASR systems in multi-speaker recordings: off-the-shelf Whisper tends to transcribe all audible speech rather than isolating a chosen speaker [2501.00114]. In the original formulation, the input is a long multi-speaker recording together with diarization labels; for each diarized speaker \(s_k\), the system constructs a target-dependent conditioning signal and decodes that speaker’s transcript independently. The resulting workflow is diarization-assisted rather than fully end-to-end: diarization is external, whereas ASR is internal and target-conditioned.

The original motivation is explicitly contrasted with target-speaker ASR systems based on x-vectors, i-vectors, enrollment speech, or other speaker embeddings. DiCoW’s authors argue that embedding-based conditioning imposes a difficult mapping from speaker-identity space into the ASR model’s internal representation space, increases data requirements, and complicates deployment because enrollment speech or a separate speaker-embedding module is required [2501.00114]. DiCoW instead conditions on **when** the target speaker is active, using diarization activity patterns as the control signal.

This design places DiCoW between classical modular diarization–ASR pipelines and fully joint speaker-attributed ASR systems. It does not emit speaker tags as native decoder outputs in its base form; speaker attribution is obtained by repeating target-conditioned decoding once per diarized speaker. The same conceptual framing was later retained in multilingual challenge systems, synthetic-data studies, and spoken-language-model integration, where DiCoW remains the encoder-side target-speaker acoustic front end rather than a full multi-output recognizer [2506.13414] [2605.15442] [2606.18134].

## 2. Conditioning mechanisms and architecture

In the original system, the acoustic input is a log-Mel spectrogram \( \mathbf{X} \in \mathbb{R}^{F\times T} \), and diarization is represented as \( \mathbf{D}\in[0,1]^{S\times T} \), where \(d(s,t)\) denotes the activity probability of speaker \(s\) at frame \(t\) [2501.00114]. For a chosen target speaker \(s_k\), DiCoW reduces the diarization output to a fixed four-class STNO representation:
\[
p_{\mathcal{S}^t} = \prod_{s=1}^S (1 - d(s, t)),
\]
\[
p_{\mathcal{T}^t} = d(s_k, t)\prod_{\substack{s=1 \\ s \neq s_k}}^S (1 - d(s, t)),
\]
\[
p_{\mathcal{N}^t} = \left(1 - p_{\mathcal{S}^t}\right) - d(s_k, t),
\]
\[
p_{\mathcal{O}^t} = d(s_k, t) - p_{\mathcal{T}^t}.
\]

These four quantities encode, per frame, silence, target-only speech, non-target-only speech, and target-involving overlap. A key architectural consequence is that the conditioning dimensionality is fixed even when the number of speakers changes.

The original paper studies three conditioning mechanisms. The simplest is **input masking**, where the waveform is multiplied by \(p_{\mathcal{T}^t}+p_{\mathcal{O}^t}\), preserving only target-active or overlap regions. The second is **query-key biasing (QKb)**, which modifies encoder self-attention and decoder cross-attention by subtracting a bias \(c\) from attention logits associated with non-target frames. The third, and best-performing, mechanism is **Frame-Level Diarization-Dependent Transformations (FDDT)**, which inserts STNO-conditioned affine transforms into the encoder between Transformer layers [2501.00114].

For encoder layer \(l\) and frame \(t\), FDDT applies
\[
\hat{\mathbf{z}_t^l} =
\sum_{i \in \{\mathcal{S}, \mathcal{T}, \mathcal{N}, \mathcal{O}\}}
(\mathbf{W}_i^l \mathbf{z}_t^l + \mathbf{b}_i^l)p_i^t.
\]
In the original implementation, the transforms are diagonally parameterized, and the initial state is deliberately suppressive: silence and non-target transforms are initialized to zero, while target and overlap transforms are initialized to identity, with all biases set to zero [2501.00114]. This keeps the initial behavior close to “preserve target, suppress non-target” and stabilizes adaptation.

The same paper also adds an optional **CTC head** on top of Whisper encoder outputs and uses joint CTC/attention decoding:
\[
\hat{C} = \arg \max_{C \in \mathcal{U}^*}
\left(
\lambda \log p_{\text{ctc}}(C|\mathbf{X}) +
(1-\lambda)\log p_{\text{att}}(C|\mathbf{X})
\right).
\]
An additional **Co-Attention** module is introduced for heavy-overlap settings so that independently decoded speaker streams can exchange contextual information and reduce redundant assignment of the same dominant speaker [2501.00114].

## 3. Training protocol and empirical behavior of the original system

The original DiCoW experiments use **Whisper-large-v3-turbo**, whose decoder is reduced from 32 to 4 layers, and train in three phases: **CTC preheat**, **FDDT preheat**, and **full fine-tuning** [2501.00114]. The reported optimization setup uses AdamW, batch size 64, weight decay \(1\times10^{-6}\), peak learning rate \(2\times10^{-7}\), 5k warm-up steps, linear decay after warm-up, maximum 50k steps, and CTC loss weight 0.3. The CTC preheat stage uses LibriSpeech 960h; multi-speaker fine-tuning uses AMI, NOTSOFAR-1, and Libri2Mix, with a multi-domain sampling ratio of 4:4:1 for the FDDT MD model [2501.00114].

The baseline contrast is large. Plain Whisper used as a target-speaker recognizer yields **tcpWER 220.0** on AMI-sdm test, **tcpWER 260.1** on NOTSOFAR-1 eval-small, and **tcpWER 588.2** on LibriCSS test [2501.00114]. Under oracle diarization, the best reported DiCoW results include **AMI-sdm cpWER 17.2** and **ORC-WER 16.5**, **NOTSOFAR-1 eval-small tcpWER 19.7** and **tcORC-WER 19.1**, **Libri2Mix test-clean cpWER 4.4**, **Libri2Mix test-both cpWER 10.9**, and **LibriCSS cpWER 5.6** [2501.00114]. Under real diarization, performance degrades to **AMI-sdm cpWER 23.6 / ORC-WER 18.0**, **NOTSOFAR-1 tcpWER 33.5 / tcORC-WER 22.6**, **Libri2Mix test-clean cpWER 6.0**, **Libri2Mix test-both cpWER 15.0**, and **LibriCSS cpWER 8.5 / ORC-WER 6.5**, confirming strong dependence on diarization quality [2501.00114].

Among conditioning mechanisms, FDDT is consistently superior to input masking and QKb. The paper also reports that FDDT converges much faster than QKb, reaching **tcpWER below 30% after 1000 steps** on NOTSOFAR-1 [2501.00114]. The optional CTC head improves both training and decoding behavior: on NOTSOFAR-1 eval, greedy/beam-5 results improve from **22.9 / 22.2** without CTC to **21.7 / 20.9** with hybrid decoding at decoding weight \(\lambda=0.2\), whereas pure CTC decoding is much worse at **46.5 / 52.8** [2501.00114].

Single-speaker robustness is largely preserved rather than destroyed. In greedy decoding, LibriSpeech clean/other WER changes from **2.5 / 4.5** for Whisper to **2.1 / 4.3** for the proposed system with \(\lambda=0\), while TED-LIUM and VoxPopuli change from **4.3 / 10.9** to **5.3 / 11.2**, indicating some cost in domain generalization [2501.00114]. The paper also demonstrates transfer beyond Whisper by applying FDDT to a Branchformer CTC-AED model, where AMI segment-group inference improves from **ORC-WER 141.2** to **26.8** [2501.00114].

## 4. Overlap ambiguity, DiCoW v3.3, and SE-DiCoW

A central limitation of original DiCoW is that STNO masks can become nearly indistinguishable for different target speakers in heavily overlapped audio [2601.19194]. If two speakers are continuously active together, both targets may be labeled mostly as overlap, so the model receives almost the same mixture and almost the same conditioning although the desired transcripts differ. SE-DiCoW addresses this by adding a second conditioning source: an automatically selected enrollment segment from elsewhere in the same recording where the target speaker is most active.

For each target speaker, SE-DiCoW searches the recording for the segment maximizing target-only activity:
\[
[t_{\text{start}}, t_{\text{end}}] =
\arg\max_{t_{\text{start}}, t_{\text{end}}}
\sum_{t=t_{\text{start}}}^{t_{\text{end}}} p_{\mathcal{T}^t}.
\]
The selected self-enrollment segment is encoded under its own STNO conditioning and injected into every encoder layer of the main input via cross-attention and an MLP fusion block [2601.19194]. Loss is computed only on the main mixture branch, not on the self-enrollment branch.

The same paper also defines **DiCoW v3.3**, which incorporates several nontrivial refinements before self-enrollment is added. These include corrected flexible data segmentation for long-form Whisper training, a new **pre-positional embedding FDDT** inserted immediately after convolutional subsampling, less aggressive initialization in which the diagonal scaling factor for non-target and silence transforms is increased from **0.1** to **0.5**, and augmentation procedures such as Gaussian noise on STNO masks with \(\epsilon^t \sim \mathcal{N}(0,0.2^2)\), dominant-class segment flips, SpecAugment, and MUSAN noise [2601.19194].

The reported effect is large. On the EMMA MT-ASR benchmark, **SE-DiCoW reduces macro-averaged tcpWER by 52.4% relative to the original DiCoW** [2601.19194]. Under oracle diarization, examples include **NOTSOFAR-1 Small-SDM 19.6 \(\rightarrow\) 16.0 \(\rightarrow\) 15.8**, **AMI SDM 17.5 \(\rightarrow\) 14.5 \(\rightarrow\) 14.3**, **Libri3Mix-Both 49.1 \(\rightarrow\) 27.7 \(\rightarrow\) 19.9**, and **Libri3Mix-Clean 39.5 \(\rightarrow\) 16.0 \(\rightarrow\) 9.7** for original DiCoW, DiCoW v3.3, and SE-DiCoW respectively [2601.19194]. Under real diarization, gains remain but are smaller, for example **AMI SDM 21.4 \(\rightarrow\) 18.6 \(\rightarrow\) 18.5** and **Libri3Mix-Clean 47.1 \(\rightarrow\) 31.6 \(\rightarrow\) 29.3**, reflecting the fact that diarization quality itself becomes the bottleneck [2601.19194].

## 5. Multilingual deployment, synthetic data, and robustness studies

DiCoW has also been studied as part of a multilingual diarization-plus-ASR pipeline in the **MLC-SLM Challenge**, where the BUT system combines DiCoW with **DiariZen**, a diarization pipeline built on top of Pyannote [2506.13414]. In that setting, the task is two-speaker single-channel multilingual ASR. On the development set with ground-truth segmentation, baseline Whisper large-v3 scores **16.8** overall, out-of-domain DiCoW scores **22.0**, and challenge fine-tuned DiCoW scores **12.9**. Under real diarization, the baseline fine-tuned Pyannote + chunked Whisper large-v3 system scores **76.1**, while DiCoW with fine-tuned DiariZen scores **28.4** out of domain and **20.8** after challenge-data fine-tuning [2506.13414]. The final system achieves a **micro-average tcpWER/CER of 16.75%** and ranks **second in Task 2** [2506.13414]. The same paper argues that encoder-side conditioning allows DiCoW to retain solid multilingual behavior even though target-speaker ASR pretraining before challenge adaptation used English-only data.

A complementary line of work examines synthetic conversational data generation for DiCoW rather than architecture. In that study, DiCoW is trained with **Whisper-large-v3-turbo**, evaluated with **tcpWER** under **ground-truth diarization** and **greedy attention-only decoding**, and used as the representative MT-ASR model against Sortformer for diarization [2605.15442]. The most prominent result is that more overlap is beneficial for DiCoW: with 500 h of synthetic conversations generated from NSF-1 close-talk source, a **flat prior** turn-taking model yields **24.8** on NSF-1 SC and **29.2** on AMI SDM, whereas **CALLHOME (OV boost)** yields **22.1** and **25.1** respectively [2605.15442]. The same paper reports that **source diversity outweighs exact domain matching**: a **Combined** synthetic source reaches **20.6** on NSF-1, **16.5** on AMI, and **10.0** macro average, outperforming the **real (AMI+NSF)** baseline at **10.9** macro average [2605.15442]. The strongest schedule is **Synthetic \(\rightarrow\) real**, which reaches **8.7** macro average and nearly matches the reported reference at **8.6** [2605.15442]. Acoustic augmentation affects DiCoW less strongly than turn-taking and source diversity; for LibriSpeech source, macro average changes from **14.0** with no augmentation to **13.7** with noise, **14.3** with reverberation, and **13.8** with noise + reverberation [2605.15442].

Taken together, these studies show that DiCoW is not only an architecture but also a training regime sensitive to diarization annotation style, overlap statistics, and source-domain composition. The multilingual challenge system further shows that better diarization error patterns can matter more than lower DER alone: in the BUT study, VAD-assisted diarization reduces test-like development **tcpWER/CER** to **17.9** and improves test-set **tcpWER/CER** from **28.6%** to **17.4%**, even when DER does not improve monotonically [2506.13414].

## 6. Acceleration, serialized decoding, and spoken-language-model integration

Later work reuses DiCoW as a modular component rather than treating it as a closed design. One direction addresses DiCoW’s inference cost, which ordinarily scales with the number of diarized speakers because the recognizer is run once per target speaker. The paper on **speaker-agnostic activity streams** leaves DiCoW’s internal STNO and FDDT machinery unchanged but replaces per-speaker masks with two speaker-agnostic **HEAT** streams, so the recognizer runs only twice under the assumption that at most two speakers overlap at a time [2510.03630]. With predicted Diarizen activity and beam search, AMI changes from **18.34 WER / 2.05 RTFx** for speaker-conditioned DiCoW to **18.99 / 4.57** for HEAT-conditioned DiCoW, while ICSI changes from **25.55 / 1.50** to **26.24 / 3.89**; the reported relative runtime gains are **123%** on AMI and **159%** on ICSI [2510.03630]. The same paper emphasizes that naive first-available merging is unusable because it causes **model collapse**, whereas **speaker-continuity** and **recency-continuity** heuristics preserve enough local conversational structure for Whisper/DiCoW compatibility [2510.03630].

A second direction converts DiCoW from per-speaker decoding into a serialized joint decoder. **SA-DiCoW** runs the DiCoW encoder once per speaker, treats each conditioned output as a speaker-channel embedding, applies a learned speaker-specific affine transform, concatenates channels along time, and uses a shared decoder to emit a serialized stream of words, timestamps, and speaker tags [2510.03723]. This architecture surpasses DiCoW on fully overlapped synthetic mixtures, with **Libri2Mix 4.8 \(\rightarrow\) 3.9** and **Libri3Mix 32.1 \(\rightarrow\) 18.0** in cpWER, further improving to **3.4** and **17.2** with 5× loss weight on speaker-timestamp tokens [2510.03723]. On real meetings, however, original DiCoW remains stronger: on **NOTSOFAR**, DiCoW scores **18.0** versus **21.0** for SA-DiCoW and **20.8** for the speaker-loss variant, which the paper attributes largely to speaker-assignment errors such as leakage and omission [2510.03723].

A third direction embeds DiCoW inside a spoken language model. In **Dixtral**, a **Diarization Conditioned Whisper (DiCoW)** encoder replaces Voxtral’s standard Whisper encoder while the modality adapter and decoder remain frozen, so multi-speaker adaptation occurs entirely on the encoder side [2606.18134]. The paper motivates this design as an alternative to Serialized Output Training for spoken LLMs, arguing that keeping the decoder frozen reduces catastrophic forgetting and preserves instruction-following, summarization, and QA behavior [2606.18134]. On macro-average cpWER, **DiCoW v3.3** as a specialized TS-ASR system achieves **14.0%**, **Dixtral** achieves **15.4%**, and the baselines **Gemini 3.0 Flash**, **VibeVoice**, and **Voxtral Mini Transcribe V2** achieve **44.4%**, **35.2%**, and **31.4%** respectively [2606.18134]. The abstract further reports absolute cpWER gains of **29.0%** over Gemini 3.0 Flash, **19.8%** over VibeVoice, and **16.0%** over Voxtral Mini Transcribe V2 [2606.18134]. On the downstream **NSF-QA** benchmark, zero-shot Dixtral reaches **54.6% content QA**, **25.4% emotion QA**, and **24.4 ROUGE-L** summarization, while task-specific fine-tuning yields **73.0% content QA**, **47.6% emotion QA**, **95.5% gender QA**, and **41.4 ROUGE-L** [2606.18134].

Across these derivative systems, DiCoW remains identifiable by the same core principle: diarization-conditioned acoustic encoding through STNO-style target/non-target structure, most often realized by FDDT inside a Whisper-compatible encoder. What changes across the literature is the surrounding system objective—constant-cost ASR, serialized speaker-attributed decoding, or spoken-language understanding—rather than the underlying role of DiCoW as a target-speaker representation mechanism.

Source: https://www.emergentmind.com/topics/dicow