DiCoW: Diarization Conditioned Whisper ASR

• The paper introduces DiCoW’s method of using diarization outputs to condition Whisper’s encoder, achieving speaker-attributed ASR without explicit speaker embeddings.
• It employs a four-way STNO mask—Separating Silence, Target, Non-target, and Overlap—to transform frame-level diarization signals into effective conditioning for accurate transcription.
• Experimental results show robust WER improvements in multi-talker and overlapping speech settings, confirming DiCoW’s scalability and effectiveness.

Searching arXiv for DiCoW and closely related papers to ground the article in the primary literature. Diarization Conditioned Whisper (DiCoW) is a family of Whisper-based target-speaker automatic speech recognition (TS-ASR) methods in which speaker diarization is used not merely to segment audio, but to condition the acoustic encoder itself. In DiCoW, a mixed single-channel recording and diarization-derived framewise speaker activity estimates are combined so that Whisper produces the transcript of one designated speaker at a time, enabling speaker-attributed ASR (SA-ASR) without relying on explicit speaker embeddings, enrollment recordings, or speaker-ID-style representations (Polok et al., 2024, Polok et al., 16 Jun 2025).

1. Conceptual scope and problem formulation

DiCoW addresses the mismatch between strong large-scale single-speaker ASR models and conversational audio containing turn-taking, overlap, and competing speakers. In standard Whisper, the model receives a mixed signal and is not explicitly told which speaker should be transcribed. DiCoW reformulates this as TS-ASR: given a recording and a chosen target speaker, output only that speaker’s transcript. SA-ASR is then obtained by running the target-speaker recognizer sequentially for each hypothesized speaker in the diarization output (Polok et al., 2024).

A defining premise of the method is that it is easier to condition ASR on frame-level diarization outputs than to learn a universal speaker-embedding space and then learn how that space should control recognition. This shifts the problem from cross-speaker identity matching to target-relative activity modeling: at each frame, the model is told whether the acoustics correspond to silence, target speech, non-target speech, or overlap involving the target speaker (Polok et al., 2024).

This framing also distinguishes DiCoW from a classic diarization-plus-ASR cascade. In a standard cascade, diarization is used to cut the recording into chunks that are then transcribed independently. In DiCoW, diarization is injected into Whisper’s encoder computation over the full recording. A common misconception is therefore that DiCoW is simply “Whisper after diarization.” The later system descriptions explicitly reject that interpretation: the official baseline for the MLC-SLM challenge transcribed diarization chunks independently, whereas the DiCoW-based system ran a long-form, diarization-conditioned Whisper whose encoder activations depended on diarization throughout acoustic encoding (Polok et al., 16 Jun 2025).

2. Target-relative diarization representation: STNO masks

The core conditioning interface in DiCoW is a target-relative four-way decomposition of diarization output, usually termed STNO: Silence, Target, Non-target, and Overlap. Let $\mathbf{D} \in [0,1]^{S \times T}$ denote frame-level diarization posteriors, where $d(s,t)$ is the probability that speaker $s$ is active at frame $t$. For a designated target speaker $s_k$, DiCoW defines

$p_S^t = \prod_{s=1}^S (1-d(s,t)),$

$$p_T^t = d(s_k,t)\prod_{\substack{s=1 \ s\neq s_k}}^S (1-d(s,t)),$$

$p_N^t = (1-p_S^t) - d(s_k,t),$

$p_O^t = d(s_k,t) - p_T^t,$

and forms the framewise conditioning vector

$$\mathbf{M}^t = \begin{bmatrix} p_S^t & p_T^t & p_N^t & p_O^t \end{bmatrix}^{T}.$$

This reduction is significant because it maps an arbitrary-speaker diarization hypothesis into a fixed four-dimensional, target-relative description that Whisper can consume without changing shape as the number of speakers varies (Polok et al., 2024).

The STNO representation is not a hard source extraction mask. It preserves uncertainty. Later descriptions emphasize that this matters because diarization is inherently uncertain, especially near boundaries and in overlap, and soft conditioning allows the ASR model to exploit that uncertainty rather than committing to brittle binary decisions (Polok et al., 16 Jun 2025).

Operationally, STNO gives the encoder a localized interpretation prior. In target-only frames, the model can preserve evidence relevant to the designated speaker; in non-target frames, it can suppress misleading competing speech; in overlap frames, it can use a distinct transformation specialized for partial contamination; and in silence, it can reduce hallucinated text. This target-relative decomposition is one of the main reasons DiCoW is especially suited to two-speaker and meeting-style ASR pipelines in which a diarizer can estimate who is active when, even when reliable enrollment material is unavailable (Polok et al., 2024).

3. Encoder-side conditioning: FDDT, minimal variants, and related mechanisms

DiCoW is architecturally still Whisper, but with encoder-side modifications. Its principal mechanism is Frame-Level Diarization-Dependent Transformations (FDDT). If $d(s,t)$0 is the framewise input to encoder transformer layer $d(s,t)$1, DiCoW defines affine transforms $d(s,t)$2, $d(s,t)$3, $d(s,t)$4, and $d(s,t)$5, and computes

$d(s,t)$6

Thus each frame is transformed into four STNO-specific versions and the actual hidden state is their soft mixture weighted by diarization posteriors. In the later BUT system description, the same idea is summarized layerwise as a framewise adaptive linear transformation driven by diarization probabilities, with no decoder changes, no speaker tokens, and no extra cross-attention path described there (Polok et al., 2024, Polok et al., 16 Jun 2025).

The original DiCoW literature also studied simpler and more aggressive parameterizations. Bias-only conditioning, diagonal-plus-bias conditioning, and full affine matrices were compared. A key empirical result was that bias-only conditioning works almost as well as richer transforms, and that even one conditioned encoder layer can approach the performance of conditioning all layers. This is the basis for the statement that “adding even a single bias term per diarization output type before the first transformer block” can transform a single-speaker ASR model into a TS-ASR model (Polok et al., 2024).

A second conditioning mechanism, Query-Key Biasing (QKb), was introduced in the broader DiCoW study as an attention-side alternative that attenuates attention to non-target frames. Input masking was also evaluated as a simpler baseline. On real datasets, however, FDDT emerged as the most robust conditioning method, while QKb was more fragile at initialization and more susceptible to hallucinations. The same paper also reports that FDDT can be transferred beyond Whisper to a Branchformer CTC-AED model, suggesting that STNO-conditioned encoder adaptation is not tied to Whisper’s decoder specifically (Polok et al., 2024).

Initialization is critical because arbitrary affine transforms can distort pretrained encoder states. Across the DiCoW papers, the added parameters are initialized so that the pretrained Whisper behavior is preserved at the start of fine-tuning; suppressive or identity-style initializations are explicitly contrasted with random initialization, which is often damaging. This suggests that DiCoW’s strong retention of general ASR behavior is closely tied to introducing diarization conditioning as a near-identity perturbation of a pretrained backbone rather than as a wholesale reparameterization (Polok et al., 2024).

4. Training and inference pipeline

DiCoW is trained by fine-tuning rather than from scratch. The original Whisper-based formulation uses a hybrid CTC-attention setup, motivated by optimization stability and hallucination control. The broader DiCoW paper adds an auxiliary CTC head on top of the encoder, and decoding can interpolate attention and CTC scores as

$d(s,t)$7

Even when CTC is not used at test time, joint training with the CTC head improves alignment and reduces hallucinations; CTC-only decoding, however, is much weaker than hybrid or attention-dominant decoding (Polok et al., 2024).

A recurring DiCoW training recipe is staged. In the BUT MLC-SLM system, the training setup follows a three-phase strategy: CTC preheat on LibriSpeech, then FDDT preheat and full fine-tuning on AMI, NOTSOFAR-1, and Libri2Mix using Whisper large-v3-turbo, followed by challenge-specific fine-tuning on the MLC-SLM dataset. Because Whisper has a 30-second input limit, training data are segmented into chunks of at most 30 seconds by iteratively concatenating consecutive provided segments until the total duration reaches 30 seconds. Inference, however, is described as long-form processing over the recording rather than independent transcription of diarization chunks (Polok et al., 16 Jun 2025).

Inference is target-speaker-centric. A diarizer first produces framewise speaker activity probabilities $d(s,t)$8. For each target speaker hypothesis $d(s,t)$9, these are converted into an STNO mask and a separate target-conditioned decoding pass is run. This naturally supports SA-ASR, but it also means inference cost scales roughly linearly with the number of speakers or speaker hypotheses because one decoding pass is required per target (Polok et al., 2024).

The later system papers add practical refinements rather than altering the core conditioning interface. In the MLC-SLM submission, for example, training uses a normalization trick in which the model computes cross-entropy both against the original cased reference and against a lowercased, punctuation-stripped version, backpropagating only the smaller loss; checkpoint selection is by development-set tcpWER; and final English decoding uses joint CTC decoding with weight $s$0 and beam size $s$1 (Polok et al., 16 Jun 2025).

5. Empirical performance, multilingual behavior, and challenge results

Early Whisper-based diarization conditioning already showed strong gains on meeting-style TS-ASR. On NOTSOFAR-1 eval-small, the strongest final system in the 2024 “Target Speaker ASR with Whisper” study reached $s$2 ORC-WER, compared with $s$3 for naive input masking and a cited NOTSOFAR baseline at $s$4, yielding an $s$5-point improvement over that baseline. The same study also reported $s$6 on AMI-sdm test and $s$7 on Libri2Mix test-both for its Whisper-large-v3 system, while showing that explicit STNO information outperformed reduced target-only variants and that even minimal bias-only conditioning remained competitive (Polok et al., 2024).

The broader DiCoW paper established the method across real and synthetic benchmarks. Using the best models reported there, DiCoW achieved $s$8 cpWER and $s$9 ORC-WER on AMI-sdm with real diarization, $t$0 and $t$1 with oracle diarization, $t$2 and $t$3 on Libri2Mix test-clean under real/oracle diarization, and $t$4 cpWER and $t$5 ORC-WER on LibriCSS with real diarization. The same paper reports that vanilla Whisper is catastrophically poor in TS-ASR settings, whereas FDDT is the most robust conditioning method and preserves much of Whisper’s single-speaker accuracy (Polok et al., 2024).

The 2025 BUT system provides a particularly clear picture of DiCoW’s behavior under realistic diarization and multilingual transfer. On the MLC-SLM development set with ground-truth segmentation, out-of-domain DiCoW was worse on average than Whisper large-v3, $t$6 versus $t$7 tcpWER/CER, but after fine-tuning on challenge data the DiCoW-based system improved to $t$8. Under realistic diarization, the baseline Pyannote-plus-chunked-Whisper pipeline scored $t$9 overall tcpWER/CER on the development set, the out-of-domain DiCoW + DiariZen system reduced this to $s_k$0, and the fine-tuned DiCoW + fine-tuned DiariZen system further improved to $s_k$1. The final challenge system achieved a micro-average tcpWER/CER of $s_k$2 and ranked second in Task 2. The same paper emphasizes that DiCoW, though previously fine-tuned on English-only data for TS-ASR, retained solid out-of-domain multilingual performance, which the authors interpret as evidence that encoder-side conditioning preserves Whisper’s multilingual capabilities (Polok et al., 16 Jun 2025).

6. Extensions, scalability variants, and limitations

A central limitation of DiCoW is its dependence on diarization quality. Because STNO masks are derived directly from diarization output, missed speech, boundary errors, speaker confusions, and overlap misestimation propagate into the encoder conditioning itself. The BUT challenge paper documents this sharply: label inconsistencies such as missing speech segments and incorrect silence annotations can cause a fine-tuned diarizer to improve DER while harming downstream DiCoW-based ASR on a modified “test-like” development set. The same study reports that combining diarization with Silero VAD using weight $s_k$3 improved speech/silence estimation and reduced tcpWER/CER from $s_k$4 to $s_k$5 on the test-like development set and from $s_k$6 to $s_k$7 on the test set (Polok et al., 16 Jun 2025).

A second limitation is ambiguity under dense overlap. SE-DiCoW identifies the failure mode explicitly: when two or more speakers are fully overlapped, their STNO masks may be nearly identical despite different target transcripts. SE-DiCoW addresses this by locating a self-enrollment segment elsewhere in the recording where the target speaker is most active, encoding that segment, and injecting it through cross-attention at each encoder layer while retaining STNO conditioning. Together with improved segmentation, initialization, and augmentation, this reduced macro-averaged tcpWER by $s_k$8 relative to the original DiCoW on the EMMA MT-ASR benchmark, with especially large gains on Libri3Mix (Polok et al., 27 Jan 2026).

The cost of one-pass-per-speaker inference has also motivated scalability reformulations. “Scaling Multi-Talker ASR with Speaker-Agnostic Activity Streams” replaces per-speaker activity with two speaker-agnostic HEAT streams and feeds stream-derived STNO masks into DiCoW unchanged. On AMI and ICSI, this keeps the number of DiCoW runs fixed at two, regardless of speaker count, and yields only small WER degradation relative to speaker-conditioned inference while substantially improving runtime in RTFx terms (He et al., 4 Oct 2025).

DiCoW has also become a reusable encoder module in larger systems. SA-DiCoW converts per-speaker DiCoW encoder outputs into a shared SOT decoder for joint multi-talker decoding, outperforming original DiCoW on dense synthetic overlap such as Libri3Mix but remaining worse on NOTSOFAR because speaker label assignment becomes the main error source (Kocour et al., 4 Oct 2025). Dixtral reuses a DiCoW encoder inside a Voxtral-style spoken LLM, conditioning the acoustic encoder on diarization masks while keeping the LLM decoder frozen; on AMI, NOTSOFAR-1, LibriSpeechMix, and Mixer6, it substantially outperforms several spoken-LM baselines on cpWER while supporting far-field multi-speaker QA and summarization (Polok et al., 16 Jun 2026).

Across these extensions, a consistent interpretation emerges. DiCoW is not merely a diarization-aware post-processor for Whisper, but a specific encoder-conditioning strategy: reduce diarization to a target-relative STNO stream, modulate hidden states with lightweight frame-conditioned transforms, and decode one speaker-conditioned transcript at a time. This design has proved effective across TS-ASR, SA-ASR, challenge systems, and spoken LLM grounding, while its main caveats remain diarization dependence, overlap ambiguity, and per-speaker inference cost.