---
title: Frame-Level Diarization-Dependent Transformations
url: https://www.emergentmind.com/topics/frame-level-diarization-dependent-transformations-fddt-e63ff8d2-85ca-4f1a-99c1-117ce3241dcc
type: topic
---

# Frame-Level Diarization-Dependent Transformations

Frame-Level Diarization-Dependent Transformations (FDDT) are a class of conditioning mechanisms for neural sequence models, particularly in speech processing, that modulate internal representations on a per-frame basis using externally provided diarization annotations. The primary context is target-speaker automatic speech recognition (ASR) and neural diarization. FDDT enables large, pre-trained, single-speaker models—such as Whisper—to function effectively in settings with overlapping or multi-speaker speech by leveraging fine-grained, frame-synchronous side information about speaker activity, rather than relying on global speaker embeddings or explicit source separation. The method is characterized by lightweight, learnable, typically affine, frame-level transformations controlled by diarization masks, supporting scenarios like target-speaker ASR, speaker-attributed ASR, and neural diarization conditioning [2409.09543][2501.00114][2303.06806].

## 1. Mathematical Formulation and Conditioning Principle

FDDT operates by conditioning each frame's hidden state on the corresponding diarization label. In ASR settings, labels are commonly encoded as a four-way STNO (Silence, Target, Non-target, Overlap) indicator or mask per frame. The general FDDT affine conditioning at layer $\ell$ is:

$$
\mathbf{z}'^\ell_t = \sum_{C \in \{\mathcal{S}, \mathcal{T}, \mathcal{N}, \mathcal{O}\}} \left( W_C^\ell \mathbf{z}_t^\ell + \mathbf{b}_C^\ell \right) p^{t}_{C}
$$

Here:
- $\mathbf{z}_t^\ell \in \mathbb{R}^{d_m}$: frame-$t$ input at layer $\ell$;
- $W_C^\ell, \mathbf{b}_C^\ell$: class-specific learnable affine transform and bias;
- $p^t_C$: probability (from diarization) that frame $t$ is in class $C$;
- $M^{t} = [p^{t}_{\mathcal{S}}, p^{t}_{\mathcal{T}}, p^{t}_{\mathcal{N}}, p^{t}_{\mathcal{O}}]^T$.

For bias-only variants, $W_C^\ell$ may be restricted to the identity or a diagonal matrix, so that FDDT reduces to a per-class bias shift.

In neural diarization, an analogous mechanism appears: at every transformer layer, a set of attractors ($A_l$) and their corresponding intermediate speaker labels ($Y_l$) are estimated via cross-attention, then injected additively (with learned fusion) into the hidden state as self-conditioning [2303.06806].

## 2. Diarization Signal Representation and STNO Mask Computation

A central element is the transformation of diarization output into a mask suitable for conditioning:

- For each frame $t$ and a fixed "target" speaker $s_k$, the frame's status is mapped to the four mutually exclusive classes via:

  - $p^t_{\mathcal{S}} = \prod_{s=1}^{S} (1 - d(s,t))$ ("silence")
  - $p^t_{\mathcal{T}} = d(s_k, t) \prod_{s \ne s_k} (1 - d(s, t))$ ("target only")
  - $p^t_{\mathcal{N}} = (1 - p^t_{\mathcal{S}}) - d(s_k, t)$ ("non-target only")
  - $p^t_{\mathcal{O}} = d(s_k, t) - p^t_{\mathcal{T}}$ ("overlap")

  $d(s,t) \in [0,1]$ is the diarization system's posterior estimated for speaker $s$ at frame $t$; $S$ is the number of speakers. In most FDDT systems, a hard (one-hot) mask is produced per frame using thresholded diarization [2501.00114].

## 3. Integration into Encoder Architectures

FDDT modules are inserted at the start of each transformer encoder layer. The generic sequence is:

- Input features $\rightarrow$ Positional Encoding $\rightarrow$ FDDT $\rightarrow$ LayerNorm $\rightarrow$ Self-Attention $\rightarrow$ LayerNorm $\rightarrow$ Feed-Forward.

Pseudocode for an encoder layer with FDDT [2501.00114][2409.09543]:

```python
def EncoderLayerWithFDDT(Z_l, M):
    Z_hat = []
    for t in range(T):
        z_hat = sum([W_C @ Z_l[:, t] + b_C) * M[C, t] for C in STNO])
        Z_hat.append(z_hat)
    Z_hat = np.stack(Z_hat, axis=1)
    H_l = LayerNorm(Z_hat)
    A_l = MultiHeadSelfAttention(H_l) + Z_hat
    F_l = LayerNorm(A_l)
    Z_next = FeedForward(F_l) + A_l
    return Z_next
```

Suppressive initialization is employed to avoid disruption of pretrained weights: $W_{\mathcal{T}}^\ell, W_{\mathcal{O}}^\ell$ initialized as identities; $W_{\mathcal{S}}^\ell, W_{\mathcal{N}}^\ell$ as zero; all biases as zero [2501.00114]. This causes non-target or silence frames to be suppressed initially.

The FDDT conditioning step is parallel across frames and does not alter sequence length, attention computation, or positional encoding. This architectural approach allows fine-grained, frame-wise modulation based on diarization status without recurrent or sequential dependencies.

## 4. Training Regimes and Optimization

FDDT is trained atop a largely frozen or pretrained encoder-decoder ASR model, usually with a joint CTC-Attention hybrid loss:

$$
L = (1 - \lambda_{\mathrm{ctc}}) \cdot L_{\mathrm{CE}} + \lambda_{\mathrm{ctc}} \cdot L_{\mathrm{CTC}}
$$

Typical training proceeds in several phases:
1. CTC-only "preheating" on single-speaker data with only the CTC head learnable.
2. FDDT preheating with only FDDT and CTC head parameters updated on multi-speaker mixtures; base model parameters frozen and higher learning rate for new modules.
3. Final joint fine-tuning of all parameters on multi-speaker datasets until convergence.

Suppressive initialization and high learning rates for FDDT parameters prevent disruption of the main model. Standard AdamW, learning rate warm-up/decay, and weight decay are used [2409.09543][2501.00114].

## 5. Empirical Results and Comparative Performance

Evaluation across multiple datasets demonstrates that FDDT outperforms input masking and embedding-based baselines for target-speaker and speaker-attributed ASR:

| Method               | AMI-sdm (%) | NOTSOFAR-1 (%) | LibriCSS (%) |
|----------------------|-------------|---------------|--------------|
| Vanilla Whisper      | 220.0       | 260.1         | 588.2        |
| Input Masking        |  52.8       |  61.6         |  56.2        |
| QK-Bias w/o shift    |  47.8       |  28.2         |  16.4        |
| FDDT init.           |  78.3       |  89.7         | 102.0        |
| FDDT (domain-tuned)  |  17.8       |  20.9         | —            |
| FDDT (multi-domain)  |  17.6       |  19.7         |   8.8        |

Results indicate that FDDT provides stronger word error rate (WER) and overlap-resolved WER (ORC-WER) reductions compared to masking or query-key biasing, particularly in overlapped and multi-speaker conditions [2409.09543][2501.00114]. Moreover, FDDT is robust to diarization imperfections (20% DER yielding only moderate degradation).

Ablations show diagonal + bias initialization outperforms full affine matrices, and four-class (STNO) masks yield better results than coarser alternatives. Gains are additive with more training data, CTC pretraining, and multi-domain adaptation.

## 6. Extensions, Limitations, and Generalization

FDDT is model-agnostic with respect to backbone: it has been successfully applied to Whisper, Branchformer, and transformer-based diarization (EEND-NA-deep+SelfCond). Limitations include reliance on external frame-level diarization—performance and robustness directly depend on diarization quality and alignment. Current approaches fix the mask to four STNO classes; more granular or multi-party masks remain an open area [2409.09543][2501.00114].

Suppressive initialization prevents catastrophic forgetting when adding FDDT atop pretrained networks but may limit expressivity relative to fully learned affine transforms. Synthetic mixture pretraining may yield further generalization to rare overlap patterns.

## 7. FDDT in Neural Diarization and Broader Context

In end-to-end neural diarization, FDDT appears as self-conditioning on intermediate attractor-derived speaker posteriors at every transformer layer [2303.06806]. This mechanism injects frame-level speaker label information (attractor-weighted residual) into each hidden representation, improving diarization error rate (DER) and training throughput over traditional EEND-EDA baselines. The generic form is:

$$
\widetilde{H}^{(l-1)}_t = H^{(l-1)}_t + W \sum_{c=1}^C a^{(l)}_c y^{(l)}_{c, t}
$$

A plausible implication is that FDDT unifies techniques in neural diarization and ASR by providing an efficient interface for integrating external or self-predicted frame-level speaker information, via lightweight and interpretable layerwise modulation.

---

**References:**

- "Target Speaker ASR with Whisper" [2409.09543]
- "DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition" [2501.00114]
- "Neural Diarization with Non-autoregressive Intermediate Attractors" [2303.06806]

Source: https://www.emergentmind.com/topics/frame-level-diarization-dependent-transformations-fddt-e63ff8d2-85ca-4f1a-99c1-117ce3241dcc