---
title: Spoof Diarization Overview
url: https://www.emergentmind.com/topics/spoof-diarization
type: topic
---

# Spoof Diarization Overview

Searching arXiv for the specified spoof diarization papers and closely related diarization work to ground the article in current literature.
Spoof diarization is the problem of determining “what spoofed when” in a partially spoofed audio utterance: a system must temporally locate bona fide and spoofed regions and assign spoofed regions to their underlying generation methods rather than merely detecting the presence of any spoofing [2406.07816][2509.13085]. In the current literature, the task is positioned as a generalization of utterance-level spoof detection and frame-level spoof localization, while remaining distinct from speaker diarization because it clusters regions by spoof method rather than speaker identity, often at short time scales such as 20 ms frames [2406.07816].

## 1. Task definition and problem scope

The initial task formulation treats an input waveform \(x_{1:T}\) as a sequence to be broken into contiguous frames or segments, each assigned a multi-class label \(c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}\), where \(\mathrm{ConP}\) is an optional label for non-speech or joined segments [2406.07816]. A later formulation expresses the same problem at frame level: given an audio utterance \(x\) or pre-trained frame embeddings \(\{F_t \in \mathbb{R}^d\}_{t=1}^T\), the system outputs a label sequence \(\{y_t\}_{t=1}^T\) with \(y_t=0\) for bona-fide speech and \(y_t \in \{1,\dots,L-1\}\) for one of \(L-1\) spoof types [2509.13085].

This framing separates spoof diarization from adjacent tasks. Utterance-level spoof detection produces a binary label per utterance, while frame-level spoof localization produces a binary label per frame; spoof diarization instead requires multi-class, per-frame assignment of spoofed frames to spoof methods [2509.13085]. The literature also distinguishes it from speaker diarization: both problems cluster temporally homogeneous regions, but speaker diarization clusters by speaker identity, whereas spoof diarization clusters by spoof method, with short 20 ms segments identified as crucial for the latter [2406.07816].

A common misconception is to treat spoof diarization as binary spoof localization followed by ordinary clustering. The benchmark formulation indicates that the task must jointly address temporal segmentation and spoof-type clustering, and the later token-based model was proposed precisely because simple structures lacked explicit reference points for separating bona fide speech from multiple spoofing types [2509.13085].

## 2. Benchmark formulation and the 3C model

The pioneering benchmark model is the Countermeasure-Condition Clustering (3C) model, which introduced a two-branch architecture for spoof diarization [2406.07816]. Its localization branch, CM-loc, is a framewise spoof countermeasure producing per-frame scores \(s_{1:M}\). Its diarization branch, CM-dia, is a framewise embedding extractor yielding \(e_{1:M} \in \mathbb{R}^D\), followed by agglomerative clustering with oracle \(K\). The inference flow is explicitly described as: uniform 20 ms framing of \(x_{1:T}\); CM-loc prediction of \(s_i\); thresholding at \(\tau_{\mathrm{EER}}\) on the development set to obtain bona/spoof decisions; CM-dia extraction of \(D\)-dimensional embeddings; computation of pairwise cosine affinity \(A_{ij}=\mathrm{cosine}(e_i,e_j)\); application of oracle-\(K\) hierarchical clustering; and label-based CM-constraint (LCM), which forces a frame labeled as bona fide by CM-loc to remain bona fide regardless of cluster assignment [2406.07816].

Training in the benchmark is based on cross-entropy objectives. CM-loc uses
\[
L_{\mathrm{loc}} = -\sum_m \sum_c 1[y_m=c]\log p_{\mathrm{loc}}(c|x_m),
\]
and CM-dia, when trained multi-class, uses
\[
L_{\mathrm{dia}} = -\sum_m \sum_c 1[y^*_m=c]\log p_{\mathrm{dia}}(c|x_m).
\]
No clustering loss is used because clustering is unsupervised under oracle \(K\) [2406.07816].

The benchmark study also explored three labeling schemes for training countermeasures. Bin-CM uses binary labels \(\{\text{bona fide},\text{spoof}\}\); Mul-CM uses multi-class labels \(\{\text{bona fide},A_1,\dots,A_N,\mathrm{ConP}\}\); and Spf-CM uses only spoof classes \(\{A_1,\dots,A_N,\mathrm{ConP}\}\), excluding bona fide [2406.07816]. Their stated trade-offs already exposed a central tension in spoof diarization: focused bona/spoof discrimination improves localization, whereas method-specific labels improve inter-method discrimination, but no single labeling scheme removes the trade-off.

## 3. Token-based attractors and cross-attention

The later model replaces the two-branch structure with a single merged-branch architecture augmented with learnable attractor tokens [2509.13085]. The front-end is Wav2Vec 2.0 XLS-R, which transforms raw audio into latent features
\[
F_{1:T}\in\mathbb{R}^{T\times d}.
\]
Two learnable tokens \(C_1,C_2\in\mathbb{R}^d\) are introduced, with \(N=2\): one represents bona-fide speech and one represents generic spoofed speech. These tokens are appended to frame embeddings,
\[
A_{1:M} = [F_1;\dots;F_T;\;C_1;\;C_2]\in\mathbb{R}^{M\times d},\quad M=T+2,
\]
and passed through a Transformer encoder, after which layer outputs are aggregated via a learned weighted sum and refined with a gMLP block to produce
\[
E_{1:M}=g(\hat A_{1:M}),\quad E_{1:M}\in\mathbb{R}^{M\times d}.
\]
The representation is then split into frame embeddings \(E_{1:T}\) and token embeddings \(E_C\) [2509.13085].

The model projects frames and tokens into separate spaces for localization and diarization:
\[
S_{1:T}=\mathrm{GELU}(E_{1:T}W_s+b_s)\in\mathbb{R}^{T\times d},
\]
\[
C_{1:2}=\mathrm{GELU}(E_CW_c+b_c)\in\mathbb{R}^{2\times d}.
\]
Cross-attention is then used in two distinct roles. For spoof-versus-bona-fide localization, an attention map \(P^{\mathrm{loc}}\in\mathbb{R}^{T\times 2}\) is computed between frames and tokens using \(\ell_2\)-normalized embeddings:
\[
P^{\mathrm{loc}}_{t,k}
=
\frac
{\exp\!\bigl((\tilde S_t W_{Q_s})(\tilde C_k W_{K_s})^\top/\sqrt{d}\bigr)}
{\sum_{k'=1}^2 \exp\!\bigl((\tilde S_t W_{Q_s})(\tilde C_{k'}W_{K_s})^\top/\sqrt{d}\bigr)}.
\]
This quantity is interpreted as frame-wise spoof/bona probabilities [2509.13085].

For spoof-type conditioning, the same token keys are reused to inject token-level context into frame representations:
\[
H_{1:T}
=
\mathrm{softmax}\!\Bigl(\frac{(\tilde E_{1:T}W_{Q_e})(\tilde C W_{K_e})^\top}{\sqrt{d}}\Bigr)(CW_{V_e})
\in\mathbb{R}^{T\times d}.
\]
The conditioned representation is concatenated with the original frame embedding,
\[
E'_t=[E_t;\,H_t]\in\mathbb{R}^{2d},
\]
and mapped to a \(d\)-dimensional space for multi-class classification. A set of \(L\) class prototypes \(O_{\mathrm{dia}}\in\mathbb{R}^{d\times L}\) is maintained, one for bona-fide and one per spoof method, and scoring is performed by cosine similarity:
\[
P^{\mathrm{dia}}_{t,\ell}
=
\tilde E'_t\cdot \tilde O_{\mathrm{dia},\ell},\quad \ell=1\ldots L.
\]
The paper characterizes the attractor tokens as explicit “reference points” that pull bona-fide and spoofed frame embeddings toward distinct regions in the feature space, while cross-attention lets each frame query these tokens to sharpen the decision boundary and enrich frame features with spoof-type context [2509.13085].

## 4. Optimization, data, and evaluation protocol

The token-based model uses the P2SGrad regression loss in cosine-similarity space:
\[
\mathcal{L}_{\mathrm{P2SGrad}}(P,Y)
=
\frac{1}{N\,K}\sum_{i=1}^N\sum_{k=1}^K(P_{i,k}-Y_{i,k})^2.
\]
Three losses are combined. The localization loss is
\[
\mathcal{L}_{\mathrm{loc}}
=
\mathcal{L}_{\mathrm{P2SGrad}}(P^{\mathrm{loc}},Y^{\mathrm{loc}}),
\]
the diarization loss is
\[
\mathcal{L}_{\mathrm{dia}}
=
\mathcal{L}_{\mathrm{P2SGrad}}(P^{\mathrm{dia}},Y^{\mathrm{dia}}),
\]
and the token supervision loss uses utterance-level prediction \(P^{\mathrm{token}}\in\mathbb{R}^{L}\), a separate prototype set \(O_{\mathrm{token}}\in\mathbb{R}^{d\times L}\), and multi-label ground truth \(Y^{\mathrm{token}}\in\{0,1\}^L\):
\[
\mathcal{L}_{\mathrm{token}}
=
\mathcal{L}_{\mathrm{P2SGrad}}(P^{\mathrm{token}},Y^{\mathrm{token}}).
\]
The total objective is
\[
\mathcal{L}_{\mathrm{total}}
=
\mathcal{L}_{\mathrm{loc}}
+
\mathcal{L}_{\mathrm{dia}}
+
\mathcal{L}_{\mathrm{token}}.
\]
This design makes token supervision an explicit training signal rather than an incidental by-product of the architecture [2509.13085].

The experiments use the PartialSpoof dataset. In the later study, PartialSpoof contains approximately 10 hours of English speech from 50 speakers, with 10 different TTS/VC systems denoted A01–A10 and train/dev/test splits containing 55.3%, 56.0%, and 60.7% bona-fide proportions, respectively [2509.13085]. The benchmark study reports PartialSpoof v1.2 with train/dev/eval splits containing approximately 55–61% bona fide and 17 TTS/VC methods balanced; oracle VAD removes non-speech except ConP, and experiments are conducted in a single-speaker scenario with oracle \(K\) known [2406.07816]. This indicates that dataset versioning and protocol differ across studies, but both works evaluate spoof diarization in controlled partially spoofed settings.

Evaluation is centered on Jaccard-based measures. The benchmark defines, per file \(j\),
\[
\mathrm{JI}_{\mathrm{bona},j}
=
\frac{\mathrm{FA}_{\mathrm{bona},j}+\mathrm{MD}_{\mathrm{bona},j}}{\mathrm{TOTAL}_{\mathrm{bona},j}},
\]
and
\[
\mathrm{JER}_{\mathrm{spoof},j}
=
\frac{1}{|\mathcal{A}_j|}\sum_{A_i\in\mathcal{A}_j}
\frac{\mathrm{FA}_{A_i,j}+\mathrm{MD}_{A_i,j}}{\mathrm{TOTAL}_{A_i,j}},
\]
followed by macro-averaging over the test set [2406.07816]. The later paper gives equivalent measures in true-positive form:
\[
\mathrm{JI}_{\mathrm{bona}}
=
1-\frac{\sum_j \mathrm{TP}_{\mathrm{bona},j}}
{\sum_j(\mathrm{TP}+\mathrm{FP}+\mathrm{FN})_{\mathrm{bona},j}},
\]
\[
\mathrm{JER}_{\mathrm{spoof}}
=
1-\frac{1}{\sum_j|A_j|}
\sum_j\sum_{a\in A_j}
\frac{\mathrm{TP}_{a,j}}
{\mathrm{TP}_{a,j}+\mathrm{FP}_{a,j}+\mathrm{FN}_{a,j}}.
\]
In both papers, lower is better for \(\mathrm{JI}_{\mathrm{bona}}\) and \(\mathrm{JER}_{\mathrm{spoof}}\) [2509.13085][2406.07816].

## 5. Reported performance and empirical behavior

The benchmark study established that the task is difficult even in restricted conditions. On the evaluation set, Bin, Mul, and Spf models yielded \(\mathrm{JI}_{\mathrm{bona}}\) / \(\mathrm{JER}_{\mathrm{spoof}}\) of \(16.85/33.13\), \(19.66/28.05\), and \(32.30/38.51\), respectively, while integrated two-branch variants such as Mul \(|\) Bin and Mul \(|\) Mul produced \(15.15/34.13\) and \(17.08/35.34\) [2406.07816]. The paper identifies Mul-CM in CM-dia combined with Bin-CM in CM-loc as the best trade-off, giving the lowest \(\mathrm{JI}_{\mathrm{bona}}\) on evaluation but higher \(\mathrm{JER}_{\mathrm{spoof}}\), and it reports substantial degradation on unseen spoof methods, with \(49.0\%\) JER on unseen methods versus \(11.98\%\) on known methods [2406.07816].

The token-based study reports the following evaluation-set comparison [2509.13085]:

| Model | JI_bona (%) | JER_spoof (%) |
|---|---:|---:|
| Baseline 3C (Zhang et al.) | 15.15 | 34.13 |
| 3C* re-impl. | 19.13 | 28.24 |
| 3C* + attractor tokens | **15.58** | **24.38** |
| Merged-branch w/o tokens | 22.64 | 32.59 |
| Merged-branch + tokens | **18.05** | **26.99** |

Relative to the 3C* baseline, adding tokens yields an 18.6% relative gain in \(\mathrm{JI}_{\mathrm{bona}}\) and 13.7% in \(\mathrm{JER}_{\mathrm{spoof}}\). In the simpler merged-branch model, the reported relative improvements are 20.3% and 17.2%, respectively [2509.13085].

Ablation results indicate that applying tokens only to localization or only to diarization still yields gains over the no-token baseline, while the best performance occurs when tokens guide both objectives, which the authors interpret as confirmation of complementary roles [2509.13085]. t-SNE visualizations further show that without tokens, bona-fide and spoof embeddings overlap heavily, whereas with tokens, clusters become more compact and well-separated [2509.13085]. This suggests that the attractor mechanism is not merely improving the final classifier but reshaping the intermediate embedding geometry in a way consistent with the stated “reference point” interpretation.

## 6. Limitations, open problems, and related challenges

The literature explicitly emphasizes that spoof diarization remains highly challenging. The benchmark paper identifies short spoofed segments down to phoneme level, open-set spoof types, unknown cluster count, and the distinction between single-speaker and multi-speaker scenarios as central difficulties [2406.07816]. The later token-based paper likewise reports that performance still degrades on unseen spoof types and that utterance-level detection measured by EER remains below specialized countermeasure systems [2509.13085].

Several future directions are stated directly. The token-based study proposes dynamically generating more than two tokens, for example one per spoof family via a Transformer decoder; incorporating open-set contrastive losses to improve generalization to unseen attacks; and jointly optimizing diarization with an explicit cluster-count estimator to remove the oracle-cluster assumption [2509.13085]. The benchmark study calls for multi-speaker spoof-speaker diarization (“who and what when”), automatic estimation of \(K\), open-set clustering and discovery of novel spoof methods, end-to-end models that unify localization, embedding extraction, and clustering, and exploration of overlapping spoof events [2406.07816].

A related but distinct line of evidence comes from diarization under natural voice alteration. A study of tabletop role-playing game recordings reports that participants routinely alter pitch range, timbre, speaking rate, and sometimes adopt accents to indicate character speech, and that two diarizers exhibit higher confusion on this material than on AMI or ICSI [2502.12714]. The paper reports, for example, DER values on TTRPG data of \(0.33\) for pyannote.audio and \(0.48\) for wespeaker, with confusion rates of \(0.12\) and \(0.29\), respectively [2502.12714]. This does not define spoof diarization, but it does indicate that diarization systems can be destabilized by acoustically induced persona shifts even without synthetic spoof generation. A plausible implication is that robust spoof diarization may require representations that separate source identity, spoof method, and style variation more explicitly than current acoustic-only pipelines do.

Across the current literature, spoof diarization is therefore characterized by a dual requirement: accurate temporal segmentation of bona fide versus spoofed speech, and reliable clustering of spoofed regions by manipulation method. The progression from 3C to token-based attractors and cross-attention shows a clear movement away from loosely coupled localization-and-clustering pipelines toward architectures that encode explicit spoof-aware reference structure directly in the frame representation space [2406.07816][2509.13085].

Source: https://www.emergentmind.com/topics/spoof-diarization