Spoof Diarization Overview

• Spoof diarization is a task that segments audio into bona fide and spoofed regions and categorizes spoofed segments by their generation methods.
• It generalizes binary detection to a multi-class, frame-level approach, integrating temporal segmentation with embedded spoof method clustering.
• Recent models, including the 3C and token-based approaches, leverage attractor tokens and cross-attention to sharpen decision boundaries and improve performance.

Searching arXiv for the specified spoof diarization papers and closely related diarization work to ground the article in current literature. Spoof diarization is the problem of determining “what spoofed when” in a partially spoofed audio utterance: a system must temporally locate bona fide and spoofed regions and assign spoofed regions to their underlying generation methods rather than merely detecting the presence of any spoofing (Zhang et al., 2024, Koo et al., 16 Sep 2025). In the current literature, the task is positioned as a generalization of utterance-level spoof detection and frame-level spoof localization, while remaining distinct from speaker diarization because it clusters regions by spoof method rather than speaker identity, often at short time scales such as 20 ms frames (Zhang et al., 2024).

1. Task definition and problem scope

The initial task formulation treats an input waveform $x_{1:T}$ as a sequence to be broken into contiguous frames or segments, each assigned a multi-class label $$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$, where $\mathrm{ConP}$ is an optional label for non-speech or joined segments (Zhang et al., 2024). A later formulation expresses the same problem at frame level: given an audio utterance $x$ or pre-trained frame embeddings $\{F_t \in \mathbb{R}^d\}_{t=1}^T$, the system outputs a label sequence $\{y_t\}_{t=1}^T$ with $y_t=0$ for bona-fide speech and $y_t \in \{1,\dots,L-1\}$ for one of $L-1$ spoof types (Koo et al., 16 Sep 2025).

This framing separates spoof diarization from adjacent tasks. Utterance-level spoof detection produces a binary label per utterance, while frame-level spoof localization produces a binary label per frame; spoof diarization instead requires multi-class, per-frame assignment of spoofed frames to spoof methods (Koo et al., 16 Sep 2025). The literature also distinguishes it from speaker diarization: both problems cluster temporally homogeneous regions, but speaker diarization clusters by speaker identity, whereas spoof diarization clusters by spoof method, with short 20 ms segments identified as crucial for the latter (Zhang et al., 2024).

A common misconception is to treat spoof diarization as binary spoof localization followed by ordinary clustering. The benchmark formulation indicates that the task must jointly address temporal segmentation and spoof-type clustering, and the later token-based model was proposed precisely because simple structures lacked explicit reference points for separating bona fide speech from multiple spoofing types (Koo et al., 16 Sep 2025).

2. Benchmark formulation and the 3C model

The pioneering benchmark model is the Countermeasure-Condition Clustering (3C) model, which introduced a two-branch architecture for spoof diarization (Zhang et al., 2024). Its localization branch, CM-loc, is a framewise spoof countermeasure producing per-frame scores $s_{1:M}$. Its diarization branch, CM-dia, is a framewise embedding extractor yielding $$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$0, followed by agglomerative clustering with oracle $$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$1. The inference flow is explicitly described as: uniform 20 ms framing of $$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$2; CM-loc prediction of $$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$3; thresholding at $$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$4 on the development set to obtain bona/spoof decisions; CM-dia extraction of $$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$5-dimensional embeddings; computation of pairwise cosine affinity $$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$6; application of oracle-$$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$7 hierarchical clustering; and label-based CM-constraint (LCM), which forces a frame labeled as bona fide by CM-loc to remain bona fide regardless of cluster assignment (Zhang et al., 2024).

Training in the benchmark is based on cross-entropy objectives. CM-loc uses

$$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$8

and CM-dia, when trained multi-class, uses

$$c_m \in \{\text{bona fide}, A_1,\dots,A_N,[\mathrm{ConP}]\}$$9

No clustering loss is used because clustering is unsupervised under oracle $\mathrm{ConP}$0 (Zhang et al., 2024).

The benchmark study also explored three labeling schemes for training countermeasures. Bin-CM uses binary labels $\mathrm{ConP}$1; Mul-CM uses multi-class labels $\mathrm{ConP}$2; and Spf-CM uses only spoof classes $\mathrm{ConP}$3, excluding bona fide (Zhang et al., 2024). Their stated trade-offs already exposed a central tension in spoof diarization: focused bona/spoof discrimination improves localization, whereas method-specific labels improve inter-method discrimination, but no single labeling scheme removes the trade-off.

3. Token-based attractors and cross-attention

The later model replaces the two-branch structure with a single merged-branch architecture augmented with learnable attractor tokens (Koo et al., 16 Sep 2025). The front-end is Wav2Vec 2.0 XLS-R, which transforms raw audio into latent features

$\mathrm{ConP}$4

Two learnable tokens $\mathrm{ConP}$5 are introduced, with $\mathrm{ConP}$6: one represents bona-fide speech and one represents generic spoofed speech. These tokens are appended to frame embeddings,

$\mathrm{ConP}$7

and passed through a Transformer encoder, after which layer outputs are aggregated via a learned weighted sum and refined with a gMLP block to produce

$\mathrm{ConP}$8

The representation is then split into frame embeddings $\mathrm{ConP}$9 and token embeddings $x$0 (Koo et al., 16 Sep 2025).

The model projects frames and tokens into separate spaces for localization and diarization: $x$1

$x$2

Cross-attention is then used in two distinct roles. For spoof-versus-bona-fide localization, an attention map $x$3 is computed between frames and tokens using $x$4-normalized embeddings: $x$5 This quantity is interpreted as frame-wise spoof/bona probabilities (Koo et al., 16 Sep 2025).

For spoof-type conditioning, the same token keys are reused to inject token-level context into frame representations: $x$6 The conditioned representation is concatenated with the original frame embedding,

$x$7

and mapped to a $x$8-dimensional space for multi-class classification. A set of $x$9 class prototypes $\{F_t \in \mathbb{R}^d\}_{t=1}^T$0 is maintained, one for bona-fide and one per spoof method, and scoring is performed by cosine similarity: $\{F_t \in \mathbb{R}^d\}_{t=1}^T$1 The paper characterizes the attractor tokens as explicit “reference points” that pull bona-fide and spoofed frame embeddings toward distinct regions in the feature space, while cross-attention lets each frame query these tokens to sharpen the decision boundary and enrich frame features with spoof-type context (Koo et al., 16 Sep 2025).

4. Optimization, data, and evaluation protocol

The token-based model uses the P2SGrad regression loss in cosine-similarity space: $\{F_t \in \mathbb{R}^d\}_{t=1}^T$2 Three losses are combined. The localization loss is

$\{F_t \in \mathbb{R}^d\}_{t=1}^T$3

the diarization loss is

$\{F_t \in \mathbb{R}^d\}_{t=1}^T$4

and the token supervision loss uses utterance-level prediction $\{F_t \in \mathbb{R}^d\}_{t=1}^T$5, a separate prototype set $\{F_t \in \mathbb{R}^d\}_{t=1}^T$6, and multi-label ground truth $\{F_t \in \mathbb{R}^d\}_{t=1}^T$7: $\{F_t \in \mathbb{R}^d\}_{t=1}^T$8 The total objective is

$\{F_t \in \mathbb{R}^d\}_{t=1}^T$9

This design makes token supervision an explicit training signal rather than an incidental by-product of the architecture (Koo et al., 16 Sep 2025).

The experiments use the PartialSpoof dataset. In the later study, PartialSpoof contains approximately 10 hours of English speech from 50 speakers, with 10 different TTS/VC systems denoted A01–A10 and train/dev/test splits containing 55.3%, 56.0%, and 60.7% bona-fide proportions, respectively (Koo et al., 16 Sep 2025). The benchmark study reports PartialSpoof v1.2 with train/dev/eval splits containing approximately 55–61% bona fide and 17 TTS/VC methods balanced; oracle VAD removes non-speech except ConP, and experiments are conducted in a single-speaker scenario with oracle $\{y_t\}_{t=1}^T$0 known (Zhang et al., 2024). This indicates that dataset versioning and protocol differ across studies, but both works evaluate spoof diarization in controlled partially spoofed settings.

Evaluation is centered on Jaccard-based measures. The benchmark defines, per file $\{y_t\}_{t=1}^T$1,

$\{y_t\}_{t=1}^T$2

and

$\{y_t\}_{t=1}^T$3

followed by macro-averaging over the test set (Zhang et al., 2024). The later paper gives equivalent measures in true-positive form: $\{y_t\}_{t=1}^T$4

$\{y_t\}_{t=1}^T$5

In both papers, lower is better for $\{y_t\}_{t=1}^T$6 and $\{y_t\}_{t=1}^T$7 (Koo et al., 16 Sep 2025, Zhang et al., 2024).

5. Reported performance and empirical behavior

The benchmark study established that the task is difficult even in restricted conditions. On the evaluation set, Bin, Mul, and Spf models yielded $\{y_t\}_{t=1}^T$8 / $\{y_t\}_{t=1}^T$9 of $y_t=0$0, $y_t=0$1, and $y_t=0$2, respectively, while integrated two-branch variants such as Mul $y_t=0$3 Bin and Mul $y_t=0$4 Mul produced $y_t=0$5 and $y_t=0$6 (Zhang et al., 2024). The paper identifies Mul-CM in CM-dia combined with Bin-CM in CM-loc as the best trade-off, giving the lowest $y_t=0$7 on evaluation but higher $y_t=0$8, and it reports substantial degradation on unseen spoof methods, with $y_t=0$9 JER on unseen methods versus $y_t \in \{1,\dots,L-1\}$0 on known methods (Zhang et al., 2024).

The token-based study reports the following evaluation-set comparison (Koo et al., 16 Sep 2025):

Model	JI_bona (%)	JER_spoof (%)
Baseline 3C (Zhang et al.)	15.15	34.13
3C* re-impl.	19.13	28.24
3C* + attractor tokens	15.58	24.38
Merged-branch w/o tokens	22.64	32.59
Merged-branch + tokens	18.05	26.99

Relative to the 3C* baseline, adding tokens yields an 18.6% relative gain in $y_t \in \{1,\dots,L-1\}$1 and 13.7% in $y_t \in \{1,\dots,L-1\}$2. In the simpler merged-branch model, the reported relative improvements are 20.3% and 17.2%, respectively (Koo et al., 16 Sep 2025).

Ablation results indicate that applying tokens only to localization or only to diarization still yields gains over the no-token baseline, while the best performance occurs when tokens guide both objectives, which the authors interpret as confirmation of complementary roles (Koo et al., 16 Sep 2025). t-SNE visualizations further show that without tokens, bona-fide and spoof embeddings overlap heavily, whereas with tokens, clusters become more compact and well-separated (Koo et al., 16 Sep 2025). This suggests that the attractor mechanism is not merely improving the final classifier but reshaping the intermediate embedding geometry in a way consistent with the stated “reference point” interpretation.

6. Limitations, open problems, and related challenges

The literature explicitly emphasizes that spoof diarization remains highly challenging. The benchmark paper identifies short spoofed segments down to phoneme level, open-set spoof types, unknown cluster count, and the distinction between single-speaker and multi-speaker scenarios as central difficulties (Zhang et al., 2024). The later token-based paper likewise reports that performance still degrades on unseen spoof types and that utterance-level detection measured by EER remains below specialized countermeasure systems (Koo et al., 16 Sep 2025).

Several future directions are stated directly. The token-based study proposes dynamically generating more than two tokens, for example one per spoof family via a Transformer decoder; incorporating open-set contrastive losses to improve generalization to unseen attacks; and jointly optimizing diarization with an explicit cluster-count estimator to remove the oracle-cluster assumption (Koo et al., 16 Sep 2025). The benchmark study calls for multi-speaker spoof-speaker diarization (“who and what when”), automatic estimation of $y_t \in \{1,\dots,L-1\}$3, open-set clustering and discovery of novel spoof methods, end-to-end models that unify localization, embedding extraction, and clustering, and exploration of overlapping spoof events (Zhang et al., 2024).

A related but distinct line of evidence comes from diarization under natural voice alteration. A study of tabletop role-playing game recordings reports that participants routinely alter pitch range, timbre, speaking rate, and sometimes adopt accents to indicate character speech, and that two diarizers exhibit higher confusion on this material than on AMI or ICSI (Remme et al., 18 Feb 2025). The paper reports, for example, DER values on TTRPG data of $y_t \in \{1,\dots,L-1\}$4 for pyannote.audio and $y_t \in \{1,\dots,L-1\}$5 for wespeaker, with confusion rates of $y_t \in \{1,\dots,L-1\}$6 and $y_t \in \{1,\dots,L-1\}$7, respectively (Remme et al., 18 Feb 2025). This does not define spoof diarization, but it does indicate that diarization systems can be destabilized by acoustically induced persona shifts even without synthetic spoof generation. A plausible implication is that robust spoof diarization may require representations that separate source identity, spoof method, and style variation more explicitly than current acoustic-only pipelines do.

Across the current literature, spoof diarization is therefore characterized by a dual requirement: accurate temporal segmentation of bona fide versus spoofed speech, and reliable clustering of spoofed regions by manipulation method. The progression from 3C to token-based attractors and cross-attention shows a clear movement away from loosely coupled localization-and-clustering pipelines toward architectures that encode explicit spoof-aware reference structure directly in the frame representation space (Zhang et al., 2024, Koo et al., 16 Sep 2025).