The Hidden Cost of Pairwise Verification in Synthetic Speech Source Tracing

Abstract: Open-set source tracing is increasingly framed as a verification problem, motivating the use of pairwise metric-learning objectives from biometrics. We thus compare global anchoring and pairwise verification under matched backbones and a fixed data and epoch budget on MLAAD (in-domain) and STOPA (out-of-domain). In our runs, global anchoring yields lower in-domain error (8.61% EER) than pairwise variants (12-15% EER), even with rival mining and XLS-R finetuning. Because pairwise objectives optimize similarity directly, they concentrate variance into fewer embedding directions, reducing resolution among closely related generators. To test if this drives the drop, we impose a similar bottleneck to the globally supervised baseline, yet the baseline remains competitive. Together with an embedding-space analysis ($k_{99}$), these results suggest that the gap is not explained by dimensionality alone, but rather by the pairwise objective's shaping of the retained directions.

• The paper demonstrates that global anchoring outperforms pairwise metric learning, achieving EERs as low as 8.61% compared to 12–15% for pairwise methods.
• The paper employs the XLS-R backbone and varied metric strategies to reveal that pairwise training causes severe embedding dimensionality collapse, reducing k99 from 121 to 13.
• The paper highlights that pairwise methods induce broader impostor-target score overlaps, compromising forensic reliability at stringent low-FPR operating points.

A Critical Assessment of Pairwise Metric Objectives for Synthetic Speech Source Tracing

Introduction

The task of attributing synthetic speech to its underlying generative source—commonly known as source tracing—has evolved as a critical audio forensics challenge, particularly in the context of increasing sophistication and prevalence of speech deepfakes. While initial efforts focused on closed-set classification, recent approaches emphasize open-set source verification, borrowing methodologies from speaker and face recognition based on metric learning. The paper "The Hidden Cost of Pairwise Verification in Synthetic Speech Source Tracing" (2606.11666) provides a rigorous comparative analysis of global anchoring (i.e., cross-entropy classification) versus several variants of pairwise metric-learning objectives for this open-set attribution setting. The experiments highlight notable limitations of pairwise training and analyze the geometric and operational implications on the resulting embedding spaces.

Comparative Framework and Methodology

The experimental protocol compares two principal objective classes:

• Global Anchoring: Implemented via closed-set cross-entropy classification on training classes (generators), this method encourages embeddings to be structured around class centers, typically resulting in compact, separable clusters for each generator. At inference, verification is conducted by comparing embeddings via cosine similarity.
• Pairwise Metric Learning: Pairwise (Siamese) objectives directly optimize the distance between sample pairs, focusing on bringing positive pairs closer while pushing negative pairs apart. Variants involve intermediate random sampling, hard-negative mining, coverage-driven selection, and rival mining that exploits metadata to shape pairwise trial selection.

To ensure a controlled comparison, both approaches utilize the XLS-R backbone (frozen or fine-tuned), with consistent pooling architectures (mainly MHFA), using the MLAAD dataset for in-domain and STOPA for out-of-domain evaluation. Representative error metrics include EER, nDCF at low prior, and TPR at strict FPRs.

Main Results: Global Anchoring vs. Pairwise Training

Global anchoring consistently produces lower in-domain EER and superior TPR at stringent operating points compared to all tested pairwise variants Figure 1, even when employing advanced mining regimes and backbone fine-tuning. For instance, the globally supervised system attains an EER of 8.61% (MLAAD, $R=1$) versus 12–15% for pairwise methods. Notably, pairwise systems exhibit only marginal gains with rival mining and backbone adaptation, remaining distinctly inferior at critical low-FPR thresholds.

Figure 1: DET curves for the best Global and Pairwise checkpoints on MLAAD (left) and STOPA (right); differences at strict operating points are dominated by low-FPR tail overlap.

Domain generalization to STOPA leads to steep performance deterioration across both paradigms; however, the relative ranking persists, with global anchoring maintaining its edge, albeit with low overall TPR in the rare-target regime.

Embedding Space Analysis

A central theoretical contribution is the analysis of the implications of the optimization objective on the embedding topology. Principal component variance analysis (quantified by $k_{99}$, the number of components explaining 99% variance) reveals a stark contrast:

• Global anchoring: Embedding variance is retained across many directions ($k_{99} \approx 121$), supporting resilience to subtle inter-generator differences.
• Pairwise objectives: Training results in steep dimensionality collapse ($k_{99} \approx 13$), concentrating variance into a few dominant directions.

However, controlled bottleneck experiments—projecting embeddings into 10–13 dimensions under global supervision—demonstrate that low dimensionality alone does not account for the large performance gap. Bottlenecked, globally trained embeddings retain competitiveness, indicating that the critical difference lies in which embedding directions are shaped and preserved by the objective, not merely in their number.

Figure 2: Cumulative variance analysis demonstrating embedding dimensionality collapse under pairwise objectives.

Score Distributions, Error Modes, and Diagnostic Findings

Score distribution analysis underscores that pairwise training, despite increasing between-class separation, produces broader tails and greater overlap between impostor and target scores at the operational thresholds mandated by forensics. This broadening leads to higher false acceptances, particularly among closely related generator pairs.

Figure 3: Score CDFs on MLAAD for Global (top) and Pairwise (bottom), visualizing target, non-target CDF overlap and the increased score spread.

False acceptance breakdown further reveals systematic weaknesses. Pairwise-trained models suffer a pronounced drop in resolution between "architectural cousins"—generators differing in configuration but sharing core architecture and data—while both paradigms fail to distinguish "digital twins" sharing both architecture and training set, limited instead by the backbone's invariance.

Binary probe experiments reinforce that generator attribution is primarily constrained by corpus and speaker overlap, rather than model architecture, reflecting backbone limitations rather than just the effect of the training objective.

Practical and Theoretical Implications

The empirical evidence refutes the assumption that approaches successful in biometrics directly translate to synthetic speech attribution. While pairwise objectives induce strong separation for gross inter-class differences, they undermine the retention of the fine-grained cues critical for distinguishing subtle generator variants. In source tracing, especially under adversarial and post-incident analysis, the increased tail overlap induced by pairwise training leads to diminished utility at the stringent thresholds required for effective forensics.

The findings advocate for:

• Preferential use of global anchoring as a default baseline for forensic source tracing tasks.
• Restriction of pairwise objectives to scenarios where performance benefits at low FPRs are empirically validated.
• The need for future work exploring alternative metric-learning losses (e.g., supervised contrastive, proxy-based) that may better preserve fine-grained manifold structure without compromising operational separation.
• Further research on backbone architectures capable of disentangling subtle model-specific artifacts independent of speaker or channel information.

Conclusion

The analysis demonstrates that, in the open-set synthetic speech attribution context, global class-based supervision outperforms pairwise metric learning under matched system configurations. The observed deficiencies in pairwise training are primarily attributed to their effect on embedding topology—steep dimensionality collapse and adverse tail behavior—rather than mere reduction in representation dimension. These insights carry both practical and theoretical weight, challenging the routine application of pairwise objectives for open-set audio source tracing in forensic pipelines, and motivating development of embedding learning strategies explicitly aligned with operational forensic requirements.

---

(2606.11666) The Hidden Cost of Pairwise Verification in Synthetic Speech Source Tracing Further context and related results: [st-resnet-aasist-configurations-loss-sampling-embedding-size], [muller2024mlaad], [firc25_interspeech], [stan25_interspeech]