Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR

Abstract: Self-supervised learning (SSL) underpins modern audio deepfake detection, yet most prior work centers on a single large wav2vec2-XLSR backbone, leaving compact under studied. We present RAPTOR, Representation Aware Pairwise-gated Transformer for Out-of-domain Recognition a controlled study of compact SSL backbones from the HuBERT and WavLM within a unified pairwise-gated fusion detector, evaluated across 14 cross-domain benchmarks. We show that multilingual HuBERT pre-training is the primary driver of cross-domain robustness, enabling 100M models to match larger and commercial systems. Beyond EER, we introduce a test-time augmentation protocol with perturbation-based aleatoric uncertainty to expose calibration differences invisible to standard metrics: WavLM variants exhibit overconfident miscalibration under perturbation, whereas iterative mHuBERT remains stable. These findings indicate that SSL pre-training trajectory, not model scale, drives reliable audio deepfake detection.

• The paper shows that compact ~100M mHuBERT backbones, especially mHuBERT-Iter2, can outperform 300M wav2vec2-XLSR and commercial systems on pooled cross-domain EER when RAPTOR and training conditions are fixed.
• The controlled methodology isolates SSL pre-training by combining six compact encoders, two training protocols, and 14 cross-domain benchmarks, demonstrating that multilingual refinement matters more than parameter count for robustness.
• The paper finds that test-time augmentation uncertainty exposes overconfident WavLM failures, while joint analysis of entropy and EER degradation supports calibration-aware evaluation beyond standard detection accuracy.

Motivation and research questions

Most high-performing audio deepfake detectors are built on the 300M-parameter wav2vec2-XLSR encoder, with architectural innovation concentrated in the downstream classifier head (graph attention, temporal convolutions, state-space models). Large-scale evaluations have shown that strong in-domain performance does not reliably transfer out of domain, raising the question of whether robustness is determined by the classifier at all. This paper isolates the SSL backbone as the variable of interest and asks three questions: (RQ1) how does pre-training strategy—particularly iterative multilingual refinement—affect cross-domain detection; (RQ2) can ~100M backbones compete with systems 5–20× larger, including commercial detectors; and (RQ3) can test-time augmentation (TTA)-derived aleatoric uncertainty expose calibration failures invisible to EER.

Methodology

The study fixes the downstream detector to RAPTOR (Representation Aware Pairwise-gated Transformer for Out-of-domain Recognition), a hierarchical layer-fusion architecture, and varies only the pretrained encoder across six compact (~95–100M) backbones: HuBERT-Base plus three checkpoints from iterative multilingual training (mHuBERT-Iter1/Iter2/Final, 90K hours, 147 languages), and WavLM-Base/Base+ (960h vs. ~60K+ hours English-centric data).

RAPTOR fuses adjacent SSL layer pairs via time-dependent softmax gates, recursively fuses pair-level representations through a second gating stage, then applies attention pooling and binary classification. A consistency regularizer ($\lambda = 0.25$) penalizes Jensen–Shannon divergence between gate distributions on clean and RawBoost-augmented views, encouraging augmentation-invariant layer routing.

For uncertainty estimation, each test utterance is augmented into $K{=}3$ views (VoIP codec simulation, additive noise, speed–pitch perturbation), distinct from all training augmentations. The paper reports both the TTA ensemble effect on EER ($\Delta$EER) and a mean prediction entropy $U_{\mathrm{ale}}$ interpreted as an aleatoric-style proxy for representation sensitivity to perturbation. Two training protocols are used: ASVspoof 2019 only (Protocol 1) and the Speech DF Arena multi-dataset recipe spanning nine corpora with offline MUSAN/RIR augmentation (Protocol 2), evaluated over 14 cross-domain benchmarks.

Pre-training trajectory drives cross-domain robustness

Under Protocol 2, mHuBERT-Iter2 is the most consistent performer among 100M systems: 1.56% EER on ITW, 7.02% on ASVspoof 2021LA, 16.01% on ASVspoof 2024, and 3.14% on FoR. Because architecture, data, and optimization are held fixed, these gains are attributable to the pre-training stage alone. The trajectory is non-monotonic: mHuBERT-Final regresses sharply on CodecFake (25.68% vs. 13.34%/14.04% for Iter1/Iter2), which the authors interpret as a sensitivity–diversity trade-off in which continued multilingual refinement trades low-level acoustic artifact sensitivity for phonetic coverage. This regression is an open question rather than an explained phenomenon.

The WavLM family shows that data volume alone does not substitute for multilingual refinement: WavLM-Base+ improves over WavLM-Base but remains weaker than mHuBERT-Iter2 in aggregate despite comparable pre-training scale. Notably, even large proprietary systems fail badly under shift—ResembleAI-2B reaches 33.04% on CodecFake and MoLEX 31.93% on ADD22-T1—implying that no single benchmark adequately characterizes detector robustness.

Compact models versus large-scale and commercial systems

Under Protocol 2, mHuBERT-Iter2 achieves the best average EER among 100M systems (7.83%) and mHuBERT-Final the best pooled EER (11.11%). Both beat the 300M wav2vec2-XLSR baselines W2V2-AASIST (12.46% pooled) and W2V2-TCM (12.88%), as well as the commercial ResembleAI-2B (12.74%) and MoLEX (12.40%), using roughly one-third or fewer of the parameters. The purpose-built DF-Arena 500M system remains strongest overall (5.78% average, 10.88% pooled), so the claim is competitiveness, not superiority over the best large model; the compact RAPTOR variants do clearly surpass the earlier DF-Arena 100M-V1 baseline by 2.20 pooled EER points.

Under Protocol 1 (ASVspoof 2019 only), all systems degrade severely out of domain: near-zero in-domain EERs (0.18–0.59%) do not transfer, with wav2vec2 systems reaching 7.79–11.19% on ITW—comparable to the spread among 100M compact systems. This supports the conclusion that cross-domain behavior depends more on pre-training trajectory and training coverage than on backbone scale.

TTA-based uncertainty exposes calibration failure modes

The central diagnostic finding is a systematic confidence–accuracy misalignment specific to the WavLM family. WavLM-Base exhibits $\Delta$EER of +13.88% on ITW and +16.57% on ASV19 while $U_{\mathrm{ale}}$ stays low (0.274, 0.190); WavLM-Base+ shows +13.14% and +9.82% with $U_{\mathrm{ale}}$ of 0.214 and 0.141. This combination—large degradation under perturbation alongside narrow posteriors—is the signature of overconfident miscalibration: such a detector would not emit uncertainty signals sufficient to trigger abstention or human review precisely when its discrimination has collapsed. In contrast, mHuBERT variants show small $\Delta$EER (+0.38% to +0.89%) with moderate-to-high $U_{\mathrm{ale}}$, consistent with well-calibrated representations. The 300M wav2vec2 systems fall between the two families (+1.70/+1.83% $\Delta$EER, $K{=}3$0 0.227/0.299), offering a calibration-based perspective on why added capacity does not close the pooled-EER gap.

An important caveat: on FoR, TTA degrades EER by more than 42 percentage points for every system without exception, indicating an incompatibility between the chosen perturbation set and that dataset's acoustic characteristics. The authors accordingly stress that $K{=}3$1 and $K{=}3$2EER must be read jointly across multiple datasets to separate backbone-level calibration properties from dataset-specific augmentation effects.

Qualitative gate-map analysis further shows spoof utterances activating lower-to-middle SSL layer pairs (indices 2–4) more strongly than bona fide speech, consistent with prior layer-wise findings that synthesis artifacts concentrate early in the SSL hierarchy.

Limitations and open questions

The paper concedes several constraints. The TTA framework uses deterministic forward passes and captures only aleatoric-style uncertainty; epistemic uncertainty would require weight-posterior inference via Bayesian approximation or ensembles. Gate-map interpretation remains qualitative, lacking quantitative layer-pair entropy or gate-consistency statistics to substantiate artifact localization. The mechanism behind the mHuBERT-Final regression on codec-based synthesis—and more generally the sensitivity–diversity trade-off in iterative multilingual pre-training—is hypothesized but not established. Finally, the fixed VoIP/noise/perturbation set fails catastrophically on FoR, motivating domain-adaptive perturbation selection, and the proprietary systems included are reference-only since their architectures and training details are undisclosed.

Conclusion

By holding the downstream architecture fixed and varying only the SSL encoder, this controlled study establishes that pre-training trajectory, not parameter count, governs cross-domain audio deepfake detection: compact 100M mHuBERT variants match or exceed 300M wav2vec2-XLSR systems and proprietary commercial detectors on pooled cross-domain metrics, while exhibiting better perturbation calibration than WavLM counterparts. The introduced TTA protocol demonstrates that standard EER conceals deployment-relevant overconfident miscalibration, arguing for calibration-aware evaluation alongside aggregate discrimination metrics.