- The paper demonstrates an automated framework that systematically discovers adversarial perturbation chains to bypass audio deepfake detectors while preserving speech quality and speaker identity.
- It employs a dual search strategy by combining exhaustive breadth-first search with a Q-learning agent to efficiently navigate the combinatorial space of audio augmentations.
- The framework exposes critical vulnerabilities in detectors, highlighting non-additive effects in sequential perturbations and an increased risk of false positives in real-world scenarios.
Automated Adversarial Robustness Testing for Audio Deepfake Detectors: The Proteus Framework
Introduction
The proliferation of audio deepfakes and their potential for malicious exploitation in authentication and evidentiary contexts necessitates resilient detection systems. However, current deepfake detectors are primarily evaluated using clean, in-distribution audio, which inadequately captures the degradations and transformations encountered in real-world deployment. Prior evaluations have established that codec artifacts, signal-processing perturbations, and simple manipulations can significantly impair detector performance, yet most studies are restricted to fixed corruption sets rather than adversarially optimized transformation sequences.
Proteus, as presented in "Proteus: Automated Adversarial Robustness Testing for Audio Deepfake Detectors" (2606.29544), addresses this gap by introducing an automated framework for systematically discovering perturbation chains that can bypass audio deepfake detectors while preserving speech quality and speaker identity. The approach transitions from passive robustness evaluation to active black-box adversarial search, with implications for both model assessment and retraining pipelines.
Framework Architecture and Methodology
Proteus is structured as a black-box testing platform, leveraging a comprehensive audio augmentation library, a perceptually motivated quality gate, and two complementary search strategies:
- Augmentation Library: The system combines 35 augmentation types across 11 categories with hyperparameterized variants, resulting in approximately 110 distinct transformation options (e.g., multiple codecs, noise types, reverberation models, filtering modes, dynamic processing, environmental overlay, and VoIP effects). This breadth permits exploration of realistic, production-relevant perturbation scenarios.
- Quality Gate: To constrain the adversarial search space to plausible attacks, Proteus applies perceptual quality filters: word error rate (WER) thresholds using Whisper ASR enforce speech intelligibility, while cosine similarity of speaker embeddings ensures preservation of speaker identity. Chains failing these criteria are pruned prior to detector evaluation.
- Search Strategies:
- Breadth-First Search (BFS): At chain depth d, all Vd possible candidates are exhaustively evaluated per sample, with top-k chains progressing between levels and early termination based on lack of score improvement. While computationally expensive, BFS yields a comprehensive mapping of augmentation efficacy and pairwise interaction patterns.
- Q-Learning Agent: Recognizing the combinatorial explosion at greater depths, Proteus formulates the sequential augmentation process as a Markov Decision Process (MDP). The RL agent employs discrete state-action encoding (augmentation types as states, augmentation transitions as actions) with UCB-based action selection and online reward shaping proportional to detector score shifts. The Q-table is warm-started from BFS statistics, allowing efficient exploitation of previously observed transitions while supporting continued exploration. Hyperparameter variants are selected by categorical bandits, adaptively updating sampling probabilities based on success rates.
Empirical Findings
Deploying Proteus against an industrial deepfake detector revealed several critical robustness insights. Evaluation involved depth-2 and depth-3 chains across eight baseline samples, of which approximately 28% passed the stringent quality gate. The following key observations emerged:
- Attack Efficacy and Asymmetry: All of the top 100 augmentation chains (by absolute detector score shift) targeted bonafide samples, consistently shifting their scores into the model's "spoof" domain while maintaining high intelligibility and speaker similarity. The most effective two-step chain, synthetic reverb followed by Opus codec, shifted bonafide scores by +0.99, flipping verdicts with minimal perceptual compromise.
- Interaction Effects: Marginal contribution analysis across chain steps highlighted substantial non-additive interactions: specific transitions, such as gain control prior to lossy coding, frequently amplified attack efficacy beyond single-step expectations, underscoring the importance of sequential dependency modeling.
- False Positive Vulnerability: The primary exploitable surface lies in inducing false positives—transforming genuine speech into detector outputs consistent with spoofed audio. This exposes the detector to the "liar's dividend": simple perturbations allow adversaries to discredit authentic evidence by causing model misclassification. Synthetic-to-bonafide escapes (reducing detector suspicion of spoofed speech) were less achievable, hinting at asymmetric decision boundaries and residual synthetic artifacts that persist through augmentation.
Implications and Future Directions
Proteus operationalizes adversarial robustness testing in the audio domain, enabling continuous threat-informed model hardening. High-shift augmentation chains identified by the system are incorporated into detector retraining regimes, augmenting bonafide and spoofed samples with adversarial transformations. This cycle—attack discovery, retraining, and re-testing—establishes an automated adversarial development loop directly targeting emergent failure modes. Over time, this process can drive detectors towards both improved invariance to real-world degradation and enhanced capacity to differentiate between synthetic artifacts and natural channel effects.
Theoretically, the use of RL-based search underlines the necessity of transition-aware attack composition, suggesting that chain-based robustness cannot be adequately characterized by independent corruption studies; sequential dependencies are key. Deeper chains and more sophisticated search algorithms (e.g., hierarchical RL, meta-optimization) may reveal further vulnerabilities or inform curriculum-based training methods.
From a practical standpoint, the results expose risks in evidentiary and authentication contexts exposed to benign channel transformations, emphasizing the criticality of both robust detector design and comprehensive evaluation under adversarially selected conditions. Integrating frameworks such as Proteus into deployment pipelines will likely become essential for vendors, regulatory compliance, and forensic auditing.
Conclusion
Proteus exemplifies a rigorous and automated approach to adversarial robustness evaluation for audio deepfake detectors. By systematically uncovering realistic and perceptually-constrained augmentation chains that severely degrade detector performance, Proteus not only quantifies operational vulnerabilities but also prescribes targeted retraining strategies. The framework's combination of exhaustive and RL-guided search establishes a robust foundation for ongoing model hardening and advances the methodological state of adversarial testing in synthetic media detection.