Audio Deepfake Detection Overview

• Audio Deepfake Detection (ADD) is the process of identifying synthetic audio manipulated by deep learning, emphasizing TTS, VC, and neural codecs.
• ADD methodologies rely on supervised classification, manipulation region localization, and open-set algorithm attribution to address forensic and generalization challenges.
• Recent research advances include self-supervised feature learning, fusion architectures, and adversarial training to improve robustness and interpretability.

Audio Deepfake Detection (ADD) is the field concerned with automatically identifying audio content generated or manipulated by deep-learning models, including text-to-speech synthesis (TTS), voice conversion (VC), and neural codecs. ADD has grown in prominence due to the proliferation of high-fidelity synthetic audio and its misuse in impersonation, fraud, and disinformation. Modern ADD research focuses on supervised classification of real vs. fake audio, robust generalization to unseen attacks, forensic localization of manipulated intervals, algorithm attribution, and interpretability of decision criteria.

1. Taxonomy of Tasks and Protocols

ADD encompasses several sub-problems, reflecting forensic, security, and generalization needs:

• Binary Real/Fake Classification: Canonical ADD systems process an audio utterance $x$ to predict $y \in \{0, 1\}$, where $y=0$ is bona fide and $y=1$ is synthetic. Equal Error Rate (EER)—the operating point where false acceptance equals false rejection—is the dominant metric (Kawa et al., 2022, Li et al., 2024, Shin et al., 2023, Gu et al., 16 May 2025).
• Manipulation Region Localization (RL): Frame-wise or segment-wise labeling of partially fake audio, identifying temporal intervals $[t_s, t_e]$ as manipulated (Yi et al., 2024), [ADD 2023]. Metrics include frame-F$_1$ and sentence accuracy.
• Algorithm Recognition (AR): Attribution of fake audio to its generative method, formulated as multi-class classification $$c \in \{\mathrm{TTS}_1, \mathrm{VC}_2, ..., \mathrm{unknown}\}$$, often requiring open-set recognition (Yi et al., 2024).
• All-Type and Cross-Domain Detection: Systems must generalize to diverse audio types (speech, music, singing, environmental sounds) and conditions (codec compression, packet loss, cross-language) (Xie et al., 6 Jan 2026, Wang et al., 4 Sep 2025, Li et al., 2024).
• Explainability and Rationalization: Integration of post-hoc or by-design interpretability methods to clarify why audio is flagged as fake (Grinberg et al., 23 Jan 2025, Xie et al., 6 Jan 2026, Zhu et al., 2024).

Recent challenge protocols (ADD 2023) and open benchmarks (AUDDT, AUDETER) have moved beyond simple classification to demand robust localization, open-set algorithm traceability, and multi-condition resilience (Yi et al., 2024, Zhu et al., 25 Sep 2025, Wang et al., 4 Sep 2025).

2. Datasets: Scale, Diversity, and Open-World Coverage

ADD datasets span controlled studio corpora, in-the-wild conversational speech, and increasingly, highly diverse synthetic domains:

• Legacy Benchmarks: ASVspoof2019/2021/2024, WaveFake, FakeAVCeleb—well-annotated, focused on traditional TTS/VC synthesis (Kawa et al., 2022, Yang et al., 2024).
• Codec/LLM-Based Fakes: Codecfake, CodecFake, AudioGen—capture ALM and neural codec artifacts, scaling to $>$1M samples in multiple languages and codecs (Xie et al., 2024).
• Open-World and Cross-Domain Corpora: AUDETER ($>3$M clips, 4.5K h), CD-ADD ($>$300 h, multi-TTS, cross-domain prompts), In-the-Wild (Wang et al., 4 Sep 2025, Li et al., 2024, Zhu et al., 25 Sep 2025).
• Challenge Datasets: ADD 2023, ADD-C—simulate channel degradation, packet loss, compression, and partial spoofing (Yi et al., 2024, Shi et al., 16 Apr 2025).

Comprehensive benchmarking now requires evaluation across diverse manipulation types (autoreg/cascaded TTS, vocoders, codecs, LLMs), recording conditions (studio, phone, broadcast), and audio genres (speech, music, non-speech) (Zhu et al., 25 Sep 2025, Wang et al., 4 Sep 2025).

3. Feature Extraction and Representation Learning

ADD feature pipelines have evolved from hand-crafted spectral features to fully-learnable and self-supervised representations:

• Cepstral Features: MFCC, LFCC, CQCC—capture spectral envelope and fine-grained frequency content, with linear-scale LFCC outperforming mel-based features in capturing high-frequency artifacts (Kawa et al., 2022, Yang et al., 2024).
• Spectrograms and Constant-Q: Log-mel, CQT, log-spec, magnitude/phase tensors—input to CNNs or transformer variants (Yang et al., 2024, Shin et al., 2023, Uddin et al., 8 Sep 2025).
• Self-Supervised Speech Models: Wav2Vec2, HuBERT, WavLM, Whisper—contextualized transformer embeddings trained on $>$100K h speech, now standard for both in-domain accuracy and OOD generalization (Yang et al., 2024, Martín-Doñas et al., 2022, Zhu et al., 2024, Xie et al., 6 Jan 2026).
• Multi-View and Fusion Strategies: Channel-attention and transformer fusion of several feature backends (e.g., XLS-R, HuBERT, WavLM) improves generalization on out-of-domain data (Yang et al., 2024, Shi et al., 2 Aug 2025).
• Style–Linguistics Dependency: SLIM models extract parallel style and linguistic features via one-class SSL, quantifying cross-subspace mismatch as an explicit anomaly signal (Zhu et al., 2024).
• Stereo and Spatial Augmentation: Mono-to-stereo conversion plus dual-branch GAT encoders enhances artifact contrast and detection accuracy on spatialized signals (Liu et al., 2023).

The dominance of speech-pretrained models reflects strong data-driven generalization; fusion and domain-aware representations remain critical for open-world robustness (Wang et al., 4 Sep 2025, Xie et al., 6 Jan 2026).

4. Model Architectures and Optimization

ADD systems utilize a range of neural architectures, often tailored for detection efficiency and granularity:

• CNNs and LCNNs: Lightweight convolutional networks (LCNN) with MFM activations achieve strong stability and competitive EER, especially when paired with LFCC features (Kawa et al., 2022).
• Transformers and Attention Networks: Conformer, AASIST, RawGAT-ST, and MGAA modules combine convolutional and attention pathways to capture local/global artifacts (Shin et al., 2023, Shi et al., 2 Aug 2025, Yang et al., 2024).
• Self-Supervised + Classifier Pipelines: Frozen pretrained transformer backbones coupled with shallow classifiers or attentive statistical pooling dominate current SOTA approaches (Martín-Doñas et al., 2022, Li et al., 2024).
• Token Aggregation and Hierarchical Pooling: Multi-level CLS tokens (HM-Conformer) and hierarchical pooling compress token redundancy, improve gradient flow, and reinforce detection at various temporal scales (Shin et al., 2023).
• One-Class and SSL Detection: SLIM-style pretraining on real-only samples induces robust anomaly detection by learning authentic style-content dependencies (Zhu et al., 2024).
• Audio LLMs (ALLMs): ALLM4ADD and FT-GRPO systems reframe ADD as audio question-answering or structured rationalization, combining audio encoder projections with LLM reasoning (Xie et al., 6 Jan 2026, Gu et al., 16 May 2025).
• Prompt Tuning: Lightweight, plug-in prompt vectors inserted into transformer layers enable low-shot, computationally efficient domain adaptation with minimal overfitting (Oiso et al., 2024).

Optimization protocols typically use binary cross-entropy loss (classification), OC-Softmax (one-class margin), or contrastive objectives (SSL), often regularized by dropout, spectral masking, or domain-balanced updates (CSAM) (Zhu et al., 2024, Xie et al., 2024).

5. Generalization, Robustness, and Attack Resilience

Domain shift and adversarial robustness represent principal challenges in ADD deployment:

• Open-Set and Cross-Domain Failures: Models trained narrowly on legacy TTS/VC data exhibit high error rates ($\mathrm{EER} > 20\%$) when faced with unseen synthesis engines, codecs, neural enhancement, or degraded audio (Wang et al., 4 Sep 2025, Li et al., 2024, Xie et al., 2024).
• Data Augmentation for Robustness: On-the-fly compression (codecs), packet loss, and noise augmentation at training time are essential to maintain accuracy under real-world transmission conditions (Shi et al., 16 Apr 2025, Shi et al., 2 Aug 2025).
• Adversarial Attack Vulnerability: State-of-the-art detectors are susceptible to both statistical and optimization-based anti-forensic attacks (e.g., PGD, C&W, DeepFool, pitch shifting, quantization), with accuracy drops up to $72$ percentage points (Uddin et al., 8 Sep 2025, Farooq et al., 21 Jan 2025). Adversarial training and hybrid architectures yield incremental gains but no current system is fully robust.
• Domain-Balanced Optimization: CSAM corrects domain ascent bias in multi-domain co-training, producing universal detectors with sub-$1\%$ average EER across conditions (Xie et al., 2024).
• Few-Shot and Prompt-Tuned Adaptation: Prompt tuning and few-shot head adaptation permit rapid specialization to new domains with $<100$ labeled examples per target, minimizing computational cost and overfitting (Oiso et al., 2024, Li et al., 2024).
• Localization and Attribution Limits: Precise manipulation region localization (frame-F$_1 \sim 60\%$) and robust attribution of unknown generative algorithms remain open problems, especially under compression and multi-edit scenarios (Yi et al., 2024).

Widely adopted systems now prioritize OOD and all-type generalization over closed-set accuracy; progress depends on dataset scale/diversity, robust optimization, and attack-aware defense schemes (Zhu et al., 25 Sep 2025).

6. Interpretability and Forensic Explainability

Model explainability is increasingly central for forensic trust and deployment:

• Structured Rationales: FT-GRPO enables ALLMs to produce chain-of-thought explanations tagged by frequency/time domain cues, facilitating transparent verdicts (Xie et al., 6 Jan 2026).
• Style-Linguistics Mismatch Visualization: SLIM computes interpretable frame-wise heatmaps of style-content divergence, offering explicit falsification evidence (Zhu et al., 2024).
• Time-Domain Relevance Attribution: GATR (gradient-average transformer relevancy) quantitatively ranks critical temporal regions; reveals dataset-dependent importance of non-speech and phonetic content (Grinberg et al., 23 Jan 2025).
• Mono-to-Stereo Artifact Amplification: M2S-ADD’s dual-branch approach exposes subtle deepfake artifacts in stereo that evade mono-only analysis (Liu et al., 2023).
• Attribution and Source Recognition: Open-set AR approaches combine classifier confidence with embedding space and thresholding (OpenMax, k-NN) for labeling unknown generation sources (Yi et al., 2024).

Explainable ADD models and post-hoc analyses support forensic validation, error analysis, and improved user trust in high-stakes scenarios (law enforcement, broadcast, content moderation) (Xie et al., 6 Jan 2026, Grinberg et al., 23 Jan 2025, Zhu et al., 2024).

7. Benchmarking, Limitations, and Future Directions

The field is marked by rapid evolution in both technology and evaluation protocols:

• Unified Evaluation Toolkits: AUDDT automates large-scale, subgroup-aware benchmarking across 28 datasets and manipulation types, diagnosing strengths and blind spots in pretrained models (Zhu et al., 25 Sep 2025).
• Dataset Gaps: Few resources cover emotional speech, singing, non-speech audio, or expressive neural enhancement; dataset creation lags generative method innovation (Zhu et al., 25 Sep 2025, Wang et al., 4 Sep 2025).
• Dynamic Adversarial Challenges: ADD 2023 and future competitions advocate for open-ended adversarial frameworks (“fake game”), continual learning, and real-time deployment simulation (Yi et al., 2024).
• Meta-Learning and Self-Supervision: Expanding multi-domain self-supervised pretraining, meta-learning adaptation, and feature regularization are vital for further robustness.
• Interpretability and Source Attribution: Fine-grained manipulation localization, algorithm/source traceability, and rationalized decision-making will shape next-generation forensic applications (Yi et al., 2024, Xie et al., 6 Jan 2026).
• Multimodal Extension: Cross-modal (audio-visual) deepfake detection and benchmarking are emerging directions as LLMs become core audio content generators.

A plausible implication is that future ADD systems will be ensemble, attack-aware, self-supervised, and interpretable by design, drawing on large open-world datasets with continuous benchmarking protocols (Wang et al., 4 Sep 2025, Yi et al., 2024, Xie et al., 6 Jan 2026, Zhu et al., 25 Sep 2025).