Audio Deepfake Detection Models

• Audio deepfake detection models are computational systems that classify audio as genuine or synthetic by analyzing spectral features, raw waveforms, and deep neural embeddings.
• They employ diverse architectures including classic signal processing pipelines, CNNs with RNNs, SSL transformers, and graph attention networks to achieve low equal error rates.
• Robustness is enhanced through multi-domain training, codec augmentation, replay attack simulation, and continual learning, ensuring reliable real-world performance.

Audio deepfake detection models are computational systems designed to classify audio signals as real or synthetically generated (“deepfakes”) by modern speech synthesis, voice conversion, or generative audio models. The goal is to detect and resist increasingly sophisticated generative attacks, generalize across synthesis methods, and in some cases provide attribution or content privacy. This field rapidly evolves with advances in both generative AI and self-supervised representation learning, and current methods span a broad spectrum: classical signal-processing approaches, deep end-to-end neural architectures, continual and cross-domain learning, multimodal fusion, privacy-preserving detectors, and model attribution pipelines.

1. Detection Model Architectures and Core Methodologies

Audio deepfake detection models utilize a wide array of feature representations and classification backbones:

Handcrafted and Classic Signal Processing Pipelines:

Early approaches operate on cepstral features such as Linear Frequency Cepstral Coefficients (LFCC), Mel-Frequency Cepstral Coefficients (MFCC), delta and double-delta features, and higher order statistics extracted from short-time spectral representations. Gaussian Mixture Models (GMMs), trained on these feature distributions, remain competitive in open-domain, out-of-distribution benchmarks due to their inherent robustness to narrow generator overfitting (Frank et al., 2021).

Deep Neural Architectures on Spectrograms:

Modern systems employ Convolutional Neural Networks (CNNs) or hybrid convolutional-recurrent networks on log-Mel, LFCC, and other spectrogram images. For instance, SpecRNet exploits a compact CNN with attention and bidirectional GRUs (277k parameters) on LFCCs, achieving EER < 0.2% on standard tasks and running at sub-10 ms inference times, which is suitable for embedded deployment (Kawa et al., 2022). Ensemble models fuse multiple spectrogram transforms (STFT, CQT, Wavelet) and classifier types for further performance gains (Pham et al., 2024).

End-to-End Raw Waveform Models:

Lightweight convolutional-recurrent (RawNet2, RawNetLite) and residual architectures applied directly to amplitude-normalized waveforms eliminate handcrafted feature steps. RawNetLite, for example, uses stacked Conv1D, residual connections, adaptive pooling, and bidirectional GRUs, achieving in-domain EER of 0.25% and demonstrating improved open-domain robustness through multi-domain training and augmentations (Pierno et al., 29 Apr 2025).

Self-Supervised Learning (SSL) Front-Ends:

Large transformer-based SSL encoders (wav2vec2, HuBERT, XLS-R, WavLM, Whisper, UniSpeech-SAT) pretrained on hundreds of thousands of hours of real speech, then fine-tuned for binary deepfake detection, now deliver the state of the art in both accuracy and cross-domain robustness (Ali et al., 2 Mar 2026, Lopez et al., 2 Jul 2025, Combei et al., 2024). These models yield highly discriminative, domain-invariant representations and demonstrate the greatest resilience to unseen attacks and corruptions.

Spectro-Temporal Graph Attention Networks:

Models such as AASIST and RawGAT-ST perform joint time–frequency relational modeling, offering strong generalization across synthesis types and languages, especially when paired with robust SSL front-ends (Marek et al., 2024).

Continual and Domain Incremental Learning:

Stateful learning frameworks address catastrophic forgetting and adapt to ever-evolving attack distributions. Universal Adversarial Perturbation (UAP) is used to encode synthetic directions in feature space and replay pseudo-attacks for continual adaptation without storing prior data, with feature-space UAP showing 40–50% EER reduction over naïve sequential fine-tuning (Li et al., 25 Nov 2025).

Model Family	Input Feature(s)	Core Structure
GMM-based	MFCC/LFCC/Δ/ΔΔ	Diag. mixture models
CNN/CNN-RNN	Spectrograms (LFCC/STFT/etc)	Conv + (GRU/LSTM)
SSL Transformer	Raw waveform	SSL, Transf., pooling
Graph Attention	Raw or spectral features	GAT/self-att., pooling
Ensemble/Fusion	Mult. features/models	Score fusion

2. Training Objectives, Loss Functions, and Augmentation

Supervised Binary Cross-Entropy is the standard training loss, with probabilities assigned to “real” and “fake” classes, and EER (Equal Error Rate) as the primary evaluation metric.

Alternate and Regularized Objectives:

• Focal Loss: Favors hard-to-classify samples for open-set generalization (Pierno et al., 29 Apr 2025, Lopez et al., 2 Jul 2025).
• Center and Hinged-Center Losses: Enforce within-class compactness; smooth or hard margins are used to prevent overfitting (Lopez et al., 2 Jul 2025).
• Knowledge Distillation: Used to transfer domain knowledge (e.g., from large ASVspoof5-trained teachers to compact models) without overfitting to codec artifacts (Lopez et al., 2 Jul 2025).
• Multi-task (Localisation & Detection): Frame-level and utterance-level losses are combined (FakeSound uses α·accuracy + (1–α)·F1) (Xie et al., 2024).

Data Augmentation Strategies:

• Signal-level noise: AWGN, environmental, and reverberant noise; SNR sampled per batch (Lopez et al., 2 Jul 2025).
• Room Impulse Response: RIR convolution injects real or synthetic playback room effects (Müller et al., 20 May 2025, Lopez et al., 2 Jul 2025).
• Codec Augmentation: Considers lossy compressions (MP3, Opus, Encodec, etc.) to simulate telephony and internet streaming (Lopez et al., 2 Jul 2025, Li et al., 2024).
• RawBoost: Complex pipeline synthesizing pitch, speed, spectral warping, and codec artifacts (Lopez et al., 2 Jul 2025).
• Waveform augmentations: Pitch-shift, time-stretch, additive noise, clipping (Pierno et al., 29 Apr 2025).
• Vocoder Re-synthesis: Vocoder-generated fakes from real data, exposing models to a variety of generator “fingerprints” (Lopez et al., 2 Jul 2025).

This comprehensive suite of augmentations and specialized loss design is essential for closing the cross-corpus generalization gap documented in major evaluations (Lopez et al., 2 Jul 2025, Ali et al., 2 Mar 2026).

3. Generalization, Robustness, and Open-World Detection

Cross-Domain and Out-of-Distribution (OOD) Evaluation:

Current detectors often overfit to known generator artifacts or data domains, failing against new synthesis techniques, codecs, language shifts, or recording conditions. Major insights include:

• Detection models trained on small or monocultural datasets (e.g., English-only) show high EER (10–45%) when evaluated on unseen languages or domains; intra-lingual fine-tuning on even a small amount (∼4k utterances) of target-language data is critical for robust deployment (Marek et al., 2024).
• Modern benchmarks (AUDETER) with >3 million clips, >6 k hours, and a systematic pairing of >20 generative models establish baselines for true open-world generalization, showing that only SSL-based detectors trained on both TTS and vocoder outputs, and validated out-of-corpus, reach EER <5% on “In-the-Wild” speech (Wang et al., 4 Sep 2025).

Codec and Compression Robustness:

Neural audio codecs (e.g., Encodec, AudioDec, FACodec) and traditional codecs (MP3, Opus) pose major challenges, with EER often increasing by 10–20 pp (percentage points) for models not explicitly trained to resist them (Li et al., 21 Mar 2025, Li et al., 2024). This suggests codec-aware augmentation or joint codec–detector training is essential.

Replay Attacks:

Physical playback and re-recording (“air-gap attacks”) erase or mask generator-specific artifacts; even state-of-the-art systems such as W2V2-AASIST see EER increase from 4.7% to 18.2% under these conditions despite adaptive RIR retraining (Müller et al., 20 May 2025). Effective defense requires explicit training with simulated and real RIRs and features invariant to channel coloration.

Model Scale and Self-Supervised Learning:

Larger self-supervised models (XLS-R, WavLM Large, HuBERT Large) trained on multi-100k hour, multilingual corpora consistently set the gold standard for clean and degraded conditions. However, robustness improvements saturate beyond 500M parameters, so a balance of model size and domain-specific augmentation is recommended (Ali et al., 2 Mar 2026, Li et al., 21 Mar 2025).

4. Multimodal and Privacy-Preserving Detection

Multimodal Detection:

Contemporary threat landscapes often involve synchronised audio-video fakes. Fusion approaches (AV-LMMDetect, ERF-BA-TFD+) employ large multimodal transformers (Qwen 2.5 Omni), joint attention, and prompt-based classification (“Is this video real or fake?”) to maximize cross-modal cues, achieving AUC of 0.92 and accuracy >85% in open-set tests (Cao et al., 25 Feb 2026, Zhang et al., 24 Aug 2025). Ablations confirm that audio-only models lag in performance, but fusion recovers most of the gap.

Content Privacy:

SafeEar separates semantic and acoustic information using a neural codec with hierarchical quantization, exposing only content-invariant acoustic tokens (timbre, prosody) to downstream detectors. This prevents leakage of speech content—confirmed by WER >93.9% for ASR models and extremely low subjective intelligibility—yet achieves EER as low as 2.02% on multilingual benchmarks, rivaling end-to-end wave detectors (Li et al., 2024). Privacy-preserving detection enables enterprise and regulatory monitoring without violating confidentiality.

5. Attribution, Model Recognition, and Continual/Adaptive Learning

Attribution and Model Recognition:

Beyond binary detection, forensic needs include identification of the synthesis technique or generator model. The LAVA framework employs a fake-only convolutional autoencoder and an attention-enhancement module to encode generator artifacts, with a two-stage classifier for technology-level (ASVspoof2021/FakeOrReal/CodecFake) and model-instance-level (six codec classes) attribution. Confidence-based rejection thresholds provide open-set robustness, with macro-F1 up to 96.31% and reliable error propagation analysis (Pierno et al., 4 Aug 2025).

Continual and Lifelong Learning:

The rapid turnover of synthesis models and attack types requires continual adaptation without catastrophic forgetting. UAP-based frameworks store and replay compact feature-level adversarial directions from prior distributions during fine-tuning. Feature-level UAPs achieved up to 48% EER reduction compared to naïve sequence fine-tuning, with marginal storage overhead and no need for direct access to historic data (Li et al., 25 Nov 2025).

Cross-Resolution and Spectral Consistency:

Resolution-aware detection fuses multi-scale spectral features via cross-scale attention and consistency learning, enforcing that bona-fide speech representations are resolution-invariant. This approach is computationally efficient (159k parameters), yet achieves EER of 0.16% on ASVspoof LA and 4.81% under real-world OOD speech (Shahriar, 10 Jan 2026).

6. Best Practices, Limitations, and Future Directions

Empirical findings and benchmarks converge on several best practices:

• Use the largest feasible, multilingual SSL front-end (XLS-R, WavLM Large) fine-tuned with extensive, balanced real+fake corpora from both modern TTS and vocoder systems (Wang et al., 4 Sep 2025, Ali et al., 2 Mar 2026).
• Curate training data to match expected deployment channel, language, and synthetic attack diversity; prioritize in-language or domain adaptation where feasible (Marek et al., 2024).
• Incorporate codec, replay, and room impulse response augmentation at fine-tuning to harden against physical and bandwidth attacks (Müller et al., 20 May 2025, Li et al., 21 Mar 2025).
• Fuse semantic, structural, and signal-level detection pipelines (multi-view/ensemble) for the strongest overall performance, especially under architectural variation in synthesis (GANs, flow-match, LLM-based) (Singh et al., 28 Jan 2026).
• Explore privacy-preservation and attribution pipelines as adoption and regulatory requirements evolve (Li et al., 2024, Pierno et al., 4 Aug 2025).
• Incrementally expand benchmark datasets and periodically reassess with new attacks and real acoustic conditions to avoid dataset overfitting (Wang et al., 4 Sep 2025).

Limitations remain in codec-duping attacks, extremely low-bitrate/telephony fakes, and continual adaptation without Euclidean drift or catastrophic forgetting. Open challenges include formalizing privacy guarantees for content-invariant detectors, understanding discriminative features under adversarial permutation, and integrating attribution with robust real/fake classification in a single streaming pipeline.

In sum, the current generation of audio deepfake detection models is defined by hybrid self-supervised architectures, comprehensive data augmentation, and a focus on cross-domain generalization and privacy. As generative models advance, robust, explainable, and adaptive detection will remain an enduring research frontier.