MixFake: Benchmarking and Enhancing Audio Deepfake Detection in Diverse Real-world Mixed Audio

Abstract: Speech deepfake detection has achieved remarkable success in clean environments but faces significant challenges in complex, real-world scenarios where speech is often mixed with background music or noise. Current state-of-the-art methods rely on semantic features from self-supervised learning (SSL) models, which often fail when processing non-speech or mixed-source audio. In this paper, we first introduce MixFake, a large-scale benchmark dataset designed to simulate diverse acoustic environments with varying SNR levels and mixed authenticity components. To address the "semantic-centric" limitation, we propose a Multi-stream Prompt Tuning framework that injects signal-level priors into SSL backbones. By integrating base, frequency, and texture streams through deep prompt injection, our model effectively captures acoustic artifacts. Experimental results demonstrate that our method significantly outperforms existing baselines, achieving a 0.95% EER in foreground detection and a substantial 7.72% absolute improvement in complex background detection tasks. Our dataset and code are available at https://github.com/saltfish233/MixFake.

• The paper presents a novel MixFake dataset and multi-stream prompt tuning framework that reduces error rates in both foreground and background audio deepfake detection.
• It integrates signal-level priors using HHT and TKEO to capture frequency anomalies and energy fluctuations, outperforming standard SSL-based models.
• The method demonstrates robust cross-domain performance and achieves up to a 7.72% absolute improvement in EER across varied SNR conditions.

MixFake: Benchmarking and Enhancing Audio Deepfake Detection in Diverse Real-world Mixed Audio

Motivation and Problem Statement

The proliferation of neural TTS and voice conversion systems has introduced significant vulnerability in voice authentication, especially as usage shifts to noisy, mixed-source real-world contexts. Prevailing deepfake detection models, primarily based on self-supervised learning (SSL) backbones, have demonstrated competitive results for clean speech but fail to generalize to scenarios where speech is intermixed with music or environmental sounds. These models are inherently semantic-centric, leading to poor performance when processing non-linguistic audio components, such as manipulated backgrounds. Attackers may exploit this by embedding synthetic audio within background noise or music, circumventing conventional detection pipelines.

MixFake Dataset Construction

MixFake comprises a large-scale benchmark specifically designed for complex audio deepfake detection in varied acoustic conditions, encompassing both single-source and mixed-source scenarios. The dataset presents a rigorous permutation matrix of genuine/fake combinations across foreground speech and background music/environmental sounds, uniformly distributed over SNR levels from $-5$~dB to $20$~dB for robustness. Synthesis diversity is ensured through integration of 19 speech algorithms (ASVspoof), 10 music algorithms (Sonics, FakeMusicCaps), and 3 environmental sound algorithms (EnvSDD). The decoupled mixing strategy allows authentic cross-pairing, dynamic SNR mixing, and temporal alignment, providing granular authenticity labeling for foreground and background components.

Figure 1: The overall framework of our proposed method. Left: The dataset construction pipeline for MixFake, highlighting the decoupled mixing strategy. Right: The Multi-stream Prompt Tuning framework, featuring the backbone model and deep prompt injection.

Multi-stream Prompt Tuning Architecture

To overcome the semantic-centric weakness of traditional SSL models, the paper introduces a Multi-stream Prompt Tuning framework, augmenting a frozen XLSR-AASIST backbone with signal-level priors via deep prompt injection. Prompt streams are injected at each transformer layer:

• Base Stream ($P_{base}$): Standard learnable prompts, maintaining fundamental adaptability.
• Frequency Stream ($\tilde{P}_{fre}$): Encodes instantaneous frequency anomalies via a multi-scale Hilbert-Huang Transform (HHT) to capture phase discontinuities and local synthetic artifacts.
• Texture Stream ($\tilde{P}_{tex}$): Exploits Teager-Kaiser Energy Operator (TKEO) and feature flux to track nonlinear energy fluctuations, distinguishing real/mixed signals especially across diverse SNR regimes.

Prompt embeddings from all streams are concatenated with input features at each layer, yielding a composite representation sensitive to both high-level semantics and low-level acoustic artifacts. The layer-wise, collaborative injection mechanism enables artifact detection irrespective of signal source or context.

Experimental Evaluation

Detection Performance

MixFake evaluation is split into foreground speech detection and background audio detection tasks. The architecture attains an EER of $0.95\%$ for foreground detection, outperforming XLSR-AASIST ($2.84\%$) and WPT-XLSR-AASIST ($2.85\%$). In the background detection task—where linguistic semantics are absent—baseline EERs degrade dramatically (e.g., XLSR-AASIST at $20.12\%$), while the proposed method achieves $12.40\%$, a 7.72% absolute improvement over XLSR-AASIST and consistently surpasses all baselines. This firmly demonstrates the necessity of signal-level priors for detection outside semantic contexts.

Generalization and Robustness

Cross-dataset evaluation leverages ASVspoof 2019 LA (training) and In-the-wild (test). The model obtains an EER of $20$0, outperforming XLSR-AASIST ($20$1) and XLSR-Mamba ($20$2), signifying robust cross-domain generalization. SNR robustness analysis reveals superior model stability across interference and signal masking scenarios: foreground detection EER declines to $20$3 at $20$4~dB SNR, while background detection EER is contained at $20$5 even as background signals become heavily masked, preventing degradation exhibited by competing models.

Figure 2: Performance comparison of baseline models and our proposed method under varying SNRs on the MixFake dataset.

Ablation Analysis

Systematic ablation demonstrates that multi-stream prompt configurations consistently outperform single-stream variants. Frequency-stream-only ($20$6) and texture-stream-only ($20$7) variants achieve lower EERs than the base stream; integrating all three yields optimal results, confirming that HHT-based frequency and TKEO-based energy features are synergistic and critical for extracting synthetic artifacts from non-speech audio.

Practical and Theoretical Implications

The MixFake dataset and Multi-stream Prompt Tuning framework provide a new paradigm for audio deepfake detection, targeting real-world scenarios where foreground/background manipulation is common. By transcending semantic-centric limitations, the approach offers practical countermeasures against sophisticated audio attacks, such as synthetic noise embedding or background music forgery, applicable to voice verification, media authentication, and forensic analysis. The architectural modularity with signal-level injection directly informs future research into adaptive and domain-agnostic anti-fraud models.

Theoretically, the integration of signal-processing techniques (HHT, TKEO) with SSL backbones through prompt tuning opens avenues for multi-modal representation learning, suggesting broader applicability in domains where semantic, spectral, and energy-level artifacts must be disentangled.

Conclusion

The presented research delivers a comprehensive benchmarking and enhancement framework for audio deepfake detection in mixed-source acoustic environments. Through MixFake and a Multi-stream Prompt Tuning architecture augmenting SSL models with signal-level priors, the system achieves strong numerical gains (up to 7.72\% absolute EER reduction), robust generalization, and resilience across harsh SNR conditions. These contributions establish a foundation for audio forensic tools capable of meeting the evolving challenge of synthetic audio manipulation, and delineate a new direction for hybrid semantic-acoustic detection strategies in artificial intelligence.