MixFake: Audio Deepfake Benchmark

Updated 5 July 2026

MixFake is an audio deepfake benchmark that evaluates authenticities of speech and background components independently in mixed-source audio.
It uses a modular dataset with dynamic SNR mixing and four authenticity pairings (RF-RB, FF-RB, RF-FB, FF-FB) to mimic realistic acoustic conditions.
The Multi-stream Prompt Tuning framework integrates frequency and texture prompts to enhance detection, especially for non-speech forgery cues.

MixFake is an audio deepfake benchmark and detection framework for mixed-source audio, introduced to study a setting in which a speech foreground is embedded in background music or environmental sound, the source-to-background ratio varies, and the authenticity of foreground and background can vary independently. It departs from the standard clean-speech anti-spoofing assumption by explicitly modeling cases in which either the spoken foreground, the non-speech background, or both are fake, and it pairs this benchmark with a Multi-stream Prompt Tuning method designed to compensate for the “semantic-centric” bias of self-supervised speech encoders such as wav2vec 2.0, HuBERT, WavLM, and XLS-R (Li et al., 22 May 2026).

1. Problem formulation and conceptual shift

MixFake targets a problem setting in which the observed signal is not treated as a single-source utterance. Instead, the audio may contain a speech foreground mixed with background music or environmental sound, and the authenticity of the two components is allowed to vary independently. The benchmark explicitly considers four authenticity pairings: real foreground with real background (RF-RB), fake foreground with real background (FF-RB), real foreground with fake background (RF-FB), and fake foreground with fake background (FF-FB). The mixture is stressed across SNR values sampled from $\{-5, 0, 5, 10, 15, 20\}$ dB, covering conditions in which the background strongly masks the speech, the speech dominates, or both are comparably audible (Li et al., 22 May 2026).

The paper’s central argument is that current state-of-the-art detectors are highly effective on clean speech precisely because they inherit strong speech representations from SSL backbones, yet this same bias becomes a liability in realistic audio. When the signal contains music, ambient sound, or mixed-source content in which the forgery cue may be located in the background rather than the spoken content, speech encoders optimized for linguistic and phonetic structure no longer have the appropriate inductive bias. MixFake therefore reframes audio deepfake detection from the conventional question “is this speech fake?” into the more general question of which component of a mixed audio scene is fake under acoustic interference (Li et al., 22 May 2026).

This reformulation also addresses a benchmark-deployment mismatch. The paper contrasts speech-centric datasets such as ASVspoof, ADD, and In-the-Wild with real recordings from social media, voice notes, podcasts, films, livestreams, and telephony, where soundtrack music, room ambience, traffic noise, and other overlapping sources are commonplace. It further emphasizes a threat model in which an attacker may synthesize or alter not only the voice but also the ambient scene or background music, making background-focused authenticity judgment a first-class task rather than a peripheral nuisance variable (Li et al., 22 May 2026).

2. Benchmark construction and data composition

MixFake contains 252,500 audio samples totaling about 673.69 hours. Of these, 510.59 hours are single-source audio and 163.10 hours are mixed-source audio. The benchmark is partitioned into train, dev, and eval splits as follows (Li et al., 22 May 2026).

Split	Samples	Hours
Train	116,000	371.02
Dev	20,500	43.67
Eval	116,000	258.99

The construction is modular. For foreground sources, MixFake uses ASVspoof 2019 LA as the exclusive speech pool, including both genuine and synthetic speech; the synthetic side includes 19 distinct spoofing algorithms. For background sources, the benchmark extends beyond speech. Real backgrounds are aggregated from FMA-Medium for music and EnvSDD for environmental sounds such as airport or street. Synthetic backgrounds come from Sonics and FakeMusicCaps for generated music and from EnvSDD for generated environmental sounds. In total, the paper reports 10 non-overlapping music-generation algorithms drawn from Sonics and FakeMusicCaps, plus 3 representative environmental-sound synthesis algorithms from EnvSDD (Li et al., 22 May 2026).

The mixed-data construction pipeline has three stated steps. First is authenticity cross-pairing: every foreground sample can be combined with backgrounds to instantiate RF-RB, FF-RB, RF-FB, and FF-FB, with $mix\_ratio = 4$ , meaning each foreground sample is paired with four unique backgrounds. Second is dynamic SNR mixing: background gain is adjusted using RMS energy so that the final mixture matches a target SNR sampled from $\{-5,0,5,10,15,20\}$ dB. Third is temporal alignment: background segments are looped or truncated to match the foreground duration. This yields controlled mixtures while keeping the evaluation centered on the target signal (Li et al., 22 May 2026).

MixFake also includes single-source data for isolated-component analysis and multi-scenario training. Across all splits, the paper reports 4,500 real foreground speech clips and 21,000 fake foreground clips; 9,000 real background clips and 42,000 fake background clips; and 44,000 samples for each mixed authenticity condition RF-RB, FF-RB, RF-FB, and FF-FB. The labels are correspondingly richer than in conventional anti-spoofing corpora: single-source audio uses a unified bona fide/fake label, while mixtures carry independent labels for foreground authenticity, background authenticity, and SNR level (Li et al., 22 May 2026).

3. Evaluation tasks and benchmark semantics

The benchmark defines two principal sub-tasks. The first is foreground speech detection, in which the system must determine whether the foreground speech is real or fake while treating the background as interference. The second is background audio detection, in which the system must determine whether the background audio is real or fake while treating the foreground speech as interference. The latter is intentionally difficult because prior speech anti-spoofing systems are not usually evaluated on non-speech authenticity (Li et al., 22 May 2026).

This decoupling is the defining property of MixFake. The benchmark does not treat the background merely as additive noise; it treats it as a potentially manipulated source with its own authenticity label. As a result, the dataset supports both speech-focused and background-focused detection, under multiple SNRs, across single-source and mixed-source regimes. A common misconception is to view MixFake as an ordinary noisy-speech extension of ASV-style benchmarks. The paper makes clear that this is not its purpose: the novelty lies in independent authenticity control over foreground and background and in explicit benchmarking of non-speech forgery detection (Li et al., 22 May 2026).

The benchmark’s semantics also make failure modes measurable. The paper reports that degradation is especially severe when the target of detection is the background component rather than the spoken foreground, because SSL speech encoders were pretrained for speech understanding or generation tasks and therefore prioritize semantic content. MixFake is designed to expose that asymmetry rather than average it away (Li et al., 22 May 2026).

4. Multi-stream Prompt Tuning framework

To address the benchmark’s challenge, the paper proposes a Multi-stream Prompt Tuning framework built on the XLSR-AASIST baseline. The backbone consists of XLS-R as the SSL pretrained encoder and AASIST as the backend. Rather than fine-tuning the full SSL model, the method freezes all original SSL encoder parameters and learns only three sets of trainable components: the multi-stream prompt vectors at each layer, the HHT and TKEO signal modules, and the backend classification network. The stated motivation is parameter-efficient adaptation that preserves broad pretrained knowledge while injecting task-specific acoustic priors (Li et al., 22 May 2026).

At every Transformer layer $i$ , the method introduces three prompt streams. The base stream $P_{base}^{(i)}$ provides generic learnable adaptation prompts. The frequency stream starts from learnable prompts $P_{fre}^{(i)}$ and processes them with a simulated multi-scale Hilbert-Huang Transform module to produce $\tilde{P}_{fre}^{(i)}$ , intended to encode instantaneous-frequency anomalies and local phase irregularities. The texture stream starts from learnable prompts $P_{tex}^{(i)}$ and uses Teager-Kaiser Energy Operator analysis plus feature flux derived from raw SSL encoder features to produce $\tilde{P}_{tex}^{(i)}$ , intended to capture nonlinear energy fluctuations and temporal texture cues (Li et al., 22 May 2026).

The deep prompt injection rule is applied at every Transformer layer:

$X^{(i)} = [P_{base}^{(i)} ; \tilde{P}_{fre}^{(i)} ; \tilde{P}_{tex}^{(i)} ; H^{(i)}].$

This per-layer concatenation is a central design decision. Instead of injecting priors only at the input, the framework gives every layer simultaneous access to generic adaptation prompts, frequency priors, and texture priors during hierarchical representation learning (Li et al., 22 May 2026).

The frequency stream is defined through multi-scale Hilbert analysis. For each scale, the prompt sequence is converted into an analytic signal

$mix\_ratio = 4$ 0

with instantaneous phase

$mix\_ratio = 4$ 1

and instantaneous frequency

$mix\_ratio = 4$ 2

The resulting transient, global, and trend IF components are concatenated and projected back into prompt space (Li et al., 22 May 2026).

The texture stream uses the raw pretrained feature sequence $mix\_ratio = 4$ 3 and computes the Teager-Kaiser Energy Operator

$mix\_ratio = 4$ 4

together with feature flux

$mix\_ratio = 4$ 5

An adaptive gating weight is then formed as

$mix\_ratio = 4$ 6

and the final texture prompt is

$mix\_ratio = 4$ 7

This gate modulates the amount of injected texture prior according to acoustic complexity, which is particularly relevant under varying SNR and overlap conditions (Li et al., 22 May 2026).

In implementation, audio is resampled to 16 kHz and normalized to 4 seconds by padding or random cropping. Optimization uses AdamW with learning rate $mix\_ratio = 4$ 8, weight decay $mix\_ratio = 4$ 9, batch size 32, and 30 training epochs on an NVIDIA H800 GPU. Evaluation uses Equal Error Rate (EER) (Li et al., 22 May 2026).

5. Empirical results and ablation structure

The main results support the paper’s thesis that SSL-based detectors degrade sharply in mixed audio and that signal-level priors partially correct this failure. On MixFake foreground speech detection, the proposed method achieves 0.95% EER, compared with 2.84% for XLSR-AASIST, 1.37% for XLSR-Mamba, and 2.85% for WPT-XLSR-AASIST. On the more difficult background audio detection task, it achieves 12.40% EER, compared with 20.12%, 17.86%, and 15.81% for the same baselines, respectively (Li et al., 22 May 2026).

Method	Foreground EER (%)	Background EER (%)
XLSR-AASIST	2.84	20.12
XLSR-Mamba	1.37	17.86
WPT-XLSR-AASIST	2.85	15.81
Proposed method	0.95	12.40

The paper emphasizes the 7.72% absolute improvement over XLSR-AASIST in complex background detection. It also reports cross-dataset generalization from ASVspoof 2019 LA train to In-the-Wild: the proposed model reaches 6.24% EER, versus 9.60% for XLSR-AASIST, 6.71% for XLSR-Mamba, and 7.35% for WPT-XLSR-AASIST. This suggests that injecting signal-level priors does not merely fit MixFake’s construction but improves robustness under distribution shift (Li et al., 22 May 2026).

The SNR analysis is especially diagnostic. In foreground detection, performance improves as SNR increases; the proposed model reaches 0.36% EER at 15 dB and still maintains 3.10% EER at the hardest $\{-5,0,5,10,15,20\}$ 0 dB condition, compared with 6.46% for XLSR-AASIST and 5.48% for WPT-XLSR-AASIST. In background detection, the trend reverses as expected: the proposed model obtains 11.24% EER at $\{-5,0,5,10,15,20\}$ 1 dB, and when speech dominates at 20 dB it limits degradation to 16.70%, whereas XLSR-AASIST degrades to 27.05% (Li et al., 22 May 2026).

The ablation study shows that each prompt stream contributes and that the full three-stream design is optimal. Base-only prompts yield 3.05% foreground EER and 14.31% background EER. Frequency-only prompts yield 2.01% and 13.50%; texture-only prompts yield 2.13% and 14.89%. The best partial combination is base+frequency at 1.50% and 12.86%, while the full model with $\{-5,0,5,10,15,20\}$ 2 reaches 0.95% and 12.40%. The paper explicitly notes that the HHT-derived frequency prior is the most useful single auxiliary cue for non-speech forgery detection (Li et al., 22 May 2026).

6. Interpretation, limitations, and broader significance

The principal insight of MixFake is that semantic cues and signal-level cues are not interchangeable. SSL speech encoders are strong in clean speech settings because they compress representations aligned with content and speaker information, but music and environmental sounds carry no linguistic semantics while still containing synthetic traces such as abnormal frequency trajectories, phase discontinuities, energy modulation, and texture irregularities. The HHT-based frequency stream and TKEO-based texture stream are intended to restore sensitivity to these non-semantic artifacts, and the experimental pattern—especially the larger gains on background detection than on foreground detection—supports that interpretation (Li et al., 22 May 2026).

The paper also identifies clear limitations. Even with the proposed model, background detection remains substantially harder than foreground detection: 12.40% EER versus 0.95% on the aggregate tasks, and 16.70% at 20 dB when the background is heavily masked by speech. The benchmark does not provide a fine-grained breakdown by individual background type, by specific music or environmental synthesis model, or by unseen attack family inside MixFake. Moreover, the construction remains synthetic in the sense that foreground-background pairing is algorithmic and duration-matched by looping or truncation; this leaves real-world reverberation, microphone effects, compression, and spatial mixing only partially modeled (Li et al., 22 May 2026).

Practically, MixFake is suited to deployment settings in which overlapped or noisy audio is the norm rather than the exception: social-media clips, podcasts, multimedia uploads, surveillance recordings, user-generated content, and scenarios in which either the voice or the soundtrack may be manipulated. The paper’s release of dataset and code fixes a reproducible reference point for this regime (Li et al., 22 May 2026).

A plausible implication is that MixFake belongs to a broader benchmark-design trend in fake-media research: replacing single-source authenticity assumptions with mixed-source, multi-pattern, or composition-aware formulations. In vision, MultiFakeVerse studies person-centric visual and conceptual manipulations rather than a single artifact family (Gupta et al., 1 Jun 2025), and FakeChain examines multi-step deepfakes in which final-stage artifacts dominate detectability (Heo et al., 20 Sep 2025). In multimodal misinformation, MMFakeBench formalizes mixed-source distortion across text, image, and cross-modal consistency (Liu et al., 2024), while AMG frames fake news as a set of heterogeneous attribution patterns rather than a binary label (Guo et al., 2024). Within audio, MixFake performs the analogous shift from clean, speech-only spoofing to independently manipulable components in realistic acoustic mixtures (Li et al., 22 May 2026).