SynSFX: Multi-Model Sound Effects Synthesis Dataset for Deepfake Detection and Evaluation

Abstract: While audio deepfake detection has advanced significantly, representative detectors show limited generalization to synthetic sound effects. Existing environmental audio datasets such as EnvSDD provide important initial resources, but remain limited in scale and generation provenance for studying isolated sound-effect deepfakes. To support this direction, we present SynSFX, a large-scale corpus of 43374 clips (26452 synthetic, 16922 real) spanning 7 popular text-to-audio models.

• The paper introduces SynSFX, a multi-model dataset with over 43K audio clips to benchmark deepfake detection in non-speech audio.
• It reveals that speech-centric detectors perform poorly on synthetic environmental sounds, emphasizing the need for generator-agnostic methods.
• Joint-domain training mitigates catastrophic forgetting but still struggles with unseen TTA generators, pointing to promising future research directions.

SynSFX: A Multi-Model Benchmark for Sound Effect Deepfake Detection

Introduction

The "SynSFX: Multi-Model Sound Effects Synthesis Dataset for Deepfake Detection and Evaluation" (2607.04848) paper addresses the critical gap in audio forensics concerning the detection of synthetic, non-speech audio, particularly environmental sound effects created by recent text-to-audio (TTA) models. While deepfake research has primarily focused on speech-centric datasets and countermeasures, the rapidly increasing fidelity and diversity of non-speech generative models necessitates new benchmarks and methodologies. SynSFX is introduced as a large, structurally controlled dataset specifically designed for benchmarking and analysis of deepfake detection in the domain of sound effects, under both seen-generator and out-of-domain (unseen-generator) conditions.

SynSFX Dataset Construction and Design

SynSFX contains 43,374 audio clips (approximately 178 hours), with 26,452 synthetic and 16,922 real recordings. The dataset's real subset draws from five reputable open-source repositories, including AudioCaps, Clotho, ESC-50, TACoS, and WavCaps, to cover a broad range of environmental and Foley sounds. The synthetic subset is generated via seven state-of-the-art TTA models—AudioLDM (v1/v2), AudioCraft/AudioGen, MMAudio, StableAudio, Make-An-Audio, and TangoFlux—selected for architectural diversity.

A critical feature is the integration of a Shared Prompt Subset, wherein 1,890 identical prompts are used across all generators to allow cross-generator, prompt-controlled evaluation of detection models. This design addresses shortcomings in prior datasets, providing the field with a tool for disentangling generator artifacts from semantic or class-dependent cues.

Prompt engineering leverages LLMs to expand concise event descriptions into context-rich, safety-filtered prompts, injecting acoustic and scene realism representative of practical TTA deployment scenarios.

Baselines and Zero-Shot Generalization Performance

The dataset is evaluated using three architectures: two speech-centric deepfake detection models (AASIST, RawNet2) and a more generalized audio detector (EAT-AASIST). Under a strict zero-shot protocol:

• AASIST and RawNet2 collapse in performance, yielding EERs of 60.84% and 49.94%, effectively random. This confirms their reliance on speech-intrinsic cues (fundamental frequency, formant structure) that are absent in non-speech effects.
• EAT-AASIST, pre-trained on environments with mixed synthetic backgrounds and speech, is more robust (EER 23.71%, ROC AUC 0.8590), but still falls far short of in-domain results.

This evidence indicates that detectors trained solely on speech data are insufficiently equipped for non-speech deepfake forensics and establishes SynSFX's necessity.

Domain Adaptation and Catastrophic Forgetting

Fine-tuning both AASIST and EAT-AASIST on the SynSFX corpus yields:

• Substantial in-domain performance gains (AASIST EER reduced to 3.23%, EAT-AASIST to 2.36% on SynSFX test set).
• Catastrophic forgetting is observed: AASIST's EER on speech deepfakes rises from ~4% pre-fine-tuning to 33.61% post fine-tuning on SynSFX. EAT-AASIST is similarly affected.

This demonstrates that naive single-domain adaptation erodes prior capabilities, reinforcing the requirement for multi-domain or joint training approaches for robust, general-purpose audio forensics.

Mitigating Forgetting via Joint-Domain Training

Inclusion of the ASVspoof 2019 speech deepfake dataset in joint fine-tuning restores speech-domain EERs to 3.61% for AASIST and 5.25% for EAT-AASIST, while maintaining low EERs on synthetic sound effect detection (3.76% and 2.45% respectively). Thus, joint training stabilizes cross-domain memory.

However, a more nuanced analysis reveals persistent deficits in generalization to unseen TTA generators, especially evidenced in diagnostic evaluation splits:

• EERs rapidly degrade when classifying fake audio from an unseen generator (e.g., EAT-AASIST joint-trained: 27.50% EER seen-real+unseen-fake).
• The main challenge is identified as generator-shift—in comparison, real-domain shifts (unseen-real+seen-fake) are much less detrimental.

Illusion of Generalization and Feature Space Collapse

t-SNE visualizations of penultimate-layer embeddings reveal:

• Linear separability for authentic versus synthetic speech is strong.
• For unseen non-speech audio, however, feature entanglement is severe, with synthetic and real samples from out-of-distribution generators forming overlapping manifolds and abrogating meaningful detector decision boundaries.

Figure 1: t-SNE visualization of the penultimate-layer embeddings for the joint-trained (a) AASIST and (b) EAT-AASIST models. Both architectures maintain linear separability for speech domains (red vs. blue) but suffer from severe feature entanglement when processing unseen non-speech audio (green vs. purple).

Beyond visual analysis, shared-prompt diagnostics with EAT-AASIST quantitatively show that, after exposure to SynSFX, generator-dependent confidence variance decreases for seen architectures (score range drops from 0.89 to 0.09), but this does not transfer to unseen generators.

Implications and Future Directions

The study’s findings highlight severe overfitting of existing detection paradigms to generator-specific artifacts, paralleling difficulties previously observed in speech deepfake detection. SynSFX offers a benchmark explicitly structured for rigorous cross-generator and cross-domain robustness evaluation, with practical significance in media authentication, gaming, streaming, and safety-critical applications where environmental sound forgeries have substantial impact.

Moving forward, the indication is clear: future advances in synthetic audio forensics must (a) target more generalized (generator-agnostic) acoustic anomaly representations, (b) leverage synthetic data design to control for content confounds, and (c) develop pre-training regimes or architectures resistant to catastrophic forgetting. The shared-prompt mechanism of SynSFX uniquely facilitates such research.

Expansion of SynSFX to encompass broader soundscapes, scaling of unseen-generator splits, and exploration of more powerful representation learning or domain adaptation methodologies (including self-supervised pretraining or invariant feature disentanglement) each represent promising research directions.

Conclusion

SynSFX sets a new standard for evaluating and developing sound effect deepfake detectors by providing a multi-generator, structurally controlled corpus specifically designed for non-speech audio forensics. While joint-domain adaptation mitigates catastrophic forgetting, robust zero-shot generalization to unseen TTA generators remains unsolved. The dataset's architecture—including its shared-prompt subset—is engineered to directly support new research on disentangling semantic, class, and generator effects, with significant implications for practitioners and theorists addressing the evolving threat landscape of audio deepfakes (2607.04848).