SingFox: Multilingual Singing Deepfake Benchmark

• SingFox is a comprehensive benchmark dedicated to singing deepfake detection and forensic source tracing using a diverse multilingual dataset.
• It features 113,802 audio clips, 126+ hours of recordings, and six evaluation tracks that address language, genre, and synthesis variability.
• The benchmark emphasizes robust generalization and explainability through open-set attribution and stress testing with unseen generative models.

SingFox is a multi-lingual, multi-track benchmark designed specifically for evaluating singing deepfake detection and source tracing/verification systems. It is introduced as a comprehensive and large-scale dataset for robust evaluation of singing deepfake detection and source tracing systems, and it targets two core research problems: reliable singfake detection under realistic, multilingual conditions, and forensic source tracing of the generative pipeline used to produce a fake (Shah et al., 17 Jun 2026). The corpus encompasses over 113,802 audio clips across 20 languages, totaling more than 126.32 hours of audio data and featuring 1,150 singers. It is organized into six tracks, T1–T6, each targeting a distinct form of novelty, including language diversity, genre- and instrument-specific material, alternative fake constructions, and a dedicated source verification protocol. The benchmark is designed for testing generalization rather than model pretraining, with emphasis on text symmetry, cross-model robustness, and explainability (Shah et al., 17 Jun 2026).

1. Research scope and distinguishing characteristics

SingFox addresses singfake detection as a binary classification problem over real versus AI-generated or AI-converted singing. The formulation explicitly accounts for singing-specific cues, including melody, vibrato, sustained $F_0$, and rhythmic structure, which differ from the operating conditions typical of speech deepfake detection. In parallel, it defines source tracing or verification as forensic attribution of a fake to its generating model in both closed-set and open-set regimes, thereby connecting detection with model explainability (Shah et al., 17 Jun 2026).

The dataset departs from prior resources through the combination of several factors within a single benchmark. It provides language coverage over 20 languages, comprising 14 international languages and 6 Indic languages. It includes a dedicated genre- and instrument-focused track with 5 instrument classes: flute, guitar, piano, tabla, and violin. It also introduces a first-of-its-kind track containing fake vocals mixed with real background music and related hybrids, with the explicit goal of stressing detectors that rely on background cues rather than vocal evidence. In addition, it consolidates multiple fake-generation families in one dataset: 3 GAN vocoders, 2 diffusion systems, 2 voice conversion systems, and 1 text-to-music model. The benchmark further includes a dedicated source tracing protocol, T6, and model-wise difficulty analysis intended to clarify what detectors learn (Shah et al., 17 Jun 2026).

A plausible implication is that SingFox is structured less as a static corpus for aggregate accuracy reporting and more as a stress test for generalization under realistic distribution shifts. That interpretation is supported by its emphasis on multilingual variability, heterogeneous synthesis families, and the explicit inclusion of open-set source rejection.

2. Corpus composition and track organization

At the corpus level, SingFox contains 113,802 clips, 126.32+ hours of audio, and 1,150 singers whose identities are anonymized. Language tags are provided per file, whereas singer IDs are not released for ethical and copyright reasons. The language inventory is described as roughly balanced across high-resource and Indic languages, with typical per-language file counts in the 9.2k–10.5k range; example tallies include en $\approx 10{,}512$, zh $\approx 10{,}391$, fr $\approx 10{,}342$, mr $\approx 10{,}027$, de $\approx 10{,}004$, ta $\approx 9{,}884$, id $\approx 9{,}733$, te $\approx 9{,}693$, hi $\approx 9{,}679$, ar $\approx 10{,}512$0, pt $\approx 10{,}512$1, pa $\approx 10{,}512$2, ru $\approx 10{,}512$3, ko $\approx 10{,}512$4, ja $\approx 10{,}512$5, bn $\approx 10{,}512$6, and es $\approx 10{,}512$7 (Shah et al., 17 Jun 2026).

The six-track structure is central to the benchmark’s design. Some tracks overlap by construction: T4 is the union of T1 and T2, and T6 is a protocol layered on T4’s test split. Consequently, per-track hours and clip counts are not additive toward the global total (Shah et al., 17 Jun 2026).

Track	Focus	Size
T1	14 major non-Indic languages; cross-lingual robustness under global conditions	47,885 files; 53.17 h
T2	6 Indic languages; robustness to underrepresented languages and regional variability	20,243 files; 22.47 h
T3	5 instrument categories; music-context variability and genre cues	12,725 files; 14.12 h
T4	Union of T1 and T2 for “global validity” testing; recommended primary cross-lingual test set	68,128 files; 75.63 h
T5	Alternative fakes, including fake vocals over real background music	32,949 files; 36.57 h
T6	Source verification protocol on T4 test audio	24,997 trials; subset duration reference 27.73 h

T5 is defined through three class configurations: real vocals plus real instruments with 12,011 trials, fake vocals plus fake instruments with 10,469 trials, and real instruments plus fake vocals with 10,469 trials. T6 introduces enrollment and evaluation tuples for source attribution, including both seen-source and unseen-source scenarios. For T3 and T5, the five instrument types are described as approximately balanced, with one track snapshot reporting around 2,545 files per instrument (Shah et al., 17 Jun 2026).

This organization suggests that the benchmark encodes several orthogonal nuisance dimensions—language, accompaniment, synthesis family, and source identity—rather than collapsing them into a single detection split. That design is consistent with the stated goal of evaluating robustness.

3. Generative families, data construction, and preprocessing

The fake data span multiple generation paradigms. The GAN vocoder group includes HiFi-GAN (Universal), BigVGAN, and UnivNet. The diffusion-based singing synthesis group includes DiffSinger and DiffRhythm, where diffusion systems produce mel-spectrograms consumed by HiFi-GAN. The voice conversion group includes Retrieval-based Voice Conversion (RVC) and So-VITS-SVC. Text-to-music generation is represented by MusicGen, used to obtain AI-generated music for relevant tracks such as fully synthetic instrumentals or background tracks (Shah et al., 17 Jun 2026).

The corpus uses many models in “universal” pre-trained form, with upstream training corpora including VCTK, LibriTTS, and LJ Speech. Whisper-large-960h is used to transcribe lyrics for synthesis pipelines that require text, with average WER $\approx 10{,}512$8 across selected languages. Mel-spectrograms are computed with librosa; GAN vocoders synthesize waveforms from these mels; diffusion systems generate mels which are then rendered by HiFi-GAN (Shah et al., 17 Jun 2026).

Real singing is sourced from copyright-free songs on open platforms such as pixabay.com/music/. Retrieval is diversified by keywording, and care is taken to avoid overlaps in singers and languages across tracks. Preprocessing converts audio to mono .flac, resamples to 16 kHz, and applies dual normalization per file: peak normalization and RMS normalization. The stated reason is to prevent shortcut learning via loudness or silence artifacts. Audio is then segmented into randomized 4-second excerpts to standardize clip length and avoid biases tied to long context. No source separation is applied; the full mix is preserved to mirror real-world conditions (Shah et al., 17 Jun 2026).

Annotations include the real/fake label for detection, per-file language tag, per-file instrument or genre tag where applicable, and track ID. For fake clips, generating-model labels are also available. In T6, the protocol file specifies claim_source, file_name, and label, where the label takes values in $\approx 10{,}512$9; it also distinguishes seen from unseen sources for open-set evaluation (Shah et al., 17 Jun 2026).

4. Experimental tasks and evaluation protocol

SingFox formalizes two tasks. Task 1 is singfake detection, where the objective is to classify each 4-second clip as real or fake. Task 2 is source tracing or verification, instantiated in T6, where the objective is attribution of a fake to its source model along with rejection of unseen sources (Shah et al., 17 Jun 2026).

For detection, the benchmark uses a train/test protocol oriented toward out-of-distribution generator evaluation. Approximately 30% of each track used in training is allocated to training and validation, with the remainder reserved for testing. Only three generators—HiFi-GAN, So-VITS-SVC, and DiffRhythm—appear in train/validation. All other generators, including subsets of training generators, are used at test time to measure cross-model generalization. Standard cross-entropy training is used, and neither lyric text nor source separation is used in the classifiers in order to avoid shortcuts (Shah et al., 17 Jun 2026).

The reported metrics are Accuracy, Precision, Recall, F1, ROC-AUC, and EER. The formulas are given as

$\approx 10{,}391$0

$\approx 10{,}391$1

and

$\approx 10{,}391$2

EER is defined at the operating point where $\approx 10{,}391$3 as the threshold $\approx 10{,}391$4 is swept over the score range. ROC-AUC is the area under the ROC curve obtained by sweeping the decision threshold over the posterior or logit score (Shah et al., 17 Jun 2026).

For source tracing in T6, the protocol has two stages. In enrollment, each source model $\approx 10{,}391$5 is represented by 20 utterances, written as $\approx 10{,}391$6. In evaluation, tuples of the form (claim_source, file_name, label) are scored, with positive and negative source claims. Closed-set attribution requires accepting the true enrolled source and rejecting the others among seen sources. Open-set rejection requires rejecting non-enrolled, unseen-source models. The same backbones used for detection can be repurposed for source verification, and the metrics include Acc, ROC-AUC, EER, and confusion across sources (Shah et al., 17 Jun 2026).

5. Baselines and empirical findings

The baseline feature and model families include LFCC with ResNet as the primary baseline, as well as MFCC and GFCC with CNN, BiLSTM, and BiGRU back-ends. SSL and end-to-end baselines evaluated on T4 and T6 include Wav2Vec 2.0, HuBERT, Whisper, XLSR, AASIST, and RawNet2. Input representations include mel-spectrograms for GAN vocoders and LFCC, MFCC, and GFCC for detection and verification (Shah et al., 17 Jun 2026).

The headline cross-dataset result is obtained when training on FMC and testing on SingFox T4: the best cross-dataset accuracy reaches 77.84%. For comparison, training on CtrSVDD and testing on SingFox T4 yields 46.06% accuracy, while training on WildSVDD and testing on SingFox T4 yields 54.17%. The benchmark interprets this as evidence that SingFox surfaces generalization gaps for models trained on speech-centric or singfake-limited corpora, whereas FMC-trained models generalize better to SingFox’s multilingual diversity (Shah et al., 17 Jun 2026).

Within SingFox, several track-level observations are highlighted. As language diversity increases, with T4 compared to T1 or T2, models gain robustness, and LFCC-based systems benefit from multilingual exposure. T5 is especially difficult: LFCC+ResNet drops to 45.13% accuracy in one setting, indicating that fake vocals with real accompaniment can fool detectors that rely on backing tracks (Shah et al., 17 Jun 2026).

For T6 verification-style evaluation, DET curves are reported over approximately 24,997 spoof trials and approximately 32,741 genuine trials. RawNet2 obtains EER $\approx 10{,}391$7, LFCC (BiLSTM) EER $\approx 10{,}391$8, AASIST EER $\approx 10{,}391$9, XLSR EER $\approx 10{,}342$0, GFCC EER $\approx 10{,}342$1, MFCC EER $\approx 10{,}342$2, Wav2Vec2 EER $\approx 10{,}342$3, and Whisper EER $\approx 10{,}342$4. The reported interpretation is that traditional acoustic cues, particularly LFCC, remain competitive under constrained training data and often outperform SSL features in this setting (Shah et al., 17 Jun 2026).

For T6 source tracing with ResNet back-ends, the aggregated seen-source attribution plus unseen-source rejection accuracies are MFCC $\approx 10{,}342$5, LFCC $\approx 10{,}342$6, and GFCC $\approx 10{,}342$7. Model-specific difficulty varies markedly: UnivNet reaches 71.17%, DiffRhythm 58.39%, BigVGAN 1.02%, and RVC 1.77% against the best classifier. DiffSinger, HiFi-GAN, and So-VITS-SVC are described as near-ceiling on the training-included models and are omitted or marked X in model-specific detection tables. The paper characterizes BigVGAN as very hard to detect or attribute and, by this measure, the most realistic among those tested (Shah et al., 17 Jun 2026).

6. Explainability, reproducibility, and limitations

Explainability in SingFox is operationalized through the source tracing protocol and model-wise performance profiling. T6 quantifies both closed-set attribution and open-set rejection, while generator-specific performance differences are used to identify detector blind spots. Spectral feature comparisons indicate that LFCC captures spoof artifacts more reliably for attribution than GFCC in this corpus. The benchmark also notes that users can apply saliency or layer-wise relevance propagation on spectrogram inputs; although the paper centers on verification metrics, the combination of hooks and file-level annotations makes post-hoc explanations feasible (Shah et al., 17 Jun 2026).

Reproducibility is a stated design objective. Code and protocols are publicly available in a repository that includes end-to-end Colab notebooks for generation, preprocessing scripts, and T6 protocol construction. Dataset access is provided for non-commercial research. The benchmark standardizes audio as mono .flac, 16 kHz, 4-second segments with dual normalization, and the documented workflow is: download audio and metadata, follow the repository scripts to build LFCC or MFCC features and train ResNet or other baselines on the designated training splits, evaluate on T1–T5 for detection and on T6 for source verification, and report Acc, ROC-AUC, EER, and F1 with language-wise and genre-wise analyses (Shah et al., 17 Jun 2026).

The limitations are also explicitly described. Although the corpus is multilingual and instrument-diverse, imbalance may remain in genre micro-styles and recording conditions. MOS testing was limited to language-known participants and does not cover all languages equally. Text symmetry is emphasized to reduce lyric-content shortcuts, but residual asymmetries may persist because transcription is imperfect, even though Whisper WER is approximately 4.9%. In addition, because T4 subsumes T1 and T2 and T6 reuses T4 test audio for verification, per-track hours should not be summed toward the global total (Shah et al., 17 Jun 2026).

The ethical framing is mitigation-oriented. Real audio is harvested from platforms offering copyright-free music; singer identities are anonymized and not distributed; and distribution is restricted to non-commercial research. At the same time, the benchmark acknowledges potential misuse and positions detection and attribution research as the intended countermeasure domain (Shah et al., 17 Jun 2026).

7. Use cases and prospective extensions

The benchmark provides explicit practical guidance. T4 is recommended as the primary multilingual detection benchmark, T5 is added to assess robustness to alternative fakes, and T6 is used for source attribution and open-set rejection. Cross-dataset tests are recommended for estimating real-world generalization, with the reported reference point being the 77.84% FMC-to-SingFox T4 accuracy. To avoid overfitting, the benchmark advises against training on generators that are intended solely for testing and recommends keeping unseen models such as BigVGAN for test-time evaluation of out-of-distribution robustness (Shah et al., 17 Jun 2026).

It also recommends avoiding reliance on background accompaniment cues, because T5 demonstrates that such cues can be misleading; a plausible implication is that robust systems should preferentially model vocal artifacts rather than accompaniment artifacts. Language-stratified validation is recommended so that performance is not dominated by a subset of languages. For reporting, the suggested protocol is to provide Acc, ROC-AUC, EER, and F1 on T4, add per-language and per-instrument breakdowns, and include cross-model breakdown by generator. For source tracing, the recommended setup is 20 enrollment clips per seen source with testing on both seen and unseen generators, reporting closed-set attribution accuracy, open-set EER, and source confusion (Shah et al., 17 Jun 2026).

The future directions identified for SingFox include expansion to more languages, dialects, micro-genres, and singing styles; language-adaptive or multilingual SSL features trained with singing data; improved source tracing metrics such as calibrated likelihood-ratio scoring and open-set AUROC for attribution confidence; multimodal approaches combining audio with lyrics, score or MIDI, or visual performance cues; countermeasures that isolate singing-specific artifacts such as vibrato statistics, pitch-contour micro-perturbations, and harmonic-noise modeling; standardized spectro-temporal saliency benchmarks; and data extensions including source-separated vocals, larger-scale training splits, and richer metadata such as tempo, key, and $\approx 10{,}342$8 statistics (Shah et al., 17 Jun 2026).

Taken together, SingFox functions as a rigorous benchmark for multilingual singfake detection and forensic source tracing. Its principal contribution is not only scale, but also the deliberate construction of evaluation settings that expose failures under unseen generators, multilingual variability, accompaniment confounds, and source attribution demands. The best reported cross-dataset accuracy of 77.84% on T4 simultaneously indicates measurable progress and underscores the remaining difficulty of building detectors that generalize across languages, genres, and synthesis families (Shah et al., 17 Jun 2026).