---
title: 'FakeMusicCaps: AI Music Forensics Dataset'
url: https://www.emergentmind.com/topics/fakemusiccaps
type: topic
---

# FakeMusicCaps: AI Music Forensics Dataset

FakeMusicCaps is a dataset for the detection and attribution of synthetic music generated via Text-to-Music (TTM) models. Introduced as a resource built by re-generating the music-caption pairs of MusicCaps with several state-of-the-art TTM techniques, it formalizes two central forensic tasks: distinguishing real music from generated music, and attributing a generated clip to its source model. In subsequent work, it has also become a benchmark for AI-generated music detection, especially for 10-second clip classification, cross-generator evaluation, and zero-shot detection settings [2409.10684].

## 1. Origin, scope, and forensic motivation

The FakeMusicCaps dataset was proposed in response to the rapid diffusion of TTM systems and the associated concerns regarding copyright violation and rightful attribution. The original study frames the problem in explicitly forensic terms: TTM models had reached superior performances to previous state-of-the-art systems, had lowered the technical proficiency needed for music generation, and had already started to be adopted for commercial uses and music production practices [2409.10684].

Its construction is anchored in MusicCaps. The original paper describes MusicCaps as containing 5,500 pairs of 10-second real music clips from AudioSet and professional musician-written captions. FakeMusicCaps reuses those captions as prompts for TTM generation, thereby creating a paired benchmark in which the textual conditioning is held constant while the audio provenance varies. This design makes the dataset suitable not only for binary “real versus fake” detection, but also for source-model attribution, since different generators can be compared under matched prompt conditions [2409.10684].

Later benchmark papers describe FakeMusicCaps as containing 10-second audio clips generated by five different TTM models together with real music recordings; one such description reports 5,373 real tracks and 27,605 AI-generated tracks. Those later descriptions emphasize the dataset’s diversity across generative models and conditioning artifacts, which is one reason it has been repeatedly used for generalization studies rather than only for closed-set classification [2509.08283].

## 2. Dataset construction and content

The original FakeMusicCaps paper states that, for every MusicCaps caption, five state-of-the-art TTM models were used to generate a corresponding synthetic audio clip. All generated tracks and original tracks are standardized as mono, 16 kHz, float32 PCM WAV files. The paper also characterizes the dataset as extensible: new TTM models can be added in a plug-and-play fashion using the same prompt set [2409.10684].

The five TTM systems used in the initial release are described as follows:

| TTM model | Description |
|---|---|
| MusicGen | Autoregressive language model using Encodec tokens |
| MusicLDM | Latent diffusion conditioned on CLAP |
| AudioLDM2 | Latent diffusion with VAE + HiFiGAN |
| Stable Audio Open | Latent diffusion with T5 text encoder |
| Mustango | Diffusion model with music-knowledge-informed UNet |

This composition is important methodologically. Because each synthetic clip is generated from a caption drawn from the same caption distribution as the real reference set, FakeMusicCaps is not merely a collection of AI music clips; it is a prompt-aligned benchmark for studying how generator-specific artifacts appear under common textual prompts. A plausible implication is that the dataset is particularly well suited to attribution settings in which the classifier must discriminate not just “real” from “fake,” but also among multiple synthetic provenance classes [2409.10684].

## 3. Detection, attribution, and evaluation protocols

The original paper defines two tasks. **Detection** is binary classification between real and TTM-generated music. **Attribution** is multi-class classification among real music and the known TTM generators. It further distinguishes **closed-set** evaluation, in which all classes are seen during training and test, from **open-set** evaluation, in which some generators are unseen during training and appear only at test time. For the open-set case, Suno-generated tracks from the SunoCaps dataset are used as “unknown” examples [2409.10684].

Three baseline neural architectures are reported: **M5**, a CNN on raw waveforms; **RawNet2**, an end-to-end anti-spoofing model operating on raw audio; and **ResNet18 + Spec**, a ResNet18 adapted to single-channel log-spectrogram input. Training uses cross-entropy loss, the Adam optimizer with learning rate \(1 \times 10^{-4}\), and early stopping after 10 epochs without loss improvement, up to a maximum of 100 epochs. The closed-set formulation uses standard softmax classification, while the open-set experiments evaluate two strategies: a thresholding method based on the ratio of the top two softmax scores with \(\theta = 2\), and a one-class SVM on softmax vectors [2409.10684].

The evaluation metrics include balanced accuracy, precision, recall, and F1 score. The balanced accuracy is given as

$$
\text{ACC}_B = \frac{1}{C}\sum_{i=1}^C \frac{TP_i}{TP_i + FN_i},
$$

where \(C\) is the number of classes. In the later MusicDET work, Equal Error Rate (EER) becomes the primary metric for cross-generator detection experiments, reflecting a shift from attribution-style multiclass reporting toward detector robustness under unseen generators [2409.10684; 2605.18072].

## 4. Empirical results and subsequent benchmark use

In the original closed-set attribution experiments, all three baselines perform strongly, with **ResNet18 + Spec** reaching **1.00** balanced accuracy, **1.00** precision, **1.00** recall, and **1.00** F1 score. The open-set setting is substantially harder. Under thresholding, **ResNet18 + Spec** reaches **0.85** balanced accuracy, **0.78** precision, **0.85** recall, and **0.8** F1 score; under the SVM approach, the same model drops to **0.48** balanced accuracy, **0.80** precision, **0.48** recall, and **0.56** F1 score. The paper reports that commercial models not included during training are particularly challenging, and that highly realistic unknown syntheses are often confused with real music or with the closest known TTM class [2409.10684].

Subsequent papers have used FakeMusicCaps as a benchmark for increasingly different detector designs:

| Paper | Use of FakeMusicCaps | Reported result |
|---|---|---|
| "Segment Transformer: AI-Generated Music Detection via Music Structural Analysis" [2509.08283] | Segment-level short-audio detection | MERT: Accuracy 0.994, F1 0.996, AUC 0.998 |
| "MusicDET: Zero-Shot AI-Generated Music Detection" [2605.18072] | Zero-shot, real-only training with cross-generator evaluation | MusicDET average EER 4.51; class-conditional variant 0.89 |
| "Detecting Musical Deepfakes" [2505.09633] | Mel-spectrogram CNN under pitch and tempo manipulation | Baseline Accuracy 0.885, F1 0.878 |

These later studies shift the emphasis from closed-set attribution toward generator-agnostic detection. The Segment Transformer paper uses an 80/10/10 split and evaluates segment-level detectors on 10-second clips, reporting that **MERT** outperforms **Wav2vec 2.0**, **Music2vec**, **FXencoder**, and a **MobileNet** baseline on FakeMusicCaps [2509.08283]. MusicDET goes further by formulating a zero-shot setting in which the detector is trained exclusively on real music and evaluated on music produced by all five generators; its reported average EER of **4.51%** markedly improves over conventional discriminative baselines in that setting [2605.18072]. A separate ResNet18 study applies pitch shifting in \([-2, +2]\) semitones and tempo scaling in \([0.8, 1.2]\), finding that performance degrades modestly rather than catastrophically under these manipulations [2505.09633].

## 5. Relation to adjacent datasets and terminological ambiguities

FakeMusicCaps occupies a specific niche within audio forensics. A useful contrast is with **FSD**, which is described as a dataset for fake song detection in the singing-voice domain. That comparison states explicitly that FakeMusicCaps focuses on text-to-music generation for music deepfake detection and is music, not singing, oriented, whereas FSD targets singing voice deepfakes mixed with real accompaniments and is therefore more directly aligned with singer impersonation [2309.02232].

The dataset has also been absorbed into broader mixed-audio benchmarks. **MixFake** includes music backgrounds drawn from **FMA-Medium**, **Sonics**, and **FakeMusicCaps**, and recombines foreground speech and background music or noise in all authentic/fake permutations at SNR levels sampled from \(\{-5, 0, 5, 10, 15, 20\}\) dB. In that setting, FakeMusicCaps is no longer the sole object of study, but a component of a more realistic mixed-source benchmark for background authenticity detection [2605.23201].

The term “FakeMusicCaps” has not always been used exclusively to denote the 2024 dataset. In the caption-poisoning literature, attacker-injected, LLM-generated adversarial caption entries are described as yielding highly plausible “FakeMusicCaps” data [2605.30365]. In multimodal image-to-music work, the phrase “FakeMusicCaps” risk is used to denote the possibility that pseudo-aligned or spurious pairs can mislead models and evaluations into overstating cross-modal feeling alignment [2512.00120]. These usages are conceptually distinct from the dataset introduced for detection and attribution of synthetic music generated via TTM models.

## 6. Limitations and open research directions

The original FakeMusicCaps paper presents the dataset and experiments explicitly as a baseline rather than as a solved benchmark. The reported limitations are concrete: only open-source TTM models are included for paired generation; commercial systems are used only as “unknown” in open-set tests; the baselines are simple off-the-shelf networks; the paired data setting may not represent the full diversity of real-world music or TTM behavior; and the study does not yet examine how caption–music dependence, musicology, or music theory might improve detection [2409.10684].

Its empirical results already indicate the main unresolved problem. Closed-set attribution appears straightforward with current models and data, but open-set detection and attribution remain substantially more difficult, especially for realistic commercial generators. This has motivated later work to emphasize cross-generator generalization and detector designs that do not rely on seeing fake examples from the target generator during training [2409.10684; 2605.18072].

The future directions stated in the original work are correspondingly broad: extending the dataset to additional TTM models, studying the effect of textual descriptions on detection accuracy, leveraging caption–music pairing in attribution, incorporating musicology or music theory, exploring partial fakes and splicing, and benchmarking stronger forensic architectures [2409.10684]. Taken together with later segment-level and zero-shot studies, this suggests that FakeMusicCaps functions less as a finished benchmark than as an initial standardized substrate for research on music-domain forensic generalization, model attribution, and authenticity verification under rapidly changing generative conditions [2509.08283; 2605.18072].

Source: https://www.emergentmind.com/topics/fakemusiccaps