Towards Assessing Data Replication in Music Generation with Music Similarity Metrics on Raw Audio (2407.14364v2)

Abstract: Recent advancements in music generation are raising multiple concerns about the implications of AI in creative music processes, current business models and impacts related to intellectual property management. A relevant discussion and related technical challenge is the potential replication and plagiarism of the training set in AI-generated music, which could lead to misuse of data and intellectual property rights violations. To tackle this issue, we present the Music Replication Assessment (MiRA) tool: a model-independent open evaluation method based on diverse audio music similarity metrics to assess data replication. We evaluate the ability of five metrics to identify exact replication by conducting a controlled replication experiment in different music genres using synthetic samples. Our results show that the proposed methodology can estimate exact data replication with a proportion higher than 10%. By introducing the MiRA tool, we intend to encourage the open evaluation of music-generative models by researchers, developers, and users concerning data replication, highlighting the importance of the ethical, social, legal, and economic consequences. Code and examples are available for reproducibility purposes.

• The paper introduces MiRA to assess data replication in generative music using audio similarity metrics like CoverID, CLAP, and DEfNet.
• It employs controlled forced-replication experiments across diverse genres to rigorously evaluate metric performance.
• Embedding-based metrics consistently detect low-level replication, while FAD underperforms, indicating directions for further research.

Towards Assessing Data Replication in Music Generation with Music Similarity Metrics on Raw Audio

The paper "Towards Assessing Data Replication in Music Generation with Music Similarity Metrics on Raw Audio" by Batlle-Roca et al. addresses a pressing concern in the domain of generative AI for music: potential data replication and its implications, particularly around intellectual property rights, ethical usage, and business models. To tackle this issue, the authors introduce the Music Replication Assessment (MiRA) tool, a model-independent evaluation methodology leveraging audio-based music similarity metrics.

The paper is structured to investigate two primary questions: the suitability of audio-based metrics to assess data replication and the development of a model-agnostic evaluation method. Five metrics are considered: four standard ones (CoverID, KL divergence, CLAP, and FAD) and a novel metric (DEfNet).

Experimental Framework

A controlled forced-replication experiment with synthetic data was conducted, focusing on six diverse music genres. This controlled environment ensured that various degrees of replication (5%, 10%, 15%, 25%, and 50%) were tested, and metrics were evaluated based on their performance in detecting these exact replications.

Results and Analysis

The paper provides a detailed analysis of each metric's performance. Here are the key findings:

1. Cover Song Identification (CoverID): Demonstrated sensitivity to data replication at a threshold of 10%, with potential sensitivity lowering to 5% in specific cases. This metric's reliance on pitch-content features and local alignment made it robust for this task.
2. KL Divergence: Showed some sensitivity in detecting replication but was ineffective in distinguishing between different degrees of replication.
3. CLAP and DEfNet Scores: Both metrics based on embeddings performed well, highlighting their ability to detect replication even at lower levels (5%). They exhibited consistent, statistically significant results across all music genres and degrees of replication.
4. Fréchet Audio Distance (FAD) Using CLAP Music Embeddings: This metric did not perform effectively in this context. It showed a contrary trend, yielding higher similarity for the baseline group and displaying inconsistencies across genres.

Implications

The findings underscore that while some metrics, such as CoverID and embedding-based metrics like CLAP and DEfNet, can effectively detect data replication, others like FAD may require further investigation or alternate classifiers for adequate performance. The successful implementation of MiRA, utilizing the validated metrics, positions it as a significant tool for evaluating generative music models on the basis of data replication.

Future Work and Limitations

Future efforts should address the robustness of these metrics against typical perturbations and data augmentation techniques. Exploring the potential of other classifiers for computing FAD remains crucial. Additionally, applying the investigated metrics to actual AI-generated music will further validate MiRA's capabilities in real-world scenarios.

Conclusion

This paper contributes valuable insights and tools necessary for the ethical and transparent development of AI in the music industry. By introducing and validating MiRA, the authors provide a practical solution for assessing exact data replication in AI-generated music, promoting ethical practices and awareness regarding the legal and social implications of generative AI. Ensuring openness and reproducibility, the release of MiRA as an open-source tool encourages widespread adoption and further research in this critical area.