Measuring Audio Prompt Adherence with Distribution-based Embedding Distances
Abstract: An increasing number of generative music models can be conditioned on an audio prompt that serves as musical context for which the model is to create an accompaniment (often further specified using a text prompt). Evaluation of how well model outputs adhere to the audio prompt is often done in a model or problem specific manner, presumably because no generic evaluation method for audio prompt adherence has emerged. Such a method could be useful both in the development and training of new models, and to make performance comparable across models. In this paper we investigate whether commonly used distribution-based distances like Fr\'echet Audio Distance (FAD), can be used to measure audio prompt adherence. We propose a simple procedure based on a small number of constituents (an embedding model, a projection, an embedding distance, and a data fusion method), that we systematically assess using a baseline validation. In a follow-up experiment we test the sensitivity of the proposed audio adherence measure to pitch and time shift perturbations. The results show that the proposed measure is sensitive to such perturbations, even when the reference and candidate distributions are from different music collections. Although more experimentation is needed to answer unaddressed questions like the robustness of the measure to acoustic artifacts that do not affect the audio prompt adherence, the current results suggest that distribution-based embedding distances provide a viable way of measuring audio prompt adherence. An python/pytorch implementation of the proposed measure is publicly available as a github repository.
- MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325.
- Ddx7: Differentiable fm synthesis of musical instrument sounds. arXiv preprint arXiv:2208.06169.
- Look, listen, and learn more: Design choices for deep audio embeddings. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3852–3856. IEEE.
- Singsong: Generating musical accompaniments from singing.
- CLAP: learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
- Neural audio synthesis of musical notes with wavenet autoencoders.
- Riffusion - Stable diffusion for real-time music generation.
- Bassnet: A variational gated autoencoder for conditional generation of bass guitar tracks with learned interactive control. Applied Sciences, 10(18:6627). Special Issue ”Deep Learning for Applications in Acoustics: Modeling, Synthesis, and Listening”.
- A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723–773.
- Adapting frechet audio distance for generative music evaluation.
- Cnn architectures for large-scale audio classification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 131–135. IEEE.
- Mulan: A joint embedding of music audio and natural language.
- Noise2music: Text-conditioned music generation with diffusion models.
- Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301.12661.
- Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In INTERSPEECH, pages 2350–2354.
- High-level control of drum track generation using learned patterns of rhythmic interaction. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, WASPAA 2019, New Paltz, NY, USA, October 20-23, 2019. IEEE.
- Audioldm: Text-to-audio generation with latent diffusion models.
- A common language effect size statistic. Psychological bulletin, 111(2):361–365.
- Comparing representations for audio synthesis using generative adversarial networks.
- Drumgan: Synthesis of drum sounds with timbral feature conditioning using generative adversarial networks. arXiv preprint arXiv:2008.12073.
- Stemgen: A music generation model that listens.
- Bass accompaniment generation via latent diffusion.
- Musika! fast infinite waveform music generation. arXiv preprint arXiv:2208.08706.
- MUSDB18-HQ - an uncompressed version of MUSDB18.
- A hierarchical latent vector model for learning long-term structure in music.
- Moûsai: Text-to-music generation with long-context latent diffusion.
- Demystifying MMD GANs. In International Conference for Learning Representations, pages 1–36.
- Music controlnet: Multiple time-varying controls for music generation.
- Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5.
- Jukedrummer: Conditional beat-aware audio-domain drum accompaniment generation via transformer vq-vae.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.