Do Music Generation Models Encode Music Theory? (2410.00872v1)

Abstract: Music foundation models possess impressive music generation capabilities. When people compose music, they may infuse their understanding of music into their work, by using notes and intervals to craft melodies, chords to build progressions, and tempo to create a rhythmic feel. To what extent is this true of music generation models? More specifically, are fundamental Western music theory concepts observable within the "inner workings" of these models? Recent work proposed leveraging latent audio representations from music generation models towards music information retrieval tasks (e.g. genre classification, emotion recognition), which suggests that high-level musical characteristics are encoded within these models. However, probing individual music theory concepts (e.g. tempo, pitch class, chord quality) remains under-explored. Thus, we introduce SynTheory, a synthetic MIDI and audio music theory dataset, consisting of tempos, time signatures, notes, intervals, scales, chords, and chord progressions concepts. We then propose a framework to probe for these music theory concepts in music foundation models (Jukebox and MusicGen) and assess how strongly they encode these concepts within their internal representations. Our findings suggest that music theory concepts are discernible within foundation models and that the degree to which they are detectable varies by model size and layer.

• The paper introduces SynTheory, a synthetic dataset and probing framework to evaluate if music generation models encode seven core music theory concepts.
• It reveals that models like Jukebox and MusicGen capture fundamental musical attributes, with smaller MusicGen models sometimes outperforming larger ones.
• The study also finds that traditional handcrafted audio features can rival model representations, suggesting new avenues for controllable music generation.

Analyzing Music Theory Encoding in Music Generation Models

The paper "Do Music Generation Models Encode Music Theory?" by Megan Wei, Michael Freeman, Chris Donahue, and Chen Sun, presents an in-depth exploration into the extent to which state-of-the-art music generation models internalize foundational Western music theory concepts. This paper introduces SynTheory, a synthetic dataset designed to probe music theory concepts in generative models such as Jukebox and MusicGen.

Research Motivation and Contributions

Past research has highlighted the ability of music generation models to capture high-level musical attributes such as genre or emotion. However, the encoding of more granular music theory concepts, such as tempo, pitch class, and chord progressions, has been under-explored. Addressing this gap, the authors of this paper investigate whether music generation models can recognize and encode fundamental music theory concepts.

The paper makes two primary contributions:

1. Introduction of SynTheory Dataset: A synthetic dataset that isolates seven core music theory concepts (tempo, time signatures, notes, intervals, scales, chords, and chord progressions). This dataset allows controlled and scalable generation of MIDI and audio clips for probing specific music theory concepts without copyright concerns.
2. Probing Framework and Analysis: The development of a probing framework to evaluate music generation models' understanding of music theory using SynTheory. They apply this framework to Jukebox and MusicGen models to assess their internal representations effectively.

Methodology

The exploration focuses on whether generative models learn representations of music theory concepts. To this end, the authors trained probing classifiers on internal representations extracted from the music generation models. The classifiers predict music theory concepts from representations obtained at various layers within the models. The evaluation metrics included classification accuracy for discrete concepts (e.g., chord types, time signatures) and regression R² scores for continuous concepts (e.g., tempo).

Results

The results indicate that music generation models do indeed encode music theory concepts, with variation in encoding strength depending on the model and its architecture. Key findings from the analysis include:

• Jukebox Model Performance: Jukebox consistently demonstrated high probing scores across all music theory tasks, indicating a strong and coherent internal representation of these concepts.
• MusicGen Model Analysis: Smaller MusicGen models outperformed their larger counterparts, particularly in their ability to encode music theory concepts. This contradicts conventional scaling laws that discuss performance improvement with increased model size.
• Handcrafted Features: Traditional handcrafted audio features like mel spectrograms, MFCC, and chroma were also evaluated. Aggregate handcrafted features showed competitive performance relative to MusicGen models, suggesting that these features can still provide valuable insights into music theory concepts.

Implications and Future Directions

The findings of this paper suggest significant practical and theoretical implications. Practically, understanding how well generative models encode music theory concepts can facilitate the development of more controllable music generation systems. It underscores the potential for fine-grained control over specific musical attributes, aiding musicians and composers in their creative processes.

From a theoretical perspective, the paper highlights areas where models perform well and areas needing improvement, particularly in isolated note recognition. This points towards the necessity for further research into more complex and entangled music theory tasks that challenge existing models beyond the capabilities of traditional handcrafted features.

Future research should also consider multi-modal probing, exploring the interactions between text and music representations, to enhance text-controllable music generation. By extending the probing framework and introducing more challenging benchmarks, the research community can advance the understanding and capabilities of music generative models further.

Conclusion

In summary, this paper contributes significantly to the understanding of music theory encoding within music generation models. Through the introduction of the SynTheory dataset and an effective probing framework, the authors reveal that these models indeed internalize fundamental music theory concepts to various extents. As research progresses, these insights will be pivotal in advancing controllable music generation, offering both practical tools for musicians and deeper theoretical understanding for researchers.