Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset (1810.12247v5)

Abstract: Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.

• The paper presents the Wave2Midi2Wave framework that factorizes piano music generation into three components—transcription, composition, and synthesis—for enhanced audio realism.
• It leverages the MAESTRO dataset, comprising over 172 hours of precisely aligned MIDI and audio recordings, to achieve state-of-the-art piano transcription and robust music modeling.
• Experimental results show competitive NLL scores and near-indistinguishable synthesized audio, underscoring its potential for automated music production and advanced generative research.

Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

This paper introduces a novel framework for generating and modeling piano music by leveraging a new dataset called MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization). The research addresses the challenges of generating musical audio with neural networks by presenting a comprehensive system termed Wave2Midi2Wave, which segments the task into transcription, composition, and synthesis sub-problems.

Overview of the Wave2Midi2Wave System

The Wave2Midi2Wave approach pioneered in this research tackles musical audio generation by explicitly factorizing the process into three well-defined components:

1. Transcription Model (Encoder): The authors utilize the Onsets and Frames model to transcribe audio into symbolic MIDI representations. This transcription model achieves state-of-the-art performance, guided by the high-quality alignment and size of the MAESTRO dataset.
2. LLM (Prior): A Music Transformer, based on self-attention mechanisms, generates new MIDI sequences by modeling the structure and long-term coherence of piano music. It is trained on the transcriptions produced by the first model.
3. Synthesis Model (Decoder): A conditional WaveNet model synthesizes audio waveforms conditioned on the generated MIDI, transforming symbolic representations back into audio. This aspect enables nuanced audio reproduction, capturing intricate timbral details.

The MAESTRO Dataset

Central to the success of the proposed system is the MAESTRO dataset, which the authors contribute as part of this work. The dataset comprises over 172 hours of paired audio and MIDI recordings of virtuoso piano performances, acquired from nine years of International Piano-e-Competition events. It represents a significant advance over existing datasets, offering precisely aligned audio-MIDI pairs with approximately 3ms accuracy, facilitating the detailed paper and generation of piano music.

Strong Numerical Results and Contributions

Through comprehensive experiments, the paper demonstrates remarkable advances:

• Transcription Performance: Leveraging the vast MAESTRO dataset, the transcription model achieves state-of-the-art results on piano transcription benchmarks, highlighting the effectiveness of high-quality input data.
• Music Transformer Evaluation: Training on both the original and transcribed MIDI from MAESTRO shows robust performance, as represented by competitive Negative Log-Likelihood (NLL) scores.
• Synthesis Realism: Listening tests reveal that models conditioned on the retrieved MIDI sequences produce audio that listeners rate as nearly indistinguishable from real piano recordings, demonstrating notable success in capturing musical nuances.

Practical and Theoretical Implications

The implications of this work extend into practical applications and further theoretical research. Practically, the approach holds potential for enhancing music production tools, enabling more nuanced and automated music creation processes. The MAESTRO dataset sets a new benchmark for future studies, enriching the community with a valuable resource for both supervised and unsupervised learning approaches.

Theoretically, the factorization method employed in Wave2Midi2Wave invites exploration into multi-modal and hierarchical generative models, paving the way for similarly structured methodologies across diverse domains beyond music. It encourages leveraging high-quality datasets and modular architectures, facilitating improved interpretability and control in generative processes.

Future Directions

This research opens multiple avenues for future exploration. Notably, extending the approach to other instruments or multi-instrument scenarios presents both challenges and opportunities in terms of dataset acquisition and model generalization. Further studies could explore cross-instrument interactions or transcription techniques, seeking datasets with equivalent alignment precision. The robustness and modularity of the Wave2Midi2Wave architecture suggest its adaptability to these complexities, hinting at broader implications for AI-driven creativity across multiple auditory domains.

In conclusion, this paper illustrates a substantial step towards sophisticated and interpretable musical audio generation, underpinned by meticulous data curation and innovative model design.