MelodyT5: A Unified Score-to-Score Transformer for Symbolic Music Processing (2407.02277v2)

Abstract: In the domain of symbolic music research, the progress of developing scalable systems has been notably hindered by the scarcity of available training data and the demand for models tailored to specific tasks. To address these issues, we propose MelodyT5, a novel unified framework that leverages an encoder-decoder architecture tailored for symbolic music processing in ABC notation. This framework challenges the conventional task-specific approach, considering various symbolic music tasks as score-to-score transformations. Consequently, it integrates seven melody-centric tasks, from generation to harmonization and segmentation, within a single model. Pre-trained on MelodyHub, a newly curated collection featuring over 261K unique melodies encoded in ABC notation and encompassing more than one million task instances, MelodyT5 demonstrates superior performance in symbolic music processing via multi-task transfer learning. Our findings highlight the efficacy of multi-task transfer learning in symbolic music processing, particularly for data-scarce tasks, challenging the prevailing task-specific paradigms and offering a comprehensive dataset and framework for future explorations in this domain.

• The paper introduces MelodyT5, a Transformer-based encoder-decoder that employs multi-task pre-training to unify score-to-score symbolic music processing.
• It utilizes an innovative bar patching technique with ABC notation to effectively capture both global and local musical patterns.
• Experimental results show improved performance over task-specific models, enhancing harmonic compatibility and segmentation accuracy.

MelodyT5: A Unified Score-to-Score Transformer for Symbolic Music Processing

Abstract:

In the domain of symbolic music processing, the work by Wu et al. proposes MelodyT5, an encoder-decoder framework that aims to overcome the conventional constraints of task-specific models. By utilizing multi-task transfer learning on a newly curated dataset, MelodyHub, MelodyT5 integrates and addresses seven distinctive melody-centric tasks. This essay provides an expert overview of the paper, highlighting its methodology, findings, and implications, as well as its impact on the future development of symbolic music models.

Introduction:

Symbolic music processing, which involves the manipulation of notated music rather than continuous audio signals, presents unique challenges and opportunities within the field of AI. Historically, AI models in this domain have been task-specific, focusing on individual applications without leveraging the potential synergies across different tasks. The scarcity of annotated symbolic music datasets further compounds these challenges, limiting the generalizability and performance of AI models in this area.

Wu et al.'s MelodyT5 addresses these challenges by conceptualizing symbolic music tasks as score-to-score transformations, akin to the text-to-text framework in NLP. This paradigm shift allows for a unified approach to symbolic music processing, making use of multi-task transfer learning to improve performance, particularly in data-scarce tasks.

Methodology:

The core of MelodyT5 is its Transformer-based encoder-decoder architecture, enhanced by the bar patching technique to handle longer sequences efficiently. ABC notation is utilized for its concise representation of musical elements, facilitating the application of NLP techniques.

Data Representation:

MelodyT5 employs ABC notation to encode musical scores textually. To manage the complexity and length of sequences, the bar patching technique groups sequences into bar patches, enabling more efficient processing and preserving semantic coherence within the music.

Model Architecture:

The model architecture includes:

• Linear Projection: Converts bar patches into dense embeddings, providing input for the patch-level encoder.
• Patch-level Encoder: Generates contextualized representations using self-attention and feed-forward networks.
• Patch-level Decoder: Uses these representations for autoregressive generation of the next bar patch.
• Character-level Decoder: Produces detailed character sequences within bar patches, reconstructing the target musical score.

This hierarchical structure allows MelodyT5 to capture both global and local patterns in musical compositions.

Pre-training Objective:

The pre-training objective relies on cross-entropy loss, focusing on next token prediction. By minimizing cross-entropy loss across tokens in the target sequence, MelodyT5 learns to perform diverse symbolic music tasks under a unified framework optimized for score-to-score transformations.

Dataset:

MelodyHub, the dataset used for pre-training MelodyT5, includes 261,900 unique melodies across over one million task instances spanning seven tasks: generation, harmonization, melodization, segmentation, transcription, cataloging, and variation. These were meticulously curated from publicly available sources, ensuring high quality and uniformity.

Experiments:

Settings:

Experiments were conducted using MelodyHub, with data split into training and validation sets. MelodyT5's configuration included extensive parameterization, processing lengthy sequences and employing complex training protocols to ensure robust performance.

Ablation Studies:

Ablation studies revealed that multi-task pre-training significantly enhances model performance across various tasks. This was evident in lower bits-per-byte (BPB) scores and improved metrics compared to task-specific pre-training or no pre-training, demonstrating improved generalization and efficiency.

Comparative Evaluations:

Comparisons with task-specific models like TunesFormer, STHarm, CMT, and Bi-LSTM-CRF showed that MelodyT5 outperforms these baselines in most tasks. Objective metrics indicated superior performance in terms of controllability, harmonic compatibility, and segmentation accuracy. Subjective evaluations through A/B testing further validated MelodyT5's advantages in generation and harmonization, although CMT outperformed in melodization preferences, highlighting areas for future optimization.

Implications and Future Directions:

MelodyT5 signifies a substantial advancement in symbolic music processing by leveraging multi-task transfer learning. Its ability to generalize across different tasks without task-specific modifications points towards the potential for developing comprehensive, versatile music models.

Conclusion:

In conclusion, MelodyT5 presents a robust and unified framework for symbolic music processing, overcoming the traditional limitations of task-specific models through multi-task transfer learning. The curated MelodyHub dataset provides a rich resource for future research, enabling advancements across various melody-centric tasks. Further work is needed to enhance the model's performance, particularly in complex compositions, aligning AI-generated music more closely with human creative processes.