Accompanied Singing Voice Synthesis with Fully Text-controlled Melody (2407.02049v1)

Abstract: Text-to-song (TTSong) is a music generation task that synthesizes accompanied singing voices. Current TTSong methods, inherited from singing voice synthesis (SVS), require melody-related information that can sometimes be impractical, such as music scores or MIDI sequences. We present MelodyLM, the first TTSong model that generates high-quality song pieces with fully text-controlled melodies, achieving minimal user requirements and maximum control flexibility. MelodyLM explicitly models MIDI as the intermediate melody-related feature and sequentially generates vocal tracks in a LLM manner, conditioned on textual and vocal prompts. The accompaniment music is subsequently synthesized by a latent diffusion model with hybrid conditioning for temporal alignment. With minimal requirements, users only need to input lyrics and a reference voice to synthesize a song sample. For full control, just input textual prompts or even directly input MIDI. Experimental results indicate that MelodyLM achieves superior performance in terms of both objective and subjective metrics. Audio samples are available at https://melodylm666.github.io.

• The paper introduces MelodyLM, a model that generates accompanied singing voices using fully text-controlled melody, achieving high key accuracy and reduced pitch discrepancies.
• It employs a three-stage framework—text-to-MIDI, text-to-vocal with multi-scale transformers, and vocal-to-accompaniment via latent diffusion models—to align vocals and instruments effectively.
• Evaluations demonstrate improved F0 frame error and prosody quality compared to baselines, making singing voice synthesis more accessible and versatile.

Overview of "Accompanied Singing Voice Synthesis with Fully Text-controlled Melody"

The paper introduces MelodyLM, a novel approach for Text-to-Song (TTSong) synthesis that emphasizes completely text-controlled melody generation for accompanied singing voice. Unlike existing methods which rely heavily on musical scores or MIDI sequences, MelodyLM sets itself apart by minimizing user requirements and maximizing control flexibility. This model represents a significant step towards making singing voice synthesis (SVS) more accessible and versatile.

MelodyLM employs a multi-stage process: initially, it models MIDI sequences as intermediate features, then sequentially generates vocal tracks conditioned on both textual and vocal prompts, and finally synthesizes accompaniment music using latent diffusion models enhanced with hybrid conditioning for temporal alignment. The approach allows the user to input merely lyrics and a reference voice to create a song, with optional inputs for enhanced control.

Methodology and Framework

MelodyLM operates through a three-stage framework:

1. Text-to-MIDI: MIDI note sequences are generated conditioned on text prompts. This step introduces melody-related features without needing prior musical scores.
2. Text-to-Vocal: Using a LLM framework, detailed in a multi-scale transformer setup, the model generates vocal tracks that align with both the MIDI and the textual inputs.
3. Vocal-to-Accompaniment: Accompaniment music is synthesized via a latent diffusion model that aligns the generated vocals with instrumental tracks.

The paper leverages significant advancements in LLMs (LMs) and introduces MIDI sequences as an intermediate feature to bridge the gap between textual descriptions and music generation. Additionally, MelodyLM harnesses the power of both pre-existing music and lyrics to influence its generative process.

Evaluation and Results

The experiments conducted demonstrate MelodyLM's superiority over existing baselines in both objective and subjective evaluations. Specifically, for MIDI generation, MelodyLM achieved a high key accuracy rate and significantly reduced pitch and duration discrepancies compared to other models. The singing voice synthesis showed improvements in F0 frame error, and subjectively, listeners rated the generated singing voices favorably in terms of prosody and quality, albeit with some challenges in singer similarity due to the data's variable quality.

When tested on generating both vocals and accompaniment, MelodyLM outperformed existing models such as Melodist, particularly when melding vocals with instruments, showing high alignment with the intended melody and text prompts.

Implications and Future Directions

The implications of this research are manifold, both from practical and theoretical perspectives. Practically, MelodyLM's framework reduces the barriers for creating synthesized music, making it more accessible for users with limited musical training. Theoretically, it opens new pathways for exploring how LLMs can interface with other models to produce coherent, high-quality artistic outputs.

Future efforts could expand the model's training dataset to include a broader variety of music styles beyond Mandarin pop, facilitating more universal applications. Additionally, streamlining the multi-stage process into a more cohesive end-to-end framework might improve efficiency and reduce complexity.

In conclusion, MelodyLM showcases promising capabilities in synthesizing high-quality accompanied songs with flexible user inputs facilitated by contemporary language and diffusion models, staking out new territory in SVS research and application.