Mozart's Touch: A Lightweight Multi-modal Music Generation Framework Based on Pre-Trained Large Models

Abstract: In recent years, AI-Generated Content (AIGC) has witnessed rapid advancements, facilitating the creation of music, images, and other artistic forms across a wide range of industries. However, current models for image- and video-to-music synthesis struggle to capture the nuanced emotions and atmosphere conveyed by visual content. To fill this gap, we propose Mozart's Touch, a multi-modal music generation framework capable of generating music aligned with cross-modal inputs such as images, videos, and text. The framework consists of three key components: Multi-modal Captioning Module, LLM understanding & Bridging Module, and Music Generation Module. Unlike traditional end-to-end methods, Mozart's Touch uses LLMs to accurately interpret visual elements without requiring the training or fine-tuning of music generation models, providing efficiency and transparency through clear, interpretable prompts. We also introduce the "LLM-Bridge" method to resolve the heterogeneous representation challenges between descriptive texts from different modalities. Through a series of objective and subjective evaluations, we demonstrate that Mozart's Touch outperforms current state-of-the-art models. Our code and examples are available at https://github.com/TiffanyBlews/MozartsTouch.

• The paper introduces Mozart's Touch, a framework that integrates pre-trained large models to convert diverse inputs into musically coherent outputs.
• The method employs a three-module architecture—multi-modal captioning, LLM bridging, and music generation—to align cross-modal descriptions with audio synthesis.
• Experimental evaluations demonstrate that the approach outperforms models like CoDi and M2UGen, achieving superior metrics such as FAD, KL divergence, and IB Rank.

Overview of "Mozart's Touch: A Lightweight Multi-modal Music Generation Framework Based on Pre-Trained Large Models"

In the paper titled "Mozart's Touch: A Lightweight Multi-modal Music Generation Framework Based on Pre-Trained Large Models," the authors address the challenges of generating multi-modal music content through a novel integration and framework. The research presented in this paper contributes to the burgeoning field of AI-Generated Content (AIGC), emphasizing the need for a system that efficiently synthesizes music from diverse inputs such as images and videos.

The core contribution of this paper lies in the development of "Mozart's Touch," a framework that combines LLMs with pre-trained models for music generation, without necessitating additional training or fine-tuning. This characteristic highlights its lightweight design and adaptability. The framework consists of three integral modules: a Multi-modal Captioning Module, an LLM Understanding Bridging Module, and a Music Generation Module. Together, these components work to convert cross-modal inputs into semantically aligned musical outputs.

Key Components and Methodology

• Multi-modal Captioning Module: This module extracts descriptive information from various input modalities (images, videos, and text) using advanced techniques like ViT and BLIP. This capability sets the groundwork for an enriched understanding of the input content.
• LLM Understanding Bridging Module: A distinctive feature of this module is its ability to bridge the heterogeneous disparity between multi-modal descriptions and the requirements of music generation models. By leveraging the interpretive capacity of LLMs, it enhances the relevance and coherence of the prompts used in subsequent music generation, effectively aligning visual, textual, and musical vocabularies.
• Music Generation Module: The framework employs MusicGen-medium, a pre-trained model, to translate the refined prompts into musical output, capturing the essence of the input imagery or video content.

Experimental Evaluation

The experimental results demonstrated that "Mozart's Touch" markedly outperforms existing models such as CoDi and M2UGen in both image-to-music and video-to-music tasks. Objective metrics like Frechet Audio Distance (FAD), Kullback-Leibler divergence (KL), and ImageBind Rank (IB Rank) underscore its superior alignment and quality of generated music. The subjective evaluation further supports the framework's capability to generate contextually relevant and high-quality music.

Implications and Future Work

The implications of this research are significant, offering pathways for more integrated and responsive multimedia creative tools. By simplifying the model's architecture while enhancing its efficacy, "Mozart's Touch" presents a viable candidate as a baseline for future developments in multi-modal AIGC systems. The paper suggests that the proposed framework not only addresses current methodological gaps but also conserves computational resources—an aspect that holds substantial promise for broadening the accessibility of such technologies.

Looking forward, potential areas for further research include refining the LLM bridging strategies and exploring additional modalities beyond the current scope. Moreover, investigation into adaptive prompting strategies could further enhance the congruence between multi-modal inputs and music outputs.

In conclusion, the paper presents a substantiated advancement in multi-modal music generation, harnessing the power of large pre-trained models and strategic use of LLMs to produce musically coherent outputs from diverse inputs, all while upholding efficiency and usability.