Audio Prompt Adapter: Unleashing Music Editing Abilities for Text-to-Music with Lightweight Finetuning (2407.16564v2)

Abstract: Text-to-music models allow users to generate nearly realistic musical audio with textual commands. However, editing music audios remains challenging due to the conflicting desiderata of performing fine-grained alterations on the audio while maintaining a simple user interface. To address this challenge, we propose Audio Prompt Adapter (or AP-Adapter), a lightweight addition to pretrained text-to-music models. We utilize AudioMAE to extract features from the input audio, and construct attention-based adapters to feedthese features into the internal layers of AudioLDM2, a diffusion-based text-to-music model. With 22M trainable parameters, AP-Adapter empowers users to harness both global (e.g., genre and timbre) and local (e.g., melody) aspects of music, using the original audio and a short text as inputs. Through objective and subjective studies, we evaluate AP-Adapter on three tasks: timbre transfer, genre transfer, and accompaniment generation. Additionally, we demonstrate its effectiveness on out-of-domain audios containing unseen instruments during training.

Audio Prompt Adapter: Unleashing Music Editing Abilities for Text-to-Music with Lightweight Finetuning

The paper under discussion introduces "Audio Prompt Adapter (AP-Adapter)", a novel approach designed to bridge the gap in the text-to-music generation domain by enabling fine-grained musical audio editing through a lightweight addition to large pre-trained models. The impetus for this work stems from the challenge of maintaining detailed control over generated music while preserving an intuitive user interface. The authors propose an innovative solution that allows users to effectuate both global and local musical alterations using a combination of original audio inputs and textual commands.

Core Contributions

The paper makes several key contributions:

1. Framework Integration: The AP-Adapter framework integrates with existing pretrained models, specifically leveraging AudioLDM2, a latent diffusion-based text-to-audio model. By employing AudioMAE for feature extraction, the AP-Adapter facilitates seamless integration of audio and text prompts within the generating model.
2. Lightweight Architecture: The proposed solution is lightweight, adding only 22 million trainable parameters, which is practical for deployment on systems with limited computational resources. Through decoupled cross-attention adapters, the framework achieves precise control in music generation, supporting detailed edits that align with user inputs.
3. Zero-shot Music Editing: One of the noteworthy claims is the framework's capacity to achieve effective zero-shot music editing, offering users the flexibility to manipulate music without extensive parameter tuning or additional training overhead.
4. Task-specific Applications: The paper extensively evaluates the AP-Adapter on various tasks, including timbre transfer, genre transfer, and accompaniment generation. These tasks showcase the adaptability and comprehensiveness of the framework in handling diverse music-editing requirements.

Evaluation and Results

Through a rigorous experimental setup, the authors present both objective metrics and subjective evaluation to validate the effectiveness of their approach. The paper contrasts the performance of AP-Adapter with that of MusicGen and SDEdit-enhanced AudioLDM2 across several parameters:

• Transferability: Evaluated using CLAP cosine similarity, the AP-Adapter maintains competitive scores indicating efficient alignment of the generated audio with textual prompts.
• Fidelity: Chroma similarity metrics suggest that AP-Adapter preserves the harmonic structures and rhythmic patterns of the original inputs well.
• Overall Audio Quality: Fréchet audio distance (FAD) scores reflect the generated audio's resemblance to real music, with AP-Adapter consistently producing high-quality results.

Subjective evaluations via a series of listening tests further affirm the AP-Adapter's superiority in achieving high transferability and fidelity concurrently. Compared to the baselines, participants rated AP-Adapter significantly higher in terms of overall preference and specific attributes of transferability and fidelity across various editing tasks.

Practical and Theoretical Implications

Practically, the AP-Adapter equips musicians and music producers with a potent tool for creative audio manipulation, supporting intricate musical edits that can enhance the human-AI co-creation process. The lightweight nature reduces the barrier to deployment, making it feasible for broader adoption without needing extensive computational resources.

Theoretically, the framework opens up promising avenues for future research. Potential extensions include exploring more diverse editing tasks, integrating with other generative model architectures like autoregressive models, and enhancing capabilities to support localized edits seamlessly. By allowing controlled manipulation of audio inputs using textual prompts, the AP-Adapter sets a precedent for future advancements in the field of music generation and editing.

In conclusion, the AP-Adapter represents a notable advancement in text-to-music generation, offering a pragmatic and efficient solution to the intricate challenge of music editing. The proposed framework's ability to balance detailed audio fidelity with the flexibility of text-driven commands marks a significant step towards more intuitive and powerful music generation tools.