MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (2412.15322v2)

Abstract: We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio. In contrast to single-modality training conditioned on (limited) video data only, MMAudio is jointly trained with larger-scale, readily available text-audio data to learn to generate semantically aligned high-quality audio samples. Additionally, we improve audio-visual synchrony with a conditional synchronization module that aligns video conditions with audio latents at the frame level. Trained with a flow matching objective, MMAudio achieves new video-to-audio state-of-the-art among public models in terms of audio quality, semantic alignment, and audio-visual synchronization, while having a low inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio also achieves surprisingly competitive performance in text-to-audio generation, showing that joint training does not hinder single-modality performance. Code and demo are available at: https://hkchengrex.github.io/MMAudio

• The paper introduces MMAudio, a framework that leverages multimodal data from text and video to produce high-quality, synchronized audio outputs.
• The paper details a conditional synchronization module that improves frame-level audio-video alignment using high frame-rate visual features and self-supervised detection.
• The paper demonstrates state-of-the-art performance with reduced inference time and a compact model size, highlighting its potential for immersive media applications.

Essay on "Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis"

The paper "Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis" presents a novel approach in the domain of video-to-audio synthesis, focusing on the synthesis of high-quality and synchronized audio contextual to provided video or text inputs. The authors introduce MMAudio, a multimodal joint training framework that capitalizes on large-scale, text-audio data to enhance the semantic and temporal alignment of generated audio with corresponding video inputs.

Key Contributions and Methodology

1. Multimodal Joint Training Framework: Unlike traditional single-modality training approaches which primarily rely on limited video data, MMAudio is trained using a vast array of text-audio datasets in conjunction with existing audio-visual datasets. This joint training is facilitated through an end-to-end framework employing a single transformer network, allowing the model to harness a unified semantic space beneficial for learning the distribution of natural audio.
2. Conditional Synchronization Module: To improve temporal alignment—critical in video-to-audio synthesis—the authors introduce a conditional synchronization module. This module utilizes high frame-rate visual features, extracted from a self-supervised audio-visual desynchronization detector, to provide frame-level synchronization. This method addresses the limitations identified in using attention layers alone for indicating precise timings.
3. Performance and Efficiency: The proposed method, MMAudio, achieves state-of-the-art results in video-to-audio synthesis, in terms of audio quality, semantic alignment, and audio-visual synchronization, all while maintaining low inference time and parameter count. Impressively, MMAudio can generate an 8-second audio clip in just 1.23 seconds and functions efficiently with a model size as small as 157 million parameters.
4. Text-to-Audio Generation: Although primarily a video-to-audio framework, MMAudio demonstrates competitive performance in text-to-audio generation tasks as well, highlighting the efficacy of the joint training paradigm in preserving single-modality capabilities.

The empirical results indicate that MMAudio significantly reduces the Fréchet distance and improves Inception and synchronization scores. These improvements are attributed to the extensive multimodal data used during training which allows the network to learn complex cross-modal interactions effectively.

Implications and Future Work

The practical implications of MMAudio extend to applications requiring high-quality audio synthesis aligned with visual media, such as in film post-production, game development, and virtual simulations, where realistic audio-visual synchrony contributes significantly to user immersion.

From a theoretical perspective, this work reinforces the potential of multimodal training in overcoming the data paucity issues commonly faced in single-modality tasks. By demonstrating that joint training on diverse datasets does not detract from, but rather enhances, the model's performance across multiple modalities, the paper paves the way for further exploration in multimodal AI.

Future research directions may involve integrating even more diverse data sources to enhance the robustness of the model across varied real-world scenarios. Additionally, exploring advancements in hardware acceleration and optimization of transformer architectures could further enhance the efficiency and scalability of such models.

In conclusion, the paper makes a compelling case for the use of a multimodal joint training framework in video-to-audio synthesis. By leveraging large datasets across modalities, the authors introduce a method that not only sets new benchmarks in audio quality and synchrony but also hints at broader implications for future multimodal research endeavors.