MGAudio: Flow-Based Video-Audio Generation

• MGAudio is a flow-based generative framework that converts silent video frames into realistic audio using dual-role alignment and a flow denoising transformer.
• It employs a Dual-Role Audio-Visual Encoder and a Flow-Based Denoising Transformer to ensure robust video and audio feature integration.
• The integrated Audio Model-Guidance objective replaces traditional classifier-free methods, boosting training efficiency and benchmark performance.

MGAudio is a flow-based generative framework for open-domain video-to-audio generation, distinguished by its model-guided dual-role alignment principle. Designed to improve cross-modal coherence and perceptual fidelity, MGAudio introduces an integrated architecture and objective that address limitations of previous classifier-based and classifier-free audio generation guidance methods. The system achieves state-of-the-art results on established benchmarks, demonstrating strong generalization and significant advances in audio realism and alignment.

1. Architectural Overview

MGAudio comprises three main modules: the Dual-Role Audio-Visual Encoder (DRAVE), the Flow-Based Denoising Transformer (FBDT), and the Audio Model-Guidance (AMG) objective.

• Input processing: Silent video frames $V$ and, during training, reference audio waveform $A$ are provided.
  • $A$ is converted to a log-mel spectrogram $X\in\mathbb{R}^{64\times816}$, then encoded via a pretrained AudioLDM VAE into $x_0\in\mathbb{R}^{8\times16\times204}$ and patchified into 816 tokens of dimension 768.
  • Video frames are mapped with a CAVP (Contrastive Audio-Visual Pretraining) video encoder to per-frame features $v\in\mathbb{R}^{L_V\times512}$, projected to 768 dimensions, and aggregated into a global vector $\bar{v}\in\mathbb{R}^{1\times768}$.
• FBDT: This Transformer U-Net operates in a flow-matching paradigm, learning a continuous invertible map from Gaussian noise $\epsilon$ to the VAE latent $x_0$, conditioned on the video representation $\bar{v}$ via Adaptive LayerNorm in each block.
• Output: At inference, MGAudio samples $A$0, reconstructs the denoised latent via reverse flow, decodes it through the VAE and a neural vocoder, yielding a waveform.

2. Dual-Role Alignment Mechanism

The DRAVE module enforces alignment in both audio–audio and video–audio domains while serving as the conditional path for video-driven generation.

• Alignment branch: Clean mel-spectrogram $A$1 is encoded by the CAVP audio encoder, generating reference features $A$2. Simultaneously, FBDT's intermediate features for noisy samples, $A$3, are mapped via an MLP $A$4 to ensure dimension compatibility.
• Audio alignment loss: For each patch $A$5,

  $A$6

This loss aligns intermediate noisy latent representations to pretrained audio features, accelerating convergence and improving semantic consistency.

• Video conditioning: The video feature $A$7 conditions the full backward flow, injected via Adaptive LayerNorm at every FBDT block.
• Cross-modal alignment: MGAudio leverages CAVP-pretrained audio–visual representations, without introducing separate $A$8 or contrastive losses, as cross-modal correspondence is inherently imposed by the dual-role encoder architecture.

3. Audio Model-Guidance Objective

MGAudio replaces classic classifier-free guidance (CFG) with AMG, avoiding the need for random condition dropout and conditional–unconditional model interpolation during inference.

• Let $A$9 denote the predicted flow with video condition, and $A$0 the unconditional flow.
• AMG target:

  $A$1

  with guidance scale $A$2 and $A$3 denoting stop-gradient.

• AMG loss:

  $A$4

• Total loss: The training objective is a weighted sum of AMG and alignment losses,

  $A$5

  where typically $A$6.

• This approach enables the model to internalize the guidance path, eliminating the inference-time CFG interpolation and yielding a unified, data-efficient conditional generator.

4. Training Regimen and Implementation Specifics

• Datasets: Main experiments are conducted on VGGSound (∼182k training, 15k test) and the open-domain UnAV-100 for zero-shot evaluation.
• Preprocessing: Audio is segmented (8.2s), mel-spectrogrammed, encoded by AudioLDM VAE into latents, and patchified. Video frames cover the same window and are processed for globally pooled visual context.
• Model architecture: MGAudio-base utilizes a 12-layer Transformer (hidden size 768, 12 heads, patch size 2), totaling 131M parameters. Scaled models (S/2 at 34M, L/2 at 464M, XL/2 at 680M) are also evaluated.
• Optimization: AdamW, learning rate $A$7, batch size 64, 1.1M training steps on a single NVIDIA A100, default guidance scale $A$8, small early-stabilization random drop probability on video condition ($A$9).
• Inference: Flow matching is performed with 50 Euler–Maruyama steps.

5. Evaluation Protocols and Quantitative Results

Performance is assessed using several standard and cross-modal metrics.

Method	Params	FAD ↓	FD ↓	IS ↑	KL ↓	Align Acc ↑
FRIEREN (CFG)	157 M	1.38	12.36	12.12	2.73	97.25%
MDSGen (CFG)	131 M	1.40	17.42	9.66	2.84	96.88%
MMAudio (CFG)	157 M	0.71	6.97	11.09	2.07	92.28%
MGAudio (MG+CFG=1.45)	131 M	0.40	6.16	12.82	2.76	95.65%

• FAD (Fréchet Audio Distance): MGAudio attains 0.40 on VGGSound, a substantial improvement over previous methods such as MMAudio (FAD=0.71) and FRIEREN (FAD=1.38).
• Other metrics: MGAudio also leads on FD (6.16) and IS (12.82), with competitive KL and alignment accuracy.
• UnAV-100 zero-shot: MGAudio generalizes with FAD=0.54, FD=5.40, IS=13.90, KL=2.00, Align Acc=97.54%, exceeding prior CFG-based models.
• Ablation findings: With only 10% of training data, MGAudio yields FAD=0.73, still competitive with full-data baselines.

6. Ablation Studies and Qualitative Insights

• Guidance mechanism comparison: AMG alone surpasses unguided models; AMG+CFG outperforms CFG alone. Dual-role alignment with CAVP encoders offers further accuracy gains over alternative feature encoders.
• Scalability: Increasing model size yields modest FAD gains (e.g., MGAudio-XL attains FAD≈0.9) but with diminishing returns.
• Feature alignment: CAVP encoders outperform CLAP or DINOv2 for alignment, supporting the selection of CAVP for the DRAVE branch.
• Qualitative evaluation: Mel-spectrogram visualizations demonstrate that MGAudio recovers richer harmonic information, sharper transients, and superior temporal alignment than comparator methods.
• Human studies: MGAudio achieves the highest realism and audio–video synchronization scores on human Likert-scale evaluations (mean 4.38, SD 0.39), outperforming FRIEREN and MMAudio.

7. Technical Contributions and Relevance

MGAudio's central advances are:

• Introduction of a flow-matching Transformer (FBDT) for denoising in latent space, supporting efficient high-fidelity audio synthesis.
• Dual-role alignment through the CAVP encoder, explicitly optimizing both conditioning and intermediate representation alignment, leading to improved cross-modal semantic fidelity.
• The AMG loss, a model-guided, self-distilling training objective that obviates the need for classifier-based guidance and offers favorable data efficiency.

These innovations yield superior generalization and sample quality on established open-domain video-to-audio benchmarks, presenting a scalable, training- and inference-efficient paradigm for conditional audio generation (Zhang et al., 28 Oct 2025).