UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions (2511.03334v1)

Abstract: Due to the lack of effective cross-modal modeling, existing open-source audio-video generation methods often exhibit compromised lip synchronization and insufficient semantic consistency. To mitigate these drawbacks, we propose UniAVGen, a unified framework for joint audio and video generation. UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which enables bidirectional, temporally aligned cross-attention, thus ensuring precise spatiotemporal synchronization and semantic consistency. Furthermore, this cross-modal interaction is augmented by a Face-Aware Modulation module, which dynamically prioritizes salient regions in the interaction process. To enhance generative fidelity during inference, we additionally introduce Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint synthesis design enables seamless unification of pivotal audio-video tasks within a single model, such as joint audio-video generation and continuation, video-to-audio dubbing, and audio-driven video synthesis. Comprehensive experiments validate that, with far fewer training samples (1.3M vs. 30.1M), UniAVGen delivers overall advantages in audio-video synchronization, timbre consistency, and emotion consistency.

• The paper introduces a dual-branch model with asymmetric cross-modal interaction that enhances lip synchronization and semantic alignment.
• It employs face-aware modulation and modality-aware classifier-free guidance to boost emotional intensity and motion dynamics.
• The framework achieves state-of-the-art performance with minimal training data, demonstrating robust generalization across diverse tasks.

UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions

Introduction

UniAVGen presents a unified framework for joint audio-video generation, addressing persistent limitations in open-source multi-modal synthesis: compromised lip synchronization, insufficient semantic consistency, and poor generalization. The architecture is anchored in a dual-branch design, leveraging parallel Diffusion Transformers (DiTs) for audio and video streams, and introduces an asymmetric cross-modal interaction mechanism to achieve precise spatiotemporal synchronization and semantic alignment. The framework is further augmented by a face-aware modulation module and a modality-aware classifier-free guidance strategy, enabling robust performance across diverse tasks such as joint generation, dubbing, and audio-driven video synthesis.

Figure 1: Multi-task compatibility of UniAVGen, supporting joint generation, dubbing, and audio-driven synthesis within a single model.

Architecture and Core Mechanisms

Dual-Branch Joint Synthesis

UniAVGen employs a symmetric dual-branch architecture, with each branch instantiated as a DiT. The video branch utilizes the Wan 2.2-5B backbone, while the audio branch adopts the Wan 2.1-1.3B template, ensuring architectural parity and facilitating a cohesive latent space for cross-modal interactions. Both branches operate in the latent domain: video is encoded via a VAE at 16 fps, and audio is represented as Mel spectrograms sampled at 24 kHz.

Figure 2: Architecture of UniAVGen, illustrating the dual-branch design and asymmetric cross-modal interaction, with face-aware modulation for region-specific attention.

Asymmetric Cross-Modal Interaction

The core innovation is the asymmetric cross-modal interaction (ATI) mechanism, which supersedes prior global or symmetric time-aligned designs. ATI comprises two specialized aligners:

• Audio-to-Video (A2V) Aligner: Injects fine-grained audio semantics into the video stream using contextualized audio windows for each video frame, enabling precise lip synchronization and emotional alignment.
• Video-to-Audio (V2A) Aligner: Embeds visual identity and temporal dynamics into the audio stream via temporal neighbor interpolation, ensuring smooth transitions and robust timbre consistency.

Both aligners employ frame-wise cross-attention, with output matrices zero-initialized to prevent early-stage interference.

Figure 3: Comparison of cross-modal interaction mechanisms, highlighting the superior convergence and performance of the asymmetric design.

Face-Aware Modulation (FAM)

FAM dynamically constrains cross-modal interaction to facial regions, mitigating spurious correlations and stabilizing training. A mask-prediction head supervises the localization of salient regions, with a decaying loss weight to gradually relax constraints and enhance flexibility. The mask guides selective updates in A2V and modulates video features in V2A, improving both convergence and alignment.

Figure 4: Visual comparisons of predicted masks with fixed and decaying $\lambda_m$, demonstrating effective facial region localization and adaptive scope.

Modality-Aware Classifier-Free Guidance (MA-CFG)

Traditional classifier-free guidance (CFG) is unimodal and fails to amplify cross-modal dependencies. MA-CFG introduces a shared unconditional estimate for both modalities, explicitly strengthening cross-modal correlation signals during inference. This results in enhanced emotional intensity and motion dynamics, improving overall realism.

Figure 5: Visual comparisons of joint generation with and without MA-CFG, showing improved emotional and motion alignment with MA-CFG.

Multi-Task Unification

UniAVGen's design supports seamless adaptation to multiple tasks:

• Joint Audio-Video Generation: Default mode, generating aligned audio and video from text and reference image.
• Joint Generation with Reference Audio: Controls speaker timbre via custom reference audio.
• Audio-Video Continuation: Conditions on existing audio and video for temporal continuity.
• Video-to-Audio Dubbing: Generates emotion- and expression-aligned audio from silent video.
• Audio-Driven Video Synthesis: Produces motion-aligned video from audio input.

(Figure 1, revisited)

Figure 1: Multi-task compatibility enables unified handling of diverse audio-video generation tasks.

Experimental Results

Quantitative Evaluation

UniAVGen is trained with only 1.3M joint samples, significantly fewer than competing models (e.g., Ovi: 30.7M). Evaluation spans three dimensions: audio quality (PQ, CU, WER), video quality (SC, DD, IQ), and audio-video consistency (LS, TC, EC). UniAVGen achieves:

• Audio Quality: PQ=7.00, CU=6.62, WER=0.151, outperforming all joint generation baselines and matching two-stage pipelines.
• Video Quality: DD=0.410, IQ=0.779, with strong subject consistency.
• Audio-Video Consistency: LS=5.95, TC=0.832, EC=0.573, demonstrating superior lip-sync, timbre, and emotion alignment.

Qualitative Evaluation

Visual comparisons reveal that UniAVGen generates high-fidelity, well-aligned audio-video pairs for both in-distribution (real human) and out-of-distribution (anime) inputs, outperforming Ovi and UniVerse-1 in generalization and semantic coherence.

Figure 6: Visual comparisons against Ovi and UniVerse-1, showing UniAVGen's superior alignment and generalization, especially for OOD anime inputs.

Ablation Studies

Ablations confirm the efficacy of ATI, FAM, and MA-CFG:

• ATI: Outperforms symmetric and global interaction in LS, TC, and EC metrics.
• FAM: Supervised, decaying mask yields best consistency scores.
• MA-CFG: Enhances emotional and motion alignment in joint generation.

  Figure 7: Training strategy comparisons, demonstrating the benefits of joint generation pre-training followed by multi-task learning.

Implementation Considerations

• Training: Three-stage protocol—audio branch pre-training, joint optimization, and multi-task fine-tuning. AdamW optimizer, batch sizes of 256 (audio) and 32 (joint), with linear decay of mask supervision.
• Inference: MA-CFG requires dual forward passes for unconditional and cross-modal estimates.
• Resource Requirements: 7.1B parameters, efficient convergence due to architectural symmetry and targeted interaction.
• Limitations: While generalization to OOD inputs is strong, further scaling and domain adaptation may be required for highly diverse visual/audio domains.

Implications and Future Directions

UniAVGen establishes a new benchmark for unified audio-video generation, demonstrating that modality-specific, temporally aligned cross-modal interaction is critical for semantic consistency and generalization. The framework's extensibility to multiple tasks within a single model has practical implications for content creation, dubbing, and avatar synthesis. Future work may explore scaling to larger datasets, domain adaptation for non-human subjects, and integration with real-time generation pipelines.

Conclusion

UniAVGen introduces a unified, efficient, and generalizable framework for joint audio-video generation, anchored in asymmetric cross-modal interaction and region-specific modulation. The model achieves state-of-the-art performance in synchronization, timbre, and emotion consistency, with strong generalization across domains and tasks. These advances pave the way for practical, versatile multi-modal generative systems in both research and industry.