Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation

Abstract: Motion, speech, and sound effects are fundamental elements of human-centric videos, yet their heterogeneous temporal characteristics make joint generation highly challenging. Existing audio-video generation models often fail to maintain consistent alignment across these modalities, leading to noticeable mismatches between motion, speech, and environmental sounds. We present Unison, a unified framework that explicitly promotes coherence across the motion, speech, and sound modalities. Within the audio stream, Unison employs a semantic-guided harmonization strategy that decouples the generation of speech and sound-effect components. Leveraging bidirectional audio cross-attention and semantic-conditioned gating for semantic-driven adaptive recomposition, this approach effectively mitigates speech dominance and enhances acoustic clarity. For audio-motion synchronization, we propose a bidirectional cross-modal forcing strategy where the cleaner modality guides the noisier one through decoupled denoising schedules, reinforced by a progressive stabilization strategy. Extensive experiments demonstrate that Unison achieves state-of-the-art performance in both audio perceptual quality and cross-modal synchronization, highlighting the importance of explicit multimodal harmonization in human-centric video generation.

• The paper's main contribution is a unified framework that decouples and harmonizes speech and sound using modules like Zipformer, Bi-ACA, and SCG.
• It employs a curriculum-based bidirectional cross-modal forcing strategy to stabilize denoising trajectories and enhance audio-video synchronization.
• Experimental results on 2 million clips show state-of-the-art perceptual quality (PQ=6.34) and reduced modal interference, outperforming several baselines.

Unison: A Unified Framework for Harmonized Motion, Speech, and Sound in Human-Centric Audio-Video Generation

Introduction and Motivation

The synthesis of temporally coherent and perceptually harmonious audio-visual content presents persistent challenges in human-centric generative modeling. Existing text-to-audio-video diffusion systems typically exhibit a critical decoupling between motion, speech, and environmental sound, leading to intra-audio interference (e.g., speech masking effects) and prominent cross-modal desynchronization. Unison addresses these limitations by introducing explicit architectural and training mechanisms that jointly harmonize heterogeneous temporal and semantic signals across motion, speech, and sound effects (2605.08729).

Semantic-Guided Audio Harmonization

A core innovation of Unison is its semantic-guided harmonization strategy for the audio branch. Standard approaches fuse all sonic elements in a monolithic pathway, causing speech to dominate and obscure subtle non-speech cues. Unison systematically decouples speech and sound-effect streams at the representation level by integrating Zipformer for high-fidelity speech and employing a Bidirectional Audio Cross-Attention (Bi-ACA) mechanism for mutual context exchange.

To adapt contextual dominance based on scene semantics, a Semantic-Conditioned Gating (SCG) module selectively modulates the interaction between speech and SFX streams. SCG computes content-aware gating coefficients from text and transcription embeddings, dynamically suppressing or amplifying interaction channels depending on whether narration or ambient events should take precedence. This ensures both phonetic purity in speech-dominant contexts and acoustic richness when environmental sound is perceptually salient. Ground truth supervision for both streams is enforced via independent flow-matching losses, eliminating ambiguities and supporting robust disentanglement.

Bidirectional Cross-Modal Forcing for Synchronization

Unison advances beyond symmetric cross-attention fusion by introducing bidirectional cross-modal forcing. Unlike rigidly synchronized denoising schedules, Unison samples independent diffusion steps for both audio and video branches. The model then upweights the loss for the modality currently at higher noise, explicitly conditioning its denoising trajectory on the semantic structure of the cleaner counterpart. This curriculum-driven strategy facilitates robust learning of cross-modal temporal dependencies and allows the system to avoid both exposure bias and modal lag effects that commonly degrade multimodal alignment.

The training regime further employs a staged curriculum, transitioning from fully synchronous updates to partial and then fully independent schedules, with loss reweighting governed by denoising directionality and capped noise disparity. This progressive relaxation stabilizes optimization and progressively enhances inter-modal synchrony, as evidenced in both ablation and qualitative video-audio correspondence analyses.

Experimental Results

Unison is trained on nearly 2 million audio-visual clips and 50 million high-fidelity audio segments. Quantitative evaluation on a curated test set demonstrates that Unison outperforms all relevant baselines—Universe-1, Ovi, UniAVGen, MOVA, LTX-2, JavisDiT—on perceptual quality (PQ), content utility (CU), audio-visual (AV) semantic alignment, and temporal correspondence (DS). Specifically, Unison achieves a PQ of 6.34 and a WER of 0.22, improving over LTX-2 despite a 4x smaller video backbone. In ablation, the removal of semantic harmonization and cross-modal forcing modules results in marked degradation of perceptual and synchronization metrics, validating the modular efficacy of the proposed strategies.

Notably, Unison supports bidirectional translation (A2V/V2A), with qualitative outputs reflecting tight synchronization of visual gestures (e.g., lip movements, instrumental transients) with time-aligned acoustic events, and aural outputs that faithfully mirror complex visual cues. The semantic gating mechanism is empirically shown to adapt content balance dynamically, prioritizing speech purity or environmental richness according to scenario.

Implications and Theoretical Perspectives

Unison demonstrates that rigorous disentanglement and adaptive recomposition of audio streams, coupled with cross-modal, direction-aware forcing, are essential for scalable, high-fidelity multimodal generation. These design principles directly address key limitations in both open-source and commercial generative frameworks, where lack of explicit modality coordination results in degraded realism and immersion.

The bidirectional, curriculum-structured training opens avenues for more flexible conditioning mechanisms in general multimodal architectures, including future text-audio-video-music integration, and cross-modal style or affect transfer. The demonstrated improvements in perceptual quality and synchronization hold promise for high-stakes applications such as digital humans, performance synthesis, and corpus-based simulation scenarios, where perceptual coherence is non-negotiable.

Future Directions

Prospective developments include the integration of large-scale, long-context temporal modeling capabilities to sustain synchrony over extended sequences, multi-agent conversational generation, and extension to more complex human-scene interactions with open-vocabulary event tracking. The semantic-conditioned harmonization approach also lends itself to controlling scene acoustic style and emphasis, which is highly relevant for interactive and controllable generative systems.

Conclusion

Unison establishes a new benchmark for harmonized audio-video synthesis by resolving motion, speech, and ambient sound misalignments through explicit decoupling, semantic context-aware gating, and bidirectional cross-modal forcing. The robust, modular design achieves state-of-the-art results in perceptual quality and synchronization, providing key mechanistic insights for the further advancement of human-centric generative models (2605.08729).