JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion

Abstract: Audio-Visual Foundation Models, which are pretrained to jointly generate sound and visual content, have recently shown an unprecedented ability to model multi-modal generation and editing, opening new opportunities for downstream tasks. Among these tasks, video dubbing could greatly benefit from such priors, yet most existing solutions still rely on complex, task-specific pipelines that struggle in real-world settings. In this work, we introduce a single-model approach that adapts a foundational audio-video diffusion model for video-to-video dubbing via a lightweight LoRA. The LoRA enables the model to condition on an input audio-video while jointly generating translated audio and synchronized facial motion. To train this LoRA, we leverage the generative model itself to synthesize paired multilingual videos of the same speaker. Specifically, we generate multilingual videos with language switches within a single clip, and then inpaint the face and audio in each half to match the language of the other half. By leveraging the rich generative prior of the audio-visual model, our approach preserves speaker identity and lip synchronization while remaining robust to complex motion and real-world dynamics. We demonstrate that our approach produces high-quality dubbed videos with improved visual fidelity, lip synchronization, and robustness compared to existing dubbing pipelines.

• The paper introduces a joint audio-visual diffusion model that unifies dubbing into a single generative process, alleviating errors from modular pipelines.
• It employs a dual-stream encoder and IC-LoRA with structured attention masks to ensure precise lip synchronization and identity preservation.
• Experimental results show 100% generation success and superior audiovisual fidelity in complex real-world scenarios, surpassing traditional methods.

Joint Audio-Visual Diffusion for Robust Video Dubbing

Motivation and Technical Challenges

The paper "JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion" (2601.22143) formulates video dubbing as a single-model, joint generative problem, departing from disjointed modular pipelines prevalent in existing work. Traditional dubbing approaches involve several specialized components, including audio translation, voice conversion, facial motion retargeting, cue-based audio-visual alignment, and post-hoc mixing of contextual sound events. This staged paradigm introduces brittleness: errors or misalignments in any component degrade the holistic quality, especially in unconstrained, real-world videos with complex motion, occlusions, and paralinguistic scene interactions.

A primary technical challenge in end-to-end joint modeling is the absence of paired, temporally aligned multilingual video data retaining identical actor identity, pose, and environment. Further, the tension between preserving speaker identity and rendering target-language pronunciation accurately imposes trade-offs. The risk of context leakage when performing spatially localized editing (e.g., inpainting of the lip region) in the presence of high-capacity VAEs compounds this challenge.

Methodology

The approach builds upon LTX-2, a foundation audio-visual diffusion transformer with dual-stream encoders and temporally aligned, bidirectional cross-attention. This backbone is adapted using In-Context LoRA (IC-LoRA), where only a lightweight set of LoRA adapters are optimized while keeping the generative prior frozen. The flow-matching training regime is preserved; the model is tasked with generating the delta between noise and data in both the video and audio streams, conditioned on high-level textual prompts that specify the target language dialogue. Notably, translation is assumed to be handled outside the model; the prompt directly contains the full translated text.

To circumvent the absence of paired data, the authors synthesize training pairs by leveraging the generative capacity of the foundation model: multilingual clips are generated featuring a speaker switching languages mid-scene, and the video is split and inpainted to produce context-paired sequences with cross-lingual alignment. Latent-Aware Fine Masking is introduced to prevent leakage through the VAE’s receptive field, ensuring that the masked region is sufficiently broad to erase all latent traces of the original lip motion.

Figure 1: The pipeline synthesizes paired audio-visual dubbing data using generation and inpainting to provide context-aligned bilingual supervision.

The methodology also introduces lip augmentation prompts which perturb the viseme distribution, increasing the expressivity and diversity of mouth movements in the synthetic training set. These strategies collectively regularize the model to avoid degenerate identity-copying or "mumbling" lip reconstructions and enable visually and acoustically rich outputs.

The fine-tuning protocol maintains strict modality-isolated cross-attention, employing structured attention masks that prevent noisy signals in one stream (e.g., masked audio) from corrupting context in the other stream (e.g., uncorrupted visual regions). Shared positional encodings further ensure that context and target tokens are precisely aligned at an intra-modal level.

Figure 2: The model is trained with paired noisy/clean AV samples using in-context learning, with LoRA adapters and modality-masked cross-attention guaranteeing robust, context-grounded edits.

Experimental Results

The empirical evaluation spans both well-curated datasets (HDFT, TalkVid) with controlled conditions and challenging, unconstrained benchmarks curated from the wild (profile views, heavy occlusions, stylized/non-human faces, complex backgrounds). Quantitative metrics encompass generation success rate, identity preservation, visual fidelity (FID), temporal coherence (FVD), mouth motion diversity (MAR), and frame-accurate AV synchronization, as well as audio metrics (duration error, voice similarity, intensity correlation, WER).

• On challenging real-world data, the proposed joint model achieves a 100% generation success rate versus 74--80% for modular baselines, which frequently fail under pose, occlusion, or stylization shifts.
• The model achieves strong results in identity preservation (CSIM), improved expressiveness (higher MAR), and state-of-the-art FVD, evidencing coherent and temporally sharp video synthesis.
• Audio alignment is precise, with minimal duration error and synchronization within 1--2 frames, outperforming CosyVoice and OpenVoice in both alignment and intensity correlation. The unified framework avoids common modular artifacts such as rolling back motion or temporal mismatches.

  Figure 3: User study shows clear preference for joint model over LatentSync and HeyGen regarding lip sync, prompt adherence, and holistic quality.

Ablation studies confirm that both the LoRA adapters and lip augmentation in training are crucial; removing either drives the model toward degenerate copy-paste behavior or degraded language rendering accuracy.

Qualitative results highlight several robust properties:

• The model gracefully handles non-frontal poses, occlusions, and stylized/non-human faces which defeat mask-based pipelines.
• Temporal structure (e.g., the synchronization of a dog bark or chewing with emitted sound) remains semantically intact, even when dialogue durations vary cross-lingually.
• Non-dialogue acoustic events are preserved and correctly grounded in the scene (e.g., laughter, sighing), in contrast with modular pipelines that often erase or desynchronize such events.

  Figure 4: In complex scenes, non-verbal cues and scene-grounded events are preserved through holistic joint generation.

Implications and Future Directions

This work empirically demonstrates the superiority of single unified generative models for tightly coupled multimodal tasks like dubbing. It addresses the core technical limitations of modularization: brittle interfaces, error propagation, and loss of temporal or semantic alignment across modalities. The success on unconstrained and challenging data suggests joint AV priors learned over large, diverse generative datasets can be robust to a much broader array of real-world conditions.

Practically, this approach enables more natural and immersive cross-lingual video experiences, especially in settings where environmental and speaker interactions are complex. The retention of subtle paralinguistic cues (e.g., breathing, interleaved environ-mental sounds) directly augments downstream applications in entertainment, accessibility, and communication.

Theoretically, the results invite future research into larger-context, multi-turn conversational generation, tighter disentanglement of voice identity from underlying phonetic realization, explicit speaker supervision, and direct modeling of translation within the joint diffusion backbone. Extending to arbitrary scene durations, group dialogue, and fine-grained style control (prosody, emotional tone) within a holistic framework are natural next steps.

Conclusion

"JUST-DUB-IT" (2601.22143) establishes joint audio-visual diffusion with minimal task-specific adaptation (IC-LoRA) as a compelling foundation for high-fidelity, fully synchronized cross-lingual dubbing. It addresses data scarcity through model-driven paired sample synthesis and introduces regularization techniques to defeat information leakage and mode collapse. The approach achieves strong quantitative and qualitative outcomes across benchmarks and user perception studies. While room for improvement remains, particularly in speaker identity disentanglement, this work motivates a paradigm shift toward jointly generative, context-grounded AV models for video editing and translation tasks.

Figure 5: Example of the method: the joint model dubs speech and lip motion, preserving identity, scene context, and non-speech events.

Figure 6: Even in profile/occluded/non-human scenarios, the method preserves identity and visual consistency while synchronizing translated audio and lips.