From Inpainting to Editing: A Self-Bootstrapping Framework for Context-Rich Visual Dubbing (2512.25066v1)

Abstract: Audio-driven visual dubbing aims to synchronize a video's lip movements with new speech, but is fundamentally challenged by the lack of ideal training data: paired videos where only a subject's lip movements differ while all other visual conditions are identical. Existing methods circumvent this with a mask-based inpainting paradigm, where an incomplete visual conditioning forces models to simultaneously hallucinate missing content and sync lips, leading to visual artifacts, identity drift, and poor synchronization. In this work, we propose a novel self-bootstrapping framework that reframes visual dubbing from an ill-posed inpainting task into a well-conditioned video-to-video editing problem. Our approach employs a Diffusion Transformer, first as a data generator, to synthesize ideal training data: a lip-altered companion video for each real sample, forming visually aligned video pairs. A DiT-based audio-driven editor is then trained on these pairs end-to-end, leveraging the complete and aligned input video frames to focus solely on precise, audio-driven lip modifications. This complete, frame-aligned input conditioning forms a rich visual context for the editor, providing it with complete identity cues, scene interactions, and continuous spatiotemporal dynamics. Leveraging this rich context fundamentally enables our method to achieve highly accurate lip sync, faithful identity preservation, and exceptional robustness against challenging in-the-wild scenarios. We further introduce a timestep-adaptive multi-phase learning strategy as a necessary component to disentangle conflicting editing objectives across diffusion timesteps, thereby facilitating stable training and yielding enhanced lip synchronization and visual fidelity. Additionally, we propose ContextDubBench, a comprehensive benchmark dataset for robust evaluation in diverse and challenging practical application scenarios.

• The paper introduces X-Dub, a self-bootstrapping framework that reframes audio-driven dubbing as context-rich video-to-video editing using diffusion transformers.
• It employs timestep-adaptive multi-phase learning to separately optimize global structure, detailed lip articulation, and texture fidelity for improved synchronization and identity retention.
• Empirical results on HDTF and ContextDubBench benchmarks demonstrate significant gains in lip sync accuracy, identity preservation, and robustness under challenging visual conditions.

Self-Bootstrapping for Context-Rich Visual Dubbing: The X-Dub Framework

Motivation and Limitations of Existing Paradigms

Audio-driven visual dubbing—modifying an input video’s lip region to match new speech while strictly preserving all other visual and contextual cues—poses a fundamental challenge due to the absence of paired videos that differ only in lip movement but are otherwise visually identical. Conventional approaches exploit a mask-based inpainting paradigm, masking the lower facial region and reconstructing it given alternative audio and sparse visual references, thus enabling self-supervised training. However, this paradigm leads to an ill-posed editing problem, as the model must simultaneously hallucinate occluded content, extract identity from pose-misaligned references, and synchronize lips, resulting in visual artifacts, identity drift, and suboptimal synchronization. These limitations are further exacerbated in unconstrained scenarios with occlusions, dynamic lighting, and out-of-domain content.

Figure 1: X-Dub formulation surpasses traditional mask-inpainting by recasting dubbing as aligned video-to-video editing, ensuring robust lip sync and identity preservation even under challenging conditions.

X-Dub: Self-Bootstrapping Contextual Video-to-Video Dubbing

X-Dub introduces a self-bootstrapping paradigm, leveraging Diffusion Transformers (DiTs) for both ideal training data generation and context-rich editing. The process consists of two key stages:

1. Paired Data Construction via DiT Generator: A pre-trained DiT-based generator synthesizes a lip-altered companion video for each real sample by inpainting masked facial regions conditioned on alternative audio and a reference frame. The focus of the generator is not perfect lip synchronization but the preservation of identity, pose, occlusion patterns, and illumination across the synthetic-real pair, with lip movements being sufficiently distinct to provide task-relevant supervision. Several guiding strategies are implemented: short-term segment processing for stability, occlusion-aware masking, synchronized lighting augmentation, quality filtering, and the inclusion of 3D-rendered videos for perfectly aligned pairs.

   Figure 2: The X-Dub pipeline involves synthesizing paired data (left), training the context-driven DiT editor (middle), and deploying timestep-adaptive multi-phase learning for structural, articulatory, and fine-grained refinement (right).

2. Context-Driven Editing via DiT Editor: The DiT-based editor is trained on the curated paired data, given the full, visually aligned companion as input alongside the target audio. This mask-free formulation precisely leverages complete spatiotemporal context, simplifying the deanonymized problem to one of focused, speech-driven lip modification. The result is enhanced fidelity, identity retention, and robustness.

Timestep-Adaptive Multi-Phase Learning

Reframing dubbing as contextual editing introduces competing objectives—global structure inheritance, local lip modification, and high-fidelity texture preservation. To address this, X-Dub incorporates a timestep-adaptive multi-phase learning strategy:

• High-Noise Phase: Full-parameter optimization under high noise fosters alignment of global structure, background, head pose, and coarse identity by inheriting features from the reference context.
• Mid-Noise Phase: Lightweight LoRA modules fine-tune for articulated lip motion, under SyncNet-based lip-sync loss.
• Low-Noise Phase: Additional LoRA experts optimize for identity and perceptual detail, supervised by ArcFace and CLIP losses. Random suppression of audio cross-attention prevents textural tuning from harming synchronization.
• Expert Activation: Each LoRA is activated within specific timestep windows, targeting the diffusion process where its respective objective is most effectively learned.

This stage-wise decoupling is critical; ablations confirm that uniform training collapses or dilutes performance, whereas progressive, phase-aligned optimization enables the editor to harness the full potential of paired contextual supervision.

Figure 3: Ablation demonstrates the impact of reference injection schemes and multi-phase learning on lip accuracy and textural fidelity.

Empirical Results: Robustness and Quality

On the HDTF benchmark and the newly introduced ContextDubBench—which features unconstrained settings, stylized and non-human subjects, challenging light, motion, and occlusion—the X-Dub editor attains new state-of-the-art results across all evaluation axes: FID, FVD, SyncNet-based lip sync, identity similarity (ArcFace, CLIP), and subjective user studies.

• Quantitative Highlights: On ContextDubBench, X-Dub achieves a SyncNet score of 7.28 (vs. 6.28 for the best baseline), CSIM (ArcFace) of 0.85 (+6.1% vs. baseline), and a 96.4% success rate—over 24 points higher than any prior method, indicating robust applicability across complex, real-world data. Visual quality (e.g., NIQE 5.78, BRISQUE 29.87) also surpasses strong diffusion and GAN-based competitors.
• Qualitative Robustness: X-Dub yields accurate lip articulation, preserves occluders and lighting dynamics, and eliminates mask-induced leakage and artifacts, even on non-human and stylized content where mask-based models fail.

  Figure 4: Qualitative comparison: X-Dub achieves superior lip sync, artifact suppression, and generalization to occlusions, side views, and non-human cases (see yellow/blue/red error annotations for baseline deficiencies).

  

  Figure 5: Example of identity drift when long-range generator segments are used without the tailored short-term processing adopted in X-Dub.

  

  Figure 6: X-Dub’s mask processing robustly preserves occluders, preventing their removal during editing.

  

  Figure 7: Lighting augmentation: Synchronized static/dynamic relighting ensures contextual alignment under real-world illumination shifts.

Benchmark and Generalization

ContextDubBench is introduced as the first benchmark purposefully constructed for rigorous assessment under real-world and generative complexity. It encompasses real, stylized, and non-human appearances, challenging lighting, occlusion, and identity-preserving modifications.

Figure 8: ContextDubBench example—non-human diversity.

Figure 9: ContextDubBench example—stylized artistic characters.

Figure 10: ContextDubBench example—real humans in the wild.

Figure 11: ContextDubBench—diverse lighting, occlusions, and identity-distinct segments.

Discussion and Implications

The strong empirical performance of X-Dub, especially its massive gain (+24% success rate) in unconstrained scenarios, underscores the central insight: the critical determinant in audio-driven visual dubbing is not incremental model improvement in inpainting, but a paradigm shift to context-rich, well-posed editing. By synthesizing ideal paired data and leveraging full-frame context, X-Dub enables the model to assign maximal capacity to the genuinely ambiguous aspects of the task—lip re-articulation—while inheriting the vast remainder of appearance, scene, and spatiotemporal content.

The timestep-adaptive multi-phase learning strategy emerges as a necessary, not merely auxiliary, component. It mitigates gradient competition, enables specialization per step, and facilitates stable contextual learning. The combination of synthetic data bootstrapping and hierarchical training is likely extensible to other conditional video editing tasks where paired data is fundamentally unattainable or expensive (e.g., CG character retargeting, video-driven translation).

Practically, the approach is computationally tractable; inference is comparable in speed and cost to other diffusion-based methods and can be further accelerated via adaptive early-stopping of high-noise denoising steps and test-time caching.

Future Prospects

Conceptually, the self-bootstrapping editing formulation leaves open several directions:

• Extending Modalities: Extension of the contextual editing paradigm to other video-to-video cases, such as gesture transfer, head pose editing, and emotional retargeting.
• Cross-Domain Adaptation: Application to out-of-distribution contexts, e.g., animation, avatars, and synthetic-real fusion scenarios.
• Data Construction Advances: Exploring more sophisticated companion generation, including improved simulation of subtle coarticulation and prosody transfer, or using generative adversarial expert filtering to optimize for rare data regimes.
• Real-Time and Low-Compute Deployment: Further compression or distillation of the DiT backbone, combined with editing-aware pruning and conditional step reduction for efficiency.

Conclusion

By fundamentally reframing audio-driven visual dubbing as a context-rich, video-to-video editing problem and introducing a robust self-bootstrapping training paradigm, X-Dub achieves state-of-the-art performance in both controlled and highly challenging settings. The approach demonstrates that judicious pairing of synthetic data with structured diffusion model training can overcome longstanding bottlenecks imposed by the absence of real paired data. These findings have broad implications, pointing toward more scalable and generalizable conditional video editing systems (2512.25066).