DreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer

Abstract: Video Face Swapping (VFS) requires seamlessly injecting a source identity into a target video while meticulously preserving the original pose, expression, lighting, background, and dynamic information. Existing methods struggle to maintain identity similarity and attribute preservation while preserving temporal consistency. To address the challenge, we propose a comprehensive framework to seamlessly transfer the superiority of Image Face Swapping (IFS) to the video domain. We first introduce a novel data pipeline SyncID-Pipe that pre-trains an Identity-Anchored Video Synthesizer and combines it with IFS models to construct bidirectional ID quadruplets for explicit supervision. Building upon paired data, we propose the first Diffusion Transformer-based framework DreamID-V, employing a core Modality-Aware Conditioning module to discriminatively inject multi-model conditions. Meanwhile, we propose a Synthetic-to-Real Curriculum mechanism and an Identity-Coherence Reinforcement Learning strategy to enhance visual realism and identity consistency under challenging scenarios. To address the issue of limited benchmarks, we introduce IDBench-V, a comprehensive benchmark encompassing diverse scenes. Extensive experiments demonstrate DreamID-V outperforms state-of-the-art methods and further exhibits exceptional versatility, which can be seamlessly adapted to various swap-related tasks.

• The paper introduces DreamID-V, which achieves high-fidelity video face swapping using a diffusion transformer and synthetic paired supervision to maintain identity consistency.
• It leverages a novel SyncID-Pipe to generate bidirectional quadruplets, integrating adaptive pose attention and enhanced background recomposition to reduce temporal artifacts.
• Empirical validation on the IDBench-V benchmark shows superior identity retention, attribute fidelity, and smooth video transitions compared to state-of-the-art methods.

DreamID-V: High-Fidelity Video Face Swapping via Diffusion Transformer

Introduction and Problem Statement

DreamID-V introduces a comprehensive solution to video face swapping (VFS), addressing the intrinsic challenges of identity consistency, attribute preservation, and temporal coherence that differentiate VFS from its image counterpart. While diffusion-based image face swapping (IFS) methods excel in generating high-fidelity, attribute-preserving results, their naive extension to video—by frame-wise processing—inevitably yields severe temporal artifacts, such as flickering and identity drift. Previous VFS frameworks have tended to improve temporal consistency at the cost of either identity similarity or attribute fidelity. The DreamID-V framework systematically bridges the methodological gap between static and dynamic domains, leveraging a purpose-built data pipeline and a scalable transformer-based generative backbone.

Figure 1: DreamID-V robustly handles complex expressions, animation, occlusion, pose variations, and small faces.

SyncID-Pipe: Data Synthesis and Paired Supervision

A core limitation of VFS research is the paucity of explicitly paired video supervision. DreamID-V addresses this via SyncID-Pipe, a new data curation pipeline that constructs bidirectional ID quadruplets. The pipeline utilizes an Identity-Anchored Video Synthesizer (IVS), built upon a first-last-frame video generation foundation modulated by adaptive pose attention, to create pose-driven synthetic video pairs. By leveraging a state-of-the-art IFS model for identity transfer on video endpoints, SyncID-Pipe generates quadruplets spanning real and synthetic modalities: $\{I_r, V_r, I_g, V_g\}$. Additionally, expression adaptation and enhanced background recomposition modules are employed to overcome identity-expression entanglement and background mismatch, respectively, both of which are critical for highly decoupled and robust VFS.

Figure 2: SyncID-Pipe combines IVS and IFS to enable bidirectional quadruplet supervision for explicit video face swapping training.

DreamID-V Architecture

The DreamID-V framework is the first to leverage the Diffusion Transformer (DiT) backbone for video-based face swapping. It employs a Modality-Aware Conditioning (MC) module to disentangle and inject semantic identity, structural pose, and spatio-temporal context independently and discriminatively. Specifically:

• The spatio-temporal context module encodes the background and lighting via direct latent concatenation with face masks.
• The structural guidance module incorporates pose sequences through adaptive pose-attention, initialized from the pre-trained IVS.
• The identity module embeds reference semantics via a dedicated identity encoder, interacting with DiT through token concatenation, enabling high-fidelity cross-frame identity propagation.

  Figure 3: DreamID-V framework employs specialized injection strategies for spatio-temporal, structural, and identity features.

Training Paradigm: Synthetic-to-Real Curriculum and RL-based Consistency

DreamID-V employs a multi-phase curriculum, first leveraging distribution-aligned synthetic paired videos for rapid convergence and superior identity learning, then progressively augmenting with real-world videos enhanced for background consistency. To further address the instability of identity preservation—especially under intense motion or non-frontal views—the framework adopts an Identity-Coherence Reinforcement Learning (IRL) strategy. IRL prioritizes "difficult" frames (where identity metrics are low) by explicit Q-weighted loss reweighting, thus directly minimizing intra-video identity variance and substantially mitigating flicker.

Benchmarking and Empirical Validation

DreamID-V is evaluated on a newly proposed benchmark, IDBench-V, which spans a wide array of dynamic, occlusive, and real-world video scenarios. Quantitative results show:

• DreamID-V surpasses all state-of-the-art baselines in identity metrics (ArcFace, InsightFace, CurricularFace similarity), attribute fidelity (pose/expression), and perceptual video quality (FVD, smoothness), with its identity metrics even exceeding DreamID (IFS) when measured in video context.
• In contrast to methods like CanonSwap, which preserve attributes by minimal modification (but lack identity transfer efficacy), DreamID-V achieves strong identity transfer with minimal trade-off in correctness or realism.

Qualitative analysis (see Figure 4) highlights DreamID-V's robustness on attributes such as occlusion and dynamic expressions, with profile views and extreme motion showing reduced identity drift, especially when IRL is enabled. Ablation studies unambiguously demonstrate that SyncID-Pipe, curriculum learning, and IRL each provide measurable and complementary gains in performance and temporal consistency.

Figure 4: Qualitative comparison with SOTA; DreamID-V exhibits high identity fidelity, attribute retention, and robustness to occlusion.

Figure 5: Ablations demonstrate the effects of SyncID-Pipe, curriculum stages, and IRL on identity and attribute metrics.

Generalization and Versatility

Notably, DreamID-V is not restricted to faces. By swapping the IFS component in SyncID-Pipe with generic image editing models, it generalizes to arbitrary part or attribute transfer (e.g., accessories, hairstyles, outfits), extending its utility to a broad class of structured, temporally coherent human-centric video manipulation tasks.

Figure 6: The method generalizes to diverse swapping tasks, including outfit and hairstyle editing, not just faces.

Qualitative, Latent, and User Study Analyses

Human evaluations further corroborate DreamID-V's numerical superiority, with consistent preference for its identity, fidelity, and smoothness. Latent-space t-SNE analysis confirms the IVS-aligned synthetic distributions facilitate the DiT's effective learning. Additional figures (including user study layouts and more in-the-wild examples) reinforce the model’s robustness and domain generalization.

Figure 7: Diverse and challenging IDBench-V benchmark samples demonstrate DreamID-V's generalization.

Figure 8: t-SNE latent space visualization validates distributional alignment of IVS-generated synthetic data with DiT outputs.

Ethical Considerations

The authors recognize the dual-use nature of high-fidelity VFS and release the model under strict licensing to mitigate malicious use, requiring explicit subject consent and prohibiting privacy-violating deployment.

Conclusion

DreamID-V constitutes a technically complete, empirically robust, and extensible solution to high-fidelity, temporally consistent video face swapping. By bridging IFS and VFS with paired synthetic supervision, a modular DiT-based architecture, and explicit reinforcement of identity coherence, DreamID-V establishes new state-of-the-art on rigorous benchmarks. Its architecture and data paradigm can likely guide the future development of personalized, dynamic video editing systems and generalized structured video-to-video transfer methods.