VFace: Diffusion Video Face Swapping

• VFace is a training-free, plug-and-play diffusion framework for video face swapping that converts an image-based model into a temporally consistent video system.
• It leverages latent inversion and attention-space interventions—Target Structure Guidance, Frequency Spectrum Attention Interpolation, and Flow-Guided Attention Temporal Smoothening—to balance identity transfer with structural fidelity.
• Evaluations on datasets like FFHQ and CelebVHQ show improved FID, pose, and temporal consistency metrics, with faster runtime compared to traditional video editing frameworks.

VFace is a training-free, plug-and-play framework for diffusion-based video face swapping that converts an image-based diffusion face-swapping model into a video system with substantially stronger temporal consistency, while preserving source identity and retaining the target video’s pose, expression, structure, lighting, and background. Its operating regime is a single source face image $x^{\text{src}}$ and a target video with frames $\{x_i^{\text{tar}}\}_{i=1}^n$, from which it generates swapped frames $\{x_i^{\text{swap}}\}_{i=1}^n$. Rather than retraining a video diffusion model, VFace modifies inference through latent inversion and attention-space interventions, specifically Target Structure Guidance, Frequency Spectrum Attention Interpolation, and Flow-Guided Attention Temporal Smoothening (Baliah et al., 8 Feb 2026).

1. Problem setting and design rationale

VFace addresses a narrow but important failure mode of diffusion-based image face swapping: when methods such as REFace or DiffSwap are applied independently to video frames, they often exhibit temporal flicker, identity drift, and frame-to-frame structural inconsistency. The difficulty is structural rather than merely implementation-specific. Diffusion sampling is iterative and stochastic, and image models do not explicitly model temporal dynamics, so framewise deployment does not preserve a coherent facial trajectory over time (Baliah et al., 8 Feb 2026).

The method is explicitly positioned as training-free because many alternatives for video-consistent diffusion editing require retraining on video datasets, architecture changes such as space-time U-Nets, full source videos rather than a single source image, or video-specific fine-tuning. VFace instead operates in the practically useful source-image to target-video setting, reuses a pretrained image face-swapping backbone, and intervenes only at inference time. This places it in contrast to training-based video face-swapping frameworks such as VividFace, which introduces an image-video hybrid training framework, a VidFaceVAE, and temporal modules inside a diffusion backbone (Shao et al., 2024).

The underlying assumption is that the image backbone already provides strong single-frame identity transfer and photorealism. VFace therefore treats video face swapping as an inference-control problem: preserve the target frame’s structure, re-inject source identity where structure guidance suppresses it, and regularize attention trajectories across neighboring frames. This decomposition is the method’s central design principle (Baliah et al., 8 Feb 2026).

2. Inference pipeline and latent formulation

The VFace pipeline is built on top of a diffusion-based image face-swapping model, with REFace as the main demonstration backbone and transfer also shown to FaceAdapter. It relies on DDIM inversion of target frames so that generation starts from latents tied to the target structure rather than from random Gaussian noise. The denoising step is written as

$z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$

and inversion uses

$$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$

Starting from an observed image latent $z_0$, iterating this procedure yields an inverted latent that approximately reconstructs the target frame when denoised again (Baliah et al., 8 Feb 2026).

After inversion, VFace runs two branches. The reconstruction branch denoises the inverted target latent to reconstruct the original target frame and records intermediate attention tensors, specifically queries $\mathbf q_t^i$ and keys $\mathbf k_t^i$, for frame $i$ at timestep $t$. The generation branch starts from the same target-inverted latent but applies the conditioning of the image face-swapping backbone so that source identity is transferred into the output. VFace then modifies the generation branch using the recorded target attention and additional temporal smoothing (Baliah et al., 8 Feb 2026).

These manipulations are applied in all cross-attention layers on the output side, or upsampling side, of the denoising U-Net. This is a crucial architectural restriction: VFace does not redefine the underlying diffusion model, but it does assume access to internal attention tensors during sampling. Videos are processed in batches of 6 frames with a sliding window, and the last frame of the previous batch is reused to preserve consistency across batches (Baliah et al., 8 Feb 2026).

3. Core mechanisms

The three inference-time mechanisms are complementary rather than redundant. Target Structure Guidance anchors geometry, Frequency Spectrum Attention Interpolation restores identity characteristics that structure guidance suppresses, and Flow-Guided Attention Temporal Smoothening enforces short-range temporal coherence.

Component	Primary mechanism	Reported effect
TSG	Plug-and-play attention injection from target reconstruction branch	Improves pose and structural alignment
FSAI	Frequency-domain interpolation of source and target attention tensors	Recovers source identity while retaining target structure
FATS	Optical-flow-guided smoothing of attention features across frames	Produces the main gain in temporal consistency

Target Structure Guidance replaces the generation branch’s queries and keys with those from the reconstruction branch:

$\{x_i^{\text{tar}}\}_{i=1}^n$0

Because both branches start from the same DDIM-inverted target latent, this attention injection transfers target-frame pose, structure, and spatial layout into the swapped generation. The ablation shows that TSG strongly improves pose, but it also weakens identity similarity if used alone (Baliah et al., 8 Feb 2026).

Frequency Spectrum Attention Interpolation is introduced precisely to correct that identity loss. VFace argues that low-frequency components encode coarse semantic information such as identity and overall appearance, while high-frequency components capture fine localized details and structure cues. Rather than mixing images directly, it interpolates attention tensors in the frequency domain, keeping low frequencies from source-guided attention and high frequencies from target-guided attention:

$\{x_i^{\text{tar}}\}_{i=1}^n$1

This is applied to both queries and keys. The appendix reports that values around $\{x_i^{\text{tar}}\}_{i=1}^n$2 provide the best balance between identity preservation and video fidelity (Baliah et al., 8 Feb 2026).

Flow-Guided Attention Temporal Smoothening operates directly in attention space rather than pixel space or initial-noise space. Given intermediate features $\{x_i^{\text{tar}}\}_{i=1}^n$3 and optical flow $\{x_i^{\text{tar}}\}_{i=1}^n$4 from frame $\{x_i^{\text{tar}}\}_{i=1}^n$5 to frame $\{x_i^{\text{tar}}\}_{i=1}^n$6, the smoothed feature is

$\{x_i^{\text{tar}}\}_{i=1}^n$7

with $\{x_i^{\text{tar}}\}_{i=1}^n$8. Optical flow is computed with RAFT-Large on a downsampled $\{x_i^{\text{tar}}\}_{i=1}^n$9 version of the target video. FATS is applied only during the first $\{x_i^{\text{swap}}\}_{i=1}^n$0 DDIM steps, with $\{x_i^{\text{swap}}\}_{i=1}^n$1 reported as preferable to longer application; extending to $\{x_i^{\text{swap}}\}_{i=1}^n$2 degrades FVD markedly (Baliah et al., 8 Feb 2026).

4. Evaluation, metrics, and ablation results

VFace is evaluated using CelebA and FFHQ as source image datasets, and VFHQ and CelebV-HQ as target video datasets. The paper reports that VFHQ contributes 50 videos and CelebV-HQ about 100 videos with durations of 3–20 seconds at $\{x_i^{\text{swap}}\}_{i=1}^n$3 resolution. Evaluation spans identity preservation, frame quality, pose and expression fidelity, temporal consistency, and runtime. Identity is assessed with ArcFace features and Top-1/Top-5 retrieval against a gallery of 1,000 source images; pose uses HopeNet; expression uses Deep3DFaceRecon; frame quality uses FID; and video quality uses FVD and Content-Debiased FVD (Baliah et al., 8 Feb 2026).

On image-based evaluation, VFace improves over REFace while preserving the source-image conditioning regime. On FFHQ, it reports FID $\{x_i^{\text{swap}}\}_{i=1}^n$4 versus REFace $\{x_i^{\text{swap}}\}_{i=1}^n$5, Top-1 identity retrieval $\{x_i^{\text{swap}}\}_{i=1}^n$6, pose $\{x_i^{\text{swap}}\}_{i=1}^n$7, and expression $\{x_i^{\text{swap}}\}_{i=1}^n$8. On CelebA, it reports FID $\{x_i^{\text{swap}}\}_{i=1}^n$9 versus REFace $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$0, pose $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$1, and expression $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$2. These image-level results are important because they show that the video-oriented interventions do not merely trade off frame quality for temporal smoothness (Baliah et al., 8 Feb 2026).

On video benchmarks, the temporal gains are more pronounced. On CelebVHQ, REFace+VFace reports CD-FVD $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$3 and FVD $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$4. On VFHQ, REFace+VFace reports CD-FVD $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$5 and FVD $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$6. The paper states that these results are substantially better than vanilla REFace and better than attaching generic video editing frameworks such as AnyV2V or Go-with-the-Flow to REFace (Baliah et al., 8 Feb 2026).

The module ablation is especially diagnostic. Without any of the three components, the system reports CD-FVD $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$7, ID similarity $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$8, and pose $z_{t-1} = m_t z_t + n_t \,\epsilon_\theta(z_t,t),$9. Adding only TSG improves pose to $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$0 but reduces ID similarity to $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$1 and does not improve CD-FVD. Adding FSAI recovers ID similarity to $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$2 while keeping pose near $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$3. Adding FATS yields the decisive temporal gain, reducing CD-FVD to $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$4 while maintaining ID similarity $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$5 and pose $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$6. The appendix further reports that $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$7 for FSAI and $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$8 for FATS work best (Baliah et al., 8 Feb 2026).

Runtime is reported per frame. REFace takes $$z_t \approx \frac{z_{t-1} - n_t\,\epsilon_\theta(z_{t-1},t)}{m_t}.$$9 s/frame, VFace $z_0$0 s/frame, Go-with-the-Flow $z_0$1 s/frame, and AnyV2V $z_0$2 s/frame. The additional cost of VFace therefore arises from DDIM inversion, dual-branch processing, optical-flow estimation, and FFT-based attention manipulation, but it remains faster than the compared video editing baselines (Baliah et al., 8 Feb 2026).

5. Relation to adjacent face-video research

VFace belongs to a broader family of face-centric video systems, but its task definition is narrower than that of most similarly named methods. It is neither a generic face-video generation model nor a compression codec nor a VR telepresence system. It is specifically a training-free framework for diffusion-based video face swapping (Baliah et al., 8 Feb 2026).

This distinction matters because nearby works with similar names solve different problems. VidFace is a full-transformer solver for video face hallucination, reconstructing a high-resolution face from multiple tiny, unaligned low-resolution snapshots rather than swapping identities (Gan et al., 2021). VividFace is a diffusion-based hybrid framework for video face swapping, but unlike VFace it relies on image-video hybrid training, a VidFaceVAE, temporal modules, and 3D reconstruction conditioning inside the generative model itself (Shao et al., 2024). FaceVR is an earlier image-based self-reenactment system for VR teleconferencing, focused on HMD occlusion, stereo consistency, and gaze-aware rendering rather than diffusion-based swapping (Thies et al., 2016).

A broader conceptual connection exists with face-centric video communication systems such as FAIVConf, which treat facial video as a hybrid of compression, reenactment, and synthesis and explicitly describe this family as “VFace”-style in the sense of semantic transmission and decoder-side facial reconstruction (Li et al., 2022). That comparison is instructive: both FAIVConf and VFace exploit semantic structure rather than raw per-frame fidelity, but the former is a face-centric conferencing codec and the latter is an inference-time face-swapping wrapper around diffusion models. The shared principle is localized control over face content, not equivalence of task or architecture.

6. Limitations, failure modes, and implications

The appendix states that some flicker remains and that severe occlusions, large appearance gaps, identity leakage, and high pose variation can still produce failures. The authors attribute many of these limitations to the underlying image swapping backbones rather than to VFace alone. This is a significant qualification: VFace improves video consistency without retraining, but it does not eliminate the failure modes of the image model on which it depends (Baliah et al., 8 Feb 2026).

The temporal regularization is also only as reliable as the optical flow used to align attention features. VFace mitigates brittleness by soft blending rather than hard replacement, but its smoothing mechanism still depends on RAFT-Large estimates on downsampled target frames. Likewise, its plug-and-play nature presupposes a diffusion backbone that exposes cross-attention tensors, supports DDIM inversion, and already conditions on source identity and target facial structure (Baliah et al., 8 Feb 2026).

A common misconception is to treat VFace as a general video-diffusion model for faces. It is more precise to regard it as an inference-time control layer over an image face-swapping diffusion model. This suggests a broader implication: temporal consistency in face-video editing need not always be learned by retraining a video generator; it can also be imposed by operating on latent trajectories and attention tensors during sampling. Within the current literature, VFace therefore occupies a distinct methodological position between framewise image swapping and fully trained video-swapping systems (Baliah et al., 8 Feb 2026).