DVFace: Spatio-Temporal Dual-Prior Diffusion for Video Face Restoration

Abstract: Video face restoration aims to enhance degraded face videos into high-quality results with realistic facial details, stable identity, and temporal coherence. Recent diffusion-based methods have brought strong generative priors to restoration and enabled more realistic detail synthesis. However, existing approaches for face videos still rely heavily on generic diffusion priors and multi-step sampling, which limit both facial adaptation and inference efficiency. These limitations motivate the use of one-step diffusion for video face restoration, yet achieving faithful facial recovery alongside temporally stable outputs remains challenging. In this paper, we propose, DVFace, a one-step diffusion framework for real-world video face restoration. Specifically, we introduce a spatio-temporal dual-codebook design to extract complementary spatial and temporal facial priors from degraded videos. We further propose an asymmetric spatio-temporal fusion module to inject these priors into the diffusion backbone according to their distinct roles. Evaluation on various benchmarks shows that DVFace delivers superior restoration quality, temporal consistency, and identity preservation compared to recent methods. Code: https://github.com/zhengchen1999/DVFace.

• The paper introduces a one-step diffusion framework that integrates dual spatial and temporal priors for enhanced video face restoration and computational efficiency.
• The methodology employs a novel spatio-temporal dual-codebook extraction and an asymmetric fusion module, achieving state-of-the-art metrics on benchmarks like VFHQ-Test.
• The approach preserves high-quality details, temporal coherence, and identity, setting a new standard for video generative restoration and potential applications to broader video tasks.

DVFace: Spatio-Temporal Dual-Prior Diffusion for Video Face Restoration

Introduction

DVFace introduces a one-step diffusion-based approach for video face restoration, explicitly addressing the simultaneous demands for high-quality detail reconstruction, temporal coherence, and identity preservation in degraded video sequences. The approach diverges from prior multi-step diffusion and codebook-based methodologies by incorporating a spatio-temporal dual-codebook prior extraction module and an asymmetric spatio-temporal fusion (ASTF) mechanism, yielding distinct improvements in both restoration efficacy and computational efficiency.

Methodology

One-Step Diffusion Framework

DVFace operates within a single-step diffusion paradigm, significantly reducing inference latency compared to iterative denoising schemes. The model leverages a pretrained text-to-video backbone (Wan2.1), with a conditional denoising trajectory determined by explicit spatial and temporal priors extracted from the degraded input. This backbone is augmented to accept dual priors, integrating them at each DiT block via a specialized fusion module rather than generic cross-modal conditioning.

Spatio-Temporal Dual-Codebook Prior Extraction

To capture the intrinsic multi-frame dynamics of face videos, DVFace introduces a dual-codebook approach:

• Spatial Codebook: Encodes canonical, high-fidelity appearance and morphological structures, robustly representing static facial patterns.
• Temporal Codebook: Encodes frame-to-frame motion and dynamic expressions, modeling evolution across short spatio-temporal segments.

Latent representations are extracted from the LQ input and computed via an encoder, then decoupled into spatial and temporal branches. Each branch utilizes Transformer-based sequence processing for accurate code selection, mitigating the distributional shift induced by severe degradation. The dual codebooks are learned and refined in a two-stage training scheme, aligning LQ-inferred codes with their HQ correspondents for consistent prior extraction.

Asymmetric Spatio-Temporal Fusion (ASTF)

Recognizing the disparate roles of spatial and temporal priors, DVFace’s ASTF module fuses these signals distinctly:

• Temporal Priors are pooled and mapped to global scale-and-shift parameters, modulating the backbone features consistently across layers to enforce stable temporal evolution.
• Spatial Priors, following temporal refinement, are injected via cross-attention mechanisms to serve as local residuals, introducing fine-grained details in a temporally compatible manner.

  Figure 1: Schematic of DVFace, highlighting the dual-codebook extraction and ASTF prior integration.

This partitioned fusion prevents the intermingling of global and local information, circumventing the artifact amplification and temporal instability observed in naive multi-prior conditioning.

Experimental Results

Quantitative and Qualitative Performance

Comprehensive evaluations are conducted on benchmarks including VFHQ-Test, HDTF, RFV-LQ, and VoxCeleb2. DVFace establishes new state-of-the-art performance across fidelity (PSNR, SSIM), perceptual (LPIPS, DISTS, CLIP-IQA, MUSIQ, NIQE, MANIQA, LIQE), temporal (VIDD, $E^*_{warp}$), and identity metrics (AKD, DOVER, FVD).

• On VFHQ-Test, DVFace yields the best results on all metrics except NIQE, with a notable PSNR of 31.81 and DOVER of 0.8703.
• On HDTF and real-world datasets, DVFace consistently surpasses previous methods, especially in perceptual and temporal measures.

  Figure 2: Quantitative performance comparison on VFHQ-Test; DVFace outperforms on all main evaluation axes.

  

Figure 3: Visual comparison on synthetic and real-world data. DVFace reconstructs sharper, more faithful details, robust to the absence of HQ reference in-the-wild videos.

Figure 4: Qualitative evaluation on challenging synthetic cases. Fine-structure restoration and temporal stability are superior in DVFace.

Figure 5: Robustness on difficult real-world examples. Competing methods show instability and blur; DVFace maintains fidelity.

Figure 6: Temporal consistency stacked across frames—DVFace achieves smooth consistency, mitigating typical framewise flicker.

Ablation Studies

Ablations substantiate:

• Necessity and Complementarity of Dual Priors: Removing either spatial or temporal prior decreases both static and temporal metrics; their joint utilization is strictly optimal.
• Design of ASTF: Shared temporal global modulation and temporally-refined spatial injection outperform independent or naive fusions.
• Prior Form: Codebook-based priors provide more direct, effective signals for restoration than fixed prompts or extracted text (e.g., DAPE).

Implications and Future Developments

DVFace demonstrates that efficient, high-fidelity video face restoration is achievable without the computational overhead of multi-step diffusion when dual priors are explicitly and asymmetrically integrated. The separation of spatial and temporal priors, and the use of a robust injection mechanism, directly addresses the entanglement that previously limited generative restoration in video domains.

The architectural recipe—decoupled prior extraction, asymmetric fusion, and one-step generative inference—has implications for other structured video restoration and generation tasks wherein detail and global consistency are simultaneously demanded. Future directions include further generalization to non-face video domains, real-time deployment, and exploration of adaptive codebook capacity for unconstrained in-the-wild degradation.

Conclusion

DVFace provides a technically rigorous framework for video face restoration, advancing the state of the art in both efficiency and quality through a one-step diffusion backbone, explicit spatio-temporal prior extraction, and asymmetric fusion. The approach sets new performance standards and offers a transferable methodology applicable to broader video generative restoration challenges.