SDTalk: Structured Facial Priors and Dual-Branch Motion Fields for Generalizable Gaussian Talking Head Synthesis

Abstract: High-quality, real-time talking head synthesis remains a fundamental challenge in computer vision. Existing reconstruction- and rendering-based methods typically rely on identity-specific models, limiting cross-identity generalization. To address this issue, we propose SDTalk, a one-shot 3D Gaussian Splatting (3DGS)-based framework that generalizes to unseen identities without personalized training or fine-tuning. Our framework comprises two modules with a two-stage training strategy. In the first stage, we incorporate structured facial priors into the reconstruction module and separately predict 3DGS parameters for visible and occluded regions, enabling complete head reconstruction from a single image. In the second stage, we introduce a dual-branch motion field to model coarse and fine facial dynamics, improving detail fidelity and lip synchronization. Experiments demonstrate that SDTalk surpasses existing methods in both visual quality and inference efficiency.

• The paper presents a one-shot 3D Gaussian Splatting framework that integrates structured facial priors with a dual-branch motion field to generalize talking head synthesis without subject-specific training.
• It achieves state-of-the-art results, including improved PSNR, superior LPIPS and SSIM metrics, and robust audio-visual synchronization at an impressive 45 FPS.
• The dual-stage training and ablation studies highlight the effectiveness of separately addressing visible and occluded facial regions for enhanced detail recovery and precise lip synchronization.

SDTalk: Generalizable Gaussian Talking Head Synthesis via Structured Facial Priors and Dual-Branch Motion Fields

Introduction and Motivation

SDTalk (2605.09956) introduces a one-shot 3D Gaussian Splatting (3DGS)-based framework for talking head synthesis that is designed to generalize to unseen identities without requiring personalized training or fine-tuning. The motivation is rooted in the persistent trade-off between visual fidelity, generalization, and inference efficiency in talking head synthesis pipelines. Prior reconstruction- and rendering-based approaches—primarily NeRF and recent 3DGS variants—rely on identity-specific optimization, which prolongs adaptation to new subjects and limits deployment flexibility. In contrast, SDTalk addresses this with a system that leverages structured facial priors and a dual-branch motion field, yielding superior generalization, high-quality synthesis, and real-time performance.

Figure 1: The SDTalk architecture integrates a reconstruction module driven by a structured facial prior and a dual-branch audio-conditioned motion field to enable efficient, high-fidelity talking head synthesis from a single image.

Core Technical Contributions

Structured Facial Priors for Head Reconstruction

SDTalk mitigates the information inadequacy of single-image reconstruction by bifurcating the process into two complementary branches:

• Visible Branch: Utilizes a dual-lifting approach to reconstruct geometry and texture for observed facial regions, guided by features from DINOv2 and convolutional networks.
• Completion Branch: Predicts the Gaussian parameters of occluded or internal regions—such as teeth and oral cavity—by leveraging geometric correspondence from FLAME mesh priors and integrating learned global features with localized mesh vertex weights.

The two branches synergistically furnish a dense and structurally complete Gaussian representation, bridging gaps typically left by methods that operate exclusively on visible regions.

Dual-Branch Motion Field for Facial Dynamics

SDTalk introduces a novel dual-branch motion field for animating reconstructed Gaussian heads under audio conditioning:

• Coarse Motion Field: Captures global, low-frequency facial motions (jaw, head, etc.) via tri-plane hash-encoded features and MLP regression.
• Fine Motion Field: Encodes high-frequency lip and mouth dynamics necessary for accurate audio-lip synchronization.

Both fields operate independently over their dedicated Gaussian subsets and are hierarchically integrated, ensuring robust global expressivity coupled with precise local articulation. Training proceeds via a two-stage scheme: scene reconstruction in the first phase (with all modules except DINOv2 trainable), and facial motion learning by only optimizing the motion field in the subsequent phase.

Experimental Evaluation

Quantitative Analysis

On the HDTF dataset, SDTalk demonstrates strong numerical superiority across all assessed metrics compared to state-of-the-art zero-shot and one-shot paradigms, as well as PAA-based methods. PSNR improvement (+1.6 dB vs. MimicTalk), lowest LPIPS, and the best SSIM affirm its image quality; LMD and SyncNet confidence indicate robust audio-visual alignment; and a remarkable inference speed of 45 FPS—substantially outperforming baselines—underscores its deployability.

Qualitative Analysis

Figure 2: Qualitative comparisons reveal that SDTalk offers improved texture fidelity and fewer synthesis artifacts compared to Real3DPortrait and NeRFFaceSpeech, particularly in boundary continuity and global facial consistency.

Figure 3: SDTalk maintains sharper facial features and more stable lip synchronization across broader pose and emotion variations, outperforming baselines on MEAD.

Qualitative comparisons further highlight the benefits of structured priors and multi-branch motion modeling. Artifact reduction, improved lip closure, and mitigation of over-smoothing distinguish SDTalk, especially in out-of-distribution poses and expressive contexts.

Ablation Studies

Ablation experiments confirm that the removal of the completion branch degrades intraoral detail recovery and that using only a single motion branch reduces overall temporal fidelity and synchronization. The dual-branch structure is instrumental in balancing stability and expressivity.

Figure 4: Ablation visualization exposes perceptible degradations in occluded and fine-motion regions when omitting key structural components, such as the completion branch or fine motion module.

Implications and Future Directions

Practically, SDTalk’s ability to synthesize high-fidelity talking head videos from a single image with no per-identity adaptation presents clear implications for content creation, virtual avatars, and real-time human-computer interaction. By decoupling scene reconstruction from animation, SDTalk offers potential for modular upgrades—e.g., substituting the motion field with emotion or multi-modal drivers.

From a theoretical standpoint, the effectiveness of explicit geometric priors (FLAME) merged with Gaussian splatting substantiates the value of combining model-based and data-driven priors in generalized neural rendering pipelines. The dual-branch motion decomposition points to the benefits of hierarchical, disentangled representations in controlling high-DOF avatars.

Future research could involve:

• Scaling up training data diversity and testing on more challenging real-world benchmarks.
• Incorporating multi-view or temporal signals for further robustness.
• Exploring plug-and-play modules for affective or cross-lingual driving.

Conclusion

SDTalk sets a new technical reference point for one-shot talking head generation by innovatively integrating structured facial priors with a dual-branch motion field atop a 3DGS backbone. Its architecture bolsters both geometric completeness and dynamical expressivity, yielding improved generalization, image and motion fidelity, and real-time inference. The work motivates further research in structured, modular neural representations for 3D avatar synthesis.