HeadGaS++: Real-Time 3D Face Reconstruction

• HeadGaS++ is an audio-driven, real-time technique that uses dynamic 3D Gaussian splatting to synchronize speech with facial animations.
• It employs a compact MLP and sinusoidal positional encoding to modulate per-Gaussian color and opacity, achieving high fidelity at 250 FPS.
• The system seamlessly integrates with full-body models and conversational pipelines, enabling robust, immersive virtual avatar interactions.

HeadGaS++ is an audio-driven, real-time, photorealistic 3D face reconstruction technique based on dynamic 3D Gaussian Splatting within the ICo3D system (Shaw et al., 19 Jan 2026). It builds on advances in Gaussian-based human modeling, integrating facial animation from speech, multi-modal feature fusion, and high-fidelity rendering suitable for real-time immersive virtual interactions. HeadGaS++ is designed for seamless integration with full-body models and conversational pipelines, yielding avatars that synchronize audio speech and facial dynamics at high frame rates. This article details its system context, theoretical foundations, architectural components, training paradigms, quantitative performance, and practical engineering considerations.

1. System-Level Context and Workflow Integration

HeadGaS++ operates as a core stream within the ICo3D virtual human architecture. The complete ICo3D system merges three main modules: HeadGaS++ for the photorealistic animatable face, SWinGS++ for a dynamic full-body model, and an LLM+TTS/ASR pipeline for conversational interaction.

At run-time:

• User input (text/audio) is transcribed via ASR (using Whisper) and processed by a quantized LLM (Qwen2 0.5 B).
• The generated textual response is converted to audio via OpenVoice V2 TTS. This audio is featurized by SyncTalk into a frame-synchronous expression parameter vector $e_i$.
• HeadGaS++ consumes $e_i$ along with head-pose $H_i$ to predict per-Gaussian dynamic color $c_i$ and opacity $o_i$.
• SWinGS++ produces body animation (either replayed or procedurally generated).
• The head and body Gaussian clouds are merged, pruned/blended, and rendered using a high-throughput 3D Gaussian Splatting renderer (≈100 FPS).

This configuration supports continuous, real-time avatar conversations in both written and oral form, facilitating avatars which synchronize facial animation to synthesized audio while maintaining geometric and photometric consistency (Shaw et al., 19 Jan 2026).

2. HeadGaS++ Model Architecture

2.1 Gaussian Splatting Representation

The static representation leverages the 3DGS primitive:

• Each Gaussian is defined by its center $\mu \in \mathbb{R}^3$, covariance $\Sigma \in \mathbb{R}^{3 \times 3}$ (factored as $\Sigma = R S S^\top R^\top$), view-dependent color coefficients $c \in \mathbb{R}^{3(k+1)^2}$ (spherical harmonics of degree $k$), and opacity $o$.
• The density function:

$$G(x) = \exp\big(-\tfrac{1}{2} (x - \mu)^\top \Sigma^{-1} (x - \mu)\big)$$

2.2 Audio-Driven Dynamics

Distinctively, HeadGaS++ enables dynamic color and opacity:

• Rather than moving Gaussian centers, per-Gaussian color $c_i$ and opacity $o_i$ are modulated by audio-visual features.
• A learned latent feature basis $F \in \mathbb{R}^{B \times f}$ ($B = 39$ blendshape/e-eye dimensions, $f = 32$) and bias $f_0 \in \mathbb{R}^f$ facilitates high-dimensional fusion.
• At each frame $i$:
  • Expression vector $e_i$ ($32$-D SyncTalk audio + $7$-D ARKit eye) is linearly blended:

  $f_i = F^\top e_i + f_0$ - A compact MLP $\phi$ ($2 \times$ Linear+LeakyReLU, hidden=64) predicts dynamic $(c_i, o_i)$ using positional encoding on $\mu$:

  $(c_i, o_i) = \phi(f_i, \gamma(\mu))$

  where $\gamma(\cdot)$ denotes sinusoidal positional encoding.

2.3 Initialization and Optimization

• Gaussian centers $\mu$ are initialized from $2\,500$ FLAME mesh vertices; covariances isotropically.

• The feature basis $F$ is zero-initialized, bias $f_0$ is learned.

• Learning rates: $\mu$, $\phi$ at $1.6 \times 10^{-4}$; $F$ at $2.5 \times 10^{-3}$; scale $S$ at $5 \times 10^{-3}$; rotation $R$ at $1 \times 10^{-3}$.

• Optimized via SGD with decay for $50,000$ iterations on a single V100 GPU (∼1 hour).

2.4 Loss Function

The composite loss incorporates photometric, structural, and perceptual objectives:

$$\mathcal{L} = \lambda_1 \|I_r - I_{gt}\|_1 + \lambda_s (1 - \text{SSIM}(I_r, I_{gt})) + \lambda_p \mathcal{L}_{\text{perceptual}}(I_r, I_{gt})$$

with $$(\lambda_1, \lambda_s, \lambda_p) = (0.8, 0.2, 0.1)$$, and perceptual loss only activated after $10,000$ iterations to promote stabilized representation.

3. Advancements over Prior Models

HeadGaS++ extends previous Gaussian face models as follows:

• Replaces offline blendshape weights (HeadGaS) with a learned audio-visual feature basis ($F$, $\phi$) predicting color/opacity directly from live audio features.

• Utilizes enhanced positional encoding for $\mu$ and increased MLP hidden dimensions, leading to sharper reconstructions.

• Performance-optimal loss scheduling: perceptual loss is temporally delayed, and loss weights ($\lambda$) are empirically adjusted for robust detail retention.

• Includes integration hooks for joint optimization: HeadGaS++ can unfreeze its final color layer to match merged head-body models in cases of lighting mismatch.

This suggests an increased modularity and adaptability relative to precedent Gaussian face approaches (Shaw et al., 19 Jan 2026).

4. Training Regime and Datasets

• The training corpus is the RenderMe-360 multi-view dataset (24 synchronized cameras at 15 FPS, neutral illumination).

• Preprocessing steps comprise facial cropping, resizing to $512^2$, and FLAME mesh tracking for geometric initialization.

• The schedule is:

  • $50,000$ iterations SGD on $1 \times$ V100.
  • Initial $10,000$ iterations omitting perceptual loss for stabilization.
  • Adaptive density control applied between iterations $500 \to 15,000$.
  • Training concludes upon plateauing $L_1+$SSIM losses.

5. Quantitative Evaluation and Benchmarking

5.1 Self-Reconstruction Metrics

HeadGaS++ achieves state-of-the-art reconstruction fidelity and performance among evaluated 3DGS-based talking head models:

Method	PSNR	SSIM	LPIPS	LMD	Sync-C	TrainTime	FPS
RAD-NeRF	26.79	0.901	0.083	–	4.988	3h	25
ER-NeRF	27.35	0.904	0.063	–	5.172	1h	35
TalkingGaussian	29.32	0.920	0.046	2.688	5.802	0.8h	110
GaussianTalker	29.13	0.911	0.085	2.814	5.350	5h	130
HeadGaS++ (Ours)	30.40	0.935	0.051	2.793	5.928	1h	250

HeadGaS++ leads in PSNR/SSIM, Sync-C, and achieves a significant throughput at 250 FPS (Shaw et al., 19 Jan 2026).

5.2 Cross-Driven Lip-Sync

When tested on cross-driven settings (e.g. Macron$\to$Obama), HeadGaS++ surpasses prior methods by a $>$1 dB margin in Sync-C, demonstrating high-fidelity cross-identity animation.

6. Implementation Guidance and Engineering Considerations

To maximize HeadGaS++ performance and reliability:

• Maintain a compact head MLP (≤2 layers) to sustain FPS.
• Apply sinusoidal positional encodings to $\mu$ for high-frequency detail.
• Delay integration of perceptual loss until after 10,000 iterations to avoid premature overfitting.
• During head-body merging, prune body Gaussians within a face centroid sphere every 100 iterations and regionally near the jaw to prevent artifacts (e.g. "double-chin"); border Gaussians from FLAME mesh smooth the facial seam.
• If head/body scans differ in lighting/intrinsics, selectively unfreeze only the final color branch of $\phi$ for joint optimization during body training.

A plausible implication is that these practical steps are essential for artifact-free, visually consistent avatar rendering in complex, multi-modal virtual environments.

7. Broader Context and Future Extensions

HeadGaS++ inherits and synthesizes advances from related 3D Gaussian Splatting frameworks:

• GGHead (Kirschstein et al., 2024) demonstrates real-time, high-res ($1024^2$) 3D head generation using UV-templated Gaussian attributes predicted via 2D CNNs, forming a scalable generative pipeline for 3D-consistent avatars.
• GaussianHeadTalk (Agarwal et al., 11 Dec 2025) incorporates temporal transformers to stably map audio to FLAME parameters, minimizing "wobble" and supporting multi-language synthesis and style conditioning.

HeadGaS++ could further benefit from embedding richer FLAME blendshape controls, diversifying audio feature basis (e.g. explicit pitch/prosody for affective modeling), and generalizing to full-body reenactment through integration with models such as SMPL-X. Continued research directions include adversarial style-transfer for emotion control and robustness against deepfake misuse by watermarking Gaussian domain parameters (Agarwal et al., 11 Dec 2025).

These convergences position HeadGaS++ as a foundational methodology for expressive, real-time, and fully-integrated 3D avatar animation in next-generation virtual interaction platforms.