Transformer-Based 3D Portrait Animation

• The paper presents transformer-based methods that integrate mesh and video latent approaches with self- and cross-attention to achieve controllable 3D portrait animation.
• It leverages diverse loss functions and regularization strategies to enhance identity preservation, motion consistency, and the disentanglement of facial, pose, and viewpoint signals.
• The models are validated across benchmark datasets, demonstrating improved PSNR, SSIM, and FID metrics, and robust performance under dynamic conditions.

Transformer-based 3D portrait animation models constitute a class of deep generative methods leveraging transformer architectures and spatio-temporal latents for the synthesis and dynamic control of 3D animated portraits. Modern instantiations span direct mesh-sequence transformers (Chen et al., 2021), video-diffusion transformer backbones (Cui et al., 1 Dec 2024), and control methods based on disentangled facial, pose, and viewpoint signals (Tang et al., 12 Dec 2025). These models integrate identity, expression, pose, and multi-view conditioning with advanced self- and cross-attention mechanisms, offering high-fidelity, temporally coherent, and controllable portrait generation.

1. Core Model Architectures and Tokenization

Transformer-based models for 3D portrait animation are instantiated either directly on mesh data or on latent video representations from causal VAEs. The mesh-based approach, as in AniFormer (Chen et al., 2021), operates on sequences of vertex sets:

• Each driving frame mesh $G^t \in \mathbb{R}^{V_1 \times 3}$ is encoded via Conv1D layers into per-vertex features $Z^t \in \mathbb{R}^{C \times V_1}$.
• Temporal positional embeddings $E_t$ are added, producing $Z = [Z^1; \dots; Z^T] + [E_1; \dots; E_T]$.
• Every per-frame vertex is treated as an independent token, totaling $T \times V_1$ tokens.

For video-diffusion transformer variants (Cui et al., 1 Dec 2024, Tang et al., 12 Dec 2025):

• Raw RGB video clips $x_0 \in \mathbb{R}^{T \times H \times W \times 3}$ are encoded into spatio-temporal latents $$z_0 \in \mathbb{R}^{T \times H' \times W' \times C}$$ via causal 3D VAEs.
• Latents are reshaped and processed as tokens within a transformer backbone (typically DiT/CogVideoX, with 48–64 layers and full self-attention over all temporal and spatial positions).
• Reference frames for identity preservation are prepended as additional tokens along the temporal axis.

Both mesh- and video-latent approaches incorporate separate streams for target/identity modulation, employing style tokens (AniFormer) or identity reference transformers (Cui et al., 1 Dec 2024) to ensure the transfer of appearance characteristics across generated frames.

2. Attention Mechanisms and Conditioning Streams

The central architectural innovation in these models is the integration of long-range self-attention with cross-attention for identity, audio, and motion or pose conditioning.

• AniFormer (Chen et al., 2021): Within each transformer block, motion tokens undergo self-attention via QKV Conv1D projections, after which style fusion (via InstanceNorm and adaptive modulation) incorporates target mesh appearance.
• Hallo3 (Cui et al., 1 Dec 2024): Video-diffusion latents receive simultaneous cross-attention from text prompts, audio embeddings (Wav2Vec2-based), identity reference tokens (via a 42-layer causal transformer), and optionally motion-frame feedback.
• FactorPortrait (Tang et al., 12 Dec 2025): Expression embeddings are extracted from a driving video and injected at every layer using AdaLN (Adaptive LayerNorm) techniques, allowing each temporal chunk an independent expression bias. Camera and pose information are provided through channel-wise concatenation of Plücker ray maps and body mesh normal maps, convolved and downsampled for compatibility with latent dimensionality.

The transformer backbone ensures global context mixing, which is essential for rendering non-frontal views, handling moving backgrounds, and maintaining temporal coherence.

3. Loss Functions and Regularization Strategies

All cited models utilize differentiated loss landscapes to encourage high-fidelity, identity preservation, motion accuracy, and disentanglement.

AniFormer (Chen et al., 2021):

• Reconstruction loss:

$$\mathcal{L}_r = \frac{1}{T V_2} \sum_{t=1}^T \| \hat{M}^t - G^t \|_2^2$$

• Motion consistency loss matching relative speeds:

$$\mathcal{L}_m = \frac{1}{T-2} \sum_{t=3}^T \left[ \frac{ \| \hat{M}^{t-1} - \hat{M}^{t-2} \|_2^2 }{ \| G^{t-1} - G^{t-2} \|_2^2 } - \frac{ \| \hat{M}^t - \hat{M}^{t-1} \|_2^2 }{ \| G^t - G^{t-1} \|_2^2 } \right]^2$$

• Appearance regularization:

$$\mathcal{L}_a = \frac{1}{T V_2} \sum_{t=1}^T \sum_{p=1}^{V_2} \sum_{u \in N(p)} ( \| p_{\hat{M}^t} - u_{\hat{M}^t} \|_2^2 - \| p_{M_0} - u_{M_0} \|_2^2 )^2$$

• Full objective: $$\mathcal{L} = \lambda_r \mathcal{L}_r + \lambda_m \mathcal{L}_m + \lambda_a \mathcal{L}_a$$.

Hallo3 (Cui et al., 1 Dec 2024):

• Diffusion loss:

$$\mathcal{L}_{\text{diff}} = \mathbb{E}_{t,z_0,\epsilon} \| \epsilon - \epsilon_\theta(z_t, t, c) \|^2$$

• VAE ELBO, identity perceptual losses, optional adversarial/background-immersion, and sync losses for audio-video alignment.

FactorPortrait (Tang et al., 12 Dec 2025):

• Denoising loss:

$$L_{\text{diff}} = \mathbb{E}_{z_0, \epsilon, t} \| \epsilon - \epsilon_\theta(z_t, t, \text{cond}) \|^2$$

• Occasional reference frame reconstruction and implicit regularization via domain-specific data splits to achieve disentanglement between facial expressions and camera views.

4. Input Modalities and Control Signal Disentanglement

Recent advances address direct control over expression, pose, and viewpoint, overcoming previous constraints due to entangled latent representations.

• FactorPortrait (Tang et al., 12 Dec 2025): Expression signals $e_i$ are extracted per driving frame via deep feature encoders and contextually aggregated with temporal attention, allowing for precise chunk-level injection via AdaLN. Camera trajectories are encoded as Plücker ray maps, while pose is captured by per-frame normal maps from tracked 3D meshes. Channel-wise fusion aligns the conditioning tokens to latent representations, facilitating explicit, disentangled control over expression, pose, and viewpoint.
• Hallo3 (Cui et al., 1 Dec 2024): Audio-driven conditioning is fused via cross-attention, producing lip-synchronous animated portraits even when extrapolating videos with moving cameras or dynamic backgrounds. Motion-frame feedback (caching last $n$ decoded frames as new latent conditions) enables progressive, continuous generation beyond the original clip.

This explicit disentanglement and conditioning enable full cross-identity reenactment and multi-view synthesis, supporting arbitrary camera trajectories and dynamic scene content.

5. Training Protocols and Data Regimes

All models utilize curated data and progressive training schedules, emphasizing large-scale multimodal datasets, adaptive curriculum, and multi-stage optimization.

• AniFormer (Chen et al., 2021): Trained on DFAUST, FAUST, MG-cloth, SMAL datasets, using 30-frame motion clips, randomizing vertex and face order, with Adam optimizer (lr=$5\mathrm{e}^{-5}$); batch size 2, sliding-window inference with $T=3$.
• Hallo3 (Cui et al., 1 Dec 2024): Employs benchmark/wild datasets for evaluation, using $T_{\max}=50$ diffusion steps, 42-layer identity transformers, motion-frame window $n=2$.
• FactorPortrait (Tang et al., 12 Dec 2025): Constructs a synthetic dataset with 802 3D head avatars, diverse camera trajectories (ViewSweep, DynamicSweep), and mixes real (PhoneCapture, StudioCapture) and synthetic sequences in a staged curriculum (Table below). Learning rates are annealed from $1\mathrm{e}^{-4}$ to $2\mathrm{e}^{-5}$ over 90k iterations.

Stage	Phone	Studio	ViewSweep	DynamicSweep	Timesteps L
1	100%	–	–	–	13
2	60%	40%	–	–	25
3	20%	20%	30%	30%	49
4	20%	20%	30%	30%	81

6. Performance Benchmarks and Observed Capabilities

Quantitative evaluation demonstrates substantial improvements in fidelity, temporal consistency, and control.

• AniFormer (Chen et al., 2021): Achieves point-wise mesh Euclidean distance (PMD) of 0.12 on DFAUST (seen motions) and 3.95 (unseen motions), outperforming LIMP (5.44/7.11) and NPT (4.37/5.31). Ablations show dramatic improvements with combined losses.
• Hallo3 (Cui et al., 1 Dec 2024): Validated on multiple data regimes, demonstrating that transformer-based video backbones generalize to non-frontal perspectives, immersive backgrounds, and dynamic objects, with robust identity preservation and lip-sync across wild datasets.
• FactorPortrait (Tang et al., 12 Dec 2025): Delivers 20–30% lift in PSNR/SSIM, 2–3× lower FID and FVD, and notably higher expressiveness and disentanglement over GAGAvatar, CAP4D, and HunyuanPortrait. Preserves detailed hair strands, skin wrinkles, and mouth interiors, supports arbitrary camera and pose trajectories.

7. Adaptation Strategies for High-Fidelity Portrait Animation

Extending mesh-centric architectures like AniFormer to detailed portrait meshes involves several procedure modifications (Chen et al., 2021):

• Mesh resolution: Increase vertex count ($V$) to 10,000–20,000, enforce consistent topology.
• Facial priors: Inject landmark features from pretrained networks, augment per-vertex features with 2D/3D positional encodings.
• Expression bases: Use blendshape models (FLAME, FaceWarehouse) and inject deformation coefficients as sequence tokens or cross-attention conditions.
• Loss tuning: Upweight appearance regularization, add perceptual geometry losses for sensitive facial features.
• Training data: Use 4D face datasets (VOCASET, Audio2Face) and synthetically augment with diverse expressions and poses.
• Implementation: Increase channel capacity, frame count, and adopt mixed precision to accommodate high-resolution portrait meshes.

Similarly, video-diffusion transformer models rely on identity latents for long-term consistency and multimodal signal fusion. These strategies suggest practical pathways to scalable, end-to-end 3D portrait animation for applications in facial reenactment, multi-modal avatar synthesis, and realistic digital humans.

---

A plausible implication is that transformer-based 3D portrait animation models now achieve controllable synthesis with fine-grained disentangled signals and generalized scene understanding, leveraging advances in causal VAEs, spatio-temporal attention, and multimodal conditioning. These developments mark a significant shift away from rigid mesh correspondence and frame-wise regression, establishing a new standard for dynamic, identity- and motion-consistent portrait animation.