3D Emotion Transfer Network

• 3D Emotion Transfer Networks are models that modify facial expressions on 3D avatars while retaining speaker identity and articulation.
• They employ techniques such as geometry mapping, continuous affect conditioning, and dual-path modulation to enable explicit emotional control.
• Key challenges include data scarcity, balancing emotion with lip synchronization, and ensuring geometric stability in synthesized faces.

A 3D Emotion Transfer Network is a class of facial animation and avatar models that attempts to preserve identity and speech-driven articulation while altering or controlling affect in a 3D representation. In current arXiv literature, closely related systems range from unpaired emotion transfer on 3D facial meshes to continuous emotion-conditioned 3D Gaussian splatting and explicit emotion control in feed-forward single-image head avatars; these works do not all expose the same notion of transfer, and several explicitly distinguish target-emotion control from merely emotion-aware synthesis (Wang et al., 2021, Cha et al., 2 Feb 2025, Xu et al., 22 Mar 2026, Gong et al., 16 Apr 2026).

1. Terminological scope and defining criteria

The term has no single canonical definition across the literature. In one line of work, it denotes a system that takes a neutral or source 3D face performance and rewrites it into a target emotion domain while preserving speech content and identity. This is the framing of 3D-TalkEmo, which formulates the task as audio-driven 3D talking head synthesis with unpaired emotion transfer and treats emotion as a separately controllable attribute layered onto speech-driven motion (Wang et al., 2021).

A second line of work uses explicit affective controls without source-expression transfer in the strict sense. EmoTalkingGaussian conditions a subject-specific 3D Gaussian splatting avatar on continuous valence and arousal while retaining lip synchronization with input audio; it is therefore an emotion-conditioned 3D renderer rather than a reference-expression transfer system (Cha et al., 2 Feb 2025). A third line of work is emotion-aware rather than emotion-controllable: EmoTaG uses Semantic Emotion Guidance and a Gated Residual Motion Network to synthesize motion coherent with emotional prosody in audio, but it does not expose an inference-time target emotion label, embedding, reference clip, or style token (Xu et al., 22 Mar 2026). A fourth formulation explicitly separates articulation from emotion in FLAME space and injects a target emotion label into both geometry and appearance branches of a feed-forward avatar backbone, making emotion a first-class control signal (Gong et al., 16 Apr 2026).

System	Emotion interface	3D substrate
3D-TalkEmo	discrete target domain label	3D mesh and geometry map
EmoTalkingGaussian	continuous valence/arousal	3D Gaussian splatting
EmoTaG	teacher-distilled emotion guidance, not user-controllable	FLAME-guided rigged 3DGS
Giving Faces Their Feelings Back	explicit discrete target emotion label	feed-forward 3D head avatar

This suggests that the most precise encyclopedic use of the term is not a specific architecture but a functional criterion: the network should preserve identity, maintain articulation or driving motion, and alter affective state in a 3D facial representation under some explicit or implicit emotional mechanism.

2. Problem formulation and factorization of identity, articulation, and affect

A recurrent design principle is factorization. In 3D-TalkEmo, the inputs are a static neutral 3D facial mesh, a speech audio sequence, and a target emotion label; the system first produces neutral talking motion and then translates that motion into one of four emotional domains: neutral or calm, happy, angry, and surprise (Wang et al., 2021). Identity is anchored by the neutral mesh and fixed identity parameters, speech content is derived from DeepSpeech features, and emotion is introduced later through a StarGAN domain label. The staged pipeline therefore decomposes the problem into neutral talking synthesis and emotion transfer rather than learning a monolithic audio-to-emotionally-expressive mesh mapping.

EmoTalkingGaussian uses a related but not identical decomposition. Its inside-mouth branch is audio-driven, its face branch is driven by audio plus action units, and its emotion branch is driven by a continuous 2D vector of valence and arousal, $e \in [-1,1]^2$ (Cha et al., 2 Feb 2025). Emotion is added as further per-Gaussian offsets in position, scale, and rotation on top of audio- and AU-driven motion. The representation is not a transferred source expression but a controllable low-dimensional affect code.

In feed-forward single-image avatars, the same separation is formulated in FLAME parameter space. “Giving Faces Their Feelings Back” defines the task as

$\mathcal{A} = \mathcal{F}(I, \mathbf{p}_{1:T}, e),$

where $I$ is a single reference RGB image, $\mathbf{p}_{1:T}$ is a sequence of FLAME-based motion coefficients, and $e$ is an explicit emotion label (Gong et al., 16 Apr 2026). Geometry modulation acts on expression and jaw parameters to preserve articulation while replacing residual emotional bias, and appearance modulation adds emotion-dependent visual cues such as wrinkles and shading changes. In this formulation, a 3D emotion transfer network is explicitly a mechanism for overriding the emotional content of a driver while retaining identity and timing.

EmoTaG occupies a narrower position within this landscape. It predicts FLAME expression and jaw parameters plus intra-oral Gaussian residuals, and it decomposes motion into a base branch, a residual branch, and a gate,

$$\boldsymbol{\delta} = \boldsymbol{\delta}_{\text{b}} + g \cdot \boldsymbol{\delta}_{\text{r}}.$$

However, its emotion variables are teacher targets rather than user controls, so it is best described as emotion-aware motion synthesis rather than explicit transfer (Xu et al., 22 Mar 2026).

3. Representational substrates and architectural patterns

The representational choice largely determines how emotion transfer is realized. In 3D-TalkEmo, the central representation is the geometry map, which converts a registered 3D facial mesh into a structured $128 \times 128 \times 3$ tensor by computing all pairwise geodesic distances on a template mesh, applying classical multidimensional scaling, keeping the first two dimensions, mapping them to a canonical image plane, and storing the original 3D coordinates in three channels (Wang et al., 2021). This permits the use of StarGAN for unpaired multi-domain translation on face geometry rather than on unordered vertex coordinates, with generator $G(x,c)$ and discriminator heads $D_{src}$ and $D_{cls}$. The emotion-transfer objective combines adversarial, domain-classification, and reconstruction terms:

$\mathcal{A} = \mathcal{F}(I, \mathbf{p}_{1:T}, e),$0

Within mesh-based transfer, this is one of the clearest examples of explicit emotion-domain rewriting.

EmoTalkingGaussian uses anisotropic 3D Gaussians with canonical inside-mouth and face regions, persistent Gaussian fields, tri-plane hash encoding, and branch-specific manipulation networks (Cha et al., 2 Feb 2025). The face representation includes position, scale, rotation, opacity, color, and a normal residual; the emotion branch operates on already deformed face Gaussians and adds further offsets conditioned on valence and arousal. Rendering follows standard alpha compositing over inside-mouth, face, and background layers. The architectural significance is that emotion is injected into the 3D representation itself, not merely into the final image-space renderer.

EmoTaG replaces direct Gaussian deformation with prediction in structured FLAME parameter space, then moves rigged Gaussians with the mesh. This introduces explicit geometric priors and a hybrid motion model: global facial motion is FLAME-driven, while fine intra-oral articulation is handled by residual Gaussian offsets (Xu et al., 22 Mar 2026). The architecture is especially notable for its Gated Residual Motion Network, which decomposes articulation and emotional deviation without exposing a direct target emotion control variable.

The feed-forward explicit-control framework of “Giving Faces Their Feelings Back” retains the backbone’s original FLAME space and adds dual-path modulation rather than a new deformation basis (Gong et al., 16 Apr 2026). Geometry modulation computes

$\mathcal{A} = \mathcal{F}(I, \mathbf{p}_{1:T}, e),$1

where $\mathcal{A} = \mathcal{F}(I, \mathbf{p}_{1:T}, e),$2 is the learned emotion token, while appearance modulation augments image features by

$\mathcal{A} = \mathcal{F}(I, \mathbf{p}_{1:T}, e),$3

This makes geometry and appearance complementary transfer channels rather than forcing all emotional variation into one space.

A notable 2D precursor is GC-GAN, which uses 68 facial landmarks as continuous conditions, embeds them into a 32-dimensional semantic expression manifold with contrastive learning, and concatenates the expression latent with an identity latent before generation (Qiao et al., 2018). Although it is not a 3D method, its geometry-conditioned latent design is a direct conceptual antecedent for 3D expression and emotion embeddings.

4. Data construction, supervision, and training regimes

Data scarcity is a central constraint in 3D emotion transfer. 3D-TalkEmo addresses the lack of large-scale synchronized 3D emotional talking data by reconstructing 3D faces from around 10,000 audio-synchronized videos from RAVDESS and LBG using 3DMM-based fitting and the FaceScape model, then augmenting non-neutral emotions because landmark-based reconstruction alone does not show sufficiently strong emotional deformation (Wang et al., 2021). Its augmentation strategy uses reference FaceScape expressions such as calm, eyebrows up, eyebrows down, and grin, and adds weighted displacement to reconstructed meshes. Training is split by stage: the neutral talking model uses paired audio and reconstructed neutral mesh motion, while the emotion transfer model uses emotion labels, realism discrimination, target-domain classification, and cycle consistency but not paired same-content cross-emotion supervision.

EmoTalkingGaussian solves a different supervision problem: short monocular subject-specific videos do not contain enough emotional diversity to train an emotion branch (Cha et al., 2 Feb 2025). It therefore constructs synthetic emotional supervision using a lip-aligned emotional face generator based on EmoStyle and StyleGAN2, with lip landmark, lip pixel, regularization, emotion, and identity losses. To reduce synthetic-real domain gap, emotion-expressive regions around the eyes and mouth are cut and pasted onto original frames with seamless cloning, and the 3D model is trained with masked RGB, LPIPS, D-SSIM, normal-map losses, and a SyncNet-based self-supervised lip-sync term. The training is staged: canonical Gaussians, mouth and face manipulation networks, emotion branch, then face-canonical fine-tuning.

EmoTaG follows a Pretrain-and-Adapt paradigm. A universal motion prior is pretrained on HDTF, using 70 videos from 70 identities, and adaptation to a new identity uses only 5 seconds of video while freezing the pretrained GRMN and fine-tuning only AdaIN modulation parameters (Xu et al., 22 Mar 2026). Emotional supervision is distilled from DeepFace as a 7-way emotion distribution and a scalar intensity score. This reduces the need for explicit dense emotion labels in 3D avatar training, but it also means that emotion enters as teacher supervision rather than a direct generative interface.

The explicit feed-forward control framework constructs a time-synchronized, emotion-consistent multi-identity dataset by using EmoTalk3D to generate frame-synchronized anchor sequences with the same person, same sentence, identical timing, and different emotions, then transferring those dynamics to many identity images with X-NeMo (Gong et al., 16 Apr 2026). This dataset supplies supervision for both emotion-induced geometry differences under fixed articulation and identity-aware appearance responses under shared emotion. The geometry and appearance branches are trained with

$\mathcal{A} = \mathcal{F}(I, \mathbf{p}_{1:T}, e),$4

and

$\mathcal{A} = \mathcal{F}(I, \mathbf{p}_{1:T}, e),$5

A 2025 abstract, “EmoDiffusion,” describes a latent-diffusion formulation with two VAEs, an Emotion Adapter, and a 3D-BEF dataset, but the supplied document itself contains only a minimal LaTeX skeleton. As a result, its method, losses, and empirical claims cannot be technically reconstructed from the provided content, even though the abstract indicates another emerging design direction for 3D emotional facial animation (Zhang et al., 14 Mar 2025).

5. Evaluation protocols and empirical behavior

Evaluation in this area typically measures geometric accuracy, emotional correctness, realism, and lip synchronization rather than a single transfer score. 3D-TalkEmo evaluates on 500 video segments from 27 unseen speakers, using Reconstruction Error, Velocity Error, and Classification Error, and reports a user study with 23 participants, 15 queries each, and 345 total responses (Wang et al., 2021). Its ablations show that mouth-local weighting improves the neutral talking stage and that emotion augmentation strengthens emotion discrimination, while comparison against a PointNet-based generator indicates that geometry map plus StarGAN yields cleaner and more plausible emotional deformation than unordered point-set processing.

EmoTalkingGaussian evaluates self-reconstruction, cross-domain audio, and an emotion-conditioned scenario on four public subjects: Macron, Obama, Lieu, and May (Cha et al., 2 Feb 2025). In the emotion-conditioned setting, it reports lower V-RMSE and A-RMSE, higher sign agreement, and higher Emotion Accuracy than ER-NeRF, GaussianTalker, and TalkingGaussian; the full model obtains V-RMSE 0.352, A-RMSE 0.383, V-SA 0.766, A-SA 0.637, and E-Acc 46.6. Ablations indicate that removing the emotion branch collapses E-Acc from 46.6 to 25.3, while combining face and emotion branches improves LPIPS but degrades Sync-C and E-Acc.

EmoTaG evaluates self-reconstruction, emotion-intensity generalization, and OOD audio-driven synthesis, with PSNR, SSIM, LPIPS, LMD, AUE-L/U, Sync-C, and Sync-E (Xu et al., 22 Mar 2026). On the emotional test set it reports PSNR 29.95, LPIPS 0.022, SSIM 0.877, LMD 2.456, AUE-L 0.702, AUE-U 0.236, Sync-C 6.147, train time 11 min, and 76.4 FPS. Its ablations show that removing the gate, residual branch, or AdaIN identity modulation sharply worsens visual fidelity, lip accuracy, and synchronization, supporting the GRMN decomposition.

The feed-forward explicit-control framework evaluates both reconstruction fidelity and emotion transfer using PSNR, SSIM, LPIPS, CSIM, AED, and APD (Gong et al., 16 Apr 2026). In self-identity reenactment, LAM augmented with the proposed emotion control improves PSNR from 18.083 to 20.437, SSIM from 0.6673 to 0.7373, LPIPS from 0.2750 to 0.2315, CSIM from 0.7557 to 0.8541, AED from 0.0337 to 0.0258, and APD from 0.2803 to 0.2272. In the emotion transfer benchmark, LAM with the method achieves PSNR 18.614, SSIM 0.6864, LPIPS 0.2678, CSIM 0.7586, AED 0.0314, and APD 0.4292. The user study of 141 participants rating 30 videos further indicates improved emotion recognizability and emotional vividness.

Taken together, these results support a consistent empirical pattern: explicit emotion pathways, whether implemented as domain translation, continuous affect conditioning, gated residual motion, or dual-path modulation, outperform architectures that leave emotion entangled with neutral articulation, reference appearance, or unstable geometric deformation.

6. Misconceptions, limitations, and adjacent paradigms

A common misconception is that any emotionally expressive 3D talking head model is a 3D emotion transfer network. The literature does not support that equivalence. EmoTaG explicitly lacks user-selectable target emotion control at inference and is therefore emotion-aware rather than transfer-based in the strong sense (Xu et al., 22 Mar 2026). EmoTalkingGaussian exposes continuous affect control, but it does not transfer a source facial expression from another identity or reference image at inference; its transfer signal is valence and arousal rather than a source-expression exemplar (Cha et al., 2 Feb 2025).

Another misconception is that emotion transfer can be reduced to geometry alone. The feed-forward explicit-control framework shows that removing the appearance branch leaves source emotional texture “baked in,” while removing the geometry branch preserves the wrong emotional geometry from the driver or reference (Gong et al., 16 Apr 2026). This indicates that successful transfer often requires simultaneous control of deformation and appearance-dependent affective cues.

The major technical limitations recur across models. 3D-TalkEmo uses only four emotions and relies on reconstructed and augmented data rather than directly captured 3D emotional performance (Wang et al., 2021). EmoTalkingGaussian remains subject-specific, depends on synthesized emotional supervision, and exhibits artifacts around the mouth under strong emotions (Cha et al., 2 Feb 2025). EmoTaG requires auxiliary pose and expression frames because audio alone carries limited information about upper-face expression and head pose (Xu et al., 22 Mar 2026). The feed-forward explicit-control framework depends on emotion-synchronized anchor sequences and degrades at large yaw or near-profile views (Gong et al., 16 Apr 2026).

Related 2D systems clarify the boundary of the field. CEM-Net formulates cross-emotion transfer as moving from the source emotion in the reference image to the target emotion implied by audio by retrieving an expression displacement from memory, but it is explicitly a 2D landmark-and-warping pipeline rather than a 3D model (Wu et al., 17 Aug 2025). GC-GAN likewise provides an important precursor through geometry-conditioned continuous expression transfer, yet it operates on 2D landmarks and 2D image generation (Qiao et al., 2018). A plausible implication is that some of the most reusable ideas for future 3D emotion transfer networks may come from these adjacent paradigms: semantic geometry embeddings, residual emotion displacements, and memory-based emotion-transition retrieval.

In this broader sense, the field is converging on a shared problem statement even when architectures differ: identity, articulation, and emotion are partially entangled in observed face motion, and a 3D emotion transfer network is valuable to the extent that it makes those factors separately controllable without sacrificing realism, synchronization, or geometric stability.