EAI-Avatar: Emotion-Aware Talking Head System

• EAI-Avatar is an emotion-aware talking head framework that couples dialogue generation, emotion propagation, and head motion synthesis for natural dyadic interactions.
• The system leverages large language models like GPT-4 to create temporally consistent avatars that seamlessly transition between speaking and listening states.
• Empirical evaluations demonstrate superior performance in metrics such as SSIM, PSNR, and FID, advancing beyond traditional lip-sync driven systems.

Searching arXiv for the named topic and closely related avatar papers. EAI-Avatar is an emotion-aware interactive talking head generation framework for dyadic interactions that couples dialogue generation, emotion propagation, head-motion synthesis, and diffusion-based rendering. In its canonical formulation, it leverages the dialogue generation capability of LLMs such as GPT-4 to produce temporally consistent virtual avatars with rich emotional variations that seamlessly transition between speaking and listening states, rather than restricting animation to one-way speech-driven portrait motion (Yang et al., 25 Aug 2025).

1. Research setting and conceptual scope

EAI-Avatar is motivated by a limitation in most existing talking-head systems: they animate a single portrait from audio, optimize for image quality and lip-sync, and typically do not show a listener, do not handle turn-taking, and treat emotional style as static or very coarse. Listener-focused systems generate non-verbal listener reactions, but only animate the listener and lack a mechanism to naturally switch between listening and speaking. Recent dyadic interaction works model both speaker and listener motion, but role switching often feels mechanical, and emotion is either ignored or modeled in a static, text-conditioned way (Yang et al., 25 Aug 2025).

The framework therefore targets two psycholinguistic properties that it treats as structurally important: progressive accumulation, in which current emotion depends on history, and the recency effect, in which recent dialogue turns affect emotion more strongly. This places EAI-Avatar in a distinct position relative to zero-shot talking-avatar systems such as GAIA, which disentangle appearance and motion for speech-driven portrait animation from a single image and a speech clip, and to real-time interactive head-avatar systems such as Avatar Forcing, which emphasize causal low-latency user-avatar interaction from audio and motion streams (He et al., 2023, Ki et al., 2 Jan 2026).

A common misconception is to equate EAI-Avatar with lip-sync alone. The technical formulation is broader: the system models dialogue state, emotional state, head motion, speaker facial motion, listener facial motion, and final frame synthesis in a single generative pipeline (Yang et al., 25 Aug 2025).

2. Interactive Talking Tree and emotion-state propagation

The core dialogue-state abstraction is the Interactive Talking Tree (ITT), defined as a six-tuple

$\mathcal{T} = (V, G, \Psi, \Lambda, S, D),$

where $V$ is the set of nodes, $G = \{\text{speak}, \text{listen}\}$ is the dialogue-state space, $\Psi = \{\psi_1, \psi_2, \dots, \psi_k\}$ is the emotion-state set, $\Lambda: V \to G \times \Psi \times [0,1]$ is the node annotation function, $S$ is the structural query operator for child/parent/left-sibling/right-sibling relations, and $D$ is the depth query (Yang et al., 25 Aug 2025).

Each node represents one dialogue turn by one character and stores at least the current dialogue state, an emotion category, a confidence value, and tree relations. Turn-taking is encoded by the tree-growth rule

$$v_{\text{next}} = \begin{cases} S_{\text{child}}(v), & \text{if } G(v) = \text{listen},\ S_{\text{rs}}(v), & \text{if } G(v) = \text{speak}. \end{cases}$$

Nodes at the same depth correspond to one dialogue round; a speaker node is followed by a right sibling, while a listener node extends the interaction deeper into the tree (Yang et al., 25 Aug 2025).

Per-node emotion is inferred by Emotion-LLaMA. When both roles are virtual, emotion is inferred from text only; when one role is a real human, Emotion-LLaMA uses text, audio, and video. If $f \in \mathbb{R}^d$ is the node feature vector, then emotion category and confidence are

$$\psi = \arg\max_k \left(\mathrm{softmax}(W^\top f + b)\right), \qquad P = \max \left(\mathrm{softmax}(W^\top f + b)\right).$$

This node-local estimate is not the final control signal for synthesis. EAI-Avatar instead defines an emotion propagation query $V$0 that aggregates historical emotional cues by reverse-level traversal and depth weighting, thereby encoding both accumulation and recency (Yang et al., 25 Aug 2025).

For a speaker node, the effective emotion is computed from the node’s own confidence plus weighted history from the previous depth; for a listener node, it is computed from weighted history over the current and previous depths. This structure is the principal mechanism by which EAI-Avatar moves beyond static text-conditioned affect. A plausible implication is that emotion is treated less as an instantaneous label than as a state variable over dialogue topology.

3. Motion synthesis, facial control, and diffusion rendering

Global head motion is produced by the Consistent Random Head Mask Generator (CRHMG), a Transformer-based module that learns temporally consistent motion features in a latent mask space and generates arbitrary-length, temporally consistent mask sequences. A head mask $V$1 is first encoded into a latent condition

$V$2

with implementation dimension $V$3. To synthesize a sequence of length $V$4, the latent is repeated, noise is injected,

$V$5

and a stack of $V$6 Transformer encoder layers with $V$7 heads produces the temporally structured latent sequence before decoding it back to masks (Yang et al., 25 Aug 2025).

Speaker and listener facial motion are generated differently. Speaker facial motion uses EAT (Efficient Emotional Adaptation) as the audio-to-expression model, fine-tuned per character so that audio and propagated emotion $V$8 are mapped to expression coefficients and then to 3DMM-driven 2D facial motion. Listener facial motion uses the Listener Emotion Expression Dictionary (LEED): for each character and emotion label, multiple expression sequences are precomputed with EAT and silent audio, then retrieved stochastically at runtime according to the required emotion and duration (Yang et al., 25 Aug 2025).

The final image synthesis stage is a ControlNet-based diffusion model conditioned jointly on head masks and facial motion. For frame sequence generation, the model is trained with the denoising objective

$V$9

where $G = \{\text{speak}, \text{listen}\}$0 is the mask sequence and $G = \{\text{speak}, \text{listen}\}$1 is the speaker or listener facial-motion sequence (Yang et al., 25 Aug 2025).

This architecture differs from speech-driven foundation models such as GAIA, which learn speech-to-motion in a disentangled latent space from a single portrait image, and from causal interactive generators such as Avatar Forcing, which model real-time user-avatar interactions through diffusion forcing with low latency for verbal and non-verbal cues. EAI-Avatar instead gives explicit architectural priority to dialogue-state transitions and emotion history (He et al., 2023, Ki et al., 2 Jan 2026).

4. Training protocol, datasets, and empirical performance

EAI-Avatar is trained on ViCo and ViCoX. ViCo provides 1.6 hours of video with single identities and responsive interactions; ViCoX contains multi-turn face-to-face performances for dyadic interaction modeling. The network trains one identity at a time, with 15 selected identities, comprising 10 male and 5 female. After training, any two trained identities can hold multi-turn conversations, with GPT-4 generating the dialogue text (Yang et al., 25 Aug 2025).

Implementation uses PyTorch on an RTX 3090 GPU. CRHMG is trained with Adam, learning rate $G = \{\text{speak}, \text{listen}\}$2, 400 epochs, and batch size 16. The conditional diffusion model is also trained with Adam at learning rate $G = \{\text{speak}, \text{listen}\}$3, for 15,000 iterations and batch size 4. Frames are resized to $G = \{\text{speak}, \text{listen}\}$4, and final synthesis combines CRHMG head masks, EAT or LEED facial motion, and diffusion rendering (Yang et al., 25 Aug 2025).

The evaluation protocol uses SSIM, PSNR, FID, LPIPS, SyncScore, SID, Var, and Temporal Consistency. The reported top-line results are summarized below (Yang et al., 25 Aug 2025).

Metric	EAI-Avatar	Reported status
SSIM	0.852	best
PSNR	32.039	best
FID	13.622	lowest
LPIPS	0.236	lowest
SyncScore	7.230	highest
Temporal Consistency total error	0.41	best

Ablation results show that removing CRHMG or ITT degrades performance, and the temporal-consistency comparison reports total error values of 1.98 for Sonic, 1.52 for DIM, 8.12 for DiffusionRig, 0.88 for the model without CRHMG, 0.81 for the model without ITT, and 0.41 for the full system (Yang et al., 25 Aug 2025).

The user study used 40 participants, 10 generated videos from 5 dialogue pairs, and 5-point Likert scales over movement naturalness, motion variety, lip-sync accuracy, emotional and contextual coherence, temporal consistency, and overall visual fidelity. The system received the highest ratings across all categories; the paper reports 87.5% top scores for movement naturalness and 90% for emotional and contextual coherence (Yang et al., 25 Aug 2025).

5. Relation to adjacent avatar-generation systems

EAI-Avatar sits within a broader technical landscape that includes avatar creation, head reconstruction, full-body embodiment, and affective sensing. Speech-driven zero-shot portrait animation is represented by GAIA, which synthesizes natural talking videos from speech and a single portrait image and removes domain priors such as 3DMMs and warping heuristics (He et al., 2023). Real-time interactive head generation is represented by Avatar Forcing, which models real-time user-avatar interactions through diffusion forcing, processes the user's audio and motion with approximately 500ms latency, and reports a 6.8X speedup compared to its baseline (Ki et al., 2 Jan 2026). Infinite-length streaming diffusion is represented by Live Avatar, which reaches 20.88 FPS end-to-end generation on 5 H800 GPUs and is designed for practical, real-time, high-fidelity long-form avatar synthesis (Huang et al., 4 Dec 2025).

At the representation layer, EAvatar provides expression-aware and deformation-aware 3D Gaussian Splatting for high-fidelity head reconstruction, with a sparse expression control mechanism, displacement-based Gaussian splitting, and real-time rendering at approximately 32 FPS on RTX 3090 (Zhang et al., 19 Aug 2025). EVA extends expressive control to the full body with an actor-specific framework that independently controls facial expressions, body movements, and hand gestures through a two-layer model combining an expressive template geometry layer and a 3D Gaussian appearance layer (Junkawitsch et al., 21 May 2025). A plausible implication is that EAI-Avatar’s dyadic dialogue model can be interpreted as complementary to these geometry- and rendering-centric systems: one emphasizes conversational state and emotion propagation, the others emphasize controllable embodiment and rendering fidelity.

Upstream avatar creation is addressed by systems such as SmartAvatar, which generates fully rigged, animation-ready 3D human avatars from a single photo or textual prompt through a VLM-agent verification loop, and EasyCraft, which maps text or image inputs to engine-ready avatar parameters across RPG engines through a feedforward multi-modal translator (Huang-Menders et al., 5 Jun 2025, Wang et al., 3 Mar 2025). Presentation-oriented avatar generation is represented by Pre-Avatar, which generates a presentation video with a talking face of a target speaker from 1 front-face photo and a 3-minute voice recording (Sun et al., 2022). Affective sensing pipelines such as EVOKE contribute lightweight EEG-based valence-arousal-dominance recognition with 87.62% accuracy and explicit mapping from VAD labels to eight emotion classes and custom 3D avatars (Nadeem et al., 2024). This suggests a broader EAI-avatar stack in which identity creation, affect recognition, motion synthesis, and rendering are separable but interoperable subsystems.

6. Deployment tensions, user acceptance, and future directions

EAI-Avatar notes several limitations directly: training per identity is required; the method relies on ViCo and ViCoX, so dataset biases in actors and recording conditions may limit generalization; head motion is constrained by masks; and interactive latency remains affected by LLM inference, TTS, and diffusion rendering (Yang et al., 25 Aug 2025). Adjacent work indicates two active engineering responses to these constraints: Avatar Forcing prioritizes causal low-latency interaction, while Live Avatar prioritizes industrial long-form streaming throughput (Ki et al., 2 Jan 2026, Huang et al., 4 Dec 2025). This suggests that present systems divide along an axis between emotion-aware dialogue modeling and runtime-optimal rendering systems, rather than fully solving both simultaneously.

A second deployment tension concerns appearance realism and acceptance. In a study of six embodied conversational agents, increased perceived humanness correlated with increased perceived eeriness, and human-controlled face-tracked avatars were more human-like but also more eerie than less realistic autonomous agents (Thaler et al., 2021). A plausible implication for EAI-Avatar is that richer emotional motion and more realistic rendering do not automatically increase social acceptability; in moderate-realism regimes, they may intensify uncanny responses if motion fidelity, expression timing, and rendering realism are not matched.

Several future directions are already suggested by the surrounding literature. One is tighter integration of real-time interaction and expressive preference optimization, as exemplified by Avatar Forcing’s label-free DPO scheme for expressive user-avatar reactions (Ki et al., 2 Jan 2026). Another is extension from head-only dyadic interaction to fully controllable digital humans, as pursued by EVA’s decoupled face-body-hand control (Junkawitsch et al., 21 May 2025). A third is broader multimodal emotion conditioning, including physiological signals such as EEG through VAD-based pipelines like EVOKE (Nadeem et al., 2024). Taken together, these developments indicate that EAI-Avatar is best understood not as a terminal solution to talking-head generation, but as a research program centered on emotionally coherent, bidirectional, and controllable avatar interaction.