EmoCAST: Emotional Talking-Head Synthesis

• EmoCAST is a diffusion-based framework that generates emotional talking-head videos from a reference image, driving audio, and emotive text prompts, ensuring identity consistency and nuanced expression.
• It employs a decoupled cross-attention mechanism to separately process facial and textual features, enhancing emotion control and speech-driven motion synchronization.
• The model leverages the new ETTH dataset and progressive training strategies to improve expression quality, lip synchronization, and out-of-domain generalization.

Searching arXiv for EmoCAST and closely related talking-head synthesis work to ground the article in recent literature. EmoCAST is a diffusion-based emotional talking portrait generation framework that synthesizes a talking-head video from a single reference image, driving audio, and an emotive text prompt, with the explicit goal of producing realistic facial motion, identity-consistent appearance, and flexible text-driven emotional expression (Jiang et al., 28 Aug 2025). It is positioned against prior talking-head systems that prioritize lip synchronization and identity preservation while treating emotion either coarsely, through discrete labels, or indirectly, through reference emotional videos. EmoCAST instead treats emotive text descriptions as the control interface for affect, and couples this interface to both appearance modeling and audio-driven motion generation.

1. Problem formulation and control paradigm

EmoCAST addresses emotional talking head synthesis under a three-condition formulation: reference portrait image $x$, driving audio $a$, and emotive text prompt $t$ (Jiang et al., 28 Aug 2025). The reference image supplies identity and appearance, the audio supplies speech content and motion cues, and the text prompt specifies the target emotion in a human-readable form. The framework is motivated by limitations identified in existing methods: control flexibility, motion naturalness, and expression quality remain restricted, and currently available datasets are primarily collected in lab settings. The paper also argues that emotion labels are coarse and inflexible, while reference emotional videos are hard to collect, restrictive, and not user-friendly.

A central design premise is that emotive text descriptions are open-ended, human interpretable, more flexible than labels, easier to customize at inference time, and suitable for real-world prompting. In this formulation, emotional control is not reduced to selecting from a fixed taxonomy. Instead, the model is intended to learn how language maps to facial appearance and expression, and how emotion should interact with audio-driven motion. This separation of roles is one of the defining conceptual features of EmoCAST.

The framework’s stated contributions consist of four elements: the EmoCAST architecture itself; two key modules, namely a text-guided decoupled emotive module and an emotive audio attention module; a new Emotive Text-to-Talking Head dataset named ETTH; and two training strategies, emotion-aware sampling training and progressive functional training (Jiang et al., 28 Aug 2025). Taken together, these components define EmoCAST not merely as a conditional generator, but as a system that treats emotional controllability, identity preservation, and speech synchronization as jointly constrained objectives.

2. Diffusion formulation and backbone architecture

EmoCAST follows a latent diffusion paradigm (Jiang et al., 28 Aug 2025). A pretrained VAE encoder $E$ maps an image into latent space,

$z_0 = E(x).$

Forward diffusion then adds noise according to a variance schedule $\{\beta_t\}$,

$$q(z_t \mid z_{t-1}) = \mathcal{N}(z_t; \sqrt{1-\beta_t}\, z_{t-1}, \beta_t I),$$

with closed-form marginal

$$q(z_t \mid z_0) = \mathcal{N}(z_t; \sqrt{\bar{\alpha}_t} z_0, (1-\bar{\alpha}_t) I),$$

where

$$\alpha_t = 1 - \beta_t, \qquad \bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s.$$

A noisy latent sample is written as

$$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1-\bar{\alpha}_t}\epsilon, \qquad \epsilon \sim \mathcal{N}(0,I).$$

The denoiser is trained with

$a$0

where $a$1 denotes the full set of conditioning signals.

The network structure is built on Stable Diffusion-style components. The architecture includes a ReferenceNet for extracting appearance from the reference image, a Denoising UNet for generating latent video frames, temporal frame-wise attention for temporal coherence, and cross-attention for injecting audio as motion control (Jiang et al., 28 Aug 2025). EmoCAST then augments this backbone with two emotion-specific modules: the text-guided decoupled emotive module and the emotive audio attention module. This yields a structured conditioning decomposition:

• image $a$2 identity and appearance,
• text $a$3 emotion,
• audio $a$4 speech-driven motion.

The significance of this decomposition lies in the fact that emotional expression in talking-head synthesis is neither purely static nor purely dynamic. Appearance-level emotion manifests in expression shape and spatial facial configuration, while motion-level emotion alters the way speech articulators and expressive regions evolve over time. EmoCAST therefore situates emotional control at multiple representational levels rather than treating it as a single global conditioning token.

3. Emotion injection through text and audio interaction

The text-guided decoupled emotive module is designed to inject emotional text while preserving identity (Jiang et al., 28 Aug 2025). The paper rejects a naive concatenation strategy in which facial features and text embeddings are merged and processed through a shared cross-attention block, arguing that such a design weakens identity-related visual information and does not learn emotional expression cleanly from text. Instead, EmoCAST adopts a decoupled cross-attention mechanism inspired by IP-Adapter-style decoupling.

Given facial embedding $a$5, text embedding $a$6, and noisy latent $a$7, the model uses two parallel branches:

$a$8

$a$9

The outputs are then added,

$t$0

with expanded form

$t$1

where

$t$2

$t$3

This architecture is intended to preserve identity through the face branch while injecting emotion through the text branch, thereby reducing entanglement between appearance and text semantics.

The emotive audio attention module addresses the interaction between emotion and audio (Jiang et al., 28 Aug 2025). Audio embedding $t$4 is extracted using pretrained wav2vec, and text embedding $t$5 modulates audio through cross-attention:

$t$6

This produces an emotion-aware audio feature $t$7. The feature then interacts with visual latent feature $t$8 through cross-attention in a region-specific manner:

$t$9

$E$0

$E$1

Here $E$2 denotes elementwise product, and the masks correspond to lips, expression, and pose. The region features are fused by a convolution layer before being passed onward in the network.

In functional terms, this module attempts to align audio content, emotion semantics, and facial motion regions. It is therefore not simply an audio-conditioning block; it is an audio-conditioning block after emotion adaptation. The intended outcome is lip movement consistent with speech, facial expression consistent with the target emotion, and improved coordination between emotion and speech dynamics.

4. ETTH dataset and training regime

EmoCAST is coupled to a new dataset, ETTH, short for Emotive Text-to-Talking Head (Jiang et al., 28 Aug 2025). The dataset is constructed to address the paper’s claim that existing emotional talking-head datasets are small and mostly lab-collected. ETTH is described as an in-the-wild emotional talking head dataset with emotion labels, fine-grained emotive text descriptions, in-the-wild identities, and improved scale.

The data sources are MEAD, HDTF, CelebV-HQ, and Hallo3. Dataset construction proceeds in three stages. First, lip-sync filtering uses SyncNet scores, specifically Syn-C and Syn-D, to select suitable talking-head clips. Second, emotion annotation uses provided labels for MEAD, while Hallo3 and CelebV-HQ are processed with Emotion-FAN, fine-tuned on MEAD, to generate abstract emotion labels and intensity values. Third, emotive text generation uses MMHead to convert abstract emotion labels into textual descriptions of emotional styles. ETTH is reported to contain 15k+ identities, 158 hours, fine-grained emotion levels, and text descriptions.

Two training strategies are introduced to exploit this dataset (Jiang et al., 28 Aug 2025). The first, emotion-aware sampling training, is applied in an initial stage of emotion-conditioned image-to-image generation. Rather than sampling both reference and target from the same emotional video, the method samples the target image from an emotional video and the reference image from the neutral expression video of the same identity. The resulting neutral-to-emotional pairing is intended to force explicit learning of expressive transformation.

The second strategy, progressive functional training, is a coarse-to-fine multi-stage schedule. Phase 1, generalization enhancement, trains on a mixed dataset of emotional talking videos with diverse identities. Phase 2, emotion refinement, excludes in-the-wild videos and trains on MEAD emotional videos and HDTF high-quality neutral talking-head videos. Phase 3, lip-sync specialization, trains on HDTF only. The stated goals of these phases are, respectively, generalization across identities and sources, refinement of expression quality and motion accuracy with cleaner supervision, and maximization of lip synchronization while reducing interference between emotion control and speech synchronization.

These design choices are significant because they relocate emotional supervision from a purely architectural problem to a data-and-curriculum problem. A plausible implication is that EmoCAST’s performance depends not only on text conditioning, but also on exposing the model to explicitly contrastive neutral-to-expressive transformations and progressively specialized supervision.

5. Evaluation protocol, quantitative results, and ablations

EmoCAST is evaluated on two test settings (Jiang et al., 28 Aug 2025). The MEAD test set contains 4 test subjects and 256 emotional videos covering all 8 emotions. The in-the-wild out-of-domain test set contains 7 reference images, 40 audio samples, and 280 generated videos spanning 8 emotional categories. The reported metrics are emotion accuracy $E$3, measured by a pretrained emotion classifier; lip-sync metrics LSE-D and LSE-C from SyncNet, where lower LSE-D and higher LSE-C are better; and FID, where lower is better. Compared methods include MakeItTalk, SadTalker, EAMM, PD-FGC, EDTalk, EAT, Aniportrait, Echomimic, Hallo, and Hallo2.

On the MEAD test set, EmoCAST achieves emotion accuracy of 83.60%, LSE-D of 8.67, LSE-C of 6.79, and FID of 35.89 (Jiang et al., 28 Aug 2025). The paper describes these results as showing much better emotional accuracy than EAT, PD-FGC, and diffusion baselines, the best FID among listed methods, and competitive synchronization. On the in-the-wild test set, EmoCAST achieves emotion accuracy of 56.43%, LSE-D of 8.12, and LSE-C of 6.94. These numbers are presented as evidence of stronger out-of-domain generalization than prior methods, especially in emotion recognition.

A user study with 18 participants evaluates audio-visual sync, video quality, and emotion quality (Jiang et al., 28 Aug 2025). EmoCAST receives 3.63 for Audio-visual Sync, 3.81 for Video Quality, and 3.71 for Emotion Quality. Within the reported comparison, these are the highest user-study scores.

The ablation study attributes performance gains to each proposed component (Jiang et al., 28 Aug 2025). Removing the decoupled emotive module reduces emotion recognition and identity preservation quality. Removing the emotive audio attention module worsens LSE-D to 9.36, lowers LSE-C to 5.80, and reduces emotion accuracy to 61.72%, indicating degraded lip sync and emotional consistency. Removing emotion-aware sampling reduces emotion accuracy to 21.09%, which the paper interprets as evidence that neutral-to-emotional pairing is critical for learning expressive transformations. Removing progressive functional training worsens LSE-D to 9.99, lowers LSE-C to 5.45, and reduces emotion accuracy to 51.56%, supporting the claim that staged training is essential.

6. Interpretation, scope, and relation to adjacent affective-computing work

EmoCAST’s primary methodological claim is that emotion control in talking-head synthesis must be solved at both the appearance and motion levels (Jiang et al., 28 Aug 2025). Text prompts provide finer control than fixed emotion labels or reference emotion videos, the decoupled cross-attention design separates identity features from emotion features, the emotion-aware audio features align speech-driven motion with the intended affect, ETTH broadens textual emotion supervision, and the neutral-to-expressive plus progressive training regime improves both emotion learning and lip synchronization.

A common misconception would be to treat EmoCAST as a label-conditioned or reference-video-conditioned system. The framework is explicitly presented as an alternative to those paradigms: its control signal is emotive text description, not merely class labels, and its synthesis pathway is structured to preserve identity while interpreting textual emotion in conjunction with audio (Jiang et al., 28 Aug 2025). Another misconception would be to view the method as concerned only with static expression rendering. The architecture instead allocates substantial modeling capacity to the interaction between emotion and speech motion, especially through the emotive audio attention module.

Adjacent work helps clarify the problem space. Research on speech codecs shows that emotional cues in speech are fragile under compression and resynthesis, particularly for emotions such as sadness, depression, fear, and disgust, and that higher bitrate generally preserves emotion better (Ren et al., 2024). This suggests that the audio pathway in emotional talking-head synthesis is not a neutral carrier of motion cues; the emotional content present in the driving audio may itself be vulnerable to upstream degradation. Likewise, a contactless affective-state dataset for remote physiological emotion recognition reports that physiological signals often outperform facial expressions alone in realistic settings, and that multimodal fusion consistently improves performance over single-modality systems (Comas et al., 8 Jul 2025). A plausible implication is that facial expression, speech motion, and broader physiological correlates of affect should be regarded as partially complementary rather than interchangeable signals.

Within that broader landscape, EmoCAST occupies a specific position: it is a generative framework for emotional talking portraits, not a codec evaluation framework and not a contactless physiological benchmark. Its importance lies in formalizing emotive text as a control interface, in operationalizing emotion-aware audio-motion interaction within latent diffusion, and in coupling those design choices to a dataset and training curriculum tailored for expressive talking-head generation (Jiang et al., 28 Aug 2025).