Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video

Abstract: Talking face generation has gained significant attention as a core application of generative models. To enhance the expressiveness and realism of synthesized videos, emotion editing in talking face video plays a crucial role. However, existing approaches often limit expressive flexibility and struggle to generate extended emotions. Label-based methods represent emotions with discrete categories, which fail to capture a wide range of emotions. Audio-based methods can leverage emotionally rich speech signals - and even benefit from expressive text-to-speech (TTS) synthesis - but they fail to express the target emotions because emotions and linguistic contents are entangled in emotional speeches. Images-based methods, on the other hand, rely on target reference images to guide emotion transfer, yet they require high-quality frontal views and face challenges in acquiring reference data for extended emotions (e.g., sarcasm). To address these limitations, we propose Cross-Modal Emotion Transfer (C-MET), a novel approach that generates facial expressions based on speeches by modeling emotion semantic vectors between speech and visual feature spaces. C-MET leverages a large-scale pretrained audio encoder and a disentangled facial expression encoder to learn emotion semantic vectors that represent the difference between two different emotional embeddings across modalities. Extensive experiments on the MEAD and CREMA-D datasets demonstrate that our method improves emotion accuracy by 14% over state-of-the-art methods, while generating expressive talking face videos - even for unseen extended emotions. Code, checkpoint, and demo are available at https://chanhyeok-choi.github.io/C-MET/

• The paper introduces C-MET that models the difference between audio and visual embeddings to transfer emotion semantics for realistic emotion editing in face videos.
• It employs pretrained encoders, transformer regression, and multimodal contrastive loss to achieve significant gains in emotion accuracy on MEAD and CREMA-D datasets.
• The method supports continuous emotion editing and plug-and-play integration, enabling fine-grained control over nuanced expressions without requiring paired data.

Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video

Introduction and Motivation

Synthesizing emotionally expressive talking face videos is a core challenge in generative modeling, with significant implications for virtual agents, digital entertainment, and human–machine interaction. While earlier work focused on visual realism, identity preservation, and precise audio–lip synchronization, more recent approaches target emotional expressiveness and nuanced facial dynamics. State-of-the-art frameworks achieve progress by conditioning on discrete emotion labels, emotional speech, or reference images; however, these strategies are fundamentally constrained. Label-based methods lack scalability and fine-grained control; image-based approaches depend on curated reference samples, making them unsuitable for extended emotion categories; and audio-based methods frequently entangle emotion and linguistic cues, resulting in failure to generalize and accurately transfer target emotion—especially for nuanced expressions such as sarcasm or charisma.

Proposed Method: Cross-Modal Emotion Transfer (C-MET)

The paper introduces C-MET, a novel architecture for emotion editing in talking face video by explicitly modeling emotion semantic vectors across audio and visual modalities. C-MET operationalizes emotion transfer as mapping the difference between encoder embeddings of two emotional states (“emotion semantic vectors”) from the speech space to the facial expression feature space.

The modeling pipeline is illustrated in (Figure 1):

Figure 1: Overview of the C-MET pipeline for learning and regressing emotion semantic vectors across modalities.

• Pretrained audio and visual encoders (emotion2vec+large for speech, EDTalk encoder for facial expression) extract modality-specific embeddings.
• The difference between embeddings for two emotional categories yields an emotion semantic vector in each modality.
• Multimodal tokenizers and contrastive learning are employed to align these token spaces.
• A stack of transformer encoder layers regresses visual semantic vectors from the audio vector, enabling seamless and plug-and-play integration with modern disentanglement-based talking face generators.
• The modified visual embedding is decoded into the edited video, yielding expressive emotion editing.

This approach is motivated by advances in expressive TTS and voice cloning which enable high-diversity emotional speech synthesis. It circumvents the need for high-quality, emotion-annotated video datasets for “extended” emotions, as emotional cues are transferred directly from expressive audio without requiring paired audiovisual data.

Comparative Analysis and Experimental Results

Extensive evaluations on MEAD and CREMA-D datasets benchmark C-MET against label-based (EAT), image-based (EAMM, EDTalk), and audio-based (FLOAT) methods. For all approaches, the task is transforming a neutral video into one displaying the target emotion as provided by a single modality (label, image, or audio).

Figure 2: C-MET delivers semantically accurate transfer especially for nuanced emotions compared to baselines constrained by modality and emotion encoding bias.

C-MET achieves significant improvement in emotion accuracy: On MEAD, C-MET attains 55.91% accuracy versus 41.99% for EDTalk, 41.56% for EAT, and 13.21% for FLOAT. It similarly maintains superiority on CREMA-D (43.47%). This gain is obtained without major degradation in visual fidelity (FID/FVD) or lip–audio synchronization.

C-MET's qualitative benefits are highlighted in facial dynamics that capture asymmetric cues for complex emotions, e.g., subtle one-sided smiles for sarcasm, and dynamic frowning/eyebrow contractions for anger, surpassing baselines that default to “closest” basic emotions or produce visually implausible results.

Figure 3: C-MET offers granular control and expressivity for both basic (angry) and extended (sarcastic) emotion editing.

Ablation studies confirm that C-MET's multimodal contrastive loss and direction loss are integral for high cross-modal alignment and discriminative emotion representation, yielding optimal emotion accuracy.

Generalization, Integration, and Practical Implications

• Plug-and-Play Integration: C-MET replaces the facial expression encoder in disentanglement architectures (e.g., EDTalk, PD-FGC), reducing inference time and boosting emotion accuracy. (Figure 4)

  Figure 4: Seamless integration into various disentanglement networks, consistently improving emotion expressiveness and inference speed.

• Continuous Emotion Editing: The model supports smooth, temporally continuous emotion transitions by interpolating semantic vectors over short time horizons, leveraging the innate structure of speech prosody and dynamism.

  Figure 5: C-MET supports fine-grained and continuous temporal emotion editing using variable-length speech-derived vectors.

• Sample Efficiency and Data Augmentation: C-MET’s performance scales with the number of expressive speech samples (“speech-shots”), reaching saturation rapidly. This robustness enables effective use of synthetic or augmented speech sources for emotion control.

  Figure 6: Emotion accuracy trends positively with the number of aggregated expressive speech samples.

• User Study: Human evaluation (N=10, AMT) demonstrates C-MET is consistently preferred for emotional accuracy, visual quality, and synchronization across both basic and extended emotion categories, with up to 91.0% preference for extended emotions.

  Figure 7: Example interface for user study reinforces the practical human-perceived advantage of C-MET.

Theoretical and Future Research Implications

The explicit modeling of emotion semantic vectors in separate latent spaces and their alignment through transformer-based regression and contrastive multi-modal objectives represents a strong paradigm shift for emotion editing and cross-modal generation:

• It enables out-of-distribution emotion transfer and supports open-vocabulary, fine-grained affect rendering without collection of dense human-labeled audiovisual datasets.
• The modality gap between linguistic, prosodic (audio), and visual affect encoding is bridged, paving the way for controllable, end-to-end multimodal affective agent design.
• This approach is robust to speaker identity, scene, and modality-specific artifacts, since learning occurs on semantic differentials rather than absolute features.

Future directions include advancing to multi-view identity editing, extending to multilingual emotional speech, and incorporating multi-scale or hierarchical semantic vector decomposition for even finer granularity. The TTS-driven pipeline also opens potential for high-throughput data augmentation for downstream affect recognition and synthetic data generation tasks.

Conclusion

C-MET establishes a new state-of-the-art for emotion editing in talking face video through explicit audio–visual semantic vector modeling and cross-modal alignment. It achieves strong improvements in both emotion accuracy and practical usability, generalizing to unseen, nuanced affect categories by leveraging expressive speech alone. Its modular, efficient architecture makes it suitable for integration into existing and emerging talking face generation frameworks, and the underlying principles extend to broader multimodal synthesis and emotion-grounded controllable generation strategies.

Reference:

"Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video" (2604.07786)