EmotionCLIP: Multimodal Emotion Recognition

• EmotionCLIP is a multimodal framework that integrates contrastive language–image modeling with data from EEG, video, and audio for emotion understanding.
• It employs a dual encoder architecture, including the SST-LegoViT for EEG, to align diverse high-dimensional inputs with textual emotion descriptions.
• The approach demonstrates superior domain generalization and robustness, outperforming traditional supervised models in affective recognition tasks.

EmotionCLIP designates a family of frameworks and pre-training paradigms in affective computing that integrate contrastive language–image/text modeling for emotion understanding across multiple modalities—including visual, audio, and neurophysiological signals. These systems are distinguished by their core use of contrastive learning, the semantic alignment of high-dimensional inputs (e.g., EEG, images, videos) with textual emotion descriptions, and the leveraging of large-scale pretrained backbones such as CLIP and its derivatives. The approach consistently demonstrates superior domain generalization and robustness compared to conventional supervised baselines.

1. CLIP-Style Contrastive Formulation for Emotion Recognition

EmotionCLIP extends the dual encoder architecture pioneered by CLIP, adapting it for diverse affective recognition tasks. The essential innovation is to cast the emotion classification problem not as a mapping of input data to discrete labels, but as a matching task between a data modality (EEG, image, video, audio) $\mathbf{x}$ and a natural language prompt or description $\mathbf{t}$. These are mapped by modality-specific and text encoders, $f_{\text{modality}}$ and $f_{\text{text}}$ respectively, into a shared space $\mathbb{R}^d$.

With $\ell_2$-normalized vectors, agreement is measured via cosine similarity $$s(\mathbf{u},\mathbf{v}) = \mathbf{u}^\top \mathbf{v}$$. Contrastive learning is employed by optimizing the (often symmetric) InfoNCE loss: $$\mathcal{L}_{\text{contrastive}} = -\frac{1}{N}\sum_{i=1}^N \log\frac{\exp(s(z_i, w_i)/\tau)}{\sum_{j=1}^N \exp(s(z_i, w_j)/\tau)}$$ where $z_i = f_{\text{modality}}(x_i)$, $w_i = f_{\text{text}}(t_i)$, and $\tau$ is a learnable or fixed temperature parameter. This formulation directly aligns samples with their semantic emotion description, enforcing robust semantic coupling.

2. SST-LegoViT Architecture for EEG-Based EmotionCLIP

For neurophysiological input such as EEG, EmotionCLIP introduces the SST-LegoViT encoder tailored to exploit the modality’s unique spatial, spectral, and temporal characteristics (Yan et al., 7 Nov 2025). The processing pipeline is as follows:

• Multi-band Feature Extraction: Raw EEG is filtered into six canonical frequency bands (δ, θ, α, β, γ₁, γ₂). For each band/channel, features such as Differential Entropy (DE) and Power Spectral Density (PSD) are computed, yielding a feature map of size $F\times C$.
• Spatial Interpolation: The electrode array is projected onto a dense 2D grid (e.g., $64\times64$) via spatial interpolation.
• Temporal Segmentation: Continuous EEG is segmented into $T$ non-overlapping time windows for dynamic analysis.
• Embedding Module: Rather than large spatial patches, a succession of 2D convolutions yields patchwise embeddings per frequency and spatial unit.
• Multi-Scale Convolutional Encoder: Within each transform block, a multi-branch convolutional module (kernels $1\times1$, $3\times3$, $5\times5$) captures spatial patterns at multiple receptive fields, with concatenation and projection to produce stable embeddings.
• Legoformer Spectral Encoder: DE and PSD inputs are processed by parallel Transformer branches ("legs"), followed by cross-attention fusion for a unified spectral representation.
• Temporal Encoder: The spectral embedding sequence is fed to a Transformer along the temporal axis. The final [CLS] token represents the full trial embedding.

This architecture ensures simultaneous exploitation of topographical (spatial), rhythmic (spectral), and dynamic (temporal) features, facilitating cross-subject and cross-session generalization.

3. Application Domains and Quantitative Benchmarks

EmotionCLIP variants demonstrate state-of-the-art performance across heterogeneous affective domains.

Modality	Dataset(s)	Task	Model/Variant	Best Metric(s)
EEG	SEED, SEED-IV	Cross-subject, cross-time emotion	EmotionCLIP-32 SST-LegoViT	88.69%, 73.50% (Acc)
Face/Vid	Aff-Wild2	EXPR/AU classification (static/dyn)	MLP+CLIP+CVaR+SAM	F1=0.36/0.43
Video	ABAW, MAFW, DFEW	Valence-Arousal, EXPR, AU (cont.)	Fine-tuned CLIP + TCN+Transformer	CCC=0.587/0.625; F1=0.465/0.580
Multimodal	MOSI/MOSEI	Multimodal emotion recognition	MER-CLIP (CLIP fusion)	F1=85.1%

For EEG (Yan et al., 7 Nov 2025), EmotionCLIP consistently exceeds graph, CNN, and Transformer-based baselines, particularly in cross-domain (subject/time) setups. In image emotion classification (Deng et al., 2022), prompt-tuning approaches (PT-DPC) achieve up to +9.29% accuracy gains over prior state-of-the-art on balanced datasets such as EmotionROI. In real-world audiovisual recognition, CLAIP-Emo (Chen et al., 18 Sep 2025) delivers 80.14% WAR on DFEW using only a small fraction (<2.5%) of available parameters, outperforming heavyweight domain-specific pipelines.

4. Prompt Engineering and Semantic Anchoring

EmotionCLIP systems depend critically on prompt design—not only generic structures (e.g., "A person feels {label} now") but also learnable "virtual" tokens and instance-/category-specific prompt blending. The process entails:

• Prompt Tuning: Prompts are jointly optimized as continuous token embeddings concatenated to class tags; only prompt parameters are updated, keeping backbone encoders frozen.
• Instance-Specific Composition: The image/EEG embedding serves to weight and blend class-specific prompts, producing input-dependent textual anchors that reflect subtle distinctions in content.
• Template Evaluation: Ablations indicate negligible sensitivity to initialization wording; diversified, instance-conditioned prompts significantly outperform simplistic and invariant schemes.

This semantic anchoring with text prompts acts as a stable reference enabling robust domain adaptation.

5. Ablation Studies and Component Analysis

Detailed ablation studies highlight the additive value of each model component:

• SST-LegoViT alone yields substantially lower accuracy than the full CLIP-style matching (e.g., EEG classifier: 61.42% vs. 88.69% with full contrastive fine-tuning).
• Removal of multi-scale spatial convolutions, spectral fusion (Legoformer), or temporal Transformer modules each lead to 3–5% drops in accuracy.
• Best trade-offs in embedding dimension ($d=512$) and temperature ($\tau=0.07$) are empirically validated.
• In multimodal settings (MER-CLIP), label encoder-guided fusion and cross-modal attention each boost F1 by several points.

These findings suggest that robust affective modeling requires coordinated architectural and algorithmic design to exploit multimodal data structures and semantic cues.

6. Discussion, Limitations, and Future Directions

EmotionCLIP’s core insight is that text-derived emotion anchors are more stable and domain-invariant than raw sensor/visual features, providing a natural pivot for multimodal alignment. Aligning EEG, visual, audio, and textual content in a shared semantic space inherits the robustness of language-image pretraining and supports transfer across subjects, sessions, or modalities.

Identified limitations concern the static nature of prompt templates, reliance on frozen backbones, and limited modality coverage. Future directions include:

• Prompt tuning or dynamic template learning to further optimize semantic anchoring.
• End-to-end language backbone fine-tuning for enhanced flexibility.
• Extension to richer textual descriptions (scenario narratives), additional physiological modalities, and applications beyond emotion (e.g., cognitive or motor imagery in BCIs).
• Hierarchical or adaptive temporal modeling for long-duration signals.
• Multilabel and dimensional emotion spaces reflecting the full spectrum of human affect.

Collectively, these lines of research indicate that EmotionCLIP frameworks are a foundational advance toward robust, generalizable, and semantically faithful emotion recognition across diverse biosignal and behavioral data types.