Cross-Modal Emotion Adapter (EmoAdapter)

• Cross-Modal Emotion Adapter (EmoAdapter) is a modular, parameter-efficient component that aligns cross-modal emotion features across diverse data types.
• It employs varied architectures such as residual MLP, linear mapping, and dynamic convolution to facilitate emotion injection with minimal disruption to pretrained models.
• Empirical results show that EmoAdapters recover unimodal performance, enable joint modality gains, and support zero/few-shot learning across multiple applications.

A Cross-Modal Emotion Adapter (EmoAdapter) is a specialized module or architectural component designed to inject, transfer, or align emotion-relevant features across heterogeneous modalities such as visual, acoustic, textual, and physiological signals. Its principal function is to augment backbone models (multimodal LLMs, transformers, or collaborative-filtering architectures) for affective computing, enabling robust, interpretable, and generalizable emotion recognition, representation transfer, or synthesis under cross-modal and cross-domain scenarios.

1. Architectural Principles and Design Variants

Cross-Modal Emotion Adapters are instantiated as lightweight, often plug-in, architectural modules that enable emotion-sensitive processing while minimizing the disturbance to pre-trained backbone representations. Several design paradigms have emerged:

• Residual MLP Adapters: As prototyped in VAEmotionLLM (Zhang et al., 15 Nov 2025), a two-stage approach freezes major encoders and applies a shared, position-wise residual MLP (Emotion Enhancer) to modality-aligned tokens before LLM ingestion. This MLP injects emotion “directions” learned across all modalities.
• Linear Mapping Adapters: For bridging embedding spaces (e.g., textual to acoustic/visual), low-rank or affine projections are trained to adapt emotion category vectors across modalities, e.g., $f_{t\to a}, f_{t\to v}$ in (Dai et al., 2020).
• Dynamic Convolutional or Prompt-Based Adapters: Video-based models such as FE-Adapter (Gowda et al., 2024) and SCPT (Luo et al., 7 May 2026) incorporate spatio-temporal message-passing or prompt-injection modules, often with sparse or low-rank adapters to preserve backbone invariances.
• Collaborative Filtering and Alignment: In EMI-Adapter (Zou et al., 2023), cross-modal emotion fusion is performed via concatenation of semantic and emotion vectors, followed by matrix factorization, NCF, or SVD$^{++}$-style fusion for audio-visual pairing.
• Domain Adaptation Adapters: Recent architectures, e.g., UF-AMA (Wang et al., 29 May 2026), utilize cross-modal transformer encoding, bi-directional cross-attention, and supervised domain adaptation (marginal/conditional alignment) within an adapterized pipeline to support robust generalization under domain shift.

Adapters are typically parameter-efficient, modular, and designed for insertion at key points—often before transformer self-attention, atop modality-specific encoders, or at fusion layers—enabling both parameter- and data-efficient emotion modeling.

2. Mathematical Formulations and Loss Functions

The mathematical mechanisms of EmoAdapters are characterized by:

• Residual Projection: Given modality embedding $z_m \in \mathbb{R}^{L_m \times d}$, the enhancer forms $$\tilde{z}_m = z_m + W_2 \operatorname{GELU}(W_1 \operatorname{LN}(z_m))$$ (Zhang et al., 15 Nov 2025).
• Emotion Supervision: Global pooling $s_m = \frac{1}{L_m} \sum_t \tilde{z}_m^{(t)}$, regression head $g_\psi$ predicts valence–arousal $\hat{y}_m = g_\psi(s_m)$, trained via $L_2$ regressions $$\mathcal{L}_{\text{emo}} = \frac{1}{|M|} \sum_{m \in M} \| \hat{y}_m - y_m \|_2^2$$.
• Cross-Modal Alignment: In text-image/video transfer (Zhang et al., 17 Jan 2026), adversarial regularization and mean-dispersion incentives disperse category clusters in a latent space; cross-entropy or multi-class contrastive losses bridge codebook-aligned text and visual features.
• Fusion and Attention: Multi-head attention and cross-attention architectures (Wang et al., 29 May 2026) implement cross-modal feature exchange with bidirectional projections and fused representations, with losses spanning classification, distribution-matching (MK-MMD), and compactness penalties.
• Composite Objectives: Training typically blends core modeling (e.g., language modeling, cross-entropy over classes, contrastive/emotion regression) with adapter-specific regularizations and alignment losses, e.g., $$\mathcal{L} = \mathcal{L}_{\mathrm{LM}} + \lambda \mathcal{L}_{\mathrm{emo}}$$ (Zhang et al., 15 Nov 2025), or

$^{++}$0

(Luo et al., 7 May 2026).

Adapters can be optimized exclusively, with backbone weights frozen, ensuring preservation of modality-specific pretraining.

3. Cross-Modal Emotion Adapter Instantiations

Key instantiations include:

Adapter	Backbone	Fusion Mechanism	Modality Coverage	Losses & Objectives
Emotion Enhancer	VLM + Audio AST	Residual 2-layer shared MLP, LoRA on LLM	Audio, Visual, Audio-Visual	$^{++}$1
FE-Adapter	Image ViT	Conv3D bottleneck + ReLU + up-proj, per-block residual	Images → Video	Cross-entropy over emotion classes
Modal Embed Mapper	LSTM	Linear $^{++}$2, dot-product distances	Text, Audio, Visual	Binary multi-label cross-entropy
EMI-Adapter	ImageBind	Semantic + emotion concat, MF/NCF/SVD++ scoring	Audio, Visual	Binary cross-entropy, ranking
DSSA (SCPT)	ViT-B/16	Shared/specific LoRA-style branch in every layer	Face, rPPG (physiology)	Multi-term: classification, sparsity, orthogonality, subject-ID
UF-AMA Adapter	Transformer	Dual-modality transformer, bidirectional cross-attention	EEG, Eye-tracking	Multilevel: CE, MMD, BCE, consistency, distillation

This table highlights the diversity of modality-bridging mechanisms, from bottleneck convolutions and residual projections, to cross-modal attention and metric learning.

4. Empirical Findings and Ablations

Extensive empirical work demonstrates that Cross-Modal Emotion Adapters can:

• Recover Unimodal Performance: The Emotion Enhancer outperforms audio-only and visual-only baselines, e.g., $^{++}$3 audio, $^{++}$4 video on ArtEmoBenchmark (Zhang et al., 15 Nov 2025).
• Enable Joint Modality Gains: Adapter-fused models achieve strong joint audio-visual accuracy gains—e.g., $^{++}$5 joint AV vs. $^{++}$6 prior (Zhang et al., 15 Nov 2025).
• Unlock Zero/Few-Shot Transfer: Modal mapping adapters generalize to unseen emotion words/classes via plug-and-play linear projections without retraining (Dai et al., 2020).
• Increase Parameter Efficiency: FE-Adapter matches or exceeds video emotion SOTA with $^{++}$7 fewer tunable parameters than full fine-tuning (Gowda et al., 2024).
• Enhance Human-Like Matching: EMI-Adapter improves human accuracy for image-music emotion retrieval over semantic-only pipelines, and SVD$^{++}$8-style fusion best captures emotional proximity (Zou et al., 2023).
• Generalize Across Domains/Subjects: Domain-adaptation adapters with confidence-aware screening achieve high cross-domain accuracy with minimal modality-specific noise (Wang et al., 29 May 2026), and DSSA branches in SCPT support subject-invariant emotion recognition (Luo et al., 7 May 2026).

Ablation studies systematically show the necessity of both enhancer (“residual injection”) and supervisor (“explicit emotion feedback”) modules in achieving full cross-modal generalization.

5. Application Domains and Extensions

Cross-Modal Emotion Adapters have been deployed in:

• Audio-Visual Artistic Emotion Models: Artistic understanding in LLMs and AVLMs is improved for fine-grained emotion inference, e.g., emotional content in music-video pairs (Zhang et al., 15 Nov 2025, Zou et al., 2023).
• Parameter-Efficient Image-to-Video Transfer: Adapters enable conventional still-image ViTs to process dynamic video for continuous emotion recognition (Gowda et al., 2024).
• Text-Driven Sentiment Transfer: Joint latent-space adapters allow text-guided alteration of visual emotion in images, supporting controllable, semantically consistent sentiment transfer (Zhang et al., 17 Jan 2026).
• Physiological-Behavioral Fusion: In video-based emotion recognition, adapters systematically bridge facial and physiological (rPPG) features, separating subject-invariant and subject-specific factors (Luo et al., 7 May 2026).
• Cross-Domain Physiological Emotion Recognition: Adapters with confidence-weighted alignment facilitate generalization across subjects and sessions for EEG/eye-tracking fusion (Wang et al., 29 May 2026).
• Low-Resource and Zero-Shot Learning: By leveraging shared semantic embeddings, adapters directly support recognition of new emotion categories in data-scarce settings (Dai et al., 2020).

A plausible implication is that adapter-based architectures will remain central as the field tackles ever greater diversity of modalities (e.g., haptics, language, physiology) and tasks (retrieval, generation, human alignment).

6. Parameter Efficiency, Modularity, and Limitations

EmoAdapters exploit modularity and parameter-sparsity to minimize retraining while maximizing cross-modal transfer:

• Parameter Reduction: Adapter-only training reduces tunable parameter count by over $^{++}$9 (e.g., FE-Adapter: $z_m \in \mathbb{R}^{L_m \times d}$0 of full model) (Gowda et al., 2024), with no significant drop in recognition performance.
• Plug-and-play Integration: Adapters are typically compatible with a wide range of backbones, from classical LSTMs to transformers and multimodal LLMs.
• Preservation of Pretrained Representations: By freezing backbone encoders and operating via residual/bottleneck/prompt mechanisms, adapters retain generalizable knowledge and reduce catastrophic forgetting.

Limitations include: dependence on labeled emotion annotations for initialization (EMI-Adapter (Zou et al., 2023)), scalability to combinations beyond two or three modalities, and the sensitivity of some architectures to domain shift/subject-variation, which continues to motivate research into robust domain-adaptive adapter designs.

7. Future Directions and Impact

Significant future directions include:

• Unified Multimodal Adapters: Expanding adapter designs to cover broader, unified latent spaces (as pursued in Nano-EmoX (Huang et al., 2 Mar 2026) and EmoLat (Zhang et al., 17 Jan 2026)) capable of supporting generalized empathy, understanding, and interaction tasks, fully parameterized for all hierarchy levels of affective cognition.
• Self-Supervised and Unsupervised Adapter Training: Reducing reliance on explicit emotion labeling via contrastive or meta-learning-based adapter initialization.
• Generative and Retrieval Models: Adoption of adapters in affect-driven generative multimedia (image, music, or video synthesis) and affective retrieval, with dynamic adjustment to user intent, domain context, or subjective preferences.
• Scalable Cross-Domain Applications: Enhanced adapters that dynamically adapt to distribution shifts across populations, devices, and environments via confidence-aware screening and hybrid domain adaptation frameworks (Wang et al., 29 May 2026).

As evidenced by a substantial body of recent work, Cross-Modal Emotion Adapters establish a foundational paradigm for parameter-efficient, flexible, and robust affective multimodal modeling, supporting both human-aligned recognition and creative, controllable sentiment transfer across an expanding inventory of data domains and tasks (Zhang et al., 15 Nov 2025, Gowda et al., 2024, Dai et al., 2020, Zou et al., 2023, Luo et al., 7 May 2026, Zhang et al., 17 Jan 2026, Wang et al., 29 May 2026).