CLAIP-Emo: Modular AVER Framework

• CLAIP-Emo is a modular framework for in-the-wild audiovisual emotion recognition that repurposes frozen CLIP and CLAP models using parameter-efficient LoRA adaptation.
• The framework achieves state-of-the-art performance on DFEW and MAFW by updating only about 4% of parameters, preserving language-aligned priors from billion-scale data.
• Asymmetric temporal modeling—with a lightweight Transformer for vision and mean pooling for audio—ensures efficient fusion and optimal utilization of modality-specific strengths.

CLAIP-Emo is a modular framework for in-the-wild audiovisual emotion recognition (AVER) that recasts the task as a parameter-efficient adaptation problem over language-supervised foundation models, specifically CLIP for vision and CLAP for audio. It is introduced as an alternative to large-scale domain-specific pre-training, which the paper characterizes as costly and often mismatched to real-world affective data. The framework freezes the original CLIP and CLAP backbones, adapts them with LoRA, allocates temporal modeling asymmetrically across modalities, and applies a simple fusion head for prediction. On DFEW and MAFW, the reported ViT-L/14 configuration reaches 80.14\% and 61.18\% weighted average recall with only 8M training parameters, and is presented as a new state of the art (Chen et al., 18 Sep 2025).

1. Problem Setting and Conceptual Position

CLAIP-Emo is designed for audiovisual emotion recognition in the wild, where pose variation, occlusion, and background noise remain persistent failure modes. The central claim of the framework is that these conditions do not necessarily require new large-scale affect-specific pre-training; instead, they can be addressed by adapting already language-supervised models whose priors were learned from billion-scale image-text or audio-text data. In that formulation, the key design goal is not maximal parameter update, but preservation of language-aligned priors under small-task adaptation (Chen et al., 18 Sep 2025).

This positioning distinguishes CLAIP-Emo from prevailing methods that primarily rely on large-scale domain-specific pre-training. The paper explicitly frames such pre-training as both expensive and frequently mismatched to real-world affective data. A plausible implication is that the method is intended not only as an accuracy-oriented model, but also as a systems-level proposal about how affective computing should reuse foundation models when labeled emotion corpora remain comparatively small.

2. Frozen Backbones and LoRA-Based Adaptation

The backbone configuration consists of a visual CLIP ViT, either B/16 or L/14, and an audio CLAP HTSAT encoder, both kept frozen during adaptation. The paper states that this preserves the rich, language-aligned priors learned from large image-text and audio-text corpora, and that emotion-oriented task adaptation is introduced through Low-Rank Adaptation (LoRA) rather than full fine-tuning (Chen et al., 18 Sep 2025).

For each linear or attention weight matrix $W_0\in\mathbb{R}^{d_{\rm out}\times d_{\rm in}}$, CLAIP-Emo injects a low-rank update:

$W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$

with

$$A\in\mathbb{R}^{r\times d_{\rm in}},\qquad B\in\mathbb{R}^{d_{\rm out}\times r}.$$

The reported hyperparameters are $r=8$ and $\alpha=32$, with LoRA dropout $0.1$. Under this setting, only about 4.0\% of CLIP’s parameters, and 2.5\% of the combined CLIP+CLAP parameters, become trainable. The abstract summarizes this as updating $\le 4.0\%$ of the total parameters (Chen et al., 18 Sep 2025).

The architectural significance of this choice is twofold. First, it keeps the optimization budget small: the best reported model uses only 8M training parameters. Second, the paper argues that freezing foundation backbones helps avoid catastrophic forgetting on small emotion datasets. The reported ablations reinforce that interpretation, because full fine-tuning is said to collapse to approximately 74\% WAR on DFEW and 42\% WAR on MAFW, despite updating 100\% of parameters.

3. Asymmetric Temporal Modeling and Prediction Head

CLAIP-Emo applies temporal modeling asymmetrically. The visual stream receives explicit sequence modeling, whereas the audio stream uses simple mean pooling. This asymmetry is justified in the paper by the inductive biases of the two backbones: CLIP frame representations benefit from a lightweight temporal head, while CLAP’s HTSAT encoder is described as already pooling over the clip through contrastive pretraining (Chen et al., 18 Sep 2025).

For vision, each frame $v_t$ is encoded into $\mathbf{f}_t^{V}\in\mathbb{R}^{d_V}$. A learnable $[\mathrm{CLS}]$ token and positional embeddings are added:

$W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$0

followed by one lightweight Transformer layer. The clip-level visual representation is the output $W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$1 embedding:

$W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$2

For audio, the frame-wise outputs $W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$3 from CLAP are mean-pooled:

$W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$4

The two clip-level representations are concatenated and passed through a single linear layer with bias:

$W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$5

Training uses standard cross-entropy loss over the $W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$6 emotion classes. The paper further reports that “Concat + Linear” outperforms additive or gated fusion while using the fewest extra weights. This is notable because the final classifier is deliberately simple; the modeling burden is shifted to the frozen priors plus low-rank adaptation rather than to an elaborate multimodal head.

4. Datasets, Preprocessing, and Optimization Protocol

The experimental evaluation is carried out on DFEW and MAFW, both described as in-the-wild datasets with five-fold splits. DFEW is presented as a video-based emotion dataset with seven or eight universal emotions, and MAFW as a multicultural in-the-wild audiovisual set. Both are said to cover neutral, happiness, sadness, surprise, fear, disgust, and anger, with sizes on the order of 10–20 K clips per split (Chen et al., 18 Sep 2025).

The preprocessing pipeline uses S4D frame sampling with standard resizing and normalization for video, and the CLAP pipeline to produce Mel spectrograms for audio. Optimization uses Adam for 100 epochs, with batch size 16 per GPU on two NVIDIA RTX A40s. The learning-rate schedule is cosine decay with a 5-epoch linear warmup and peak learning rate $W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$7. Evaluation follows both Unweighted Average Recall (UAR) and Weighted Average Recall (WAR), computed on each fold and then averaged.

These design choices matter because they align with the broader thesis of the paper: a frozen-backbone, low-update regime is only compelling if it can be trained under a comparatively lightweight protocol. The reported implementation details are consistent with that objective.

5. Quantitative Performance and Ablation Findings

The principal quantitative results are summarized below (Chen et al., 18 Sep 2025).

Setting	DFEW	MAFW
CLAIP-Emo (ViT-L/14), WAR	80.14\%	61.18\%
CLAIP-Emo (ViT-L/14), UAR	69.52\%	46.65\%
Prior best WAR, AVF-MAE++(H)	77.45\%	60.24\%

The paper states that these results establish a new state of the art while tuning only 8M parameters, compared with 521M for the prior best system. That comparison is central to the paper’s argument: the gain is not presented as a consequence of larger adaptation capacity, but as evidence that language-supervised priors can be repurposed more effectively than heavy task-specific pipelines.

The ablation study sharpens that interpretation. With LoRA rank sweep on DFEW fold 1 and MAFW fold 1, the frozen setting $W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$8 yields 74.30\% and 46.68\% WAR, while $W \;=\; W_0 \;+\;\frac{\alpha}{r}\,B\,A,$9 reaches 79.84\% and 51.14\% WAR, a gain of $$A\in\mathbb{R}^{r\times d_{\rm in}},\qquad B\in\mathbb{R}^{d_{\rm out}\times r}.$$0 and $$A\in\mathbb{R}^{r\times d_{\rm in}},\qquad B\in\mathbb{R}^{d_{\rm out}\times r}.$$1. Increasing to $$A\in\mathbb{R}^{r\times d_{\rm in}},\qquad B\in\mathbb{R}^{d_{\rm out}\times r}.$$2 yields a slight drop, and full fine-tuning collapses to approximately 74\% and 42\% WAR. This indicates that moderate low-rank adaptation is more effective than either no adaptation or unconstrained full updating.

The temporal-design ablation reports 77.11\% and 49.56\% WAR for $$A\in\mathbb{R}^{r\times d_{\rm in}},\qquad B\in\mathbb{R}^{d_{\rm out}\times r}.$$3, and 79.84\% and 51.14\% WAR for $$A\in\mathbb{R}^{r\times d_{\rm in}},\qquad B\in\mathbb{R}^{d_{\rm out}\times r}.$$4, which the paper calls the best trade-off. Adding a Transformer on both streams, $$A\in\mathbb{R}^{r\times d_{\rm in}},\qquad B\in\mathbb{R}^{d_{\rm out}\times r}.$$5, changes performance only marginally while adding 62\% more parameters. This supports the framework’s asymmetric temporal design.

The modality analysis reports approximately 47.5\% WAR for audio-only on DFEW, approximately 76.6\% for vision-only, and approximately 79.8\% for audio+vision, corresponding to a complementary gain of roughly 3 points. The pretraining-prior analysis is also explicit: replacing CLIP-ViT with a random or ImageNet-21k-pretrained ViT degrades WAR to approximately 49\% or 72\%, respectively. The paper interprets this as confirmation that language-supervised priors are advantageous for emotion recognition.

Finally, the t-SNE analysis reports that full fine-tuning produces heavy cluster overlap, whereas CLAIP-Emo’s learned features are more compact and better separated, with the paper connecting that qualitative structure to the higher WAR and UAR.

6. Relation to Adjacent CLIP/CLAP-Based Emotion Research

CLAIP-Emo belongs to a broader line of affective models that reuse language-supervised or contrastive audio-text priors rather than relying exclusively on conventional supervised emotion backbones. Within the supplied literature, several neighboring directions are visible. “Learning Emotion Representations from Verbal and Nonverbal Communication” introduces EmotionCLIP, a visual emotion pre-training framework that derives supervision from verbal and nonverbal communication using uncurated data and employs subject-aware context encoding together with sentiment-guided contrastive learning (Zhang et al., 2023). “GEmo-CLAP” adapts contrastive language-audio pretraining to speech emotion recognition and adds gender-aware objectives (Pan et al., 2023). “SmoothCLAP” modifies CLAP training with softened targets derived from intra-modal similarity and paralinguistic features to better reflect graded emotional relationships (Jing et al., 18 Jan 2026). “MER-CLIP” uses CLIP encoders together with a label encoder-guided multimodal decoder for multimodal emotion recognition on CMU-MOSI and CMU-MOSEI (Song et al., 1 Jun 2025).

Seen in that context, CLAIP-Emo is more specific in scope than these broader pre-training or zero-shot alignment programs. Its focus is in-the-wild AVER with frozen CLIP/CLAP backbones, LoRA adaptation, and intentionally simple fusion. This suggests that the method occupies a particular point in the design space: it is not primarily a new pre-training recipe, nor a label-semantic decoder, nor a soft-target contrastive objective, but a parameter-efficient downstream adaptation framework for audiovisual emotion recognition.

The supplied summaries also indicate some terminological overlap around CLIP/CLAP-based emotion models. A plausible implication is that the surrounding literature is still stabilizing its nomenclature, while the underlying research directions remain distinguishable by modality, supervision type, and adaptation strategy.

7. Limitations, Interpretation, and Future Directions

The limitations identified for CLAIP-Emo are tightly coupled to its minimalistic design. The current audio stream uses mean pooling, and the paper states that future work might explore lightweight temporal heads that still respect CLAP’s priors. Fusion is also deliberately simple; more structured cross-modal interaction, such as attention-based fusion, is suggested as a possible extension (Chen et al., 18 Sep 2025).

The paper additionally points to extensions toward more fine-grained or continuous affect labels, and to combinations with video-LLMs. These are natural continuations of the present framework because the current formulation uses clip-level discrete emotion prediction with a single linear fusion head. A plausible implication is that the same adaptation strategy could be re-used in settings where richer label spaces or broader multimodal priors become available, provided that the frozen-backbone assumption remains beneficial.

A common misconception would be to treat CLAIP-Emo as a conventional large-scale affective model whose performance derives from scaling optimization or adding heavy temporal and fusion modules. The reported evidence points in the opposite direction: the model’s contribution is the claim that preserving language-supervised priors, updating only a small fraction of parameters, and matching temporal complexity to each modality’s pretraining bias can outperform heavier domain-specific pipelines on challenging in-the-wild AVER benchmarks.