MoodifyCLIP: Emotion-Driven Visual Alignment

• MoodifyCLIP is a CLIP-based vision-language model fine-tuned to translate abstract emotional descriptions into specific visual signals using global and local contrastive alignment.
• It integrates global, fine-grained, and optimal transport alignment mechanisms to enhance zero-shot emotion recognition and image-text retrieval performance.
• Within the Moodifier framework, it leverages an 8M+ image MoodArchive with hierarchical emotional annotations to drive precise emotion-driven image editing.

Searching arXiv for the specified paper and closely related work to ground the article. MoodifyCLIP is a specialized vision-LLM introduced in "Moodifier: MLLM-Enhanced Emotion-Driven Image Editing" (Ye et al., 18 Jul 2025). It is built on the CLIP architecture and fine-tuned to bridge abstract emotions and concrete visual attributes. Within the broader Moodifier system, MoodifyCLIP functions as the representation module that translates emotional descriptions into image-relevant visual signals, enabling both improved zero-shot emotion recognition and precise guidance for emotion-driven image editing. The model is trained on MoodArchive, an 8 million+ image dataset with hierarchical emotional annotations, and is designed to connect global emotional semantics with fine-grained region-level cues in images (Ye et al., 18 Jul 2025).

1. Position within the Moodifier framework

MoodifyCLIP is one of three complementary components in the integrated approach described in (Ye et al., 18 Jul 2025). The first component is MoodArchive, an 8M+ image dataset with detailed hierarchical emotional annotations generated by LLaVA and partially validated by human evaluators. The second component is MoodifyCLIP itself, which is fine-tuned to translate abstract emotions into specific visual attributes. The third component is Moodifier, a training-free editing model leveraging MoodifyCLIP and multimodal LLMs to enable precise emotional transformations while preserving content integrity.

In this architecture, MoodifyCLIP serves as the alignment layer between linguistic expressions of emotion and visually localized image evidence. The paper states that this alignment is central to making emotion-driven image editing precise despite the abstract nature of emotions and their varied manifestations across contexts (Ye et al., 18 Jul 2025). A plausible implication is that MoodifyCLIP is intended not merely as an emotion classifier, but as an intermediate representation model whose embedding space is structured for downstream editing control.

2. Architecture and alignment mechanisms

MoodifyCLIP is built on CLIP-style dual encoders. Its vision encoder $E_i$ is either ViT-B/16 or ViT-L/14, using patch embeddings of dimension $d$. Its text encoder $E_t$ matches the CLIP text tower and is extended via positional-embedding interpolation to handle long, structured captions (Ye et al., 18 Jul 2025).

The model introduces three alignment components. The first is global contrastive alignment. The inputs are the image embedding $I^f = E_i(I) \in \mathbb{R}^d$, the full five-sentence caption embedding $T^f = E_t(T^f) \in \mathbb{R}^d$, and the summary caption embedding $T^s = E_t(T^s) \in \mathbb{R}^d$, where the summary comprises sentences 1 and 5. The objective is two-way InfoNCE loss between $\{I^f\}$ and $\{T^f\}$, and between $\{I^f\}$ and $\{T^s\}$ (Ye et al., 18 Jul 2025).

The second component is fine-grained region-emotion alignment. The inputs are $d$0, representing $d$1 image patch embeddings, and $d$2, representing the three local “stimuli” text embeddings. Cross-attention weights are computed as

$d$3

where $d$4 comes from $d$5 and $d$6 comes from $d$7. Region-aware vectors are then formed as

$d$8

The objective is an InfoNCE contrastive loss across each region-text pair (Ye et al., 18 Jul 2025).

The third component is optimal transport filtering. It begins from the similarity matrix $d$9 and the cost matrix $E_t$0. The transport plan is computed as

$E_t$1

The loss is

$E_t$2

which encourages a globally consistent matching between all image regions and emotion stimuli (Ye et al., 18 Jul 2025).

The total loss is

$E_t$3

where $E_t$4 and $E_t$5 are the global-contrastive losses on full and summary captions, $E_t$6 is the fine-grained region loss, and $E_t$7 is the OT loss (Ye et al., 18 Jul 2025).

3. MoodArchive and hierarchical emotional supervision

MoodifyCLIP is trained on MoodArchive, which contains 8 million+ images annotated with 27 discrete emotions from GoEmotions across 4 contexts: facial expressions, natural scenery, urban scenery, and object classes (Ye et al., 18 Jul 2025). This dataset structure is important because the model is explicitly trained to treat emotion as a context-dependent visual phenomenon rather than a single homogeneous label space.

For each image, LLaVA-NeXT produces a structured five-sentence annotation. Sentence 1 is a global summary of scene or subject, sentences 2 through 4 describe three local “emotional stimuli,” and sentence 5 provides the overall emotion assessment (Ye et al., 18 Jul 2025). This annotation schema is directly reflected in MoodifyCLIP’s training objectives: full-caption alignment uses all five sentences, summary-caption alignment uses sentences 1 and 5, and fine-grained alignment uses the three local stimuli.

Human validation was performed on a random sample of 10 K images, in which AMT workers compared original alt-text against the LLaVA captions. The paper reports that 85% of LLaVA captions were judged superior, while 15% were rejected for inaccuracies such as cultural misalignment or overly dramatic wording (Ye et al., 18 Jul 2025). This indicates that the supervision source is rich but imperfect. This suggests that the model’s emotional grounding inherits both the descriptive breadth and the noise characteristics of LLaVA-generated annotations.

4. Fine-tuning procedure and optimization

The fine-tuning procedure is stated explicitly. Let $E_t$8 be batch size and $E_t$9 the contrastive temperature. The global full-caption InfoNCE loss is defined as

$I^f = E_i(I) \in \mathbb{R}^d$0

and

$I^f = E_i(I) \in \mathbb{R}^d$1

with

$I^f = E_i(I) \in \mathbb{R}^d$2

The summary-caption loss $I^f = E_i(I) \in \mathbb{R}^d$3 is defined analogously on $I^f = E_i(I) \in \mathbb{R}^d$4 (Ye et al., 18 Jul 2025).

The fine-grained region loss is

$I^f = E_i(I) \in \mathbb{R}^d$5

with $I^f = E_i(I) \in \mathbb{R}^d$6 defined symmetrically and

$I^f = E_i(I) \in \mathbb{R}^d$7

The OT loss $I^f = E_i(I) \in \mathbb{R}^d$8 is as specified in the architecture description (Ye et al., 18 Jul 2025).

The reported hyperparameters are AdamW, learning rate $I^f = E_i(I) \in \mathbb{R}^d$9 with warmup then cosine decay, batch size 512, 10 epochs, weight decay 0.01, temperature $T^f = E_t(T^f) \in \mathbb{R}^d$0, and data augmentation consisting of random crop and horizontal flip. Positional-embedding interpolation is applied to handle five-sentence captions. Example loss weights are $T^f = E_t(T^f) \in \mathbb{R}^d$1, $T^f = E_t(T^f) \in \mathbb{R}^d$2, $T^f = E_t(T^f) \in \mathbb{R}^d$3, and $T^f = E_t(T^f) \in \mathbb{R}^d$4 (Ye et al., 18 Jul 2025).

5. Empirical evaluation

The empirical evaluation covers zero-shot emotion classification, image-text retrieval, and ablation of loss components (Ye et al., 18 Jul 2025).

In zero-shot emotion classification with 5-fold CV, the datasets are Emotion6, EmoSet, and Emotic, and the metrics are Top-1, Top-2, and Top-3 Accuracy. The comparison includes CLIP_B/16, CLIP_L/14, and MoodifyCLIP. On Emotic using ViT-L/14, MoodifyCLIP achieves Top-1 38.4% versus 21.5% for the LAION-pretrained model, which the paper describes as a gain of more than 16 points (Ye et al., 18 Jul 2025).

In image-text retrieval, the datasets are SentiCap, Affection, and MoodArchive-5k, and the metrics are Recall@1 and Recall@5 in both directions. On Affection with ViT-L/14, MoodifyCLIP yields image-to-text $T^f = E_t(T^f) \in \mathbb{R}^d$5, reported as $T^f = E_t(T^f) \in \mathbb{R}^d$6 over LAION, and text-to-image $T^f = E_t(T^f) \in \mathbb{R}^d$7 (Ye et al., 18 Jul 2025).

The ablation results distinguish the effects of the alignment components. Fine-grained alignment alone produces modest gains, and OT alone produces small gains. The combination of FG + OT is reported as the best combined setting, including examples such as image-to-text $T^f = E_t(T^f) \in \mathbb{R}^d$8 up to 44.7% and classification Top-1 up to 54.8% on Emotion6 (Ye et al., 18 Jul 2025). This suggests that local region-text coupling and global assignment consistency are complementary rather than redundant.

Evaluation setting	Reported result
Emotic, ViT-L/14, zero-shot Top-1	38.4% vs. 21.5% for LAION-pretrained
Affection, ViT-L/14, I2T R@1	46.3% (+5% over LAION)
Affection, ViT-L/14, T2I R@1	48.6%
FG + OT ablation	I2T R@1 up to 44.7%, Top-1 up to 54.8% on Emotion6

6. Role in emotion-driven image editing

The paper characterizes MoodifyCLIP as the module that enables Moodifier to perform precise emotional transformations while preserving content integrity across domains including character expressions, fashion design, jewelry, and home decor (Ye et al., 18 Jul 2025). Its function is not only to associate an emotion label with an image, but to identify which visual elements must change in order to convey the target emotion.

A qualitative example of region-stimulus alignment is given for the local text “raised corners of mouth” associated with joy, where MoodifyCLIP focuses attention on the mouth patch (Ye et al., 18 Jul 2025). This indicates that the learned representation is sensitive to localized facial correlates of emotion.

Additional qualitative editing examples are reported through Moodifier. Applying “Joy” to a cocktail dress triggers richer hues and vibrant patterns. Applying “Melancholy” to a home décor vase desaturates color and introduces drooping silhouettes (Ye et al., 18 Jul 2025). These cases are presented as evidence that the embeddings capture the concrete visual attributes associated with emotional transformation. A plausible implication is that MoodifyCLIP provides a semantically structured control signal that is sufficiently specific for editing models to modify style and affect while preserving identity and structure.

7. Limitations, evaluation challenges, and future directions

Several limitations are stated explicitly. Hallucinations in LLaVA-generated captions introduce noise, and 15% of samples required human correction. Emotions remain culturally nuanced, so a single set of descriptors may not generalize globally. The paper also notes a lack of standardized emotion-editing benchmarks, which makes evaluation challenging (Ye et al., 18 Jul 2025).

These limitations bear directly on the interpretation of MoodifyCLIP’s results. The model’s supervision depends on generated captions, which can encode inaccuracies or culturally specific framings. Its improvements in classification and retrieval therefore do not eliminate the broader issue that emotional semantics are contingent and context-sensitive. This suggests that the model’s alignment space should be understood as a learned operationalization of emotion grounded in the MoodArchive annotation pipeline rather than as a universal ontology of affect.

The stated future directions are to increase the human-verified portion of MoodArchive, develop paired datasets of original versus emotionally edited images validated by humans, and extend mood understanding to more complex or mixed emotions (Ye et al., 18 Jul 2025). Within the logic of the framework, these directions point toward denser supervision, stronger evaluation protocol design, and broader affective coverage.