Moodifier: Emotion-Driven Image Editing

• Moodifier is an emotion-driven image editing framework that uses MoodArchive, MoodifyCLIP, and a training-free diffusion pipeline to translate emotions into visual modifications.
• It decomposes editing into hierarchical annotation, emotion-to-attribute grounding, and mask-controlled diffusion, ensuring precise local changes and overall content preservation.
• Empirical evaluations show significant improvements in human preference and CLIP similarity compared to baselines across applications like fashion, jewelry, and home décor.

Searching arXiv for the cited Moodifier and closely related mood-modeling papers across image, text, music, and multimodal systems. Moodifier is an MLLM-enhanced, emotion-driven image editing framework that links abstract emotions to concrete visual modifications through three coupled components: MoodArchive, MoodifyCLIP, and a training-free editing pipeline (Ye et al., 18 Jul 2025). It is designed to perform precise emotional transformations while preserving identity, structure, and background, and is evaluated across character expressions, fashion design, jewelry, and home décor (Ye et al., 18 Jul 2025). In adjacent literature, related “moodifier” systems appear in music recommendation, mobile sensing, social media, and multimedia generation, but the 2025 Moodifier system is specifically an image-editing architecture that combines large-scale emotional annotation, vision-language fine-tuning, and attention-controlled diffusion (Ye et al., 18 Jul 2025).

1. Problem setting and conceptual design

Emotion-driven image editing is difficult because emotions are abstract and their visual manifestations are context-dependent. Moodifier addresses this by decomposing the problem into three stages. MoodArchive supplies large-scale, hierarchically structured emotional supervision; MoodifyCLIP translates emotion terms into concrete visual attributes; and the final editing framework uses an MLLM together with diffusion-time attention control to localize and apply the edit (Ye et al., 18 Jul 2025).

The design motivation is explicit. Existing vision-language datasets are described as lacking the scale and emotional granularity needed for nuanced affective reasoning, standard CLIP models are described as not being tuned for fine-grained emotional subtleties, and text-guided diffusion editors are described as lacking precise emotion-specific control while often damaging scene integrity (Ye et al., 18 Jul 2025). Moodifier therefore treats emotion editing not as a single prompt-engineering problem but as a coupled data-model-inference problem.

This system sits within a broader mood-conditioned design space. Related work includes mood-consistent multimedia generation across script, visuals, and audio in MoodSmith (Menon et al., 2024) and continuous Valence–Arousal-driven affective image editing in MooD (Yin et al., 4 May 2026). This suggests that Moodifier occupies the discrete-emotion, training-free end of a larger spectrum that also includes dimensional affect control and cross-modal mood alignment.

2. MoodArchive: large-scale emotional supervision

MoodArchive is the data substrate underlying Moodifier. It adopts the 27-category GoEmotions taxonomy and organizes each emotion across four contexts: facial expressions, natural scenery, urban scenery, and object classes such as clothing, jewelry, and home décor (Ye et al., 18 Jul 2025). The dataset is intended to capture the fact that the same emotion can manifest differently across domains.

Its construction pipeline has four main stages. First, ChatGPT is prompted to expand each abstract emotion into dozens of concrete descriptors. Second, those descriptors are used to crawl approximately 8 million images from free-use photo sites including Unsplash, Pexels, Pixabay, Envato Elements, Openverse, Burst, and StockSnap. Third, NSFW or corrupt files are removed and the bottom 20% are dropped according to CLIP-score similarity to the emotion phrases. Fourth, each surviving image is sent to LLaVA-NeXT, which produces a structured caption containing a global summary, three local emotional stimuli, and an overall emotion assessment (Ye et al., 18 Jul 2025).

The annotation scheme is hierarchical rather than flat. Each image receives 5 sentences in 3 tiers: global, stimuli, and assessment. This hierarchy is central to the later MoodifyCLIP objectives, because the model is trained not only on whole-image and whole-caption alignment but also on local region-to-stimulus correspondences (Ye et al., 18 Jul 2025).

Human validation is partial but nontrivial. A random sample of 10,000 images was evaluated by Mechanical-Turk workers, who were asked whether the original alt-text or the LLaVA-generated caption better matched both content and emotion; 85% preferred the LLaVA-generated captions (Ye et al., 18 Jul 2025). The dataset statistics reported for MoodArchive are: more than 8 million images, 27 discrete emotions, 4 contexts per emotion, and 5-sentence annotations (Ye et al., 18 Jul 2025).

3. MoodifyCLIP: emotion-to-attribute grounding

MoodifyCLIP is the vision-LLM that operationalizes the mapping from emotion words to concrete visual attributes. It starts from CLIP’s vision encoder $E_i(\cdot)$ and text encoder $E_t(\cdot)$, supports ViT-B/16 and ViT-L/14 backbones, and applies positional-embedding interpolation as in Long-CLIP so that the text encoder can process the longer 5-sentence captions in MoodArchive (Ye et al., 18 Jul 2025).

Its training objective is composite. The global contrastive component operates on the full caption and on the summary caption, producing losses $L_f$ and $L_s$. The fine-grained contrastive component $L_{fg}$ parses the three local stimuli, computes cross-attention weights between textual tokens and image patches, forms region-specific image features, and applies InfoNCE at the region level. The optimal transport term $L_{ot}$ computes a similarity matrix between fine-grained image regions and stimuli texts, converts it to a cost matrix, and uses Sinkhorn to obtain a transport plan. The total loss is

$$L_{\text{total}}=\lambda_f L_f+\lambda_s L_s+\lambda_{fg}L_{fg}+\lambda_{ot}L_{ot}.$$

This formulation is intended to couple coarse emotional semantics with localized affective cues. In effect, MoodifyCLIP does not merely classify an image as matching an emotion label; it learns correspondences between emotion-bearing textual fragments and specific visual regions (Ye et al., 18 Jul 2025).

Training uses 5-fold cross-validation on held-out MoodArchive-5k, with AdamW, learning rate approximately $10^{-5}$, batch size approximately 256, and approximately 10 epochs (Ye et al., 18 Jul 2025). Reported zero-shot performance for ViT-L/14 includes Emotion6 Top-1 accuracy of 54.8%, EmoSet Top-1 of 56.3%, Emotic Top-1 of 38.4%, retrieval $R@1$ of 44.7% on SentiCap, 46.3% on Affection, and 86.2% on MoodArchive-5k (Ye et al., 18 Jul 2025). Ablation results indicate that removing the fine-grained or optimal transport losses reduces retrieval by 2–3% and classification by 2–3 points (Ye et al., 18 Jul 2025).

These results position MoodifyCLIP as the semantic bridge within Moodifier. MoodArchive provides the annotation structure; MoodifyCLIP learns to turn that structure into a usable emotion-conditioned representation.

4. Editing pipeline and attention-controlled diffusion

The editing framework is zero-shot and training-free. Given a source image $V$ and target emotion $E_t(\cdot)$0, the pipeline first extracts visual features $E_t(\cdot)$1, then calls LLaVA-NeXT to produce an emotion-rich editing prompt $E_t(\cdot)$2 and a spatial attention mask $E_t(\cdot)$3 (Ye et al., 18 Jul 2025). The prompt is 1–2 sentences, while the mask identifies the emotion-stimulus region to be edited.

MoodifyCLIP then refines $E_t(\cdot)$4 so that high-level emotion words are grounded in concrete visual-attribute phrases. The source image is inverted into diffusion latent $E_t(\cdot)$5 via non-iterative inversion. During diffusion from $E_t(\cdot)$6 down to $E_t(\cdot)$7, the system computes a source denoising path and an emotion-conditioned denoising path, blends attention maps according to the mask in early steps, and merges latents so that only the masked region receives the emotion-conditioned modification (Ye et al., 18 Jul 2025).

The key cross-attention expression is

$E_t(\cdot)$8

with $E_t(\cdot)$9 and $L_f$0 (Ye et al., 18 Jul 2025). For early steps, when $L_f$1, the blended attention map uses $L_f$2 inside $L_f$3 and $L_f$4 outside it. For later steps, when $L_f$5, the system uses the target attention map. The latent merge is

$L_f$6

This mechanism is the basis for the framework’s content-preservation claim. By blending latents only inside $L_f$7, Moodifier is intended to avoid global style shifts or inpainting artifacts. Early-step structural attention retains scene geometry; late-step emotional weighting enriches emotional details (Ye et al., 18 Jul 2025).

The architecture therefore assigns distinct roles to the two semantic subsystems. The MLLM answers where and how to edit through the mask and initial prompt; MoodifyCLIP sharpens the emotional semantics; the diffusion editor executes the change under explicit spatial control.

5. Empirical performance and qualitative behavior

Moodifier is evaluated on a test set of 240 source images, defined as 30 images per category across 8 categories: face, bag, bracelet, clothes, earring, necklace, ring, and vase, with 27 emotions considered for editing (Ye et al., 18 Jul 2025). The evaluation protocol includes structure distance, PSNR, LPIPS, MSE, and SSIM for preservation, OpenAI-CLIP similarity for emotional accuracy, and MTurk human preference for visual appeal, emotion accuracy, and overall quality (Ye et al., 18 Jul 2025).

The compared methods include ControlNet, Instruct-Pix2Pix, Instruct-Diffusion, MasaCtrl, Plug-and-Play, EditFriendly, StyleDiffusion, Null-text inversion, Direct inversion, and Inversion-Free Editing (Ye et al., 18 Jul 2025). The reported results emphasize the trade-off between structural fidelity and emotional specificity: some baselines preserve structure more aggressively, but do so at the cost of weak emotional transformation.

Metric	Moodifier	Comparison reported
LPIPS $L_f$8	94.9	next best 96.8
CLIP similarity	16.13	14.23 for Inversion-Free; 13–14 in others
Human overall preference	80.0%	next best 43.7%
Structure distance $L_f$9	17.4	best baseline 13.9 for Plug-and-Play

All major gains in CLIP similarity and human preference are reported as statistically significant with $L_s$0 using a t-test across 240 images (Ye et al., 18 Jul 2025). Editing ablations further state that both $L_s$1 and $L_s$2 are necessary: prompts alone raise emotion but break structure; masks alone preserve structure but yield weak emotion; the combined system provides the best trade-off (Ye et al., 18 Jul 2025).

Qualitative examples make the editing behavior concrete. A neutral cartoon face edited toward joy adds upward lip curvature and bright eyes, while sadness adds drooping eyelids and blue tear streaks, and anger adds furrowed brows and red accents. In fashion, a black dress edited toward excitement acquires vibrant geometric patterns and a warmer palette, whereas melancholy produces muted tones and slight droop in skirt form. In jewelry and home décor, the edits act through filigree, highlights, roughness, warping, cracks, and color palette shifts (Ye et al., 18 Jul 2025). These examples illustrate that Moodifier treats “emotion” as a structured change in local visual attributes rather than as a global style filter.

6. Broader mood-aware systems and the wider use of “Moodifier”

The term “moodifier” is used more broadly in adjacent work for systems that classify, predict, surface, or manipulate mood across media and interaction settings. The common technical pattern is the translation of affective variables into operational representations: bag-of-words lyric vectors, keystroke time series, conversation emotion logits, valence–energy coordinates, or continuous Valence–Arousal controls.

In music-text classification, MusicMood starts from a random subsample of 10,000 tracks in the Million Song Dataset, reduces to 2,773 songs after filtering, manually labels happy versus sad, and compares Naive Bayes variants; the best cross-validation result is ROC AUC approximately 0.75 for 1-gram tf-idf Multinomial NB with stop-word removal and Porter stemming, with validation precision for the happy class of 88.9% (Raschka, 2016). In mobile mental-health sensing, dpMood uses keystroke metadata and accelerometer values, a two-block 1D CNN plus bi-directional GRU, and a user-specific circadian calibration term $L_s$3; its best reported RMSE values are $L_s$4 for HDRS and $L_s$5 for YMRS with controls (Huang et al., 2018).

In on-device text analysis, LEAPMood combines Emotion Recognition in Conversation with K-means clustering, uses character embedding, phonetic hashing, attention, and CRF, reports a Micro F1 score of 62.05% with a memory footprint of 1.67MB on DailyDialog, and reports a Macro F1-score of 72.12% for mood prediction on a curated dataset (S et al., 2022). In social-media awareness tools, Tweet Moodifier is a Google Chrome extension that colors and filters tweets using Sentiment140; among Treatment 1 users, neutral tweets rose from 62.8% in week $L_s$6 to 79.7% in week $L_s$7, with $L_s$8 (Saldias et al., 2019).

In multimodal generation, MoodSmith represents mood by valence, arousal, and color warmth, uses GPT-4 for script and style suggestion, Stable Diffusion v1.5 for visuals, and Spotify valence–energy metadata for audio retrieval. Its technical evaluation reports exact match accuracy of 37.5%, valence-match accuracy of 75.0%, arousal-match accuracy of 56.2%, and a cross-modal consistency score of 1.61 versus 1.02 for a baseline, with $L_s$9 (Menon et al., 2024). In continuous affective image editing, MooD directly conditions on continuous Valence–Arousal coordinates $L_{fg}$0, introduces VA-aware retrieval and ASP-Net, and reports Time/MP of 4.50, PSNR of 17.51, CLIP-I of 0.825, V-Err of 0.537, and A-Err of 0.287 (Yin et al., 4 May 2026).

Mood-aware recommendation forms another branch of the literature. One line maps both user mood and songs to the energy–valence plane, then ranks or samples by proximity; a single-blind evaluation reports control mean 2.67 versus mood-assisted mean 3.59, with $L_{fg}$1 (Zeng, 12 Mar 2026). A closely related ranking formulation defines $L_{fg}$2 over candidate songs in the energy–valence space and reports the same mean-rating improvement and significance pattern in a real-time web application (Zeng et al., 11 Jun 2026). Another direction, MeloTune, deploys two closed-form continuous-time networks on iPhone hardware, learns a Personal Arousal Function per listener, broadcasts only structured Cognitive Memory Blocks, and reports trajectory MAE 0.414, pattern accuracy 96.6%, and intent accuracy 69.4% (Xu, 12 Apr 2026).

Taken together, these systems show that “Moodifier” names not a single methodology but a recurrent research program: the conversion of affect into operational variables that can guide classification, retrieval, generation, or recommendation. The 2025 image-editing Moodifier is distinctive within this program because it combines hierarchical emotional annotation, emotion-specialized vision-language alignment, and mask-guided diffusion editing in a single pipeline (Ye et al., 18 Jul 2025).

7. Limitations and research directions

Several limitations are explicitly identified for Moodifier. LLaVA-NeXT-generated captions can hallucinate or misinterpret cultural context; CLIP-based evaluation can overlook subtle affective distinctions such as nostalgia versus sentimentality; and there is no standardized benchmark of ground-truth emotional edits, so human evaluation remains subjective (Ye et al., 18 Jul 2025). These limitations affect both the annotation substrate and the downstream editor.

The future directions named for the system are correspondingly concrete: increase human verification coverage of MoodArchive, possibly via active learning; develop a paired dataset of source images and human-validated emotional variants as a gold standard; extend beyond 27 emotions to multi-label or dimensional affect such as valence and arousal; and incorporate user-in-the-loop controls such as sketch or color pickers for finer personalization (Ye et al., 18 Jul 2025).

A broader methodological tension also appears across the literature. Moodifier uses 27 discrete GoEmotions categories, whereas MooD is organized around continuous Valence–Arousal control (Yin et al., 4 May 2026). This suggests a continuing split between categorical affect editing and dimensional affect editing. A plausible implication is that future systems may combine Moodifier’s localized, MLLM-guided semantic control with continuous affect parameterizations, or with personalization mechanisms of the sort used in on-device mood prediction and recommendation systems (Huang et al., 2018). The current Moodifier framework already provides the modular components needed for such convergence: a large emotional dataset, an emotion-grounding vision-LLM, and a spatially controlled editing engine (Ye et al., 18 Jul 2025).