Controllable Emotional Image Generation

• C-EICG is a technique that synthesizes or edits images by integrating detailed semantic descriptors with explicit emotional cues.
• It employs specialized diffusion models, custom loss functions, and multi-modal embeddings to ensure both content fidelity and emotion alignment.
• Applications include personalized image editing, emotional design in products, and psychological interventions, with future work targeting multi-dimensional affect control.

Controllable Emotional Image Content Generation (C-EICG) enables the synthesis or editing of images in a manner that jointly preserves specified semantic content while imparting precise, user-determined emotional characteristics. This field is defined by the explicit disentanglement and control of two output axes: visual semantics (objects, scenes, compositional attributes) and affective tone (emotion, mood, valence, arousal, or discrete categories). C-EICG frameworks have evolved from text-prompt engineering atop generic diffusion models to specialized, modular architectures optimizing for both content and emotion, utilizing custom loss functions, emotion embeddings, structured datasets, and multi-modal feedback mechanisms.

1. Task Definition and Core Objectives

C-EICG systems receive as input a semantic content descriptor (either a text prompt, image, or both) and an explicit emotion signal, either as a discrete category (e.g., “anger,” “contentment”), continuous affect vector (e.g., Valence-Arousal), or an enriched descriptor. The objective is to generate an output image $I$ that optimally satisfies two conditions:

1. Content Faithfulness: $I$ should accurately realize the intended scene, objects, and spatial arrangements described by the semantic condition $c$.
2. Emotion Alignment: $I$ must evoke the specified target emotion $e$ in the viewer, as measured by human assessments or automated predictors.

Formally, for generator $G$, the optimization targets: $$\min_\theta\; \mathbb{E}_{c,e}\left[ L_\mathrm{content}(G(c,e;\theta), c) + \lambda\, L_\mathrm{emotion}(G(c,e;\theta), e) \right]$$ where $\lambda$ balances content and emotion (Yang et al., 27 Dec 2025, Yuan et al., 5 Aug 2025). In continuous settings, the emotion is a point or region in valence–arousal (V–A) space (He et al., 10 Jan 2025, Jia et al., 25 Nov 2025).

2. Conditioning Mechanisms and Model Architectures

2.1. Explicit Emotion Embedding and Injection

Contemporary C-EICG models employ explicit emotion embeddings rather than restricting emotion control to prompt engineering. For discrete emotions, learnable tokens (e.g., EmoCtrl’s $v_t^k$, $v_v^k$ for each emotion $k$) are injected at both the text encoding—via LoRA or prompt concatenation—and inside the diffusion model’s cross-attention modules (Yang et al., 27 Dec 2025). For continuous emotion, such as Valence–Arousal, specialized neural modules map these coordinates into token embeddings, subsequently fused with semantic prompt features (He et al., 10 Jan 2025, Jia et al., 25 Nov 2025). Architectures frequently involve:

• Cross-attention over emotion and content features at each U-Net layer (Lin et al., 2024, Yang et al., 27 Dec 2025).
• Low-rank adapters decomposed into polarity-shared and emotion-specific factors (HiLoRA) for scalable, hierarchical control (Yuan et al., 5 Aug 2025).
• Multi-modal emotional condition fusion via scene/object expert queries in ViT (Zhu et al., 31 Jul 2025).
• Dual-branch pipelines coupling local (scene structure from image or prompt) and global (emotion) guidance (Lin et al., 2024).

2.2. Multimodal and Agent-Based Enhancement

Recent systems incorporate multimodal using large language–vision models (LVLMs), both for semantic–affective prompt rewriting (MC-Agent, chain-of-concept rewriting by ensembles of LLM agents (Jia et al., 22 Dec 2025)) and for understanding–feedback loops (LVLM reward or textual suggestion (Jia et al., 25 Nov 2025)). Visual prompt banks, constructed by clustering image representations per emotion, are used to inject fine-grained polymodal cues (Jia et al., 22 Dec 2025).

2.3. Specialized Training Objectives

Losses in C-EICG are tailored for both pixel/latent and semantic/affective supervision. Key terms include:

• Denoising loss: Standard LDM/UNet regression to diffusion noise.
• Emotion-Consistency Loss: Contrastive or cosine alignment between learned emotion tokens and CLIP-embedded affective descriptors (Lin et al., 2024, Jia et al., 22 Dec 2025).
• Emotion Prediction Loss: Negative log-likelihood under a pretrained or jointly trained emotion classifier.
• Semantic and Affective Attribute Losses: KL or cross-entropy towards object/scene and emotion labels jointly (Yang et al., 2024, Yuan et al., 5 Aug 2025).
• Test-Time Optimization: Online gradient-based search over small sets of learnable tokens to maximize target emotion and suppress inherent/similar emotions (e.g., MUSE loss combining target, inherent, and wheel-adjacent suppression) (Xia et al., 26 Nov 2025).

3. Dataset Strategies and Annotation Protocols

Groundtruth data for C-EICG requires images labeled with both semantic content and emotional attributes. Notable pipelines include:

• EmoSet: Large-scale datasets with image–emotion pairs, sometimes extended with human-verified affective captions and concept tokens using LLMs or manual curation (Yang et al., 27 Dec 2025, Yuan et al., 5 Aug 2025, Yang et al., 2024).
• Continuous Valence–Arousal Annotations: Datasets incorporating numerical V–A labels derived from lexicons (e.g., Warriner), by regression annotation, or from multimodal LLM reasoning (He et al., 10 Jan 2025, Jia et al., 25 Nov 2025).
• Sentence-Level Caption Augmentation: Rich, context-focused captions generated by MLLMs and filtered via CLIP similarity to ensure semantic–affective grounding (Yuan et al., 5 Aug 2025).
• Iterative Data Feedback: Synthetic samples are filtered for semantic and emotion accuracy (Dual-Metric filtering) and fed back into the generator or discriminator (Zhu et al., 31 Jul 2025).

4. Evaluation Metrics and Benchmarking

Assessing C-EICG models requires metrics sensitive to both content and emotion. Core metrics are:

• Emotion Accuracy (Emo-A): Proportion of generated images classified with the intended target emotion by a strong emotion classifier (Yang et al., 2024, Yuan et al., 5 Aug 2025).
• Semantic Clarity (Sem-C), Diversity (Sem-D): Retrieval-based measures for unambiguous content representation and diversity of visual realization under the same emotion (Yang et al., 2024, Yuan et al., 5 Aug 2025).
• Valence/Arousal Errors: $\mathbb{E}[|V_{pred} - V_{target}|]$ and analogously for Arousal, under continuous models (He et al., 10 Jan 2025, Jia et al., 25 Nov 2025).
• Joint Alignment (EC-A): Fraction of samples matching both semantic and emotional targets (Yang et al., 27 Dec 2025).
• Other: FID, LPIPS, PickScore (aesthetic/semantic alignment), and user studies reporting preference rates or human psychophysics 2AFC (Yang et al., 27 Dec 2025, Xia et al., 26 Nov 2025).

Table: Representative Quantitative Results for C-EICG Methods

Method	Emo-A (%)	Sem-C	Sem-D	V/Error ↓	FID ↓
EmoCtrl	61.68	0.662	—	—	—
CoEmoGen	80.15	0.641	0.0349	—	40.66
EmoGen	76.25	0.633	0.0335	—	41.60
EmotiCrafter	—	—	—	1.510	—
MUSE	68.38	—	—	—	43.53
UniEmo	81.7	—	—	—	26.6
EmoFeedback²	—	—	—	0.521	—

Context: Emo-A evaluated on emotion-annotated test sets; V-Error and FID as reported per respective model and benchmark (Yang et al., 27 Dec 2025, Yuan et al., 5 Aug 2025, Xia et al., 26 Nov 2025, He et al., 10 Jan 2025, Jia et al., 25 Nov 2025, Yang et al., 2024, Zhu et al., 31 Jul 2025).

5. Systemic and Methodological Innovations

5.1 Hierarchical and Multiscale Conditioning

C-EICG models have demonstrated gains by stratifying emotional features by scale, particularly with expert queries that extract scene-level and object-level embeddings in hierarchical transformers, weighted by emotional correlation coefficients derived from emotion classifier statistics. These embeddings are fused and used as conditioning for diffusion generation; margin-based emotional condition losses shape alignment (Zhu et al., 31 Jul 2025).

5.2 Feedback and Reinforcement Loops

Recent frameworks employ joint learning and explicit synthetic data feedback between an emotional understanding chain (predictor or classifier) and the generation module, improving not only controllability but also emotion recognition performance via joint or reinforcement fine-tuning (Zhu et al., 31 Jul 2025, Jia et al., 25 Nov 2025). LVLMs (large vision-LLMs) serve as both reward assigners (Group-Relative Policy Optimization) and iterative prompt rewriters, facilitating closed-loop emotional optimization (Jia et al., 25 Nov 2025).

5.3 Multimodality and Multi-Agent Prompting

Instead of relying solely on text prompts, advanced systems introduce multimodal prompt banks (derived from clustered visual features), which provide strong cross-modal emotional priors. Chain-of-concept rewriting, involving ensembles of LLM agents simulating different personas, generates more expressive and human-like affective prompts, which better survive emotion-neutralizing effects during diffusion (Jia et al., 22 Dec 2025).

6. Applications, Limitations, and Future Directions

C-EICG systems have been evaluated in story visualization (Chen et al., 2023), emotional editing/design for product images, personalized portrait generation with simultaneous identity-expression disentanglement (Liu et al., 2024), affective filter applications (Zhang et al., 19 Dec 2025), and psychological interventions. Noted limitations include:

• Restriction to a small set of discrete emotions in most frameworks; continuous V–A control is only found in select models (He et al., 10 Jan 2025, Jia et al., 25 Nov 2025).
• Dependence on the quality and diversity of emotion-labeled datasets.
• Subjectivity and ambiguity in emotion perception—not fully addressed by current classifiers or scoring (Jia et al., 25 Nov 2025, Yuan et al., 5 Aug 2025).
• Subtlety of affective cues in dense or crowded scenes.
• Computational cost, particularly where test-time optimization or feedback chains are required.

Future directions prioritize expanding controllable emotion spectrum (multi-dimensional, continuous, or user-calibrated emotion embeddings); integrating personalized or user-affect feedback; refining metrics via human-in-the-loop protocols; developing lightweight, real-time architectures; and extending frameworks to multi-modal affect control (e.g., with style references or via brain signals) (He et al., 10 Jan 2025, Jia et al., 25 Nov 2025, Yuan et al., 5 Aug 2025). The field is evolving rapidly toward scalable, generalizable, and semantically rich affective generation tools with direct user-in-the-loop control.