CoEmoGen: Emotional Image Synthesis

Updated 4 July 2026

CoEmoGen is a conditional latent diffusion pipeline that uses sentence-level, emotion-focused captions to guide emotionally faithful image synthesis.
It integrates a Neuro-symbolic Mapper, Visual-Perception Encoder, and HiLoRA to enhance semantic coherence and scalability across diverse domains.
Empirical results show improved FID, LPIPS, and Emo-A metrics over EmoGen, demonstrating robust performance in emotional image content generation.

Searching arXiv for CoEmoGen and closely related work to ground the article in current literature. CoEmoGen is a pipeline for Emotional Image Content Generation (EICG), the task of generating semantically clear, emotionally faithful images conditioned on a target emotion category (Yuan et al., 5 Aug 2025). It is positioned as a successor to EmoGen (Yang et al., 2024), but departs from earlier EICG systems by replacing word-level attribute supervision with sentence-level, emotion-focused captions and by introducing a Hierarchical Low-Rank Adaptation (HiLoRA) mechanism for emotion-conditioned diffusion. Within the literature, CoEmoGen addresses open-domain emotional image synthesis rather than facial-expression generation alone, distinguishing it from unified facial-affect control frameworks such as C2A2 (Paskaleva et al., 2024).

1. Problem formulation and conceptual scope

CoEmoGen inherits the EICG task formulation introduced by EmoGen: given an emotion such as awe, contentment, or anger, the model should synthesize an image whose content is recognizable and whose affective impression matches the target category (Yang et al., 2024). The paper follows the Mikels emotion model with eight emotions: four positive-polarity categories—amusement, awe, contentment, excitement—and four negative-polarity categories—anger, disgust, fear, sadness (Yuan et al., 5 Aug 2025).

The central technical premise is that abstract emotions are not reducible to a single visual entity. The paper argues that emotions are conveyed through a combination of low-level cues such as brightness and colorfulness, mid-level content such as objects and scenes, and high-level semantic factors such as facial expressions, actions, and interactions (Yuan et al., 5 Aug 2025). This contrasts with standard text-to-image diffusion, which is highly effective for concrete concepts but much weaker on abstract concepts.

A core implication is that EICG is not merely prompt-based image generation with emotion words inserted into the prompt. The paper explicitly emphasizes a psychological view in which visual emotions are strongly correlated with specific semantics, so low-level style manipulation alone is insufficient (Yuan et al., 5 Aug 2025). This emphasis also differentiates CoEmoGen from systems centered on continuous facial-expression control, such as C2A2, where the primary object of generation is a face conditioned on a unified affect representation rather than general emotional image content (Paskaleva et al., 2024).

2. Limitations of prior EICG and the rationale for CoEmoGen

The principal baseline is EmoGen, which conditions generation using word-level attribute labels derived from EmoSet (Yang et al., 2024). CoEmoGen identifies three structural deficiencies in that strategy (Yuan et al., 5 Aug 2025).

First, word-level labels lack contextual association. Isolated tokens do not encode how visual elements relate to one another, so they provide weak guidance for coherent scene composition. Second, the correlation between many labels and the target emotion is weak or ambiguous. Some labels are poor emotional indicators, while genuinely emotion-triggering details may be absent from annotation. Third, dependence on manual attribute labels creates missing annotations and poor scalability; incomplete labeling limits the usable corpus and reduces diversity.

The paper characterizes the downstream effect as semantically incoherent collage-like outputs, in which emotions become associated with stereotyped object sets rather than relational scene structure (Yuan et al., 5 Aug 2025). Its qualitative example is that anger may collapse into a conjunction of “fire, tigers, wide-open mouths,” producing implausible composites such as a flaming tiger. EmoGen had already argued that emotion-conditioned generation requires explicit semantic grounding and proposed an attribute loss to connect emotion to content (Yang et al., 2024). CoEmoGen extends that line of work by asserting that sentence-level coherence is the missing supervisory unit.

This reframing is the paper’s main conceptual contribution. Instead of treating emotion supervision as a bag of emotionally associated words, it treats it as emotion-triggering visual semantics expressed as a single coherent sentence. This suggests that the main failure of earlier EICG is not only insufficient conditioning strength, but also the wrong granularity of semantic representation.

3. Architecture and conditioning pipeline

CoEmoGen is organized as a conditional latent diffusion pipeline built on Stable Diffusion v1.5 with CLIP ViT-L/14 as both image encoder and text encoder (Yuan et al., 5 Aug 2025). The trainable modules are the Neuro-symbolic Mapper, the Visual-Perception Encoder, and HiLoRA; the CLIP image encoder, CLIP text encoder, and base U-Net weights remain frozen (Yuan et al., 5 Aug 2025).

For each training sample,

$\mathcal{D}_i = \{I_i, y_i, c_i\},$

where $I_i$ is the image, $y_i$ the emotion label, and $c_i$ the generated caption (Yuan et al., 5 Aug 2025).

The emotion label $y_i$ is first converted to a one-hot vector $y_i^o$ , which is passed through the Neuro-symbolic Mapper to obtain an emotional descriptor $e_i$ (Yuan et al., 5 Aug 2025). The mapper is implemented with fully connected layers and nonlinearities. The image is then encoded by the CLIP image encoder to yield a visual embedding $v_i$ , and CoEmoGen fuses $e_i$ and $v_i$ with a cross-attention-based Visual-Perception Encoder:

$I_i$ 0

where $I_i$ 1, $I_i$ 2, and $I_i$ 3 are learnable weights and $I_i$ 4 is the feature dimension (Yuan et al., 5 Aug 2025).

The visually enhanced descriptor $I_i$ 5 is then passed into the CLIP text encoder to obtain the final emotion condition $I_i$ 6 (Yuan et al., 5 Aug 2025). This condition guides a frozen Stable Diffusion U-Net equipped with HiLoRA adapters. At inference time, following EmoGen, visual embeddings are sampled from a pre-constructed Gaussian emotion cluster to increase diversity (Yuan et al., 5 Aug 2025, Yang et al., 2024).

An ablation on emotion representation reports that one-hot + Neuro-symbolic Mapper outperforms both learnable class vectors and frozen text encodings, achieving FID 40.66, LPIPS 0.732, Emo-A 80.15%, Sem-C 0.641, and Sem-D 0.0349 (Yuan et al., 5 Aug 2025). The paper interprets this as a balance between semantic consistency and diversity, whereas learnable vectors form a bottleneck and frozen text encodings over-constrain semantics.

4. Emotion-focused caption construction

CoEmoGen’s semantic supervision is built from multimodal LLM (MLLM) captions rather than manual attributes (Yuan et al., 5 Aug 2025). The MLLM is used strictly for data construction, not as the generative backbone. For each EmoSet image, the model is prompted with the known target emotion and asked to produce a one-sentence caption centered on emotion-relevant visual details, including brightness, colorfulness, scene type, object classes, facial expressions, and human actions (Yuan et al., 5 Aug 2025).

The prompt used in the paper is:

<Image> This image evokes a strong emotion of {<emotion>}. Provide a one-sentence caption that vividly describes the visual details, focusing on elements like brightness, colorfulness, scene type, object classes, facial expressions, and human actions that effectively convey and express this emotion.

The paper’s argument for captions over attributes has three parts (Yuan et al., 5 Aug 2025). Sentence-level captions provide context-rich semantics, because they encode logical relations among elements. They provide better alignment with emotional causality, because they can describe the actual visual details that evoke emotion rather than merely naming objects. They also improve scalability, because MLLMs can caption any emotion-rich image collection using a standardized pipeline.

Because MLLMs can hallucinate, CoEmoGen introduces a CLIP-based filtering step. For each image-caption pair, the model computes CLIP similarity, ranks samples within each emotion category, and discards the bottom 20% with the lowest similarity (Yuan et al., 5 Aug 2025). No additional human validation or manual refinement stage is described in the paper.

Prompt ablations show two consistent findings: including the emotional prior helps, and constraining the output to one concise sentence works better than allowing longer descriptions (Yuan et al., 5 Aug 2025). The paper attributes the latter to reduced hallucination noise and fewer irrelevant details. This suggests that the gain from caption supervision is not merely extra text, but structured text with tightly controlled semantic scope.

5. HiLoRA, hierarchical adaptation, and optimization

The second principal innovation is HiLoRA, a hierarchy of low-rank adapters motivated by the claim that emotions with the same polarity share low-level appearance while differing in high-level semantics (Yuan et al., 5 Aug 2025). Standard LoRA adapts a pretrained weight matrix by a low-rank update,

$I_i$ 7

with $I_i$ 8 and $I_i$ 9 (Yuan et al., 5 Aug 2025).

CoEmoGen replaces a single shared LoRA with two coordinated branches (Yuan et al., 5 Aug 2025):

Eight emotion-specific LoRAs $y_i$ 0
Two polarity-shared LoRAs $y_i$ 1

For a target emotion, only the matching emotion-specific LoRA and the corresponding polarity-shared LoRA are activated. For example, for amusement under positive polarity,

$y_i$ 2

The paper interprets $y_i$ 3 as modeling low-level positive-polarity cues such as brightness and colorfulness, and $y_i$ 4 as modeling amusement-specific high-level semantics (Yuan et al., 5 Aug 2025). The exact insertion points inside the U-Net are not specified beyond attachment to the denoising network.

Training combines a standard latent diffusion loss with a semantic consistency loss. The denoising objective is

$y_i$ 5

where $y_i$ 6 is the latent encoder and $y_i$ 7 the denoising U-Net (Yuan et al., 5 Aug 2025).

To prevent semantic collapse, the paper adds a cosine semantic loss between the learned condition and the caption embedding:

$y_i$ 8

where $y_i$ 9 is the CLIP text encoder (Yuan et al., 5 Aug 2025). The paper states that $c_i$ 0 and $c_i$ 1 are combined, but does not provide an explicit weighted-sum coefficient.

The reported implementation details are specific: Neuro-symbolic Mapper hidden size 512, Visual-Perception Encoder feature dimension $c_i$ 2, HiLoRA rank $c_i$ 3, batch size 1, AdamW, $c_i$ 4, $c_i$ 5, weight decay $c_i$ 6, learning rate $c_i$ 7, 130,000 training iterations, two NVIDIA RTX 4090 GPUs, and random oversampling for imbalance handling (Yuan et al., 5 Aug 2025). No stage-wise training schedule is described; the paper states that trainable modules are optimized jointly.

6. EmoArt, scalability, and domain extension

A major part of CoEmoGen’s claim is scalability, defined in two senses: scalable supervision and scalable domains (Yuan et al., 5 Aug 2025). Scalable supervision refers to replacing manual word-level attributes with a standardized MLLM captioning + CLIP filtering pipeline. Scalable domains refers to transfer from general emotional imagery to artistic emotional imagery.

To demonstrate the latter, the paper introduces EmoArt, described as the first large-scale emotional art image dataset (Yuan et al., 5 Aug 2025). Construction begins with about 100,000 artistic images collected from WikiArt, covering 129 artists, 11 genres, and 27 styles. An emotion classifier pretrained on EmoSet predicts emotion categories, and only samples with emotion confidence greater than 0.75 are retained, resulting in 13,633 strongly emotion-representative samples (Yuan et al., 5 Aug 2025).

EmoArt contains six emotion categories, excluding excitement and disgust because each accounts for less than 1% in the collected art data (Yuan et al., 5 Aug 2025). The same MLLM captioning and CLIP-based filtering procedure used for EmoSet is then applied to EmoArt.

The paper frames artistic imagery as especially suitable for EICG because it is emotion-rich, stylistically diverse, and useful for demonstrating cross-domain scalability (Yuan et al., 5 Aug 2025). It also presents application examples for emotion transfer and emotion fusion, including combinations such as amusement + awe and awe + fear. The paper does not claim zero-shot support for unseen emotion categories; its scalability claim is explicitly about easier expansion of datasets and training corpora rather than unseen-class generalization (Yuan et al., 5 Aug 2025).

7. Empirical performance, ablations, and limitations

Evaluation follows EmoGen and generates 1,000 images per emotion (Yang et al., 2024, Yuan et al., 5 Aug 2025). The five reported metrics are FID, LPIPS, Emo-A, Sem-C, and Sem-D (Yuan et al., 5 Aug 2025).

Method	FID $c_i$ 8	LPIPS $c_i$ 9	Emo-A $y_i$ 0	Sem-C $y_i$ 1	Sem-D $y_i$ 2
Stable Diffusion	44.05	0.687	70.77%	0.608	0.0199
Textual Inversion	50.51	0.702	74.87%	0.605	0.0282
DreamBooth	46.89	0.661	70.50%	0.614	0.0178
EmoGen	41.60	0.717	76.25%	0.633	0.0335
CoEmoGen	40.66	0.732	80.15%	0.641	0.0349

Relative to EmoGen, CoEmoGen improves FID from 41.60 to 40.66, LPIPS from 0.717 to 0.732, Emo-A from 76.25% to 80.15%, Sem-C from 0.633 to 0.641, and Sem-D from 0.0335 to 0.0349 (Yuan et al., 5 Aug 2025). The paper highlights at least a 3.9% improvement in Emo-A over competing methods. Qualitatively, it reports more natural compositions, richer colors, smoother visuals, and less reliance on stereotyped object bundles (Yuan et al., 5 Aug 2025).

The user study included 17 participants—8 female, 9 male, aged 13–51—who compared CoEmoGen with baselines on realism, emotion faithfulness, semantic diversity, and semantic coherence (Yuan et al., 5 Aug 2025). CoEmoGen achieved overwhelming preference on all four criteria, with an average selection rate of 88.42% on semantic coherence and an average user preference rate of 78.67% in direct pairwise comparison with EmoGen (Yuan et al., 5 Aug 2025).

Ablations attribute most of the gain to the semantic and hierarchical components. Removing $y_i$ 3 degrades performance sharply to FID 50.32, LPIPS 0.698, Emo-A 65.90%, Sem-C 0.562, and Sem-D 0.0255 (Yuan et al., 5 Aug 2025). Removing emotion-specific LoRAs yields FID 45.30, Emo-A 75.37%, and Sem-D 0.0308, which the paper interprets as evidence of semantic entanglement among same-polarity emotions (Yuan et al., 5 Aug 2025). Removing polarity-shared LoRAs produces a smaller quantitative drop—FID 41.47, Emo-A 78.83%—but the paper reports qualitatively less natural brightness and color saturation (Yuan et al., 5 Aug 2025). A separate loss-design ablation finds that cosine semantic loss outperforms MAE, MSE, and K-L alternatives (Yuan et al., 5 Aug 2025).

The paper does not include a dedicated limitations section, but several limitations are explicit or implicit (Yuan et al., 5 Aug 2025). Caption generation remains vulnerable to MLLM hallucination, even if CLIP filtering reduces noise. Emotion imbalance is sufficiently strong to require random oversampling. EmoArt omits excitement and disgust, indicating reduced coverage in the artistic domain. The ablations show that the method is sensitive to semantic supervision: without $y_i$ 4, semantic collapse becomes pronounced. The explicit future direction is to investigate the denoising process in greater depth and analyze dynamic shifts of attention toward emotion-related attributes across denoising stages (Yuan et al., 5 Aug 2025).

Taken together, CoEmoGen redefines EICG around coherent sentence-level semantic supervision and hierarchically shared adaptation. Its contribution is not only a stronger set of benchmark numbers, but a clearer statement of what emotional image generation requires: emotionally meaningful content relations, scalable supervision beyond manual attributes, and an architectural distinction between polarity-shared appearance and emotion-specific semantics (Yuan et al., 5 Aug 2025).