---
title: 'CoEmoGen: Emotional Image Synthesis'
url: https://www.emergentmind.com/topics/coemogen
type: topic
---

# CoEmoGen: Emotional Image Synthesis

Searching arXiv for CoEmoGen and closely related work to ground the article in current literature.
CoEmoGen is a pipeline for **Emotional Image Content Generation (EICG)**, the task of generating **semantically clear, emotionally faithful images conditioned on a target emotion category** [2508.03535]. It is positioned as a successor to **EmoGen** [2401.04608], but departs from earlier EICG systems by replacing word-level attribute supervision with **sentence-level, emotion-focused captions** and by introducing a **Hierarchical Low-Rank Adaptation (HiLoRA)** mechanism for emotion-conditioned diffusion. Within the literature, CoEmoGen addresses open-domain emotional image synthesis rather than facial-expression generation alone, distinguishing it from unified facial-affect control frameworks such as **C2A2** [2404.01243].

## 1. Problem formulation and conceptual scope

CoEmoGen inherits the EICG task formulation introduced by EmoGen: given an emotion such as *awe*, *contentment*, or *anger*, the model should synthesize an image whose content is recognizable and whose affective impression matches the target category [2401.04608]. The paper follows the **Mikels emotion model** with eight emotions: four positive-polarity categories—**amusement, awe, contentment, excitement**—and four negative-polarity categories—**anger, disgust, fear, sadness** [2508.03535].

The central technical premise is that abstract emotions are not reducible to a single visual entity. The paper argues that emotions are conveyed through a combination of **low-level cues** such as brightness and colorfulness, **mid-level content** such as objects and scenes, and **high-level semantic factors** such as facial expressions, actions, and interactions [2508.03535]. This contrasts with standard text-to-image diffusion, which is highly effective for concrete concepts but much weaker on abstract concepts.

A core implication is that EICG is not merely prompt-based image generation with emotion words inserted into the prompt. The paper explicitly emphasizes a psychological view in which **visual emotions are strongly correlated with specific semantics**, so low-level style manipulation alone is insufficient [2508.03535]. This emphasis also differentiates CoEmoGen from systems centered on continuous facial-expression control, such as C2A2, where the primary object of generation is a face conditioned on a unified affect representation rather than general emotional image content [2404.01243].

## 2. Limitations of prior EICG and the rationale for CoEmoGen

The principal baseline is **EmoGen**, which conditions generation using **word-level attribute labels** derived from EmoSet [2401.04608]. CoEmoGen identifies three structural deficiencies in that strategy [2508.03535].

First, word-level labels lack **contextual association**. Isolated tokens do not encode how visual elements relate to one another, so they provide weak guidance for coherent scene composition. Second, the correlation between many labels and the target emotion is weak or ambiguous. Some labels are poor emotional indicators, while genuinely emotion-triggering details may be absent from annotation. Third, dependence on manual attribute labels creates **missing annotations and poor scalability**; incomplete labeling limits the usable corpus and reduces diversity.

The paper characterizes the downstream effect as **semantically incoherent collage-like outputs**, in which emotions become associated with stereotyped object sets rather than relational scene structure [2508.03535]. Its qualitative example is that *anger* may collapse into a conjunction of “fire, tigers, wide-open mouths,” producing implausible composites such as a **flaming tiger**. EmoGen had already argued that emotion-conditioned generation requires explicit semantic grounding and proposed an attribute loss to connect emotion to content [2401.04608]. CoEmoGen extends that line of work by asserting that **sentence-level coherence** is the missing supervisory unit.

This reframing is the paper’s main conceptual contribution. Instead of treating emotion supervision as a bag of emotionally associated words, it treats it as **emotion-triggering visual semantics expressed as a single coherent sentence**. This suggests that the main failure of earlier EICG is not only insufficient conditioning strength, but also the wrong granularity of semantic representation.

## 3. Architecture and conditioning pipeline

CoEmoGen is organized as a conditional latent diffusion pipeline built on **Stable Diffusion v1.5** with **CLIP ViT-L/14** as both image encoder and text encoder [2508.03535]. The trainable modules are the **Neuro-symbolic Mapper**, the **Visual-Perception Encoder**, and **HiLoRA**; the CLIP image encoder, CLIP text encoder, and base U-Net weights remain frozen [2508.03535].

For each training sample,
$$
\mathcal{D}_i = \{I_i, y_i, c_i\},
$$
where $I_i$ is the image, $y_i$ the emotion label, and $c_i$ the generated caption [2508.03535].

The emotion label $y_i$ is first converted to a **one-hot vector** $y_i^o$, which is passed through the **Neuro-symbolic Mapper** to obtain an emotional descriptor $e_i$ [2508.03535]. The mapper is implemented with fully connected layers and nonlinearities. The image is then encoded by the CLIP image encoder to yield a visual embedding $v_i$, and CoEmoGen fuses $e_i$ and $v_i$ with a cross-attention-based **Visual-Perception Encoder**:
$$
e_i^v=\text{Softmax}\!\left(\frac{e_iW_q(v_iW_k)^T}{\sqrt{d_0}}\right)v_iW_v,
$$
where $W_q$, $W_k$, and $W_v$ are learnable weights and $d_0$ is the feature dimension [2508.03535].

The visually enhanced descriptor $e_i^v$ is then passed into the CLIP text encoder to obtain the final **emotion condition** $e_i^c$ [2508.03535]. This condition guides a frozen Stable Diffusion U-Net equipped with HiLoRA adapters. At inference time, following EmoGen, visual embeddings are sampled from a pre-constructed **Gaussian emotion cluster** to increase diversity [2508.03535; 2401.04608].

An ablation on emotion representation reports that **one-hot + Neuro-symbolic Mapper** outperforms both learnable class vectors and frozen text encodings, achieving **FID 40.66**, **LPIPS 0.732**, **Emo-A 80.15%**, **Sem-C 0.641**, and **Sem-D 0.0349** [2508.03535]. The paper interprets this as a balance between semantic consistency and diversity, whereas learnable vectors form a bottleneck and frozen text encodings over-constrain semantics.

## 4. Emotion-focused caption construction

CoEmoGen’s semantic supervision is built from **multimodal large language model (MLLM)** captions rather than manual attributes [2508.03535]. The MLLM is used strictly for **data construction**, not as the generative backbone. For each EmoSet image, the model is prompted with the known target emotion and asked to produce a **one-sentence caption** centered on emotion-relevant visual details, including brightness, colorfulness, scene type, object classes, facial expressions, and human actions [2508.03535].

The prompt used in the paper is:

> `<Image>` This image evokes a strong emotion of `{<emotion>}`. Provide a one-sentence caption that vividly describes the visual details, focusing on elements like brightness, colorfulness, scene type, object classes, facial expressions, and human actions that effectively convey and express this emotion.

The paper’s argument for captions over attributes has three parts [2508.03535]. Sentence-level captions provide **context-rich semantics**, because they encode logical relations among elements. They provide **better alignment with emotional causality**, because they can describe the actual visual details that evoke emotion rather than merely naming objects. They also improve **scalability**, because MLLMs can caption any emotion-rich image collection using a standardized pipeline.

Because MLLMs can hallucinate, CoEmoGen introduces a CLIP-based filtering step. For each image-caption pair, the model computes CLIP similarity, ranks samples within each emotion category, and **discards the bottom 20%** with the lowest similarity [2508.03535]. No additional human validation or manual refinement stage is described in the paper.

Prompt ablations show two consistent findings: **including the emotional prior helps**, and constraining the output to **one concise sentence** works better than allowing longer descriptions [2508.03535]. The paper attributes the latter to reduced hallucination noise and fewer irrelevant details. This suggests that the gain from caption supervision is not merely extra text, but structured text with tightly controlled semantic scope.

## 5. HiLoRA, hierarchical adaptation, and optimization

The second principal innovation is **HiLoRA**, a hierarchy of low-rank adapters motivated by the claim that emotions with the same polarity share low-level appearance while differing in high-level semantics [2508.03535]. Standard LoRA adapts a pretrained weight matrix by a low-rank update,
$$
W^\prime = W + \Delta W = W + A \cdot B,
$$
with $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times d^\prime}$ [2508.03535].

CoEmoGen replaces a single shared LoRA with two coordinated branches [2508.03535]:

- **Eight emotion-specific LoRAs** $\{\Delta W_j^e\}_{j=1}^{8}$
- **Two polarity-shared LoRAs** $\{\Delta W_k^p\}_{k=1}^{2}$

For a target emotion, only the matching emotion-specific LoRA and the corresponding polarity-shared LoRA are activated. For example, for **amusement** under **positive polarity**,
$$
W^\prime = W + \Delta W_1^p + \Delta W_2^e
= W + A_1^p \cdot B_1^p + A_2^e \cdot B_2^e.
$$
The paper interprets $\Delta W_1^p$ as modeling low-level positive-polarity cues such as brightness and colorfulness, and $\Delta W_2^e$ as modeling amusement-specific high-level semantics [2508.03535]. The exact insertion points inside the U-Net are not specified beyond attachment to the denoising network.

Training combines a standard latent diffusion loss with a semantic consistency loss. The denoising objective is
$$
\mathcal{L}_{\text{LDM}}
=
\mathbb{E}_{\mathcal{E}(\cdot), I_i, e_i^c, \epsilon, t}
\left[
\left\|
\epsilon - \epsilon_{\theta}(z_t, t, \mathcal{E}(I_i), e_i^c)
\right\|_2^2
\right],
$$
where $\mathcal{E}(\cdot)$ is the latent encoder and $\epsilon_\theta(\cdot)$ the denoising U-Net [2508.03535].

To prevent semantic collapse, the paper adds a cosine semantic loss between the learned condition and the caption embedding:
$$
\mathcal{L}_{\text{SEM}}
=
1 - \frac{e_i^{c} \cdot \mathcal{T}(c_i)}{\|e_i^{c}\| \|\mathcal{T}(c_i)\|},
$$
where $\mathcal{T}$ is the CLIP text encoder [2508.03535]. The paper states that $\mathcal{L}_{\text{LDM}}$ and $\mathcal{L}_{\text{SEM}}$ are combined, but does not provide an explicit weighted-sum coefficient.

The reported implementation details are specific: **Neuro-symbolic Mapper hidden size 512**, **Visual-Perception Encoder feature dimension $d_0 = 768$**, **HiLoRA rank $r = 4$**, **batch size 1**, **AdamW**, $\beta_1 = 0.9$, $\beta_2 = 0.999$, **weight decay $1e^{-2}$**, **learning rate $1e^{-3}$**, **130,000 training iterations**, **two NVIDIA RTX 4090 GPUs**, and **random oversampling** for imbalance handling [2508.03535]. No stage-wise training schedule is described; the paper states that trainable modules are optimized jointly.

## 6. EmoArt, scalability, and domain extension

A major part of CoEmoGen’s claim is **scalability**, defined in two senses: scalable supervision and scalable domains [2508.03535]. Scalable supervision refers to replacing manual word-level attributes with a standardized **MLLM captioning + CLIP filtering** pipeline. Scalable domains refers to transfer from general emotional imagery to artistic emotional imagery.

To demonstrate the latter, the paper introduces **EmoArt**, described as the **first large-scale emotional art image dataset** [2508.03535]. Construction begins with about **100,000 artistic images** collected from **WikiArt**, covering **129 artists**, **11 genres**, and **27 styles**. An emotion classifier pretrained on EmoSet predicts emotion categories, and only samples with **emotion confidence greater than 0.75** are retained, resulting in **13,633 strongly emotion-representative samples** [2508.03535].

EmoArt contains **six emotion categories**, excluding **excitement** and **disgust** because each accounts for less than 1% in the collected art data [2508.03535]. The same MLLM captioning and CLIP-based filtering procedure used for EmoSet is then applied to EmoArt.

The paper frames artistic imagery as especially suitable for EICG because it is **emotion-rich**, stylistically diverse, and useful for demonstrating cross-domain scalability [2508.03535]. It also presents application examples for **emotion transfer** and **emotion fusion**, including combinations such as **amusement + awe** and **awe + fear**. The paper does not claim zero-shot support for unseen emotion categories; its scalability claim is explicitly about easier expansion of datasets and training corpora rather than unseen-class generalization [2508.03535].

## 7. Empirical performance, ablations, and limitations

Evaluation follows EmoGen and generates **1,000 images per emotion** [2401.04608; 2508.03535]. The five reported metrics are **FID**, **LPIPS**, **Emo-A**, **Sem-C**, and **Sem-D** [2508.03535].

| Method | FID $\downarrow$ | LPIPS $\uparrow$ | Emo-A $\uparrow$ | Sem-C $\uparrow$ | Sem-D $\uparrow$ |
|---|---:|---:|---:|---:|---:|
| Stable Diffusion | 44.05 | 0.687 | 70.77% | 0.608 | 0.0199 |
| Textual Inversion | 50.51 | 0.702 | 74.87% | 0.605 | 0.0282 |
| DreamBooth | 46.89 | 0.661 | 70.50% | 0.614 | 0.0178 |
| EmoGen | 41.60 | 0.717 | 76.25% | 0.633 | 0.0335 |
| **CoEmoGen** | **40.66** | **0.732** | **80.15%** | **0.641** | **0.0349** |

Relative to EmoGen, CoEmoGen improves **FID from 41.60 to 40.66**, **LPIPS from 0.717 to 0.732**, **Emo-A from 76.25% to 80.15%**, **Sem-C from 0.633 to 0.641**, and **Sem-D from 0.0335 to 0.0349** [2508.03535]. The paper highlights at least a **3.9% improvement in Emo-A** over competing methods. Qualitatively, it reports more natural compositions, richer colors, smoother visuals, and less reliance on stereotyped object bundles [2508.03535].

The user study included **17 participants**—**8 female**, **9 male**, aged **13–51**—who compared CoEmoGen with baselines on realism, emotion faithfulness, semantic diversity, and semantic coherence [2508.03535]. CoEmoGen achieved overwhelming preference on all four criteria, with an **average selection rate of 88.42%** on semantic coherence and an **average user preference rate of 78.67%** in direct pairwise comparison with EmoGen [2508.03535].

Ablations attribute most of the gain to the semantic and hierarchical components. Removing $\mathcal{L}_{\text{SEM}}$ degrades performance sharply to **FID 50.32**, **LPIPS 0.698**, **Emo-A 65.90%**, **Sem-C 0.562**, and **Sem-D 0.0255** [2508.03535]. Removing **emotion-specific LoRAs** yields **FID 45.30**, **Emo-A 75.37%**, and **Sem-D 0.0308**, which the paper interprets as evidence of semantic entanglement among same-polarity emotions [2508.03535]. Removing **polarity-shared LoRAs** produces a smaller quantitative drop—**FID 41.47**, **Emo-A 78.83%**—but the paper reports qualitatively less natural brightness and color saturation [2508.03535]. A separate loss-design ablation finds that **cosine** semantic loss outperforms **MAE**, **MSE**, and **K-L** alternatives [2508.03535].

The paper does not include a dedicated limitations section, but several limitations are explicit or implicit [2508.03535]. Caption generation remains vulnerable to **MLLM hallucination**, even if CLIP filtering reduces noise. **Emotion imbalance** is sufficiently strong to require random oversampling. **EmoArt** omits **excitement** and **disgust**, indicating reduced coverage in the artistic domain. The ablations show that the method is sensitive to semantic supervision: without $\mathcal{L}_{\text{SEM}}$, semantic collapse becomes pronounced. The explicit future direction is to investigate the **denoising process** in greater depth and analyze **dynamic shifts of attention toward emotion-related attributes across denoising stages** [2508.03535].

Taken together, CoEmoGen redefines EICG around **coherent sentence-level semantic supervision** and **hierarchically shared adaptation**. Its contribution is not only a stronger set of benchmark numbers, but a clearer statement of what emotional image generation requires: emotionally meaningful content relations, scalable supervision beyond manual attributes, and an architectural distinction between **polarity-shared appearance** and **emotion-specific semantics** [2508.03535].

Source: https://www.emergentmind.com/topics/coemogen