---
title: 'UniEmo: Unified Emotion Recognition & Generation'
url: https://www.emergentmind.com/topics/uniemo
type: topic
---

# UniEmo: Unified Emotion Recognition & Generation

UniEmo is a unified framework for **emotional understanding** and **emotional image generation** that treats the two as complementary rather than isolated problems. It was introduced to jointly solve supervised image-level emotion recognition and **Emotional Image Content Generation (EICG)** by learning a shared hierarchical emotional representation through **learnable expert queries**, then reusing that representation to condition a diffusion model [2507.23372]. In this formulation, emotional understanding predicts an emotion label for an input image, while emotional generation produces an image whose content is semantically clear and whose global affect matches a target emotion. The framework is built around a Vision Transformer, a **hierarchical emotional understanding chain**, an emotion-conditioned diffusion generator, and two generation-driven feedback mechanisms that improve the understanding component during joint training and data augmentation [2507.23372].

## 1. Conceptual scope and problem formulation

UniEmo addresses two tasks that earlier work commonly handled separately. The first is **emotional understanding**, defined as supervised **image-level emotion recognition**: the input is an image and the output is an emotion label from a discrete set, with performance reported by top-1 classification accuracy. The second is **emotional generation**, defined in the **EICG** setting: the input is an emotion category \(e\), and the output is an image whose affect matches \(e\) while retaining semantically recognizable content [2507.23372].

The framework’s central premise is that these tasks are “inherently complementary and can mutually enhance each other.” The paper attributes the main technical difficulty to “the abstract nature of emotions,” which makes it necessary to learn visual representations useful for both discrimination and synthesis. UniEmo therefore introduces a **hierarchical emotional understanding chain with learnable expert queries** that progressively extracts **multi-scale emotional features**, then fuses those queries with the final emotional representation to guide a diffusion model [2507.23372].

The representation learned by UniEmo is not restricted to a single backbone stage or a single semantic granularity. Scene-level structure, object-level triggers, and the ViT class token all contribute to the final emotional condition used for generation. A plausible implication is that UniEmo treats emotional semantics as distributed across global context, local saliency, and backbone-level summary tokens rather than as a single latent code.

## 2. Hierarchical emotional understanding chain

The understanding module is built on a Vision Transformer whose layers are divided into three stages. The input image is split into patches, projected into token embeddings, and prepended with a learnable class token. UniEmo then introduces two sets of **learnable expert queries**: **scene-level queries** and **object-level queries** [2507.23372].

In the first stage, layers \(1 \dots L_1\) process the initial token sequence together with the scene-level query:
\[
\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right]
= E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).
\]
The scene query is then refined by an **Interactive Block** that uses patch tokens as context:
\[
\begin{aligned}
\bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \\
\tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \\
\hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} .
\end{aligned}
\]
This stage is intended to capture **global scene-level understanding** [2507.23372].

The second stage introduces the object-level query at layer \(L_1+1\), concatenating it with the refined scene query and image tokens. Layers \(L_1+1 \dots L_2\) then process the combined sequence, and a second Interactive Block refines the object query. This stage is described as focusing on **object-level understanding**. The final stage, covering layers \(L_2+1 \dots N\), jointly refines the emotional representation with both expert queries active, and the final class token \(T_N^{\text{Cls}}\) is used for emotion classification [2507.23372].

The expert queries are explicitly anchored to semantic attributes through contrastive supervision with CLIP text embeddings. For scene queries, the paper defines
\[
\mathcal{L}_{\text{S}}
= -\frac{1}{K} \sum_{i=1}^{K}
\log
\frac{
\exp\left(\mathrm{sim}\left(\hat{Q}_{L_1,i}^{\text{Scene}}, z_i^{\text{Scene}}\right)/\tau\right)
}{
\sum_{j=1}^{K}
\exp\left(\mathrm{sim}\left(\hat{Q}_{L_1,i}^{\text{Scene}}, z_j^{\text{Scene}}\right)/\tau\right)
},
\]
with an analogous \(\mathcal{L}_{\text{O}}\) for object queries and temperature \(\tau = 0.07\) [2507.23372]. Scene attributes include examples such as “bushland” and “stadium,” while object attributes include examples such as “lion” and “fire.” This supervision makes the hierarchical chain a coarse-to-fine emotional reasoning mechanism: scene queries capture broad environmental context, object queries capture emotionally salient triggers, and the class token integrates them for final prediction.

## 3. Emotional generation and condition fusion

The generation module is a **text-to-image diffusion model** in the Stable Diffusion family, instantiated with **Stable Diffusion v1.5** or **Stable Diffusion XL base-1.0** in experiments. Rather than conditioning the generator on a natural-language prompt, UniEmo conditions it on a fused emotional representation built from the understanding module [2507.23372].

The fusion uses three sources: the refined scene query \(\hat{Q}_{L_1}^{\text{Scene}}\), the refined object query \(\hat{Q}_{L_2}^{\text{Object}}\), and the final class token \(T_N^{\text{Cls}}\). Each is mapped through a two-layer MLP with ReLU, and the outputs are combined as
\[
c =
\alpha_{\text{Scene}} \cdot \phi_1(\hat{Q}_{L_1}^{\text{Scene}})
\oplus
\alpha_{\text{Object}} \cdot \phi_2(\hat{Q}_{L_2}^{\text{Object}})
\oplus
\phi_3(T_N^{\text{Cls}}),
\]
where \(\oplus\) denotes element-wise addition [2507.23372]. The coefficients \(\alpha_{\text{Scene}}\) and \(\alpha_{\text{Object}}\) are **emotional correlation coefficients** that weight scene and object features by their relevance to the target emotion.

To compute emotional correlation, UniEmo builds an attribute base from all scene and object attributes in the dataset, identifies the attributes most semantically related to a query, and averages the log-probabilities of a pretrained emotion classifier over images containing each attribute:
\[
\alpha_{j,e} = \frac{1}{M_j}\sum_{k=1}^{M_j}\ell(I_k,e).
\]
Here \(I_k\) is an image containing attribute \(j\), \(\ell(I_k,e)\) is the classifier log-probability for emotion \(e\), and \(M_j\) is the number of images with attribute \(j\) [2507.23372]. The stated purpose is to up-weight emotionally diagnostic attributes such as “fire” for anger or fear and down-weight relatively neutral attributes.

The fused condition \(c\) is inserted as the embedding of a special emotion token and passed through a CLIP text transformer \(\tau\), so the denoising network is trained with the standard diffusion objective
\[
\mathcal{L}(\theta)=
\mathbb{E}_{t,\mathbf{x}_0,\epsilon,c}
\left[
\left\|
\epsilon - \epsilon_\theta(\mathbf{x}_t,t,\tau(c))
\right\|^2
\right].
\]
At inference time, UniEmo estimates a Gaussian in the feature space of \(\hat{Q}_{L_1}^{\text{Scene}}\), \(\hat{Q}_{L_2}^{\text{Object}}\), and \(T_N^{\text{Cls}}\) for each emotion, samples features from those Gaussians, fuses them into \(c\), and generates images via the diffusion model [2507.23372].

To ensure that the condition itself stays emotionally aligned, the paper introduces an **emotional condition loss**
\[
\mathcal{L}_{\text{cond}}=
\max\bigl(0,\,
d(e^p,e^+) - d(e^p,e^-) + \xi
\bigr),
\]
where \(e^p\) is the predicted emotion distribution from a classifier on top of \(c\), \(e^+\) is the ground-truth emotion, \(e^-\) is a negative category, \(d(\cdot,\cdot)\) is cosine distance, and \(\xi=0.1\) [2507.23372]. The negative category is chosen as the second-highest scoring class if the model predicts correctly, or as the misclassified category otherwise. This explicitly separates the target emotion from confusable alternatives in the fused condition space.

## 4. Joint training and generation-driven feedback

UniEmo trains in two stages. The first stage optimizes the understanding branch alone with
\[
\mathcal{L}_{\text{U}}=
\lambda_1\mathcal{L}_{\text{S}}
+\lambda_2\mathcal{L}_{\text{O}}
+\lambda_3\mathcal{L}_{\text{Cls}},
\]
using \(\lambda_1=0.25\), \(\lambda_2=0.25\), and \(\lambda_3=0.5\) [2507.23372]. This stage establishes the hierarchical emotional representation before generation is introduced.

The second stage jointly trains understanding and generation with
\[
\mathcal{L}_{\text{Joint}}=
\gamma_1\mathcal{L}_{\text{U}}
+\gamma_2\mathcal{L}_{\text{cond}}
+\gamma_3\mathcal{L}(\theta),
\]
using \(\gamma_1=0.3\), \(\gamma_2=0.3\), and \(\gamma_3=0.4\) [2507.23372]. Because the generative and condition losses backpropagate into the Vision Transformer, the expert queries, and the fusion MLPs, the paper argues that the generation component provides **implicit feedback** to the understanding component. Figure-based ablation results reported in the paper show that adding joint training improves understanding accuracy across **ViT-B/32**, **ViT-B/16**, and **ViT-L/14** [2507.23372].

UniEmo further introduces **explicit feedback** through synthetic data augmentation. After joint training, the model generates many emotional images and filters them using two scores: **Emo-A**, which measures whether the generated image matches the intended emotion, and **Sem-C**, which measures semantic clarity. Images are ranked by both metrics, and the paper keeps samples whose ranks fall within the top \(\eta\%\) to \(\delta\%\) for both, with \(\eta=20\%\) and \(\delta=80\%\) selected empirically [2507.23372]. The filtered images are then added back to the real dataset for further training of the understanding module.

This dual-feedback design is a distinguishing property of UniEmo. A plausible implication is that the framework uses generation not only as a downstream task but also as a mechanism for representation shaping and data-space expansion.

## 5. Datasets, evaluation protocol, and empirical results

UniEmo evaluates emotional understanding on **EmoSet**, **FI**, **EmotionROI**, **Twitter I**, and **Twitter II**. EmoSet contains **3.3M images total** and **118,102 manually labeled by humans**, with **8 emotion categories** and attributes including brightness, colorfulness, scene type, object class, facial expression, and human action. FI contains **22,683 images** with **8 emotion categories**. EmotionROI contains **1,980 images** from Flickr with **6 balanced emotion categories**: joy, surprise, anger, disgust, fear, and sadness. Twitter I contains **1,269 images** with binary positive/negative labels, and Twitter II contains **603 images** with the same binary setup [2507.23372].

Generation experiments use EmoSet as the common training set and report **FID**, **Emo-A**, **Sem-C**, **LPIPS**, and **Sem-D**, following the evaluation setting of EmoGen. Understanding is evaluated by **top-1 accuracy** [2507.23372].

The main reported results are summarized below.

| Setting | UniEmo result | Notes |
|---|---:|---|
| EmoSet understanding, ViT-L/14 | **85.30%** | Top-1 accuracy |
| FI understanding, ViT-L/14 | **87.65%** | Top-1 accuracy |
| FI zero-shot, ViT-L/14 | **71.22%** | Pretrain on EmoSet, test on FI |
| EmotionROI, ViT-L/14 | **76.78%** | Top-1 accuracy |
| Twitter I, ViT-L/14 | **95.63%** | Top-1 accuracy |
| Twitter II, ViT-L/14 | **91.68%** | Top-1 accuracy |
| EmoSet generation, SD 1.5 | FID **27.73**, Emo-A **79.66%** | LPIPS **0.793**, Sem-C **0.640**, Sem-D **0.0383** |
| EmoSet generation, SD XL | FID **26.61**, Emo-A **81.74%** | LPIPS **0.807**, Sem-C **0.642**, Sem-D **0.0392** |

On large-scale understanding benchmarks, UniEmo with **ViT-B/32** reports **83.52%** on EmoSet and **85.22%** on FI, while **ViT-L/14** reaches **85.30%** on EmoSet and **87.65%** on FI. On smaller datasets, **ViT-L/14** reports **76.78%** on EmotionROI, **95.63%** on Twitter I, and **91.68%** on Twitter II. In zero-shot transfer from EmoSet to FI, UniEmo with **ViT-L/14** achieves **71.22%** [2507.23372].

For emotional generation, UniEmo with **SD 1.5** reports **FID 27.73**, **LPIPS 0.793**, **Emo-A 79.66%**, **Sem-C 0.640**, and **Sem-D 0.0383**. With **SD XL**, it reports **FID 26.61**, **LPIPS 0.807**, **Emo-A 81.74%**, **Sem-C 0.642**, and **Sem-D 0.0392** [2507.23372]. The paper states that these results “significantly outperform” comparison methods such as Stable Diffusion, DreamBooth, Textual Inversion, and EmoGen under the same backbones.

Ablation results further isolate the effect of the fusion design. Using only the class token yields **FID 30.02** and **Emo-A 71.35%**. Adding scene and object representations without emotional correlation improves results modestly, while adding emotional correlation coefficients improves them further. With both scene and object coefficients plus emotional condition loss, the paper reports **FID 27.87**, **LPIPS 0.790**, and **Emo-A 79.77%** in the ablation setting, outperforming the corresponding cross-entropy variant [2507.23372].

## 6. Position within unified emotion research

The term “UniEmo” sits within a broader research movement toward unified affective modeling, but the 2025 paper uses the exact name for a specific visual framework that unifies **emotional understanding** and **emotional image generation** through a shared representation and dual feedback [2507.23372]. It should not be conflated with adjacent systems that unify different task families.

Several neighboring efforts illustrate the broader landscape. **UniEmoX** addresses **universal scene emotion perception** through cross-modal semantic-guided large-scale pretraining for scene-centric and person-centric visual emotion analysis [2409.18877]. **EmoVerse** turns a standard multimodal large language model into a unified engine for **multimodal sentiment analysis, emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction** via multistage multitask instruction tuning [2412.08049]. **UniMEEC** unifies **multimodal emotion recognition in conversation** and **emotion-cause pair extraction** through a causal prompt template and hierarchical graph interactions [2404.00403]. **ProEmoTrans** formulates **Unseen Emotion Recognition in Conversation (UERC)**, where training and test emotion sets are disjoint, and performs prototype-based transfer to unseen conversational emotions [2508.19533]. **EmoOmni** and **Nano-EmoX** extend the unification agenda into multimodal dialogue and empathy, respectively, by structuring emotional reasoning across perception, intention, strategy, and expression or across perception, understanding, and interaction hierarchies [2602.21900] [2603.02123].

These systems do not solve the same problem as UniEmo. UniEmo is specifically concerned with the bidirectional coupling of **recognizing what emotion an image evokes** and **generating an image that evokes a target emotion**. By contrast, UniEmoX is a pretraining framework for visual perception, EmoVerse is an MLLM for affective task multiplexing, and UniMEEC and ProEmoTrans are conversation-centered. This suggests that “unified emotion” has become an umbrella design principle rather than a single task definition.

The paper mainly emphasizes performance gains and architectural novelty, but its own formulation indicates some boundaries. Training relies on datasets with rich **scene** and **object** attributes, the emotion space is **discrete**, and the emotional correlation coefficient depends on an external pretrained emotion classifier and attribute statistics [2507.23372]. A plausible implication is that extending UniEmo to continuous affective dimensions, richer taxonomies, or cross-modal settings would require new supervision schemes rather than only architectural scaling.

In the literature on affective generation and understanding, UniEmo is therefore best understood as a framework that operationalizes unification through three linked commitments: a **hierarchical coarse-to-fine emotional representation**, **diffusion conditioning derived from the same representation**, and **generation-driven feedback** that improves the discriminative model [2507.23372].

Source: https://www.emergentmind.com/topics/uniemo