UniEmo: Unified Emotion Recognition & Generation

• UniEmo is a unified framework for emotional understanding and generation that uses a shared hierarchical representation to address recognition and synthesis simultaneously.
• It employs a Vision Transformer with learnable expert queries to capture multi-scale emotional cues and conditions a diffusion model for affect-matched image generation.
• Joint training with generation-driven feedback improves both top-1 classification accuracy and the semantic clarity of generated images.

UniEmo is a unified framework for emotional understanding and emotional image generation that treats the two as complementary rather than isolated problems. It was introduced to jointly solve supervised image-level emotion recognition and Emotional Image Content Generation (EICG) by learning a shared hierarchical emotional representation through learnable expert queries, then reusing that representation to condition a diffusion model (Zhu et al., 31 Jul 2025). In this formulation, emotional understanding predicts an emotion label for an input image, while emotional generation produces an image whose content is semantically clear and whose global affect matches a target emotion. The framework is built around a Vision Transformer, a hierarchical emotional understanding chain, an emotion-conditioned diffusion generator, and two generation-driven feedback mechanisms that improve the understanding component during joint training and data augmentation (Zhu et al., 31 Jul 2025).

1. Conceptual scope and problem formulation

UniEmo addresses two tasks that earlier work commonly handled separately. The first is emotional understanding, defined as supervised image-level emotion recognition: the input is an image and the output is an emotion label from a discrete set, with performance reported by top-1 classification accuracy. The second is emotional generation, defined in the EICG setting: the input is an emotion category $e$, and the output is an image whose affect matches $e$ while retaining semantically recognizable content (Zhu et al., 31 Jul 2025).

The framework’s central premise is that these tasks are “inherently complementary and can mutually enhance each other.” The paper attributes the main technical difficulty to “the abstract nature of emotions,” which makes it necessary to learn visual representations useful for both discrimination and synthesis. UniEmo therefore introduces a hierarchical emotional understanding chain with learnable expert queries that progressively extracts multi-scale emotional features, then fuses those queries with the final emotional representation to guide a diffusion model (Zhu et al., 31 Jul 2025).

The representation learned by UniEmo is not restricted to a single backbone stage or a single semantic granularity. Scene-level structure, object-level triggers, and the ViT class token all contribute to the final emotional condition used for generation. A plausible implication is that UniEmo treats emotional semantics as distributed across global context, local saliency, and backbone-level summary tokens rather than as a single latent code.

2. Hierarchical emotional understanding chain

The understanding module is built on a Vision Transformer whose layers are divided into three stages. The input image is split into patches, projected into token embeddings, and prepended with a learnable class token. UniEmo then introduces two sets of learnable expert queries: scene-level queries and object-level queries (Zhu et al., 31 Jul 2025).

In the first stage, layers $1 \dots L_1$ process the initial token sequence together with the scene-level query: $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$ The scene query is then refined by an Interactive Block that uses patch tokens as context: $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$ This stage is intended to capture global scene-level understanding (Zhu et al., 31 Jul 2025).

The second stage introduces the object-level query at layer $L_1+1$, concatenating it with the refined scene query and image tokens. Layers $L_1+1 \dots L_2$ then process the combined sequence, and a second Interactive Block refines the object query. This stage is described as focusing on object-level understanding. The final stage, covering layers $L_2+1 \dots N$, jointly refines the emotional representation with both expert queries active, and the final class token $T_N^{\text{Cls}}$ is used for emotion classification (Zhu et al., 31 Jul 2025).

The expert queries are explicitly anchored to semantic attributes through contrastive supervision with CLIP text embeddings. For scene queries, the paper defines

$$\mathcal{L}_{\text{S}} = -\frac{1}{K} \sum_{i=1}^{K} \log \frac{ \exp\left(\mathrm{sim}\left(\hat{Q}_{L_1,i}^{\text{Scene}}, z_i^{\text{Scene}}\right)/\tau\right) }{ \sum_{j=1}^{K} \exp\left(\mathrm{sim}\left(\hat{Q}_{L_1,i}^{\text{Scene}}, z_j^{\text{Scene}}\right)/\tau\right) },$$

with an analogous $e$0 for object queries and temperature $e$1 (Zhu et al., 31 Jul 2025). Scene attributes include examples such as “bushland” and “stadium,” while object attributes include examples such as “lion” and “fire.” This supervision makes the hierarchical chain a coarse-to-fine emotional reasoning mechanism: scene queries capture broad environmental context, object queries capture emotionally salient triggers, and the class token integrates them for final prediction.

3. Emotional generation and condition fusion

The generation module is a text-to-image diffusion model in the Stable Diffusion family, instantiated with Stable Diffusion v1.5 or Stable Diffusion XL base-1.0 in experiments. Rather than conditioning the generator on a natural-language prompt, UniEmo conditions it on a fused emotional representation built from the understanding module (Zhu et al., 31 Jul 2025).

The fusion uses three sources: the refined scene query $e$2, the refined object query $e$3, and the final class token $e$4. Each is mapped through a two-layer MLP with ReLU, and the outputs are combined as

$e$5

where $e$6 denotes element-wise addition (Zhu et al., 31 Jul 2025). The coefficients $e$7 and $e$8 are emotional correlation coefficients that weight scene and object features by their relevance to the target emotion.

To compute emotional correlation, UniEmo builds an attribute base from all scene and object attributes in the dataset, identifies the attributes most semantically related to a query, and averages the log-probabilities of a pretrained emotion classifier over images containing each attribute: $e$9 Here $1 \dots L_1$0 is an image containing attribute $1 \dots L_1$1, $1 \dots L_1$2 is the classifier log-probability for emotion $1 \dots L_1$3, and $1 \dots L_1$4 is the number of images with attribute $1 \dots L_1$5 (Zhu et al., 31 Jul 2025). The stated purpose is to up-weight emotionally diagnostic attributes such as “fire” for anger or fear and down-weight relatively neutral attributes.

The fused condition $1 \dots L_1$6 is inserted as the embedding of a special emotion token and passed through a CLIP text transformer $1 \dots L_1$7, so the denoising network is trained with the standard diffusion objective

$1 \dots L_1$8

At inference time, UniEmo estimates a Gaussian in the feature space of $1 \dots L_1$9, $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$0, and $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$1 for each emotion, samples features from those Gaussians, fuses them into $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$2, and generates images via the diffusion model (Zhu et al., 31 Jul 2025).

To ensure that the condition itself stays emotionally aligned, the paper introduces an emotional condition loss

$$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$3

where $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$4 is the predicted emotion distribution from a classifier on top of $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$5, $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$6 is the ground-truth emotion, $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$7 is a negative category, $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$8 is cosine distance, and $$\left[T_{L_1}^{\text{Cls}}, T_{L_1}^{\text{Patch}}, Q_{L_1}^{\text{Scene}}\right] = E_{1:L_1}\bigl(\left[T_0^{\text{Cls}}, T_0^{\text{Patch}}, Q_0^{\text{Scene}}\right]\bigr).$$9 (Zhu et al., 31 Jul 2025). The negative category is chosen as the second-highest scoring class if the model predicts correctly, or as the misclassified category otherwise. This explicitly separates the target emotion from confusable alternatives in the fused condition space.

4. Joint training and generation-driven feedback

UniEmo trains in two stages. The first stage optimizes the understanding branch alone with

$$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$0

using $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$1, $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$2, and $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$3 (Zhu et al., 31 Jul 2025). This stage establishes the hierarchical emotional representation before generation is introduced.

The second stage jointly trains understanding and generation with

$$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$4

using $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$5, $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$6, and $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$7 (Zhu et al., 31 Jul 2025). Because the generative and condition losses backpropagate into the Vision Transformer, the expert queries, and the fusion MLPs, the paper argues that the generation component provides implicit feedback to the understanding component. Figure-based ablation results reported in the paper show that adding joint training improves understanding accuracy across ViT-B/32, ViT-B/16, and ViT-L/14 (Zhu et al., 31 Jul 2025).

UniEmo further introduces explicit feedback through synthetic data augmentation. After joint training, the model generates many emotional images and filters them using two scores: Emo-A, which measures whether the generated image matches the intended emotion, and Sem-C, which measures semantic clarity. Images are ranked by both metrics, and the paper keeps samples whose ranks fall within the top $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$8 to $$\begin{aligned} \bar{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + A(Q_{L_1}^{\text{Scene}}, T_{L_1}^{\text{Patch}}), \ \tilde{Q}_{L_1}^{\text{Scene}} &= \bar{Q}_{L_1}^{\text{Scene}} + F(\bar{Q}_{L_1}^{\text{Scene}}), \ \hat{Q}_{L_1}^{\text{Scene}} &= Q_{L_1}^{\text{Scene}} + \beta \, \tilde{Q}_{L_1}^{\text{Scene}} . \end{aligned}$$9 for both, with $L_1+1$0 and $L_1+1$1 selected empirically (Zhu et al., 31 Jul 2025). The filtered images are then added back to the real dataset for further training of the understanding module.

This dual-feedback design is a distinguishing property of UniEmo. A plausible implication is that the framework uses generation not only as a downstream task but also as a mechanism for representation shaping and data-space expansion.

5. Datasets, evaluation protocol, and empirical results

UniEmo evaluates emotional understanding on EmoSet, FI, EmotionROI, Twitter I, and Twitter II. EmoSet contains 3.3M images total and 118,102 manually labeled by humans, with 8 emotion categories and attributes including brightness, colorfulness, scene type, object class, facial expression, and human action. FI contains 22,683 images with 8 emotion categories. EmotionROI contains 1,980 images from Flickr with 6 balanced emotion categories: joy, surprise, anger, disgust, fear, and sadness. Twitter I contains 1,269 images with binary positive/negative labels, and Twitter II contains 603 images with the same binary setup (Zhu et al., 31 Jul 2025).

Generation experiments use EmoSet as the common training set and report FID, Emo-A, Sem-C, LPIPS, and Sem-D, following the evaluation setting of EmoGen. Understanding is evaluated by top-1 accuracy (Zhu et al., 31 Jul 2025).

The main reported results are summarized below.

Setting	UniEmo result	Notes
EmoSet understanding, ViT-L/14	85.30%	Top-1 accuracy
FI understanding, ViT-L/14	87.65%	Top-1 accuracy
FI zero-shot, ViT-L/14	71.22%	Pretrain on EmoSet, test on FI
EmotionROI, ViT-L/14	76.78%	Top-1 accuracy
Twitter I, ViT-L/14	95.63%	Top-1 accuracy
Twitter II, ViT-L/14	91.68%	Top-1 accuracy
EmoSet generation, SD 1.5	FID 27.73, Emo-A 79.66%	LPIPS 0.793, Sem-C 0.640, Sem-D 0.0383
EmoSet generation, SD XL	FID 26.61, Emo-A 81.74%	LPIPS 0.807, Sem-C 0.642, Sem-D 0.0392

On large-scale understanding benchmarks, UniEmo with ViT-B/32 reports 83.52% on EmoSet and 85.22% on FI, while ViT-L/14 reaches 85.30% on EmoSet and 87.65% on FI. On smaller datasets, ViT-L/14 reports 76.78% on EmotionROI, 95.63% on Twitter I, and 91.68% on Twitter II. In zero-shot transfer from EmoSet to FI, UniEmo with ViT-L/14 achieves 71.22% (Zhu et al., 31 Jul 2025).

For emotional generation, UniEmo with SD 1.5 reports FID 27.73, LPIPS 0.793, Emo-A 79.66%, Sem-C 0.640, and Sem-D 0.0383. With SD XL, it reports FID 26.61, LPIPS 0.807, Emo-A 81.74%, Sem-C 0.642, and Sem-D 0.0392 (Zhu et al., 31 Jul 2025). The paper states that these results “significantly outperform” comparison methods such as Stable Diffusion, DreamBooth, Textual Inversion, and EmoGen under the same backbones.

Ablation results further isolate the effect of the fusion design. Using only the class token yields FID 30.02 and Emo-A 71.35%. Adding scene and object representations without emotional correlation improves results modestly, while adding emotional correlation coefficients improves them further. With both scene and object coefficients plus emotional condition loss, the paper reports FID 27.87, LPIPS 0.790, and Emo-A 79.77% in the ablation setting, outperforming the corresponding cross-entropy variant (Zhu et al., 31 Jul 2025).

6. Position within unified emotion research

The term “UniEmo” sits within a broader research movement toward unified affective modeling, but the 2025 paper uses the exact name for a specific visual framework that unifies emotional understanding and emotional image generation through a shared representation and dual feedback (Zhu et al., 31 Jul 2025). It should not be conflated with adjacent systems that unify different task families.

Several neighboring efforts illustrate the broader landscape. UniEmoX addresses universal scene emotion perception through cross-modal semantic-guided large-scale pretraining for scene-centric and person-centric visual emotion analysis (Chen et al., 2024). EmoVerse turns a standard multimodal LLM into a unified engine for multimodal sentiment analysis, emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction via multistage multitask instruction tuning (Li et al., 2024). UniMEEC unifies multimodal emotion recognition in conversation and emotion-cause pair extraction through a causal prompt template and hierarchical graph interactions (Hu et al., 2024). ProEmoTrans formulates Unseen Emotion Recognition in Conversation (UERC), where training and test emotion sets are disjoint, and performs prototype-based transfer to unseen conversational emotions (Peng et al., 27 Aug 2025). EmoOmni and Nano-EmoX extend the unification agenda into multimodal dialogue and empathy, respectively, by structuring emotional reasoning across perception, intention, strategy, and expression or across perception, understanding, and interaction hierarchies (Tian et al., 25 Feb 2026, Huang et al., 2 Mar 2026).

These systems do not solve the same problem as UniEmo. UniEmo is specifically concerned with the bidirectional coupling of recognizing what emotion an image evokes and generating an image that evokes a target emotion. By contrast, UniEmoX is a pretraining framework for visual perception, EmoVerse is an MLLM for affective task multiplexing, and UniMEEC and ProEmoTrans are conversation-centered. This suggests that “unified emotion” has become an umbrella design principle rather than a single task definition.

The paper mainly emphasizes performance gains and architectural novelty, but its own formulation indicates some boundaries. Training relies on datasets with rich scene and object attributes, the emotion space is discrete, and the emotional correlation coefficient depends on an external pretrained emotion classifier and attribute statistics (Zhu et al., 31 Jul 2025). A plausible implication is that extending UniEmo to continuous affective dimensions, richer taxonomies, or cross-modal settings would require new supervision schemes rather than only architectural scaling.

In the literature on affective generation and understanding, UniEmo is therefore best understood as a framework that operationalizes unification through three linked commitments: a hierarchical coarse-to-fine emotional representation, diffusion conditioning derived from the same representation, and generation-driven feedback that improves the discriminative model (Zhu et al., 31 Jul 2025).