---
title: 'varEmotion: Ambiguous Facial Expression Dataset'
url: https://www.emergentmind.com/topics/varemotion-dataset
type: topic
---

# varEmotion: Ambiguous Facial Expression Dataset

Searching arXiv for the varEmotion paper and closely related survey context.
varEmotion is a facial-expression dataset constructed to expose **high perceptual variability** in human emotion recognition by concentrating on **ambiguous, boundary-near stimuli** rather than prototypical expressions. It was introduced in the study "Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering Human Perceptual Variability on Facial Expressions" [2507.14549]. The dataset is based on the hypothesis that facial expression samples that are ambiguous for ANN classifiers also elicit divergent perceptual judgments among human observers. Its images are generated through a perceptual boundary sampling procedure guided by ANN uncertainty, then validated through large-scale human behavioral experiments. The resulting resource contains photorealistic synthetic faces paired with empirical human response distributions over six basic emotion categories, enabling analysis of group-level and individual-level perceptual ambiguity as well as human–model alignment [2507.14549].

## 1. Conceptual motivation and dataset scope

The central problem addressed by varEmotion is that people often disagree about which emotion a face expresses, even for the six “basic” categories of **surprise, fear, disgust, happiness, sadness,** and **anger**. The source study characterizes this as **high perceptual variability** and notes that it has been underexplored in visual emotion recognition, despite modern ANNs achieving high accuracy on standard benchmarks [2507.14549].

The dataset is novel in that it does **not** primarily collect clear or prototypical expressions. Instead, it deliberately concentrates on **ambiguous, boundary-near stimuli** that maximize human uncertainty and inter-individual disagreement. This differentiates it from datasets such as **AffectNet, RAF-DB,** and **FER2013**, which are described as mainly containing clearly labeled, prototypical expressions. The core hypothesis is that ANN decision boundaries can serve as a computational probe for finding stimuli that expose human perceptual ambiguity at scale [2507.14549].

This design places varEmotion at the intersection of affective computing, cognitive science, and behavioral modeling. A plausible implication is that the dataset is less suited to conventional single-label benchmarking than to analyses of uncertainty, disagreement, and personalization. That interpretation is consistent with the recommended uses reported for the dataset: uncertainty-aware emotion recognition, human–model alignment, and individual-specific fine-tuning [2507.14549].

## 2. Perceptual boundary sampling and image synthesis

varEmotion is built with a **two-stage generation pipeline** inspired by **DALL·E 2**. In the first stage, the method performs **boundary-targeted embedding sampling with diffusion and uncertainty guidance**. It starts from **original image embeddings** used as a prior, following **CoCoG**, and applies a guided diffusion denoising process to steer samples toward ANN decision boundaries between a target pair of emotions such as **anger vs fear** [2507.14549].

The classifier used for guidance is an **MLP trained on RAF-DB image embeddings**, producing six emotion probabilities for **[surprise, fear, disgust, happiness, sadness, anger]**. The uncertainty guidance objective reported in the study is

$$
loss(x,y) = -p(y \mid x) * q(y)
$$

where $p(y \mid x)$ is the classifier’s predicted distribution for embedding $x$, and $q(y)$ is the target distribution, for example mass concentrated on two target emotions. The guided reverse diffusion step is

$$
x_{t-1} = DDPM^-(x_t) - \gamma \nabla_{x_t} loss(x_t,y)
$$

with guidance strength $\gamma = 0.5$. The first-stage diffusion backbone is **U-ViT** [2507.14549].

In the second stage, the sampled embeddings are converted into images using an **SDXL-turbo** model **without a classifier**, using only prompt guidance. The reported purpose of this stage is to provide a strong natural-image prior that regularizes the generation process, producing realistic faces while preserving the ambiguity induced in the embedding space [2507.14549].

Because the second stage introduces randomness, generated candidates are filtered to ensure that they remain boundary-near for the ANN. Let $p_{emotion1}$ and $p_{emotion2}$ denote the classifier’s activations for the two target emotions, and let $k_{emotion1}$ and $k_{emotion2}$ denote the corresponding **75th-percentile activation values** computed over RAF-DB. The retained images satisfy

$$
p_{emotion1} > k_{emotion1} \quad \text{and} \quad p_{emotion2} > k_{emotion2}
$$

The study interprets this criterion as producing a set that reliably confuses the ANN and is therefore expected to evoke human uncertainty [2507.14549].

The paper also lists alternative boundary objectives for context, including **entropy maximization on classifier outputs**, **margin-based targeting between classes**, and **generator-space optimization with regularization**, but explicitly states that these were **not** the objectives used in the reported experiments [2507.14549].

## 3. Dataset composition and behavioral protocol

The dataset comprises **1,678 photorealistic images** generated by the two-stage pipeline and filtered according to the boundary-nearness criterion. It covers six basic emotion categories: **surprise, fear, disgust, happiness, sadness,** and **anger** [2507.14549].

The human study recruited **100 participants online** under ethics approval with informed consent, with no personal data collected and with participants free to withdraw. Data were retained from **66 participants** whose sentinel-trial accuracy exceeded **70%**. Each participant completed **400 trials**, consisting of **390 random** trials and **10 sentinel** trials. The total number of retained trials was **22,450**, corresponding to an average of approximately **13.4 judgments per image** [2507.14549].

The behavioral task used a forced-choice protocol. Each trial presented a **fixation cross for 300 ms**, followed by **image presentation for 200 ms**, followed by a response screen with **six buttons**, one for each emotion category. Participants selected the perceived emotion, and **response time** was recorded. Trials ended upon choice [2507.14549].

| Component | Reported value |
|---|---|
| Emotion categories | 6 |
| Stimuli | 1,678 photorealistic generated face images |
| Participants recruited | 100 |
| Participants retained | 66 |
| Trials per participant | 400 |
| Total retained trials | 22,450 |
| Average judgments per image | ~13.4 |

The paper does not report **demographics**, and it does not report **Fleiss’ $\kappa$** or similar inter-rater agreement coefficients. It also does not specify **file formats**, a **metadata schema**, or **per-image annotations beyond the analyses reported**, namely choice distributions, entropy, and response times [2507.14549].

## 4. Labels, uncertainty, and outcome taxonomy

For each image, varEmotion aggregates human forced-choice responses across the six categories to form an empirical categorical distribution $q(y \mid x)$, obtained by normalizing counts by the number of judgments. This means that the primary annotation is not a single hard label but a **distribution over emotion categories** [2507.14549].

The principal uncertainty metric used in the paper is the **entropy of human responses**:

$$
H\!\left(q(y \mid x)\right) = -\sum_{k} q_k(y \mid x)\log q_k(y \mid x)
$$

The distribution of this entropy across images is used to characterize perceptual ambiguity. The reported entropy values are mostly between **0.5 and 2.0** and are described as **approximately normal**, indicating broad variability across participants [2507.14549].

To summarize whether guided ambiguity was achieved, the study defines a **guidance outcome taxonomy** for each guided emotion pair $(emotion1, emotion2)$. Let $p_1$ and $p_2$ be the fractions of human choices assigned to those two emotions for a given image. The reported categories are:

- **success**: $\min(p_1,p_2) > 0.25$ and $(p_1+p_2) > 0.6$
- **bias**: $\min(p_1,p_2) < 0.25$ and $(p_1+p_2) > 0.6$
- **failure**: $(p_1+p_2) < 0.6$

Nearly **80% of images** are reported as **success** or **bias**, meaning that the generated stimuli predominantly drove choices toward the guided pair of emotions [2507.14549].

The paper evaluates human–model divergence primarily through **correlations of entropies** rather than KL-based divergences. It notes that KL formulations are common for aligning models to behavioral distributions, but the reported training procedure uses **CrossEntropyLoss for six-way classification** rather than explicit KL training [2507.14549].

## 5. Empirical findings on perceptual variability and human–model alignment

The main empirical result is that **ANN-confusing stimuli**—that is, stimuli near classifier decision boundaries—also provoke **heightened perceptual uncertainty in human participants**. This is supported by the elevated entropy of human responses and by the high proportion of stimuli categorized as **success** or **bias** with respect to the intended guided emotion pair [2507.14549].

Response times are reported to be concentrated between **500 and 1,500 ms** with a **heavy tail**, and **entropy and reaction time are positively correlated** with **Spearman $\rho = 0.42$**. This suggests that the behavioral ambiguity captured by the dataset is associated not only with disagreement across observers but also with slower decisions [2507.14549].

The paper further argues that ANN representations can be aligned to both **group-level** and **individual-level** human perceptual patterns through fine-tuning on behavioral data. The reported objective is not only higher classification accuracy on varEmotion but also stronger correspondence between model-predicted and human-measured uncertainty [2507.14549].

A plausible implication is that varEmotion functions as a testbed for modeling **personal perceptual boundaries** rather than merely aggregate categorical labels. That interpretation is reinforced by the individual-specific fine-tuning results discussed in the source study [2507.14549].

## 6. Fine-tuning protocols, baselines, and reported performance

The study evaluates three architectures: **CLIP with an added MLP head**, **DAN**, and **ResEmoNet**. These are analyzed as **BaseNet** models before fine-tuning, **GroupNet** models after group-level fine-tuning, and **IndivNet** models after individual-level fine-tuning [2507.14549].

For **group-level fine-tuning**, the training data are a mixture of **RAF-DB and varEmotion at a 1:2 ratio (RAF-DB:varEmotion)**, with the stated goal of preserving RAF-DB performance while adapting to human variability. For **individual-level fine-tuning**, training initializes from the group-level model and uses a mixture of **varEmotion-i and varEmotion at a 2:1 ratio**. The reported data split is **4:1 train/validation**, with the constraint that the individual-level validation set does not overlap the group validation set. Optimization uses **Adam**, **learning rate $1e{-4}$**, **batch size 128**, **15 epochs**, and **CrossEntropyLoss** for six-class prediction [2507.14549].

The reported gains on **varEmotion** relative to the corresponding base model are:

| Architecture | Group-level gain on varEmotion | Individual-over-group gain on varEmotion-i |
|---|---:|---:|
| CLIP | +5% | +2.5% |
| DAN | +14% | +3.5% |
| ResEmoNet | +35% | +1% |

The source also reports that **RAF-DB accuracy remained comparable across BaseNet, GroupNet, and IndivNet**, indicating that fine-tuning for human variability did not meaningfully degrade standard-dataset performance [2507.14549].

For uncertainty alignment, the most specific reported result is for **DAN**: the **Spearman rank correlation** between **model-predicted entropy** and **human entropy** increased from **$\rho = 0.26$** before fine-tuning to **$\rho = 0.85$** after group-level fine-tuning. The study interprets this as strong alignment in uncertainty structure [2507.14549].

## 7. Position within affect datasets, limitations, and access

varEmotion is explicitly contrasted with conventional facial-expression datasets such as **AffectNet, RAF-DB,** and **FER2013**, which the source describes as mainly containing clearly labeled, prototypical expressions. By contrast, varEmotion is curated to concentrate on **ambiguous, boundary-near cases that provoke human disagreement** [2507.14549].

It is not a **valence–arousal** dataset. A survey of datasets for valence and arousal inference published between 2008 and 2024 does **not** mention **varEmotion** or any variant spelling in its overview table, narrative, or references [2510.00738]. This indicates that varEmotion occupies a distinct niche centered on **categorical perceptual variability** rather than continuous VA annotation.

The study identifies several limitations. **No demographic data** were collected, preventing stratified fairness or generalization analyses by age, gender, culture, or related axes. The stimuli are **synthetic**, and although they are photorealistic, they may not cover the full ecological range of real-world expressions, **micro-expressions, lighting, occlusions,** or **cultural display rules**. The generation and filtering procedure is driven by an ANN trained on **RAF-DB embeddings**, so biases in RAF-DB or in the classifier may shape which ambiguous faces are generated and selected. The paper also notes reporting constraints, including the absence of inter-rater agreement statistics, metadata schemas, file-format specifications, and licensing details [2507.14549].

Regarding ethics and privacy, the behavioral study was approved by an institutional ethics committee, used informed consent, and collected **no personal data**. The use of **synthetic faces** avoids privacy concerns associated with real subjects, although the paper notes that **cultural sensitivity** and **potential misinterpretation of expressions** remain relevant considerations [2507.14549].

The paper does not provide a **public download link**, **repository URL**, or **licensing terms**. Interested researchers are directed to **contact the authors** for information about availability, file formats, metadata, or planned release [2507.14549].

Source: https://www.emergentmind.com/topics/varemotion-dataset