varEmotion: Ambiguous Facial Expression Dataset

• varEmotion is a synthetic facial expression dataset focused on ambiguous, boundary-near stimuli to reveal high perceptual variability in human emotion judgments.
• It employs a two-stage generation pipeline that uses ANN uncertainty guidance and human validation to generate photorealistic, ambiguous facial images.
• The dataset facilitates uncertainty-aware emotion recognition and human–model alignment by providing empirical response distributions across six basic emotion categories.

Searching arXiv for the varEmotion paper and closely related survey context. varEmotion is a facial-expression dataset constructed to expose high perceptual variability in human emotion recognition by concentrating on ambiguous, boundary-near stimuli rather than prototypical expressions. It was introduced in the study "Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering Human Perceptual Variability on Facial Expressions" (Deng et al., 19 Jul 2025). The dataset is based on the hypothesis that facial expression samples that are ambiguous for ANN classifiers also elicit divergent perceptual judgments among human observers. Its images are generated through a perceptual boundary sampling procedure guided by ANN uncertainty, then validated through large-scale human behavioral experiments. The resulting resource contains photorealistic synthetic faces paired with empirical human response distributions over six basic emotion categories, enabling analysis of group-level and individual-level perceptual ambiguity as well as human–model alignment (Deng et al., 19 Jul 2025).

1. Conceptual motivation and dataset scope

The central problem addressed by varEmotion is that people often disagree about which emotion a face expresses, even for the six “basic” categories of surprise, fear, disgust, happiness, sadness, and anger. The source study characterizes this as high perceptual variability and notes that it has been underexplored in visual emotion recognition, despite modern ANNs achieving high accuracy on standard benchmarks (Deng et al., 19 Jul 2025).

The dataset is novel in that it does not primarily collect clear or prototypical expressions. Instead, it deliberately concentrates on ambiguous, boundary-near stimuli that maximize human uncertainty and inter-individual disagreement. This differentiates it from datasets such as AffectNet, RAF-DB, and FER2013, which are described as mainly containing clearly labeled, prototypical expressions. The core hypothesis is that ANN decision boundaries can serve as a computational probe for finding stimuli that expose human perceptual ambiguity at scale (Deng et al., 19 Jul 2025).

This design places varEmotion at the intersection of affective computing, cognitive science, and behavioral modeling. A plausible implication is that the dataset is less suited to conventional single-label benchmarking than to analyses of uncertainty, disagreement, and personalization. That interpretation is consistent with the recommended uses reported for the dataset: uncertainty-aware emotion recognition, human–model alignment, and individual-specific fine-tuning (Deng et al., 19 Jul 2025).

2. Perceptual boundary sampling and image synthesis

varEmotion is built with a two-stage generation pipeline inspired by DALL·E 2. In the first stage, the method performs boundary-targeted embedding sampling with diffusion and uncertainty guidance. It starts from original image embeddings used as a prior, following CoCoG, and applies a guided diffusion denoising process to steer samples toward ANN decision boundaries between a target pair of emotions such as anger vs fear (Deng et al., 19 Jul 2025).

The classifier used for guidance is an MLP trained on RAF-DB image embeddings, producing six emotion probabilities for [surprise, fear, disgust, happiness, sadness, anger]. The uncertainty guidance objective reported in the study is

$loss(x,y) = -p(y \mid x) * q(y)$

where $p(y \mid x)$ is the classifier’s predicted distribution for embedding $x$, and $q(y)$ is the target distribution, for example mass concentrated on two target emotions. The guided reverse diffusion step is

$$x_{t-1} = DDPM^-(x_t) - \gamma \nabla_{x_t} loss(x_t,y)$$

with guidance strength $\gamma = 0.5$. The first-stage diffusion backbone is U-ViT (Deng et al., 19 Jul 2025).

In the second stage, the sampled embeddings are converted into images using an SDXL-turbo model without a classifier, using only prompt guidance. The reported purpose of this stage is to provide a strong natural-image prior that regularizes the generation process, producing realistic faces while preserving the ambiguity induced in the embedding space (Deng et al., 19 Jul 2025).

Because the second stage introduces randomness, generated candidates are filtered to ensure that they remain boundary-near for the ANN. Let $p_{emotion1}$ and $p_{emotion2}$ denote the classifier’s activations for the two target emotions, and let $k_{emotion1}$ and $k_{emotion2}$ denote the corresponding 75th-percentile activation values computed over RAF-DB. The retained images satisfy

$p(y \mid x)$0

The study interprets this criterion as producing a set that reliably confuses the ANN and is therefore expected to evoke human uncertainty (Deng et al., 19 Jul 2025).

The paper also lists alternative boundary objectives for context, including entropy maximization on classifier outputs, margin-based targeting between classes, and generator-space optimization with regularization, but explicitly states that these were not the objectives used in the reported experiments (Deng et al., 19 Jul 2025).

3. Dataset composition and behavioral protocol

The dataset comprises 1,678 photorealistic images generated by the two-stage pipeline and filtered according to the boundary-nearness criterion. It covers six basic emotion categories: surprise, fear, disgust, happiness, sadness, and anger (Deng et al., 19 Jul 2025).

The human study recruited 100 participants online under ethics approval with informed consent, with no personal data collected and with participants free to withdraw. Data were retained from 66 participants whose sentinel-trial accuracy exceeded 70%. Each participant completed 400 trials, consisting of 390 random trials and 10 sentinel trials. The total number of retained trials was 22,450, corresponding to an average of approximately 13.4 judgments per image (Deng et al., 19 Jul 2025).

The behavioral task used a forced-choice protocol. Each trial presented a fixation cross for 300 ms, followed by image presentation for 200 ms, followed by a response screen with six buttons, one for each emotion category. Participants selected the perceived emotion, and response time was recorded. Trials ended upon choice (Deng et al., 19 Jul 2025).

Component	Reported value
Emotion categories	6
Stimuli	1,678 photorealistic generated face images
Participants recruited	100
Participants retained	66
Trials per participant	400
Total retained trials	22,450
Average judgments per image	~13.4

The paper does not report demographics, and it does not report Fleiss’ $p(y \mid x)$1 or similar inter-rater agreement coefficients. It also does not specify file formats, a metadata schema, or per-image annotations beyond the analyses reported, namely choice distributions, entropy, and response times (Deng et al., 19 Jul 2025).

4. Labels, uncertainty, and outcome taxonomy

For each image, varEmotion aggregates human forced-choice responses across the six categories to form an empirical categorical distribution $p(y \mid x)$2, obtained by normalizing counts by the number of judgments. This means that the primary annotation is not a single hard label but a distribution over emotion categories (Deng et al., 19 Jul 2025).

The principal uncertainty metric used in the paper is the entropy of human responses:

$p(y \mid x)$3

The distribution of this entropy across images is used to characterize perceptual ambiguity. The reported entropy values are mostly between 0.5 and 2.0 and are described as approximately normal, indicating broad variability across participants (Deng et al., 19 Jul 2025).

To summarize whether guided ambiguity was achieved, the study defines a guidance outcome taxonomy for each guided emotion pair $p(y \mid x)$4. Let $p(y \mid x)$5 and $p(y \mid x)$6 be the fractions of human choices assigned to those two emotions for a given image. The reported categories are:

• success: $p(y \mid x)$7 and $p(y \mid x)$8
• bias: $p(y \mid x)$9 and $x$0
• failure: $x$1

Nearly 80% of images are reported as success or bias, meaning that the generated stimuli predominantly drove choices toward the guided pair of emotions (Deng et al., 19 Jul 2025).

The paper evaluates human–model divergence primarily through correlations of entropies rather than KL-based divergences. It notes that KL formulations are common for aligning models to behavioral distributions, but the reported training procedure uses CrossEntropyLoss for six-way classification rather than explicit KL training (Deng et al., 19 Jul 2025).

5. Empirical findings on perceptual variability and human–model alignment

The main empirical result is that ANN-confusing stimuli—that is, stimuli near classifier decision boundaries—also provoke heightened perceptual uncertainty in human participants. This is supported by the elevated entropy of human responses and by the high proportion of stimuli categorized as success or bias with respect to the intended guided emotion pair (Deng et al., 19 Jul 2025).

Response times are reported to be concentrated between 500 and 1,500 ms with a heavy tail, and entropy and reaction time are positively correlated with Spearman $x$2. This suggests that the behavioral ambiguity captured by the dataset is associated not only with disagreement across observers but also with slower decisions (Deng et al., 19 Jul 2025).

The paper further argues that ANN representations can be aligned to both group-level and individual-level human perceptual patterns through fine-tuning on behavioral data. The reported objective is not only higher classification accuracy on varEmotion but also stronger correspondence between model-predicted and human-measured uncertainty (Deng et al., 19 Jul 2025).

A plausible implication is that varEmotion functions as a testbed for modeling personal perceptual boundaries rather than merely aggregate categorical labels. That interpretation is reinforced by the individual-specific fine-tuning results discussed in the source study (Deng et al., 19 Jul 2025).

6. Fine-tuning protocols, baselines, and reported performance

The study evaluates three architectures: CLIP with an added MLP head, DAN, and ResEmoNet. These are analyzed as BaseNet models before fine-tuning, GroupNet models after group-level fine-tuning, and IndivNet models after individual-level fine-tuning (Deng et al., 19 Jul 2025).

For group-level fine-tuning, the training data are a mixture of RAF-DB and varEmotion at a 1:2 ratio (RAF-DB:varEmotion), with the stated goal of preserving RAF-DB performance while adapting to human variability. For individual-level fine-tuning, training initializes from the group-level model and uses a mixture of varEmotion-i and varEmotion at a 2:1 ratio. The reported data split is 4:1 train/validation, with the constraint that the individual-level validation set does not overlap the group validation set. Optimization uses Adam, learning rate $x$3, batch size 128, 15 epochs, and CrossEntropyLoss for six-class prediction (Deng et al., 19 Jul 2025).

The reported gains on varEmotion relative to the corresponding base model are:

Architecture	Group-level gain on varEmotion	Individual-over-group gain on varEmotion-i
CLIP	+5%	+2.5%
DAN	+14%	+3.5%
ResEmoNet	+35%	+1%

The source also reports that RAF-DB accuracy remained comparable across BaseNet, GroupNet, and IndivNet, indicating that fine-tuning for human variability did not meaningfully degrade standard-dataset performance (Deng et al., 19 Jul 2025).

For uncertainty alignment, the most specific reported result is for DAN: the Spearman rank correlation between model-predicted entropy and human entropy increased from $x$4 before fine-tuning to $x$5 after group-level fine-tuning. The study interprets this as strong alignment in uncertainty structure (Deng et al., 19 Jul 2025).

7. Position within affect datasets, limitations, and access

varEmotion is explicitly contrasted with conventional facial-expression datasets such as AffectNet, RAF-DB, and FER2013, which the source describes as mainly containing clearly labeled, prototypical expressions. By contrast, varEmotion is curated to concentrate on ambiguous, boundary-near cases that provoke human disagreement (Deng et al., 19 Jul 2025).

It is not a valence–arousal dataset. A survey of datasets for valence and arousal inference published between 2008 and 2024 does not mention varEmotion or any variant spelling in its overview table, narrative, or references (Schneider et al., 1 Oct 2025). This indicates that varEmotion occupies a distinct niche centered on categorical perceptual variability rather than continuous VA annotation.

The study identifies several limitations. No demographic data were collected, preventing stratified fairness or generalization analyses by age, gender, culture, or related axes. The stimuli are synthetic, and although they are photorealistic, they may not cover the full ecological range of real-world expressions, micro-expressions, lighting, occlusions, or cultural display rules. The generation and filtering procedure is driven by an ANN trained on RAF-DB embeddings, so biases in RAF-DB or in the classifier may shape which ambiguous faces are generated and selected. The paper also notes reporting constraints, including the absence of inter-rater agreement statistics, metadata schemas, file-format specifications, and licensing details (Deng et al., 19 Jul 2025).

Regarding ethics and privacy, the behavioral study was approved by an institutional ethics committee, used informed consent, and collected no personal data. The use of synthetic faces avoids privacy concerns associated with real subjects, although the paper notes that cultural sensitivity and potential misinterpretation of expressions remain relevant considerations (Deng et al., 19 Jul 2025).

The paper does not provide a public download link, repository URL, or licensing terms. Interested researchers are directed to contact the authors for information about availability, file formats, metadata, or planned release (Deng et al., 19 Jul 2025).