---
title: 'EMOTIC Dataset: Emotion Recognition in the Wild'
url: https://www.emergentmind.com/topics/emotic-dataset
type: topic
---

# EMOTIC Dataset: Emotion Recognition in the Wild

The EMOTIC dataset is a large-scale resource for the study of emotion recognition in images "in the wild," with rigorous multi-label categorical and continuous affective annotation schemes. Designed to support machine learning approaches that jointly exploit personal, bodily, and rich contextual cues, EMOTIC has become a primary benchmark in affective computing for recognizing human emotions beyond facial expression—enabling models to interpret affect in unconstrained, real-world scenarios and across a broad spectrum of emotion categories and affective states [2312.07500, 2308.00228, 2305.03500, 2003.13401].

## 1. Dataset Composition and Annotation Schema

EMOTIC comprises approximately 23,571 natural, unconstrained images, sourced from web, social media, and public image collections; these images are annotated with 34,320 person instances, where each individual is independently labeled [2003.13401, 2312.07500]. For each annotated person, the dataset provides:

- **Discrete Labels**: Multi-label assignments over 26 emotion categories, including but not limited to Peace, Affection, Engagement, Disapproval, Anger, Sensitivity, Fear, Suffering. Label selection is non-exclusive, with each person typically assigned 1–9 applicable categories [2312.07500, 2502.06875].
- **Continuous Labels**: Three-dimensional Valence–Arousal–Dominance (VAD) annotations, each on a scale from 1–10 (or equivalently [0,10], integer-valued), reflecting the person's pleasantness (V), activation (A), and control (D). In alternative annotation protocols, VAD may be in [–1, 1] or in [0, 1], dependent on paper-specific normalization [2312.07500, 2305.03500, 2502.06875].
- **Metadata**: Per-person gender (M/F) and age group (child, teenager, adult) labels are also recorded. Global context metadata such as image source (e.g., COCO, ADE20K) is included for each sample [2305.03500].
- **Localization**: Each person is localized by an axis-aligned bounding box in COCO format, with uncropped full-scene context images available for context-aware modeling [2312.07500, 2003.13401].

Annotation is performed by crowdworkers (typically via Amazon Mechanical Turk). For each person, annotators are presented with both the cropped bounding box and the whole-scene context; discrete categories are collected via checklist, and VAD scores via sliders or integer selection, with the number of annotators per instance varying by data split (see Section 3). Final labels are obtained by per-category voting (for discrete) and mean-aggregation (for VAD) [2312.07500, 2003.13401].

## 2. Emotion Representations and Label Distribution

The 26 discrete emotion categories are derived to represent a broad, nuanced taxonomy beyond basic Ekman emotions, encompassing both primary and social/complex states. The full category list includes: Affection, Anger, Annoyance, Anticipation, Aversion, Confidence, Disapproval, Disconnection, Disquietment, Doubt/Confusion, Embarrassment, Engagement, Esteem, Excitement, Fatigue, Fear, Happiness, Pain, Peace, Pleasure, Sadness, Sensitivity, Suffering, Surprise, Sympathy, Yearning [2312.07500, 2003.13401].

Label distribution is highly imbalanced, with “Engagement” the most prevalent (≈55% of person-instances) and “Embarrassment” among the least common (≈1%), resulting in a long-tailed frequency structure [2312.07500, 2003.13401]. VAD annotations approximately uniformly cover the 1–10 integer range, but clustering around moderate (central) values is observed [2003.13401]. 

| Category           | Example Frequency (Person-Instances) |  
|--------------------|--------------------------------------|  
| Engagement         | ~28,000                              |  
| Confidence         | ~22,000                              |  
| Excitement         | ~20,000                              |  
| Embarrassment      | ~1,500                               |  
| Pain               | ~1,900                               |  

A plausible implication is that evaluation protocols and losses must explicitly address this class imbalance to avoid bias in multi-label settings.

## 3. Data Splits and Annotation Protocol

While some works treat EMOTIC as a single corpus, the canonical split (Kosti et al. 2019) is widely used and consists of:

- **Training Set**: ≈18,300 person-instances (or ~12,821 images), annotated by a single worker per person.
- **Validation Set**: ≈2,860 person-instances (or ~2,350 images), each labeled by five independent annotators for inter-rater agreement estimation.
- **Test Set**: ≈6,900 person-instances (or ~4,700 images), with three annotators per person; this split is held out for standardized benchmarking and challenge submissions [2312.07500, 2305.03500, 2502.06875, 2003.13401].

Quality control includes annotator qualification tests, in-batch sentinels, and outlier filtering. Ground-truth discrete labels are set by union or mode across annotators; for VAD, the (integer or continuous) mean is used [2003.13401].

## 4. Evaluation Metrics and Loss Functions

The standard metric for discrete emotion prediction on EMOTIC is mean Average Precision (mAP) over the 26 categories. Given category $i$, let average precision $\text{AP}_i$ be computed from the precision-recall curve; then

$$
\mathrm{mAP} = \frac{1}{26} \sum_{i=1}^{26} \mathrm{AP}_i
$$

For continuous VAD regression, mean absolute error (MAE) per dimension is used:

$$
\mathrm{MAE} = \frac{1}{M} \sum_{j=1}^M |\hat{y}_j - y_j|
$$

Some works also report the Jaccard Index for multilabel accuracy, Average Absolute Error (AAE), or use margin/Hinge/Smooth-L1 (Huber) variants for regression objectives [2312.07500, 2308.00228, 2003.13401, 2305.03500].

Weighted Euclidean loss ($L^{disc}$) for the discrete branch uses per-class weights $w_i = 1 / \ln(c + p_i)$, where $p_i$ is the class frequency and $c$ a smoothing hyperparameter, enforcing greater penalty for underrepresented classes [2312.07500, 2305.03500].

## 5. Impact on Affective Computing and Model Development

The EMOTIC dataset enables and benchmarks a range of architectures that incorporate both personal and contextual cues. Baseline systems typically employ dual-branch convolutional networks, processing person and scene context separately before fusion [2003.13401]. Recent advances include:

- **Multi-branch architectures** leveraging face, body, and context branches independently (input sizes e.g., 224×224 for context/body, 48×48 for face crops), supporting robust multi-cue fusion [2312.07500].
- **Context-aware models** integrating high-level semantic scene encoders (e.g., ViT-style and GIN context encoders) and object/attribute relationships to boost performance on ambiguous cases [2308.00228, 2305.03500].
- **Pose/depth augmentation** for handling occlusions and partial person visibility, often employing graph-based methods (e.g., spatial–temporal GCNs) and EmbraceNet-style random-modality fusion [2308.00228].
- **Loss weighting and ablation** for addressing severe class imbalance and quantifying the contributions of body, face, context, scene, and depth streams.

An explicit finding is that context features (global scene, object semantics, depth) confer the largest mAP gains (+5–7%) over body/face-only models, validating the psychological insight that emotion perception in real-world images crucially depends on situational context [2308.00228].

Quantitative performance has advanced from mean AP of 27.38% (body+context CNN, Kosti et al. 2019) to over 40% with modern multi-modal and scene-semantic architectures [2308.00228, 2312.07500].

## 6. Specialized Applications and Benchmark Usage

EMOTIC is routinely used to benchmark:

- **Emotion recognition models**: discrete category multi-label and VAD regression, in person-localized and context-integrated settings [2312.07500, 2308.00228, 2305.03500].
- **Vision-language systems**: recent works investigate the capacity of LLMs to infer emotions from structured VAD values (e.g., using EMOTIC to map valence/arousal to category labels or semantic captions) [2502.06875].
- **Real-time and on-device inference**: efficient single-stream context models achieve >90 frames/sec on consumer GPUs, supporting edge deployment scenarios [2305.03500].

Notably, LLMs exhibit poor discrete category mapping from VAD alone but perform well on free-text affective description when conditioned solely on VAD [2502.06875]. A plausible implication is that nuanced emotion understanding requires richer multimodal input than raw VAD, but VAD still serves as a privacy-preserving input for generative affective description.

## 7. Access, Licensing, and Extensions

The EMOTIC dataset and baseline code are open-sourced for non-commercial research via the project repository (http://sunai.uoc.edu/emotic/, https://github.com/rkosti/emotic) [2003.13401]. Community extensions include fine-tuning splits, specialized context encoders, and expanded annotation studies in combination with other emotion recognition resources.

A plausible implication is that EMOTIC will remain central to advancing context-rich computer vision models for affective computing, and its annotation schema serves as a template for future datasets that capture the nuanced, socially embedded nature of emotion "in the wild."

Source: https://www.emergentmind.com/topics/emotic-dataset