EmoCap100K: Facial Emotion Caption Dataset

• EmoCap100K is a large-scale facial emotion caption dataset featuring 107,134 image–caption pairs with rich, free-form descriptions sourced from diverse films.
• The dataset employs a structured annotation protocol that captures global, local, and summary cues to depict nuanced affective states.
• Optimized for contrastive and vision-language learning, EmoCap100K boosts zero-shot and few-shot performance in facial emotion recognition tasks.

EmoCap100K is a large-scale facial emotion caption dataset designed to advance facial emotion representation learning by moving beyond the constraints of fixed emotion categories or low-dimensional affective scales. Consisting of 107,134 image–caption pairs, EmoCap100K provides semantically rich, structured natural language descriptions that encode both holistic and granular affective information extracted from diverse cinematic sources. The dataset is optimized for deep learning frameworks leveraging multi-level natural language supervision, and it is released under a CC-BY-4.0 license to support broad research in computer vision, affective computing, and vision-language modeling (Sun et al., 28 Jul 2025).

1. Motivation and Conceptual Advances

Traditional facial emotion recognition (FER) datasets typically restrict images to a predefined set of emotional classes (e.g., Ekman’s six basic emotions) or represent affect with low-dimensional coordinates (such as valence–arousal). This simplification inhibits the modeling of complex, nuanced, or atypical expressions and restricts generalization. EmoCap100K addresses these limitations by employing free-form, semantically dense captions for each face, fostering new forms of supervision—specifically, contrastive learning with natural-language–structured targets. The dataset aims to:

• Provide a corpus of over 100,000 face–caption pairs where captions detail both the global affective “gist” and fine-grained local facial cues.
• Enable paradigms that exploit the compositional structure and semantic richness of natural language, thus enhancing performance on zero-shot and few-shot FER tasks and related benchmarks.

2. Data Collection Methodology

Face images were sourced from over 1,000 publicly available films spanning genres (drama, comedy, thriller, documentary), processed via automated face detection to extract individual frames of human heads. Human volunteers exercised manual sampling to ensure dataset diversity along the axes of head pose (frontal to extreme profile), scene context (e.g., indoor/outdoor; group/solo), and variety of affective states, particularly emphasizing compound and atypical emotions. Preprocessing entailed cropping faces around the detected box, resizing to 224×224 pixels, and optional alignment using eye-corner coordinates. All images are stored in JPEG format within a flat directory structure. Collection rules prevented over-representation by any single actor or film.

3. Annotation Protocol and Caption Structure

Captions were generated using Google’s Gemini-1.5-Flash via a prompt-engineered, fully automated loop. For each detected face, the LLM received a prompt designed to elicit:

1. One sentence summarizing the global facial configuration.
2. Three sentences, each describing a distinct local facial cue (e.g., eyebrow lift, mouth movement, cheek tension).
3. A final sentence synthesizing global and local cues into a coherent summary.

The prompt was tuned to ensure captions of 200–400 words, capturing muscle activations (e.g., “left corner of the mouth tugs upward”), intensity (“slightly annoyed” versus “enraged”), blends (“happily surprised”), and contextual nuances (“in a dimly lit room”). Quality assurance was performed via manual spot-checks on 5% of samples; captions missing local cues or containing repeated tokens were regenerated.

The caption structure formalizes each annotation as $$T^i = [T^i_{Global}; T^i_{Local\{1…M\}}; T^i_{Summary}]$$, with $M=3$ local cues in the released dataset. This multi-level format supports supervised, contrastive, or retrieval-based learning paradigms.

4. Composition, Format, and Licensing

EmoCap100K comprises 107,134 samples, each pairing a 224×224 RGB face image and a linguistically rich caption. Caption statistics:

• Average words per caption: 267
• 703 unique emotion-descriptive terms
• Each caption contains explicit global, local, and summary components

Dataset splits follow an 80/10/10 protocol: | Split | Sample Count | |:-----------|------------:| | Train | 85,707 | | Validation | 10,714 | | Test | 10,713 |

The dataset is distributed as a .zip archive with:

• /images/: Flat directory of JPEG face crops
• captions.json: Structured with keys: id, split, image_path, global, local (list), summary

The resource is licensed under CC-BY-4.0 and is accessible via https://github.com/sunlicai/EmoCapCLIP or through institutional request.

5. Example Entries

Selected illustrative samples from the dataset highlight the range and granularity of semantic annotations:

Image ID	Global Description	Local Cues (sample)	Summary
000527	“A broad, open-mouthed grin dominates…”	“Upper and lower teeth are fully exposed…”<br>“Eyes are wide open, with pupils centered…”	“This expression combines joyful surprise…”
002341	“The jaw tenses and lips purse in a thin line.”	“Nostrils flare slightly…”<br>“Brow ridge presses downward…”<br>“Cheeks are sucked in…”	“Overall, an expression of quiet irritation…”

This granular annotation enables nuanced modeling of affect, capturing both micro and macro expressive features in natural language form.

6. Benchmarking, Supervised Tasks, and Learning Paradigms

Recommended usage scenarios include:

• Contrastive pre-training exploiting both global and local caption components
• Zero- and few-shot static FER (benchmarks: RAF-DB, AffectNet-7/8)
• Zero-shot dynamic FER (DFEW, AFEW, CREMA-D) via per-frame encoding with temporal pooling
• Zero-shot action unit (AU) detection by aligning local cue sentences with video frames
• Expression–caption retrieval tasks

Baseline results underscore the benefit of language-rich supervision: training a CLIP-style dual encoder (Vision Transformer ViT-B/32) on WIT-400M + EmoCap100K yields zero-shot UAR = 62.9%, WAR = 67.0% on RAF-DB, outperforming FLIP and Exp-CLIP baselines by over 20% on UAR.

Dynamic FER evaluation on DFEW yields UAR = 42.2% (prior video-CLIP: 36.8%).

Key losses for training include:

• Global contrastive loss:

$$\mathcal{L}_g = -\frac{1}{N} \sum_{i=1}^{N} \biggl[ \log \frac{e^{\mathrm{cos}(g_I^i,g_T^i)/\tau}}{\sum_{n=1}^N e^{\mathrm{cos}(g_I^i,g_T^n)/\tau}} + \log \frac{e^{\mathrm{cos}(g_T^i,g_I^i)/\tau}}{\sum_{n=1}^N e^{\mathrm{cos}(g_T^i,g_I^n)/\tau}} \biggr]$$

• Local intra-sample contrastive loss with corresponding positive mining expansions (see Section 4.1 and Eq 6 in (Sun et al., 28 Jul 2025)).

7. Implications and Research Directions

EmoCap100K enables new directions in facial emotion understanding by facilitating:

• Modeling of affect as a spectrum using open-vocabulary, compositional linguistic representations rather than pre-defined classes.
• Transfer learning and zero/few-shot generalization through vision-language pre-training on semantically dense captions.
• Multi-modal and multi-task learning, given the data structure supports AU detection, dynamic sequence modeling, and caption retrieval as auxiliary objectives.

A plausible implication is that integrating structured natural language supervision at scale addresses known bottlenecks in FER—specifically, overfitting to narrow categorical labels and poor cross-domain generalization. The architecture and prompt-engineering protocol exemplified in EmoCap100K may serve as a reference template for similarly structured datasets in related domains (action recognition, intent decoding, etc.).

EmoCap100K is positioned as a “drop-in” corpus for next-generation FER systems, retrieval-based expression search, and open-vocabulary affective computing (Sun et al., 28 Jul 2025).