EmoNet-Face: Dual Systems for Emotion Analysis

• The paper introduces a dual-artifact approach featuring a real-time pipeline using HoG+SVM and a VGG-style CNN for emoji masking at 13 fps with 71.5% accuracy on FER2013.
• The benchmark suite presents a 40-category emotion taxonomy with expert annotations and demographically balanced synthetic datasets to capture nuanced facial affect.
• EmpathicInsight-Face ensembles achieve near human-expert performance, offering robust performance metrics and a new baseline for facial emotion inference.

EmoNet-Face denotes two distinct but related research artifacts centered on facial emotion analysis. In one usage, it refers to a real-time system for facial expression emoji masking that combines face detection using Histogram of Gradients and Support Vector Machine, facial expression categorization using a convolutional neural network trained on FER2013, and emoji masking via homography estimation (Wang et al., 2020). In a later usage, it refers to an expert-annotated benchmark suite for synthetic emotion recognition built around a 40-category emotion taxonomy, three AI-generated full-face datasets, and the EmpathicInsight-Face model (Schuhmann et al., 26 May 2025). The shared theme is facial affect inference, but the two lines of work differ in objective, granularity, and evaluation regime.

1. Scope and nomenclature

The name "EmoNet-Face" is associated with both a deployment-oriented real-time pipeline and a benchmark-oriented dataset-and-model suite. This dual use is important because the 2020 work is primarily an end-to-end computer vision system operating on live video, whereas the 2025 work is primarily a benchmark for synthetic emotion recognition with a broader emotion taxonomy and explicit expert annotation (Wang et al., 2020, Schuhmann et al., 26 May 2025).

Artifact	Primary purpose	Core elements
Real-time EmoNet-Face system	Emoji masking of a student’s face in live video	HoG + SVM face detection, CNN expression classification, homography-based overlay
EmoNet-Face benchmark suite	Synthetic emotion recognition benchmark	40-category taxonomy, EmoNet HQ/Binary/Big, expert labels, EmpathicInsight-Face

A common misconception is to treat these as a single evolving codebase or dataset family. The available record instead shows two separate projects that share a concern with facial emotion analysis but operate at different levels: one performs real-time expression-to-emoji rendering, and the other formalizes a fine-grained benchmark for emotion recognition.

2. Real-time expression-to-emoji system

The real-time system is organized as a three-stage pipeline: live video frame, face detection, cropped face patch, expression classification, predicted emotion label, and emoji masking via homography, producing a final video frame with overlaid emoji (Wang et al., 2020). The reported data-flow and real-time targets are an input resolution of 1280×720 px at 30 fps and a measured throughput of 13 fps, approximately 0.076 s per frame on an Intel i7-9750H CPU.

Face detection uses Dlib’s shape_predictor_68_face_landmarks, which internally implements a HoG + SVM detector. The standard Dlib implementation parameters reported in the technical blueprint are a cell size of 8×8 px, block size of 2×2 cells, block stride 8 px, and 9 orientation bins spanning 0–180°. The HoG feature vector per detection window is formed by concatenation of normalized block histograms, and the classifier is a linear-kernel SVM trained on HoG features with decision rule

$$w^\top \phi(x) + b \ge 0 \Rightarrow \text{“face”}.$$

The Dlib defaults listed are $C = 0.01$ and standard hinge loss with $L2$-regularization.

Facial expression classification is performed by a compact VGG-style network, "VGG BA SMALL". Preprocessing consists of cropping to the face bounding box, conversion to grayscale, resizing to 44×44 px, normalization of pixel values to $[0,1]$, and no further data augmentation in the published results. The architecture uses 3×3 convolutions with stride $= 1$ and padding $= 1$; each convolution is followed by BatchNorm2D and ReLU. After the final convolution block, the network applies 2D average-pool to collapse spatial dimensions and a single fully connected layer to 7 outputs. Training uses cross-entropy loss,

$L = -\sum_{c=1}^7 y_c \log p_c,$

with SGD, momentum $= 0.9$, weight decay $(L2) = 5 \times 10^{-4}$, initial learning rate $= 1 \times 10^{-2}$, decay by $C = 0.01$0 every 5 epochs from epoch 40 through epoch 250, gradient clipping, batch size 128, and total epochs 250.

The training dataset is FER2013 with splits as in the original dataset: approximately 28 709 for training, 3 589 public test used for validation, and 3 589 private test. The class distribution is described as heavily imbalanced, with “disgust” at approximately 437 training samples versus at least 3 000 for others, and no special rebalancing was performed. The best validation accuracy is 70.10%, and the best test accuracy is 71.50%. The reported per-class recall is approximately 80% for Neutral, 78% for Happy, 76% for Surprise, 68% for Sadness, 65% for Anger, 60% for Fear, and 52% for Disgust.

Emoji masking is implemented through planar homography. The system estimates

$C = 0.01$1

to map source emoji landmarks $C = 0.01$2 to target face landmarks $C = 0.01$3 such that

$C = 0.01$4

The solution is obtained by the standard DLT linear system $C = 0.01$5, with $C = 0.01$6 and solution by SVD, optionally with point normalization. No explicit RANSAC was used in the published pipeline. Six planar control points are manually marked on each emoji: left-eye outer corner, right-eye outer corner, left-mouth corner, right-mouth corner, philtrum point, and chin point. After estimating $C = 0.01$7, the chosen emoji image is warped into the face region and alpha-blended onto the original frame, producing a perspective-correct masking effect in real time.

Implementation uses PyTorch 1.x for CNN training and inference, Dlib for HoG+SVM face-landmarks, and OpenCV for image I/O and warping. On the Intel Core i7-9750H CPU, with no GPU used in the demo, the reported model sizes are 23.98 MB for VGG BA SMALL and approximately 100 kB for the Dlib face-landmark predictor. Measured end-to-end latency on 373 frames at 720 p is 28.21 s, corresponding to 13 fps and approximately 76 ms per frame, with approximately 20 ms for face detection plus landmarks, 45 ms for CNN inference, and 11 ms for homography and warping.

3. Emotion taxonomy and benchmark design

The 2025 benchmark introduces EmoNet-Face as a comprehensive benchmark suite for synthetic emotion recognition (Schuhmann et al., 26 May 2025). Its central design decision is a 40-category emotion taxonomy intended to capture finer details of human emotional experiences. The methodology starts from the 4th-ed. Handbook of Emotions (Lewis et al. 2016), OCR’d into approximately 946 pages. The text was split into 500-word blocks, and GPT-4 was used to extract candidate emotion nouns, yielding 170 unique terms. The taxonomy was then refined through three rounds of independent listing, deduplication, and expert consultation with psychologists, grounded in the Theory of Constructed Emotion (TCE), Barrett 2017.

The final taxonomy spans basic, social, cognitive, and bodily states, ranging from Amusement and Elation to Intoxication and Sexual Lust. The benchmark explicitly distinguishes pairs that are often conflated in affective computing. Embarrassment versus Shame follows Tangney and Fischer (1996): embarrassment arises from minor norm violations, is usually short-lived, and focuses on the social audience, whereas shame involves a negative global self-evaluation, often moral in nature, and is more intense and lasting. Other fine distinctions include Envy versus Jealousy, and Bitterness versus Sadness. This taxonomy broadens the emotional spectrum beyond coarse six- or seven-class schemes.

Dataset construction uses state-of-the-art text-to-image models, specifically Midjourney v6 and Flux-Dev/Pro, to generate three complementary, demographically balanced, full-face datasets with neutral backgrounds and strict visibility checks. Prompt templates enforced exact full-face, sharp close-ups, and neutral background. Demographic balancing was enforced per batch at 45% men, 45% women, 10% non-binary; ages 20–80 years in 10-year increments; and 14 regional backgrounds including “South Asian,” “Middle Eastern,” “Latinx,” “White/Caucasian,” and “Black/African.” Manual filtering removed images with occlusions, artifacts, or inappropriate content, and one image was removed in Binary.

Subset	Size	Labels
EmoNet-Face Big	203 201	automatically annotated
EmoNet-Face Binary	19 999	expert labels, binary
EmoNet-Face HQ	2 500	expert labels, continuous

The three subsets are differentiated by annotation granularity. EmoNet-Face Big contains approximately 203 k images and is automatically annotated by Gemini-2.5-Flash via a two-stage “hinting” strategy to ensure at least 5 k non-zero samples per emotion. EmoNet-Face Binary contains 19 999 images with multi-stage binary expert annotation. EmoNet-Face HQ contains 2 500 images with high-fidelity expert continuous annotations on a 0–7 scale. All three are released under CC-BY and are synthetic.

4. Annotation protocol and agreement structure

Annotation involved 13 expert annotators, each holding a PhD or Master’s in psychology, aged 18–44, from diverse ethnic and linguistic backgrounds (Schuhmann et al., 26 May 2025). All annotation followed detailed, standardized guidelines on a custom web UI. EmoNet-Face HQ uses 4 experts per image across 2 500 images, with 40 emotion intensities rated on a 0–7 scale and independent batch size of 250 images, with no group discussion during rating.

EmoNet-Face Binary uses a multi-stage “affirmative” pipeline. Batch 1 processes 19 999 candidate images: Annotator A positive, then reviewed by B, then C, and accepted only if all 3 agree. Batch 2 and Batch 3 repeat the procedure on positives from the previous batch with 4 annotators. Batch 4 is contrastive and performs 19 999 true-negative checks with 4 annotators. Agreement is reported as 82.83% for Batch 1, 75.11% for Batch 2, 82.37% for Batch 3, and 94.29% for Batch 4.

Inter-annotator agreement is quantified with weighted Cohen’s $C = 0.01$8 using quadratic weights and with Krippendorff’s $C = 0.01$9 at interval level. For EmoNet-Face HQ, mean pairwise $L2$0 and mean $L2$1 with 95% CI $L2$2; the highest $L2$3 is for Elation at 0.58 and the lowest for Interest at –0.08. For EmoNet-Face Binary, overall $L2$4, and the top five emotions by binary $L2$5 are Fatigue at 0.49, Pain at 0.45, Malevolence at 0.44, Teasing at 0.43, and Intoxication at 0.40.

These agreement values are lower than what might be expected for coarse facial-expression taxonomies. A plausible implication is that the benchmark is explicitly targeting nuanced or closely related states for which expert disagreement remains substantial even under standardized protocols. The reported methodology does not present this as annotation failure; rather, it quantifies the difficulty of the label space.

5. EmpathicInsight-Face and benchmark performance

EmpathicInsight-Face is the benchmark’s task model and is trained as two ensembles, Large and Small, of multi-task MLP heads on SIGLIP2-400M embeddings (Schuhmann et al., 26 May 2025). The backbone is the SIGLIP2-400M image embedder with 1 152-dimensional features. For each emotion, the Small head uses two hidden layers of 128 and 32 units with dropout 0.1 and approximately 151 k parameters per head. The Large head uses three hidden layers of 1 024, 512, and 256 units with dropout 0.2 and approximately 1.8 M parameters per head.

Training proceeds in two stages. First, the model is pre-trained on EmoNet-Face Big, where continuous labels are binarized for presence/absence, using binary cross-entropy loss,

$L2$6

Adam with learning rate $L2$7, ReduceLROnPlateau with factor 0.2 and patience 10, balanced sampling by downsampling the “0” bucket to match the “1” bucket, and 5 epochs. Second, the model is fine-tuned on EmoNet-Face Binary with the same loss and optimizer for 200 epochs, with a validation split of 5% per bucket.

Evaluation compares EmpathicInsight-Face against human experts and against general-purpose vision and vision-LLMs. Reported binary metrics per emotion are Accuracy, Precision, Recall, F1 score, Spearman’s $L2$8, and Cohen’s $L2$9. On mean pairwise $[0,1]$0 with human annotators, EmpathicInsight-Face Large reaches 0.1795, EmpathicInsight-Face Small reaches 0.1443, HumeFace reaches 0.1087, Gemini 2.5 Flash in zero-shot reaches 0.0007, and a random baseline reaches 0.0003. The Large model’s top-1 emotion by $[0,1]$1 is Astonishment at 0.498, while the Small model’s top-1 emotion is Elation at 0.659.

The human-versus-model comparison uses Mann–Whitney U tests with bootstrapped 95% confidence intervals. EmpathicInsight-Face Large versus human yields $[0,1]$2, $[0,1]$3, reported as non-significant. Proprietary VLMs versus human yield $[0,1]$4. Mean Spearman’s $[0,1]$5 across 40 emotions is approximately 0.56 (±0.04) for EmpathicInsight-Face Large, approximately 0.51 (±0.05) for EmpathicInsight-Face Small, 0.10–0.20 for Gemini variants with high variability, and approximately 0.00 for the random baseline. The paper characterizes this as human-expert-level performance on the benchmark.

The earlier real-time system uses a different evaluation regime and a different target: seven-way facial-expression categorization on FER2013 with best test accuracy 71.50%, together with deployment at 13 fps and homography-based emoji masking (Wang et al., 2020). It is therefore not directly comparable to the 40-category benchmark results, although both concern facial emotion inference.

6. Limitations, fairness, and comparative context

The real-time system documents several failure cases (Wang et al., 2020). Extreme yaw or pitch angles can confuse both detection and classification, leading to flickering emoji or mis-classification, often as “anger” or “sadness.” Very subtle expressions may be recognized as “neutral.” Low-light or very low face resolution below 80×80 px degrades HoG detection. Class imbalance causes under-performance on “disgust,” which has only approximately 50% recall in the qualitative limitations section and 52% recall in the reported per-class recall summary. In qualitative examples, the emoji are described as correctly following the user’s head pose and expression in nearly all frontal-face cases.

The 2020 paper also includes a comparison to prior “EmoNet”-style work. Duncan et al. (2016) used a custom VGG_S and simple superposition masking, with test accuracy 57% on their proprietary set and real-time speed of 2.5 fps. The reported pipeline achieves 71.5% on FER2013, 13 fps, and natural masking via homography. Proposed extensions include finer-grained emotions with an enlarged emoji library, automatic emoji selection via learned embedding or user preference model, temporal smoothing by majority-vote or RNN, concurrency or multithreading for pipelined face detection and CNN inference, integration of full-body pose or shoulder cues, and emoji animation driven by key-point deltas.

The benchmark suite foregrounds demographic balance and safety (Schuhmann et al., 26 May 2025). Equalized sampling across gender, age, and ethnicity is enforced at generation time, and the world-map visualization is reported to show that EmoNet-Face is far more globally diverse than typical web-scraped datasets. Manual screening removed images with stereotypes or harmful content, and one image was flagged and removed in EmoNet-Face Binary. Fairness findings report that no single demographic subgroup dominated any emotion category in EmoNet-Face HQ or Binary by more than 5%, and preliminary subgroup $[0,1]$6 analysis found no significant differences, with ANOVA $[0,1]$7, across ethnicity or gender in model performance. Future work is stated as rigorous intersectional fairness testing and mitigation.

A further interpretive issue concerns synthetic data. EmoNet-Face Big, Binary, and HQ are all synthetic and are designed around explicit, full-face expressions with neutral backgrounds. This directly addresses limitations attributed to existing datasets, including narrow emotional spectrum, occluded faces, and lack of demographic diversity. A plausible implication is that the benchmark emphasizes controlled facial visibility and label granularity over the uncontrolled variability typical of web-scraped imagery. That choice makes the suite particularly suitable for evaluating fine-grained emotion discrimination under controlled visual conditions, while the real-time system addresses low-latency deployment with coarse expression classes.