Occlu-FER: Occlusion-Robust FER Benchmark

• Occlu-FER is an occlusion-focused FER benchmark featuring naturally occurring occlusions to address the limitations of synthetic occlusion methods.
• It comprises 7,718 images split into training and validation sets, annotated across eight emotion categories with noticeable class imbalances.
• The benchmark leverages semantic segmentation, facial landmarks, and multi-scale fusion to enhance differentiation of expressions under occlusion.

Occlu-FER is an occlusion-focused facial expression recognition (FER) benchmark specifically constructed to facilitate the rigorous evaluation of FER robustness under real-world partial facial occlusion and extraneous face interference. Unlike conventional FER benchmarks, which mainly lack naturally occluded examples or employ unrealistic synthetic strategies (e.g., rectangular masks, random erasing), Occlu-FER is expressly curated to reflect authentic, semantically meaningful occlusion phenomena encountered in unconstrained visual environments (Zhai et al., 21 Jul 2025).

1. Dataset Design and Composition

Occlu-FER comprises 7,718 facial images, partitioned into a training set of 6,838 and a validation set of 880 samples. The dataset addresses a critical deficit in FER research: the scarcity of challenging, occlusion-rich imagery. Image sources are drawn from occluded samples in public "in-the-wild" datasets and further augmented with real-world facial photographs retrieved from the internet. The composition results in a dataset that prioritizes natural, semantic occlusions over artificially composited artifacts.

Class labeling adheres to an eight-category taxonomy:

Label	Anger	Disgust	Fear	Happy	Neutral	Sad	Surprise	Contempt
Code	AN	DI	FE	HA	NE	SA	SU	CO

The dataset is inherently imbalanced, with class frequencies ranging from 417 samples for Contempt to 1,273 for Surprise. Explicit subject-level metadata, including the number of unique identities, is not reported, nor are stratified distributions over occlusion subtypes.

2. Occlusion Characteristics

Occlu-FER targets two central occlusion phenomena: (1) partial facial occlusion caused by objects such as glasses, hands, and masks, and (2) extraneous face interference—i.e., images containing non-primary, background faces. The occlusions are naturally occurring, reflecting varied real-world imaging conditions. The dataset does not offer a formal taxonomy or quantitative inventory of occlusion types (such as mask, sunglasses, or hair), nor does it annotate occlusion location, facial region affected, or occlusion severity. All available annotations correspond strictly to facial expression class.

Occlusions are not synthetically generated for Occlu-FER; instead, images exhibiting naturally occurring occlusion events are collected directly, distinguishing the dataset from earlier FER benchmarks that rely on algorithmic occlusion imposition.

3. Annotation and Preprocessing Pipeline

The specific protocol for assigning expression class labels is not disclosed; possibilities such as manual expert annotation or label inheritance from source datasets are not addressed in the primary source. No metadata are offered assigning attributes to occlusions themselves (type, position, severity).

In terms of model-related preprocessing for Occlu-FER experiments, the reported pipeline makes use of:

• Pre-trained SegFace to generate dense semantic segmentation maps,
• Pre-trained MobileFaceNet for extracting facial landmarks (serving as sparse geometric priors),
• An IR50 backbone for global facial feature extraction.

Explicit details about standard image preprocessing steps (face alignment, resizing, normalization) are absent.

Given an input image $X_{in}$, the system computes: $$\mathbf{X}_{img}, \mathbf{X}_{seg}, \mathbf{X}_{lm} = G_{img}(\mathbf{X}_{in};\theta),\; G_{seg}(\mathbf{X}_{in}),\; G_{lm}(\mathbf{X}_{in})$$ where $G_{img}$, $G_{seg}$, and $G_{lm}$ denote the trainable backbone, segmentation generator, and landmark detector, respectively.

4. Evaluation Protocol and Benchmark Results

Occlu-FER is evaluated using overall prediction accuracy (%), as per standard classification assessment: $$\text{Overall Accuracy} = \frac{\text{Number of Correctly Classified Samples}}{\text{Total Number of Samples}}$$

Per-class accuracy and confusion matrices, though standard for FER, are not provided for this dataset. No formal subject-independent or cross-validation protocol is described; evaluation reports results on the validation split exclusively.

Comparative benchmarking against strong natural, multi-modal, and occlusion-specialized FER models demonstrates the dataset's difficulty:

Method	Occlu-FER Accuracy (%)
FDRL	65.86
Face2Exp	65.79
EAC	66.36
POSTER	66.82
POSTER V2	66.09
CLEF	66.74
CLIPER	67.61
EfficientFace	65.96
ORSANet	68.07

ORSANet achieves the highest recorded accuracy of 68.07%. Additional ablation experiments confirm the critical contribution of dense semantic segmentation prior, multi-scale cross-interaction, and DARELoss for handling ambiguities between confusable classes. No statistical significance measures (e.g., confidence intervals, p-values) are reported.

5. Failure Modes and Analysis

The Occlu-FER benchmark exposes several characteristic failure modes for current FER systems:

• Activation on occluded or non-discriminative facial regions, frequently resulting in misclassification,
• Susceptibility to extraneous/background faces, leading to ambiguous or erroneous region-of-interest selection,
• Pronounced sensitivity to ambiguous classes with subtle inter-class boundaries (addressed by ORSANet's DARELoss),
• Performance degradation on low-quality or anomalous images, likely due to limitations in semantic segmentation reliability.

No per-category breakdown of errors is provided; however, pronounced class imbalance suggests that underrepresented emotions (e.g., Contempt) could be particularly challenging, though this is not quantified.

6. Implications for Occlusion-Robust FER Methodology

Empirical results on Occlu-FER underscore the inadequacy of traditional strategies such as rectangular mask augmentation for occlusion-robust FER training and suggest the superiority of incorporating semantically informed priors. The ablation studies vindicate the importance of:

• Semantic segmentation maps (dense prior) to distinguish between occluder and expressive facial regions,
• Facial landmarks (sparse prior) to suppress identity and nuisance factors and focus on expression-relevant geometry,
• Multi-scale fusion architectures to accommodate the fine-grained nature of FER cues,
• Objective functions (e.g., DARELoss) that explicitly mitigate inter-class confusion in imbalanced, ambiguous settings.

The dataset further demonstrates that even state-of-the-art methods attain only mid- to high-60s accuracy, establishing real-world occlusion as a primary open problem for FER research.

7. Limitations and Prospective Directions

Occlu-FER provides a benchmark for evaluating models under authentic occlusion settings, but has several limitations:

• Lack of subject identity information, occlusion label granularity, and occlusion region annotation,
• Pronounced class imbalance with no reported data-level balancing strategy,
• No details regarding the labeling or image quality control process,
• Absence of statistical significance analysis in model comparisons.

Recommended advancements include the adoption of more capable semantic segmentation networks, explicit handling of anomalous facial inputs, and systematic evaluation protocols incorporating both real and synthetic occlusions.

Occlu-FER thus serves as a challenging, purpose-built resource to catalyze advances in FER, especially in designing architectures and loss functions responsive to nuanced, semantically complex occlusion phenomena (Zhai et al., 21 Jul 2025).