ORSANet: Occlusion-Robust Facial Expression Recognition

• ORSANet is an occlusion-robust facial expression recognition architecture that integrates dense semantic segmentation and sparse facial landmarks.
• It employs a multi-scale cross-modal fusion with a confidence-adaptive adversarial loss to mitigate challenges from occlusion and dataset biases.
• The system, evaluated on benchmarks like Occlu-FER, achieves state-of-the-art performance even under significant occlusion conditions.

ORSANet (Occlusion-Robust Semantic-Aware Network) is a facial expression recognition (FER) architecture introduced to address the persistent challenges of real-world facial occlusion and dataset biases, including those arising from identity, gender, and age. ORSANet is notable for its two-prior semantic guidance (dense semantic segmentation, sparse geometric landmarks), its multi-scale cross-modal fusion, and a confidence-adaptive adversarial loss. The supporting dataset, Occlu-FER, is the first released benchmark specifically constructed for occlusion-oriented FER. ORSANet achieves state-of-the-art (SOTA) performance on all major FER and occlusion-aware benchmarks (Zhai et al., 21 Jul 2025).

1. Architectural Foundations and Motivation

Conventional FER systems exhibit degraded robustness under partial occlusion (e.g., glasses, masks, hands, non-primary faces) and are prone to attribute leakage from identity, gender, age, or illumination. ORSANet addresses these problems by incorporating:

• Dense semantic priors: pixel-wise facial segmentation maps, capturing semantic facial regions, and
• Sparse geometric priors: facial landmarks, encoding structural geometry less sensitive to appearance. Input images are processed by a standard feature extraction backbone, and both priors are generated by frozen pretrained models. The pipeline fuses these modalities for an occlusion-robust semantic representation (Zhai et al., 21 Jul 2025).

Latent-Variable Perspective

ORSANet is motivated by an explicit latent-variable formalism: $$p(Y \mid X_N) = \int p\bigl(Y \mid X_{N2S}, X_{N2L}, X_N\bigr) \cdot p(X_{N2S}, X_{N2L} \mid X_N)\, dX_{N2G}$$ where $X_{N2S}$ and $X_{N2L}$ are semantic and geometric priors, respectively.

2. Multi-modal Semantic Guidance

2.1 Dense Semantic Prior: Facial Segmentation

Semantic segmentation maps, produced by a fixed SegFace model, are used as external dense priors. They provide spatially-aligned, pixel-level semantic labels to suppress occlusion noise and delineate meaningful facial structures. ORSANet employs a two-stage SPADE (Spatially-Adaptive Normalization) mechanism via the Spatial-Semantic Guidance Module (SSGM). The first SPADE aligns global structure, and the second further extracts fine-grained semantic cues through coarse-to-fine enhancement: $$\begin{aligned} \mathbf{X}'_{seg} &= \mathrm{ReLU}(\mathrm{Conv}(\mathbf{X}_{seg}))\ \mathbf{X}'_{img} &= \mathrm{Norm}(\mathbf{X}_{img})\cdot \mathrm{Conv}_\gamma(\mathbf{X}'_{seg}) + \mathrm{Conv}_\beta(\mathbf{X}'_{seg}) \end{aligned}$$

2.2 Sparse Geometric Prior: Landmarks

Facial landmarks extracted by a MobileFaceNet-based detector serve as sparse geometric priors. This modality is less sensitive to photometric and demographic nuisance factors. Landmarks guide attention to canonical spatial configurations relevant for expression and are reintegrated into downstream fusion blocks to suppress nuisance variation (Zhai et al., 21 Jul 2025).

3. Multi-scale Cross-interaction and Representation Fusion

The Multi-scale Cross-interaction Module (MCM) adaptively fuses semantics-enhanced features and landmark geometry. This is implemented as stacked multi-level Cross-Fusion Blocks (CFB), acting across multiple spatial scales. Each block involves:

• Linear projections: $\mathbf{Q}_{lm}$, $\mathbf{K}_{img}$, $\mathbf{V}_{img}$ for landmark (query), image features (key, value)
• Cross-attention:

$$\mathbf{F}_{att} = \mathrm{Softmax}\left( \frac{\mathbf{Q}_{lm} \cdot \mathbf{K}_{img}^T}{\sqrt{\mathbf{d}_i}} \right) \mathbf{V}_{img}$$

• Residual fusion and MLP: $$\mathbf{X}_{fuse} = \mathrm{MLP}\bigl( \mathrm{Norm}( \hat{\mathbf{X}}_{img} + \mathbf{F}_{att} ) \bigr)$$
• Adaptive landmark reintegration: $$\mathbf{X}_{out} = \mathrm{Conv} \left( \mathrm{Concat}( \mathbf{X}_{fuse} + s \cdot \mathbf{X}_{lm}) \right)$$ The adaptive factor $X_{N2S}$0 functions as a gate controlling the contribution of landmark information. Its parameterization is not specified in the paper, beyond being learnable (Zhai et al., 21 Jul 2025).

Multi-scale stacking enables hierarchical, progressive disentangling of expression-relevant features from nuisance and occlusion, without resorting to a fixed attention scope.

4. Confidence-Adaptive Learning: DARELoss

The Dynamic Adversarial Repulsion Enhancement Loss (DARELoss) enhances class boundary separation, focusing especially on ambiguous negatives. For a sample, the most competitive negative class is amplified according to the model's lack of target confidence: $X_{N2S}$1 DARELoss is then: $X_{N2S}$2 where $X_{N2S}$3 is the true-class logit, $X_{N2S}$4 the hardest negative, and $X_{N2S}$5 the predicted probability for the target. The final training loss combines cross-entropy with DARELoss: $X_{N2S}$6 The approach does not use explicit class-pair dynamic margins, but adapts the hardest negative's logit in a confidence-aware manner (Zhai et al., 21 Jul 2025).

5. Occlu-FER: Dedicated Occlusion FER Benchmark

Occlu-FER is introduced as the first dataset targeting robustness under real-world, semantics-driven occlusion and extraneous-face interference. Sources include:

1. Occluded samples from in-the-wild FER datasets
2. Internet-derived real facial images with natural occlusions

Occlusion types are not fine-categorized; the emphasis is on realistic coverage (glasses, hands, masks, non-primary faces). Eight emotion classes are annotated, but detailed annotation protocol is unreleased.

	Angry	Disgust	Fear	Happy	Neutral	Sad	Surprise	Contempt	Total
Train	669	533	927	1130	1114	960	1125	380	6838
Validation	86	69	136	133	144	127	148	37	880
Total	755	602	1063	1263	1258	1087	1273	417	7718

Occlu-FER is positioned as a new standard for benchmarking occlusion-aware FER (Zhai et al., 21 Jul 2025).

6. Empirical Results and Ablation Analysis

ORSANet demonstrates consistent SOTA performance on mainstream FER datasets and under increasing levels of both synthetic and real occlusion.

• RAF-DB: 92.28% (vs. POSTER V2 92.21%, CLIPER 91.61%)
• AffectNet (7-classes): 66.69% (vs. CLIPER 66.29%, POSTER V2 66.20%)
• Occlu-FER: 68.07% (vs. CLIPER 67.61%, POSTER 66.82%)

Performance under artificial occlusion (on RAF-DB) degrades minimally relative to baselines; ORSANet's accuracy at 30% occlusion is 84.02%, besting the next runner-up by 1.49 percentage points.

Ablation shows all architectural components contribute to robustness. Removing segmentation priors, cross-interaction, multi-scale fusion, landmark reintegration, or DARELoss individually each leads to a measurable drop, particularly under occlusion (Zhai et al., 21 Jul 2025).

7. Implementation and Computational Aspects

All segmentation and landmark models are frozen (SegFace, MobileFaceNet). The FER backbone is IR50. Training uses Adam with learning rate $X_{N2S}$7, batch 20, 400 epochs. Model complexity is 60.2M parameters and 6.9G FLOPs, below typical SOTA models such as POSTER, while outperforming them in robustness (Zhai et al., 21 Jul 2025).

8. Summary and Significance

ORSANet is a semantically-informed, occlusion-robust FER system integrating dense and sparse priors via targeted architectural modules and a confidence-weighted adversarial loss. Its empirical dominance is established on both mainstream FER benchmarks and dedicated occlusion scenarios, including the release of Occlu-FER as a milestone dataset. Both methodological and empirical contributions are positioned to guide future research in occlusion-insensitive affective computing (Zhai et al., 21 Jul 2025).