Residual Masking Network for FER

• Residual Masking Network (RMN) is a facial expression recognition architecture that combines residual learning with a learned masking mechanism to focus on key facial areas.
• It integrates four Residual Masking Blocks into a ResNet-34 backbone, using a U-Net-like subnetwork to generate attention maps that refine intermediate feature maps.
• RMN achieves state-of-the-art results on FER2013 and robust performance on the VEMO dataset, demonstrating its effectiveness in handling noisy, imbalanced data.

Searching arXiv for the specified paper and closely related work to ground the article in current literature. Residual Masking Network (RMN) is a facial expression recognition (FER) architecture that combines a ResNet-style residual classifier with a learned masking or attention mechanism implemented by a small U-Net-like segmentation network. Its central purpose is to make a convolutional network focus more strongly on facial regions that carry emotion cues—especially the eyes, eyebrows, nose, and mouth—while suppressing less useful areas such as hair, background, or jawline. In the formulation reported for FER, the model reuses ResNet-34 as the backbone, inserts four Residual Masking Blocks into the feature hierarchy, and refines intermediate feature maps by a learned spatial mask, achieving state-of-the-art accuracy on FER2013 and strong results on the private VEMO dataset (Pham et al., 6 Mar 2026).

1. Conceptual basis and motivation

The motivation for RMN begins from the observation that facial expressions are not uniformly distributed across the face. Most discriminative information comes from a few localized regions, whereas standard CNNs treat the whole image more uniformly. The architecture was proposed in the context of automatic FER, a task of interest because of its applications in human-computer interaction (Pham et al., 6 Mar 2026).

A central design choice is the rejection of explicit landmark dependence. In the wild, landmark-based methods often fail because facial landmark detection is unstable under occlusion, pose variation, and illumination changes. RMN therefore replaces explicit landmark localization with a learned attention mechanism. The paper’s intuition is that a localization or segmentation network can refine feature maps by estimating where the important regions are, in a manner analogous to how U-Net produces a segmentation mask in medical imaging (Pham et al., 6 Mar 2026).

This makes RMN a residual-attention architecture in which masking is not an auxiliary visualization device but a trainable computational pathway. The model does not merely classify from globally pooled deep features; it actively estimates an importance map over intermediate activations and uses that map to amplify expression-relevant structure. The authors describe this as a novel masking idea to boost CNN performance in the facial expression task (Pham et al., 6 Mar 2026).

2. Architectural organization

RMN is built from four Residual Masking Blocks. Each block combines a Residual Layer for feature extraction or transformation with a Masking Block for generating an attention mask over those features. The full network operates on an input image of $224 \times 224$, then applies a $7 \times 7$ convolution with stride 2 to obtain $64 \times 112 \times 112$, followed by a $3 \times 3$ max-pooling layer with stride 2 to obtain $64 \times 56 \times 56$. The four Residual Masking Blocks then produce the following stage outputs: $64 \times 56 \times 56$, $128 \times 28 \times 28$, $256 \times 14 \times 14$, and $512 \times 7 \times 7$. Average pooling and a fully connected layer with softmax yield predictions over 7 classes: anger, disgust, fear, happiness, sadness, surprise, and neutral (Pham et al., 6 Mar 2026).

The backbone is explicitly ResNet-34. The masking idea is embedded into that residual hierarchy rather than appended only at the classifier head. This placement is significant because it lets attention act at multiple resolutions. Since each residual stage operates at a different spatial scale, the masking subnetwork is adapted accordingly. A plausible implication is that the architecture attempts to preserve coarse semantic localization at deep stages while still retaining sufficient spatial resolution to isolate facial parts.

The architecture is therefore not a generic U-Net, not a plain residual network, and not a landmark-conditioned model. It is a residual classifier in which each major stage includes a learned localization module that refines the stage’s own feature map. The resulting design can be read as a staged interaction between representation learning and spatial reweighting.

3. Masking formulation and Residual Masking Block

The core input to a masking stage is a feature map

$F \in \mathbb{R}^{C \times W \times H}.$

The residual layer transforms it into a coarse feature map

$7 \times 7$0

The Masking Block then produces an attention map of the same spatial size,

$7 \times 7$1

where $7 \times 7$2 has values in $7 \times 7$3. The block output is defined as

$7 \times 7$4

with $7 \times 7$5 denoting element-wise multiplication (Pham et al., 6 Mar 2026).

This equation is the central mechanism of RMN. The residual path keeps the original transformed features, the mask path highlights important regions, and the final output adds the weighted features back to the original ones. If a region receives a high mask value, its activations are amplified; if a region is irrelevant, the mask suppresses its contribution. The residual addition is especially important because the mask is learned and may be imperfect. Rather than replacing the underlying representation, the mask modulates it without discarding the base residual feature map.

The Masking Block itself is described as the main novelty. It is a small variant of U-Net with a contracting path, an expansive path, and skip-like refinement behavior that produces a same-sized spatial mask. Its task is not semantic segmentation of objects but production of an importance map for facial feature refinement. The authors argue that this deeper localization network can more accurately identify the “important score” of feature maps than a shallow mask branch. The design is similar in spirit to residual attention learning, but the mask branch is specifically built as a U-Net-like localization module (Pham et al., 6 Mar 2026).

4. Optimization procedure, data, and evaluation protocol

RMN is trained as a standard 7-class classifier with softmax output. The paper does not introduce a special custom loss for the mask itself; the classification objective is the usual cross-entropy training setup implied by the softmax layer. The reported evaluation metric is accuracy,

$7 \times 7$6

Training uses images resized to $7 \times 7$7, converted to RGB, with horizontal flip and rotation in $7 \times 7$8 as augmentation. The batch size is 48, the maximum number of epochs is 50, early stopping is triggered if validation accuracy does not improve for 8 steps, the initial learning rate is $7 \times 7$9, the learning rate is reduced by a factor of 10 if validation accuracy does not improve for 2 consecutive epochs, momentum is 0.9, and weight decay is 0.001. Implementation was done in PyTorch on a GTX 1080 Ti (Pham et al., 6 Mar 2026).

Two datasets are described. FER2013 is a standard public facial expression dataset with 35,887 grayscale images of size $64 \times 112 \times 112$0, partitioned into 28,709 training, 3,859 validation, and 3,589 test images over 7 classes. The paper notes that FER2013 is noisy and imbalanced, with many more happy examples than disgust examples, and that it includes invalid or improperly cropped faces. VEMO is a private Vietnam Emotion dataset created by the authors, containing 36,470 multi-resolution images: 6,470 color images collected from YouTube, Google Image, and Flickr, and 30,000 images labeled by professional annotators, again over the same 7 expression categories. The authors state that VEMO will be released in the future (Pham et al., 6 Mar 2026).

These details clarify that RMN is evaluated under conditions typical of in-the-wild FER: class imbalance, annotation noise, and variable image quality. This is relevant to the masking premise, because a learned attention mechanism is intended to suppress nuisance regions and concentrate capacity on subtle local expression cues.

5. Empirical results and diagnostic analyses

On FER2013, RMN is compared with VGG19, EfficientNet-B2, GoogLeNet, ResNet34, ResAttNet56, Inception-v3, BAM-ResNet50, DenseNet121, ResNet152, and CBAM-ResNet50. The reported accuracy is 74.14% without ensemble and 76.82% for an RMN ensemble. The non-ensemble result is claimed to outperform previous single-model methods, including CNN-SIFT at 73.4%, while the ensemble result is reported as state of the art among ensemble methods on FER2013. On VEMO, the compared models are ResAttNet56 at 60.82%, ResNet18 at 63.94%, ResNet34 at 64.84%, and RMN at 65.94% (Pham et al., 6 Mar 2026).

The ensemble experiment uses a simple unweighted average ensemble of 7 CNNs, including RMN, selected on the basis of validation accuracy. The ensemble improves the FER2013 result to 76.82%, indicating that RMN is complementary to other CNNs in ensemble settings. This suggests that the masking mechanism may provide feature emphasis that is not redundant with standard architectural diversity.

Although the paper does not provide a formal ablation table isolating every component, it reports several analyses supporting the masking idea. RMN outperforms ResNet34, DenseNet121, CBAM-ResNet50, BAM-ResNet50, and ResAttNet56, which the authors interpret as evidence that the improvement is not merely due to deeper residual learning. Feature maps before and after the third Masking Block show stronger focus on salient facial parts after masking, and Grad-CAM visualizations indicate that the highest activations correspond to expression-relevant areas such as the eyes, nose, and mouth. Confusion matrices show that RMN performs well despite class imbalance, with easy emotions such as happiness and surprise scoring highest, while harder emotions such as fear and sadness remain more difficult, consistent with human difficulty. The error analysis attributes many mistakes to ambiguous expressions or noisy labels rather than obviously wrong model behavior (Pham et al., 6 Mar 2026).

6. Interpretation, scope, and relation to adjacent residual-masking models

A common misunderstanding is to treat RMN as a landmark-driven FER system or as a semantic segmentation network repurposed for classification. The paper argues otherwise. RMN does not depend on explicit landmarks, and its U-Net-like component is not used for semantic segmentation of objects; it is used to produce an importance map aligned with the intermediate feature map. Another possible misunderstanding is to regard the mask as a hard filter. In RMN, the mask is soft, bounded in $64 \times 112 \times 112$1, and incorporated through residual addition, so the model amplifies salient regions without discarding the base representation (Pham et al., 6 Mar 2026).

The term “residual masking” also appears in adjacent contexts, but with different operational meanings. In speech modeling, RT-MAE is described as a residual masking or residual reconstruction network in which explicit supervised attributes are complemented by residual trainable tokens inside a masked autoencoder; the residual pathway captures variability such as timbre, emotion, micro-prosody, and noise, and specific residual factors can be activated or deactivated at inference (Sadok et al., 27 Jan 2026). In image denoising, SRMNet is not a literal mask-prediction network, but its selective kernel feature fusion is characterized as soft residual-attention or soft residual masking, because residual and mainstream feature streams are selectively reweighted before aggregation (Fan et al., 2022).

This broader usage suggests that “residual masking” names a family of architectures in which a residual pathway is not merely added but selectively modulated. Within that family, RMN occupies the explicit spatial-mask end of the spectrum: it predicts same-sized attention maps over intermediate features, ties them to a U-Net-like localization module, and applies them to FER in a stagewise residual hierarchy. Its distinctive contribution is therefore not only the use of attention, but the integration of localization-style masking into a ResNet-34 classifier for facial expression recognition (Pham et al., 6 Mar 2026).