Architectural Ensemble for FER

• Architectural Ensemble (AE) is a multi-architectural feature fusion strategy that leverages diverse CNN backbones to capture complementary facial representations for FER.
• It integrates heterogeneous models like VGG16, DenseNet121, SE-ResNet34, and SE-ResNeXt50 to mitigate individual biases and improve robustness under varied conditions.
• The approach employs a non-linear meta-classifier on concatenated feature vectors, yielding enhanced accuracy on datasets such as FER2013 and RAF-DB.

Searching arXiv for the specified paper and closely related FER ensemble work. Architectural Ensemble (AE), in the sense used in "Recognizing Facial Expressions in the Wild using Multi-Architectural Representations based Ensemble Learning with Distillation" (Momin et al., 2021), is a feature-level ensemble design for facial expression recognition (FER) in the wild that combines heterogeneous convolutional backbones to learn complementary facial representations and fuses them through a meta-classifier. The formulation is motivated by the observation that FER is affected by varying pose, illumination, occlusion, image quality, and label noise, and that single CNN backbones can be biased toward their own inductive biases. In this setting, AE is implemented as EmoXNet, while EmoXNetLite is a distilled lightweight student trained from the ensemble using label-smoothen soft labels; the reported test accuracies are 85.07% on FER2013 with FER+ annotations and 86.25% on RAF-DB for EmoXNet, and 82.07% and 81.78% respectively for EmoXNetLite (Momin et al., 2021).

1. Definition and rationale

Architectural Ensemble denotes an ensemble strategy based on architectural diversity rather than repeated training of a single backbone family. In the reported FER system, the objective is to learn multi-architectural, complementary facial representations and to fuse them through a non-linear head. The stated rationale is that diverse architectures emphasize different spatial and channel-wise cues, including edges, landmarks, and region saliency, and that this diversity can reduce variance, average out individual model errors, and improve robustness and generalization across wild conditions and mislabeled examples (Momin et al., 2021).

The core problem setting is FER in the wild. The paper identifies multiple sources of uncertainty: varying pose, illumination, occlusion, image quality, and label noise. Within that setting, AE is presented as a remedy to blind spots associated with individual inductive biases. VGG-style models emphasize large receptive fields and a plain topology; DenseNet emphasizes dense feature reuse; SE-enhanced residual families emphasize channel-wise attention. The ensemble is therefore not merely a computational aggregation device, but a representational mechanism intended to capture complementary facial evidence that no single architecture captures consistently under distribution shift.

A common misconception is to treat such an ensemble as equivalent to averaging predictions from several independently trained classifiers. That is not the construction used here. The reported system is explicitly feature-level, not probability-level or logit-level. This distinction matters because the fusion head is trained to exploit cross-architecture interactions rather than to smooth already formed decisions.

2. Constituents and architectural diversity

EmoXNet integrates four ImageNet-pretrained CNN backbones: VGG16, DenseNet121, SE-ResNet34, and SE-ResNeXt50 (Momin et al., 2021). In the paper’s terminology, these correspond to a “sequential block,” a “dense block,” an “SE-ResNet block,” and an “SE-ResNeXt block.” Each backbone is adapted for FER by removing the original classifier, retaining the convolutional trunk, and applying Global Average Pooling (GAP) to the last convolutional feature maps.

The diversity mechanism is explicit. It mixes:

• Topology: sequential (VGG), dense (DenseNet), residual (ResNet), and aggregated residual (ResNeXt).
• Attention: SE and non-SE variants.
• Depth, width, and cardinality: different layer counts and branching patterns.

These differences are said to translate into complementary receptive fields and feature emphases. The paper also evaluates additional single backbones—ResNet34, ResNeXt50, and EfficientNet-B0—but the ensemble itself uses only the four architectures listed above.

Backbone	Role in AE
VGG16	Sequential block; large receptive fields; hierarchical abstraction
DenseNet121	Dense block; feature reuse; compact multi-scale descriptors
SE-ResNet34	Residual block with Squeeze-and-Excitation channel attention
SE-ResNeXt50	Split–transform–merge topology with SE attention

The paper does not report parameter counts or FLOPs for the implemented ensemble or backbones. It notes orientation values from canonical implementations, but explicitly states that exact counts for the authors’ implementations are not provided. A plausible implication is that the work prioritizes representational complementarity and empirical accuracy over a formal efficiency accounting for the teacher ensemble.

3. Fusion formulation and learning objective

The AE is implemented as feature-level fusion followed by a learned non-linear meta-classifier (Momin et al., 2021). Let $M=4$ denote the number of backbones. For an input image $x$, each backbone $i$ produces a convolutional feature map $F_i(x)$; GAP yields a vector $h_i(x) \in \mathbb{R}^{d_i}$. These vectors are concatenated into a joint representation,

$h(x) = [h_1(x); h_2(x); \ldots; h_M(x)].$

The concatenated representation is then passed to a multi-layer perceptron (MLP) meta-classifier $g(\cdot)$ to produce logits $z \in \mathbb{R}^{K}$ for $K=7$ classes:

$z = g(h(x)), \qquad p = \mathrm{softmax}(z).$

The class probabilities are

$x$0

and the cross-entropy loss is

$x$1

where $x$2 is the target distribution, either one-hot or label-smoothed.

The paper is explicit that this is not probability fusion, logit fusion, or voting over independently trained heads. Those alternatives are mentioned only for completeness and are not used. The significance of the chosen formulation is that it allows the MLP to learn non-linear interactions across architecture-specific descriptors. This suggests that AE is intended to operate as a second-stage representation learner rather than as a simple consensus mechanism.

4. Data, preprocessing, regularization, and distillation

The reported experiments use two datasets: FER2013 with FER+ annotations and RAF-DB (single-label subset) (Momin et al., 2021). FER2013 with FER+ annotations contains 35,249 grayscale $x$3 images in 7 classes: angry, disgust, fear, happy, sad, surprise, and neutral. RAF-DB contains approximately 15,339 RGB images originally sized $x$4, resized to $x$5 for consistency.

Preprocessing is minimal in the sense that no face detection or alignment pipeline is described. Inputs are resized crops from the datasets, and the figure indicates a rescaling step prior to GAP in each feature extraction block. Training augmentation includes horizontal flip, vertical flip, rotation, width and height shifts, shear, and zoom, with fill_mode set to nearest. The stated purpose is robustness to geometric perturbations and mild photometric variation. At inference, Test-Time Augmentation (TTA) uses 10 augmentation steps and averages predictions across steps; improved accuracy is reported, although no exact delta is given.

Regularization includes label smoothing with

$x$6

using $x$7 and $x$8. The cross-entropy is computed against $x$9. The paper states that smooth targets “relax confidence on the labels and hold more information,” and associates this with improved generalization and better transfer to the lightweight student.

Knowledge distillation produces EmoXNetLite from EmoXNet. The teacher is the ensemble with the MLP meta-classifier; the student is described only as a lightweight neural network, and the exact backbone is not concretely named in the text. Although the paper references Hinton et al.’s KD and the role of a temperature $i$0, it does not report a temperature-scaled KL divergence loss, nor does it specify $i$1 or a blending coefficient $i$2. Instead, the implemented distillation uses categorical cross-entropy with label smoothing and “label-smoothen soft labels.” This point is important because it distinguishes the reported method from canonical KD formulations that rely explicitly on $i$3.

5. Optimization protocol and empirical results

The training setup uses Nesterov SGD with momentum $i$4, initial learning rate $i$5, weight decay $i$6, batch size $i$7, and $i$8 epochs; early stopping and learning rate scheduler callbacks are used to prevent overfitting and aid convergence (Momin et al., 2021). Adam and AMSGrad were tried but are reported to underperform SGD in generalization and convergence. Calibration methods beyond label smoothing are not described.

Single-backbone baselines and ensemble results are reported on both datasets.

Model	FER2013 (FER+)	RAF-DB
VGG16	79.68%	77.38%
DenseNet121	82.32%	78.75%
EfficientNet-B0	78.78%	70.21%
ResNet34	81.16%	76.56%
SE-ResNet34	80.31%	74.58%
ResNeXt50	81.84%	77.41%
SE-ResNeXt50	82.77%	76.99%
EmoXNet	85.07%	86.25%
EmoXNetLite	82.07%	81.78%

The ensemble consistently outperforms all single architectures. On FER2013, the best single backbone is SE-ResNeXt50 at 82.77%, whereas the ensemble reaches 85.07%. On RAF-DB, the ensemble reaches 86.25%, substantially above the reported single-backbone results. Within the paper’s comparisons, this is presented as validation of complementary representations.

Class-wise F1 scores further characterize the error profile. On FER2013, EmoXNet versus EmoXNetLite yields: Angry 0.79 / 0.79, Disgust 0.58 / 0.67, Fear 0.60 / 0.59, Happy 0.93 / 0.92, Sad 0.68 / 0.68, Surprise 0.88 / 0.88, Neutral 0.87 / 0.89. On RAF-DB, the corresponding values are: Angry 0.72 / 0.70, Disgust 0.47 / 0.50, Fear 0.58 / 0.70, Happy 0.92 / 0.90, Sad 0.78 / 0.75, Surprise 0.80 / 0.81, Neutral 0.79 / 0.81. The reported interpretation is that fear and disgust are hardest, likely because of lower support and possibly higher label noise, while happy, surprise, and neutral are easier due to more distinctive visual cues and higher support.

Qualitative class activation maps indicate that the models attend to edges and key facial regions, and that classes with fewer samples show more distorted activation patterns. This is discussed as a manifestation of class imbalance rather than as a failure specific to a given backbone.

6. Ablations, practical guidance, and limitations

The paper reports several analyses relevant to AE as a methodological construct (Momin et al., 2021). The most direct ablation concerns architecture inclusion: although SE-ResNeXt50 is the strongest single model on FER2013, combining it with VGG16, DenseNet121, and SE-ResNet34 and then learning fusion via the MLP yields the main gain. No ablations are reported for probability fusion, logit fusion, or voting, because the implemented system uses only feature concatenation with an MLP head. Likewise, label smoothing is fixed at $i$9; robustness gains are noted, but no sweep over $F_i(x)$0 is provided.

The paper presents practical guidance for constructing an AE for FER. The recommended principle is to select diverse backbones to maximize representational complementarity, use feature-level fusion via GAP and concatenation, regularize with label smoothing, employ early stopping and learning-rate scheduling, and use strong geometric augmentation together with 10-step TTA. It also warns about class imbalance, especially for fear and disgust, and mislabeled examples. Reweighting or label cleaning is suggested only conditionally—“if available”—rather than reported as part of the implementation.

Efficiency claims are deliberately qualified. EmoXNetLite is proposed for real-time systems and resource-constrained settings because replacing the ensemble with a single lightweight student reduces latency. However, exact parameter counts, FLOPs, FPS or latency measurements, and hardware details are not reported. The real-time claim is therefore qualitative and relative to the ensemble teacher.

The reported robustness evidence is also qualified. Attention and saliency visualizations suggest focus on salient facial regions despite pose and illumination variability, but explicit occlusion or pose stress tests are not reported. Similarly, the claim that the ensemble and student transfer well across FER2013 with FER+ annotations and RAF-DB is grounded in performance across those two datasets; broader domain generalization beyond those settings is not established in the text.

The acronym “AE” is not unique across arXiv usage. In a different domain, "DMS, AE, DAA: methods and applications of adaptive time series model selection, ensemble, and financial evaluation" uses AE to mean Adaptive Ensemble, a rolling, loss-driven convex combination scheme for time-series forecasting under regime shifts (Yang et al., 2021). That usage is algorithmically unrelated to the FER method discussed here. In the present sense, Architectural Ensemble refers specifically to multi-architectural feature fusion for facial expression recognition, not to adaptive online forecast weighting in financial time series.