Hy-Facial: Hybrid Facial Expression Recognition

• Hy-Facial is a hybrid facial expression recognition system that fuses deep (VGG19) and local (SIFT, ORB) features to capture both global structure and fine-grained details.
• The framework uses a class-wise K-means selection mechanism and evaluates multiple dimensionality reduction methods, with UMAP emerging as the best for preserving manifold geometry.
• Demonstrating an accuracy of 83.3% on FER2013Plus, Hy-Facial outperforms traditional pipelines by effectively integrating heterogeneous feature representations before classification.

Hy-Facial most specifically denotes a hybrid feature extraction framework for facial expression recognition that integrates deep learning and traditional image processing, then treats dimensionality reduction as a central component of the recognition pipeline rather than a terminal compression step. In its formal instantiation, the method fuses features from VGG19, SIFT, and ORB, applies a class-wise K-means selection mechanism, and evaluates multiple dimensionality reduction operators before classification; the best reported configuration is VGG19 + SIFT + ORB, followed by K-means, UMAP, and Random Forest, with a test accuracy of 83.3% on FER2013Plus (Li et al., 30 Sep 2025). In adjacent facial-computing literature, the label has also been used in broader descriptive senses for hyperspectral facial skin analysis, high-fidelity face swapping, high-fidelity 3D reconstruction, at-home facial capture, and remote selfie-based skin assessment (Ng et al., 2023, Wang et al., 2021, Chai et al., 2023, Han et al., 2023, Soh et al., 8 Sep 2025).

1. Concept and scope

Hy-Facial is formulated as a multi-class facial expression recognition system over face images, with the input-output mapping

$$f: \mathbb{R}^{H \times W \times 3} \rightarrow \{0,1,\dots,C-1\}, \qquad \hat{y} = f(I).$$

The target label space in FER-Plus comprises eight emotion categories: angry, contempt, disgust, fear, happy, sad, surprise, and neutral (Li et al., 30 Sep 2025).

The framework is motivated by three coupled difficulties. First, facial imagery is high-dimensional even at modest spatial resolution. Second, expression classes occupy nonlinear manifolds shaped by identity, illumination, pose, and occlusion. Third, feature fusion amplifies redundancy and noise. The paper contrasts deep-only and handcrafted-only pipelines: VGG19 captures hierarchical and global semantics but can miss fine local deformations, whereas SIFT and ORB emphasize local geometry and texture but do not by themselves model the nonlinear variability of unconstrained facial expression recognition (Li et al., 30 Sep 2025).

Accordingly, Hy-Facial is not only a feature-fusion system. Its central thesis is that discriminative performance depends on jointly designing feature extraction, representative feature selection, and manifold-preserving dimensionality reduction. This places it closer to a hybrid statistical-learning pipeline than to an end-to-end trainable FER network.

2. Problem formulation and hybrid pipeline

The full Hy-Facial pipeline is described as a four-stage deterministic pipeline: $$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$

$$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$

This decomposition makes the framework analytically transparent: the feature extractor defines the ambient representation, K-means reduces within-class redundancy, dimensionality reduction organizes the manifold geometry, and a comparatively simple classifier performs the final decision rule (Li et al., 30 Sep 2025).

The feature extractor combines three sources. VGG19 provides a global deep representation. Its layer-wise computation is written as

$$Z^l = W^l A^{l-1} + b^l,\qquad A^l = \sigma(Z^l),\qquad A^0 = I,$$

with the extracted global feature denoted

$\mathbf{f}_v := A^L \in \mathbb{R}^{d_v}.$

The exact layer index and $d_v$ are not specified. The model is used as a pretrained feature extractor, and no fine-tuning is described (Li et al., 30 Sep 2025).

SIFT contributes local keypoint descriptors through Difference-of-Gaussian scale-space extrema: $$D(x,y,\sigma) = \big[G(x,y,k\sigma) - G(x,y,\sigma)\big] * I(x,y),$$ yielding a descriptor set

$\mathbf{f}_s \in \mathbb{R}^{n_s \times d_s}.$

ORB contributes binary local descriptors after FAST keypoint detection and BRIEF-style comparisons, producing

$\mathbf{f}_o \in \mathbb{R}^{n_o \times d_o}.$

The paper does not specify $n_s$, $$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$0, $$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$1, or $$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$2, and does not use a Bag-of-Words codebook; instead, per-keypoint descriptors are flattened directly (Li et al., 30 Sep 2025).

Fusion is straight concatenation: $$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$3 No weighting scheme is reported. This design deliberately maximizes representational diversity at the expense of dimensionality, which is then controlled downstream by prototype formation and dimensionality reduction.

3. Feature selection and dimensionality reduction

After fusion, the framework applies a class-wise K-means-like selection mechanism. For each class $$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$4, with feature set $$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$5, the class prototype is

$$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$6

Collecting these prototypes gives

$$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$7

with $$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$8 for FER-Plus. Although the paper frames this step as inspired by K-means, the operational mechanism is effectively class-wise centroid formation. This reduces redundancy and concentrates the representation around class-discriminative centers (Li et al., 30 Sep 2025).

The dimensionality reduction stage then maps

$$I \xrightarrow{\text{feature extraction}} \mathbf{f}_m \in \mathbb{R}^{D}, \qquad \mathbf{f}_m \xrightarrow{\text{K-means selection}} \mathbf{f}_c \in \mathbb{R}^{K \times D},$$9

Six methods are evaluated: PCA, t-SNE, UMAP, Isomap, MDS, and LLE. PCA is the linear baseline, optimizing

$$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$0

Isomap preserves geodesic structure through shortest-path distances on a $$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$1-NN graph; MDS minimizes stress,

$$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$2

LLE preserves local linear reconstruction weights,

$$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$3

and t-SNE minimizes a KL divergence over neighborhood probabilities (Li et al., 30 Sep 2025).

UMAP is the decisive component in the reported best configuration. Its objective is given as a cross-entropy between high-dimensional and low-dimensional fuzzy simplicial sets: $$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$4 The reported interpretation is that UMAP preserves both local and global structure better than t-SNE and produces embeddings suitable not only for visualization but also for downstream learning. The dimensionality analysis further identifies medium dimensions, especially around 16D, as the strongest trade-off between information retention and redundancy suppression (Li et al., 30 Sep 2025).

4. Classification stage and decision behavior

Hy-Facial evaluates three classifier families on the reduced representation $$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$5: Random Forest, KNN, and MLP. Random Forest predicts by majority vote across trees,

$$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$6

KNN uses neighborhood voting,

$$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$7

and MLP applies stacked affine-nonlinear transforms,

$$\mathbf{f}_c \xrightarrow{\text{DR}} \mathbf{f}_r \in \mathbb{R}^{K \times d}, \qquad \mathbf{f}_r \xrightarrow{\text{classifier}} \hat{y} \in \{0,\dots,C-1\}.$$8

The exact MLP architecture and Random Forest hyperparameters are not specified (Li et al., 30 Sep 2025).

Random Forest is the best-performing classifier in every reported configuration. Under RF + K-means + PCA, the feature-ablation results are 41.85% for the baseline with no VGG19, no SIFT, and no ORB; 60.14% for VGG19 only; 72.58% for VGG19 + ORB; 71.65% for VGG19 + SIFT; and 77.50% for VGG19 + SIFT + ORB. These numbers establish two points. First, VGG19 contributes the dominant initial improvement. Second, the addition of both local descriptors is complementary rather than redundant (Li et al., 30 Sep 2025).

With the full hybrid representation VGG19 + SIFT + ORB + K-means, the dimensionality reduction comparison yields the following accuracies. For Random Forest: PCA 80.50%, t-SNE 78.20%, UMAP 83.30%, Isomap 79.80%, MDS 78.50%, and LLE 78.30%. For KNN: PCA 79.50%, t-SNE 74.80%, UMAP 81.50%, Isomap 78.20%, MDS 77.80%, and LLE 76.90%. For MLP: PCA 77.30%, t-SNE 70.70%, UMAP 79.00%, Isomap 77.10%, MDS 76.50%, and LLE 75.80% (Li et al., 30 Sep 2025).

These results support the paper’s central claim that dimensionality reduction is not merely a post hoc compactification step. In this framework it is the main operator that converts a large, noisy hybrid feature vector into a classifier-friendly embedding.

5. Dataset, protocol, and empirical interpretation

The reported experiments use the Kaggle FER2013Plus dataset. Images are 48×48 grayscale faces, with 28,709 training images and a 3,589-image public test set. Faces are pre-aligned and centered. Preprocessing consists of grayscale conversion, tensor transformation, and normalization by mean and standard deviation. No data augmentation is reported (Li et al., 30 Sep 2025).

The primary metric is test accuracy. The protocol follows the provided train/test split, and cross-validation or subject-independent partitioning is not described. The analysis includes confusion matrices and dimensionality sweeps over 2, 4, 8, 16, and 32 dimensions (Li et al., 30 Sep 2025).

The best configuration is VGG19 + SIFT + ORB, followed by K-means, UMAP, and Random Forest, reaching 83.3% accuracy. The confusion analysis indicates highest accuracy for Happy, attributed to strong global cues, while Fear and Angry remain confusable and Disgust shows weak performance due to subtle cues and few samples. The dimensionality sweep shows that 2D and 4D embeddings lose too much information, dimensions around 8 and 16 are optimal, and 32D degrades performance as noise and redundancy re-enter the representation. This suggests that the useful expression manifold in the hybrid representation is neither extremely low-dimensional nor well served by retaining a large number of coordinates (Li et al., 30 Sep 2025).

The paper’s interpretation is that the hybrid representation improves intra-class compactness and inter-class separability once dimensionality reduction is appropriately chosen. In that sense, UMAP is not simply the best reducer among several candidates; it is the mechanism that makes the fused VGG19-SIFT-ORB representation operationally discriminative.

6. Broader facial-computing usage and related research

Hy-Facial sits within a wider family of facial systems that combine heterogeneous priors, modalities, or representations. A clear precursor on the expression-recognition side is the salient-patch literature, which used learning-free landmark detection, landmark-anchored active patches, LBP descriptors, PCA + LDA, and one-against-one SVMs for six-class expression recognition on CK+ and JAFFE (Happy et al., 2015). This suggests a methodological continuity: region-selective local evidence remains important, but the exact Hy-Facial framework replaces handcrafted-only patch descriptors with a fused deep-plus-local representation and places manifold learning at the center (Li et al., 30 Sep 2025).

In adjacent arXiv work, the label has also been used in broader descriptive senses for problems outside FER. Hyper-Skin addresses facial skin-spectra reconstruction from RGB images, using paired hyperspectral and synthetic RGB images over VIS and NIR bands, with 330 hyperspectral cubes from 51 subjects and benchmarked reconstruction models such as HSCNN+, HRNet, and MST++ (Ng et al., 2023). A different strand concerns high-fidelity facial manipulation: HifiFace is a single-stage, end-to-end GAN-based face-swapping framework built around a 3D shape-aware identity representation and a Semantic Facial Fusion module (Wang et al., 2021), while HifaFace targets high-fidelity and arbitrary face editing through wavelet-based high-frequency injection and an additional high-frequency discriminator (Gao et al., 2021).

Other usages are geometric or physiological rather than expressive. HiFace reconstructs animatable 3D faces from a single image by separating static details from dynamic details through SD-DeTail (Chai et al., 2023). “High-Quality Facial Geometry and Appearance Capture at Home” uses a hybrid representation for full-face capture, combining explicit eyeball meshes with neural SDF and neural BRDF fields from a single smartphone flashlight sequence (Han et al., 2023). Remote facial skin assessment from selfies estimates skin hydration and TEWL through anchor localization, patch regression, and a Skin-Prior Adaptive Vision Transformer with texture, position, and symmetry priors (Soh et al., 8 Sep 2025).

Taken together, these works indicate that “Hy-Facial” has acquired a broader descriptive resonance around hybridization in facial computing: hybrid features in FER, hybrid geometry-and-semantic priors in swapping, hybrid static-dynamic detail models in reconstruction, hybrid spectral-RGB pairings in skin analysis, and hybrid explicit-implicit representations in facial capture. This broader usage is interpretive rather than terminologically canonical, but it is consistent with the supplied literature.

7. Limitations and future directions

The exact Hy-Facial FER framework has several explicit limitations. It depends on a pretrained VGG19 trained on generic image data; domain-specific fine-tuning is not explored. Its evaluation is restricted to FER2013Plus, whose faces are aligned, centered, and relatively clean. It does not include systematic robustness experiments for occlusion, severe pose variation, lighting extremes, or strong class imbalance beyond the observed confusion patterns. The dimensionality reduction stage also complicates interpretability, because the reduced embedding is not easily mapped back to specific facial regions or original feature families (Li et al., 30 Sep 2025).

The future directions proposed for the framework are correspondingly direct: integration with modern deep architectures such as transformers, diffusion models, or FER-specific CNNs; robustness evaluation on more challenging in-the-wild datasets; and combination with explainable AI methods such as saliency maps and feature attribution. A plausible implication is that later facial systems could merge the Hy-Facial emphasis on hybrid cues and manifold-aware compactification with the richer priors used elsewhere in facial computing, such as 3D geometry, semantic masks, spectral measurements, or physiology-oriented patch regression. That implication goes beyond the formal FER pipeline, but it is consistent with the broader research landscape reflected by contemporaneous work on face swapping, 3D reconstruction, hyperspectral skin analysis, and remote skin assessment (Wang et al., 2021, Chai et al., 2023, Ng et al., 2023, Soh et al., 8 Sep 2025).

In its strict sense, however, Hy-Facial remains a facial expression classification framework whose defining contribution is the combination of VGG19, SIFT, ORB, class-wise prototype selection, and especially UMAP-based dimensionality reduction. Its main result is not simply that hybrid features help, but that feature heterogeneity becomes most effective when redundancy is aggressively reorganized into a compact manifold representation before classification (Li et al., 30 Sep 2025).