Micro-Expression Recognition (MER)

• MER is the automated analysis of extremely brief, subtle facial expressions that reveal genuine, often involuntary, emotions.
• It employs specialized techniques like optical flow, dynamic imaging, and landmark-based methods to amplify and capture low-intensity muscle movements.
• The field applies advanced deep learning and phase-aware modeling to enhance applications in psychological assessment, law enforcement, and human–computer interaction.

Facial micro-expression recognition (MER) is the automated analysis and classification of extremely brief, low-intensity, and involuntary facial expressions that reveal genuine emotions. Micro-expressions typically last less than 500 ms and are defined by subtle muscular activations in localized facial regions. Unlike macro-expressions, which can be voluntary and often exaggerated, micro-expressions are spontaneous, difficult to consciously produce or suppress, and typically go unnoticed by human observers. MER is a principal area of affective computing with applications spanning psychological assessment, law enforcement, lie detection, security screening, mental health diagnostics, and human–computer interaction. The MER problem is inherently challenging due to (i) the subtlety and transience of the motion signals, (ii) substantial variation between subjects, (iii) the limited volume and diversity of annotated datasets, and (iv) the pronounced intra-class ambiguity and inter-class similarity in facial movements.

1. Core Challenges in Micro-Expression Recognition

The difficulties of MER derive from the fundamental properties of micro-expressions:

• Low Intensity and Brief Duration: Micro-expressions manifest as minute facial muscle movements with durations often ranging from 1/25 to 1/3 seconds, typically covering only small facial regions (Peng et al., 2019).
• Subtleness and Variability: The signal-to-noise ratio is typically low, with subtle deformations easily being submerged by identity information, background clutter, or acquisition noise (Li et al., 2022).
• Data Scarcity and Class Imbalance: Most available databases (e.g., SMIC, CASME II, SAMM) are small (≤300 labeled samples), limiting the effectiveness of data-hungry deep models (Li et al., 2021, Wang et al., 18 Apr 2024).
• Temporal Localization Ambiguity: Micro-expression sequences can be too short for precise temporal segmentation, making frame selection (especially for “apex” or key motion points) error-prone (Zhu et al., 2023).
• Inter- and Intra-Class Overlap: Subtle differences in muscle movements between classes (e.g., “disgust” vs. “fear”) and large variations within a class due to subject differences complicate robust classification (Xie et al., 2020, Zhang et al., 5 Jan 2025).

Addressing these issues necessitates specialized pre-processing, feature extraction, data augmentation, and transfer learning strategies uniquely tailored to the micro-expression domain.

2. Principled Representations and Data Pipelines

Representation design in MER has evolved in response to the limitations imposed by the micro-expression signal. The principal strategies include:

• Optical Flow (OF) and Dynamic Imaging: Optical flow extracted between onset–apex or onset–apex–offset frames encodes pixel-level motion and is effective for highlighting subtle displacements (Zhang et al., 5 Jan 2025, Shao et al., 7 Sep 2025). Dynamic image representations produced via (approximate) rank pooling accumulate a video’s temporal evolution in a single image, revealing the trajectory of facial muscular changes (Liu et al., 2020, Khuong et al., 14 Oct 2025).
• Eulerian Motion Magnification and Temporal Interpolation: Eulerian frameworks formulate motion magnification and temporal interpolation (i.e., “motion boosting” and “time stretching”) in a unified, linear mapping: $\hat{V} = V \cdot W$, with $W = W^M \cdot W^I$ (Peng et al., 2019). This joint approach amplifies low-intensity movements and simultaneously increases video length, mitigating the challenge of insufficient temporal resolution for feature extraction.
• Landmark Trajectories and Geometric Graphs: Facial landmarks provide a lower-dimensional, compact representation for MER, capturing the geometry of muscle movement while discarding redundant background information (Wei et al., 2022, Wei et al., 2023). Graph neural networks model spatial and temporal dependencies between landmarks, often incorporating learnable adjacency structures and action unit (AU) constraints.
• Action Unit (AU)-Centric Features: The Facial Action Coding System (FACS) organizes muscle activations into AUs. AU localization (with or without graph modeling) provides discriminative cues, supporting both single-stage detection/classification pipelines and joint AU–category recognition (Zhou et al., 2020, Li et al., 28 Jul 2025, Liu et al., 9 May 2025).

Pre-processing typically involves face detection, alignment, normalization, and either cropping regions of interest or calculating OF/DIs on temporally selected frame pairs. Data augmentation strategies range from spatial–temporal transformations to localized blending (e.g., LocalStaticFaceMix (Liu et al., 9 May 2025)) that increase diversity while preserving critical micro-expression cues.

3. Model Architectures and Learning Strategies

MER models are typically categorized according to the treatment of spatial and temporal features, level of supervision, and mechanisms for incorporating prior knowledge. Representative approaches include:

• Dual-Stream or Multi-Branch Models: These architectures separately process appearance/motion, onset–apex/apex–offset phases, or geometric/appearance cues before fusion. For instance, DIANet processes onset–apex and apex–offset phase-aware dynamic images in parallel, integrating information via a cross-attention fusion module to capture asymmetric temporal dynamics (Khuong et al., 14 Oct 2025).
• Attention Mechanisms: Channel and spatial attention (e.g., multi-scale attention in AHMSA-Net (Zhang et al., 5 Jan 2025), continuous attention in MMNet (Li et al., 2022), vertical/single-orientation attention in FaceSleuth (Wu et al., 3 Jun 2025)) enhance sensitivity to subtle muscle activations, often focusing on empirically dominant motion directions.
• Transformer-Based Temporal Modeling: Hierarchical space–time attention modules (HSTA (Hao et al., 6 May 2024)) and local–global feature-aware transformers (Shao et al., 7 Sep 2025) provide expressive mechanisms for capturing long- and short-range temporal dependencies, integrating multi-modal frame information and managing special frames.
• Graph Neural Networks: Identification and message passing among landmark nodes or AU nodes, enriched with learnable or psychologically-driven priors, are leveraged for spatial structure learning and region-specific aggregation (Wei et al., 2022, Zhou et al., 2020, Wei et al., 2023, Li et al., 28 Jul 2025).
• Meta-Learning and Auxiliary Tasks: Meta-auxiliary learning paradigms (e.g., LightmanNet (Wang et al., 18 Apr 2024)) apply dual-branch and bi-level optimization to learn robust knowledge from scarce and imbalanced data, by aligning micro- and macro-expression features and refining task-specific and generalizable representations.
• Transfer Learning and Domain Adaptation: Several models address the scarcity and bias of micro-expression data by pre-training on large macro-expression datasets before adaptation (e.g., MA2MI and MIACNet (Li et al., 26 May 2024)). Pre-training tasks may focus on frame reconstruction or position/action decoupling, rather than naïve fine-tuning across domains.

Loss functions are commonly adapted to the data characteristics, including combined cross-entropy and AU losses (Wei et al., 2022), deviation enhancement (Liu et al., 2020), margin-based metric learning (triplet, center loss (Xie et al., 2020, Ma et al., 11 Jun 2025)), and domain adaptation objectives.

4. Temporal Segmentation and Phase-Aware Modeling

Temporal phase modeling is central to modern MER. Micro-expression sequences conform to a bell-shaped intensity curve: an onset phase (neutral to peak), apex (peak), and offset (peak to neutral). Approaches include:

• Apex-Based Methods: Many pipelines extract the apex frame (where motion intensity peaks) as most discriminative (Xie et al., 2020). Some methods build features solely from onset–apex or onset–apex–offset pairs, reducing redundancy.
• Flexible Occurring Frame Schemes: LTR3O replaces strict apex spotting with a flexible three-frame onset–occurring–offset representation by randomly segmenting the video and sampling the “occurring” frame, followed by calibration modules that enforce macro-expression-like expressivity patterns (Zhu et al., 2023).
• Phase-Specific Dynamic Images: DIANet explicitly separates onset–apex and apex–offset into phase-aware dynamic images using ARP with directionally reversed coefficients, processed by dual CNN streams with cross-attentive fusion and a phase-consistency regularizer (Khuong et al., 14 Oct 2025).
• Temporal Attention and Fusion: Hierarchical temporal encoding, often cascaded with crossmodal or multi-scale attention, enables flexible handling of spatial and temporal cues from special and global frames (Hao et al., 6 May 2024).

5. Evaluation, Benchmarking, and Experimental Findings

MER research primarily adopts leave-one-subject-out (LOSO) cross-validation or composite evaluation protocols (e.g., MEGC benchmarks) on major datasets such as CASME II, SAMM, SMIC, CAS(ME)², CAS(ME)³, and MMEW. The principal evaluation metrics include classification accuracy, F1-score, Unweighted F1 (UF1), and Unweighted Average Recall (UAR).

Salient experimental outcomes from representative studies include:

Model	Dataset(s)	Accuracy (%)	F1-score (if reported)	Notable Findings
ME-Booster (Peng et al., 2019)	SMIC-subHS	87.32	—	7%–10% accuracy gain, ×5 speed
SMA-STN (Liu et al., 2020)	CASME II	82.59	0.7946	Outperforms TSCNN-II
MMNet (Li et al., 2022)	CASME II	—	—	+7.23% acc, +10.94% F1 over ResNet
LTR3O (Zhu et al., 2023)	CASME II	78.95	76.46	No apex needed, flexible / robust
FaceSleuth (Wu et al., 3 Jun 2025)	CASME II	95.1	0.918	Verifies vertical orientation optimality
MPFNet-C (Ma et al., 11 Jun 2025)	CASME II	92.4	—	Multi-prior fusion; also strong on SMIC, SAMM
AHMSA-Net (Zhang et al., 5 Jan 2025)	CASME³	77.08	—	Balanced across databases
DIANet (Khuong et al., 14 Oct 2025)	CASME II	70.00	—	+11.2% acc over DI-only baselines
MER-CLIP (Liu et al., 9 May 2025)	CAS(ME)³	—	0.7832 (3-class UF1)	AU-guided CLIP alignment
FDP (Shao et al., 7 Sep 2025)	CASME II	—	+4.05 (ΔF1)	Fine-grained dynamic perception

These results establish a trend toward progressive integration of temporal phase awareness, cross-modal attention/fusion, and incorporation of multiple priors (motion, geometry, AU structure) for enhanced generalization, discriminability, and efficiency. Notably, the confirmatory evidence for vertical motion as the dominant axis in MER (Wu et al., 3 Jun 2025) and the pivotal role of phase separation (Khuong et al., 14 Oct 2025) represent recent strong empirical findings.

6. Open Challenges and Future Directions

Despite significant advances, open issues remain in MER:

• Generalization to In-the-Wild Data: Most benchmarks are laboratory-controlled. Robustness against uncontrolled lighting, occlusion, and natural head pose remains weak (Xie et al., 2020, Li et al., 2021).
• Data Augmentation and Synthesis: Synthetic data generation, using GANs, morphable models, or macro-to-micro transfer learning, is essential for overcoming data scarcity but calibration of synthetic realism and label fidelity remains unresolved (Xie et al., 2020, Li et al., 26 May 2024).
• Cross-Group and Socio-Cultural Bias: Recognition accuracy varies with group membership; future systems must control for or adapt to social/cultural variability (Xie et al., 2020).
• Interpretability of Feature Representations: While attention and AU-aligned models offer some explainability, full interpretability—down to the contribution of individual facial action patterns—remains an open goal (Liu et al., 9 May 2025).
• Integration with Multimodal Signals: The fusion of facial micro-expression with additional modalities (audio, physiological signals) is recognized as a direction for richer emotion understanding (Xie et al., 2020, Li et al., 2021).
• Temporal Uncertainty Modeling: Improved mechanisms for handling frame ambiguity, especially around apex estimation, can further mitigate errors stemming from noisy annotation (Khuong et al., 14 Oct 2025).

Practical deployments demand not only state-of-the-art performance but also lightweight architectures, privacy preservation (e.g., via federated learning (Li et al., 28 Jul 2025)), and ethical safeguards in sensitive application scenarios.

7. Summary Table of Representative Methods

Method	Key Innovation	Temporal Modeling	Performance/Highlight
ME-Booster (Peng et al., 2019)	Joint mag.+interpolation (linear)	Eulerian, ARP	87.3% (SMIC-subHS), speedup
SMA-STN (Liu et al., 2020)	DSSI, STMA, DE-loss	Segment/ARP, ST-attention	82.6% (CASME II LOSO)
MMNet (Li et al., 2022)	CA block, ViT PC module	2-branch diff./pos. fusion	+7–11% F1 over ResNet
MPFNet (Ma et al., 11 Jun 2025)	Dual prior-encoder, prog. train	I3D + coordinate attention	92.4% (CASME II)
FaceSleuth (Wu et al., 3 Jun 2025)	CVA/SOA vertical pooling	Swin Transformer + AU	95.1% ACC, vertical optimal
DIANet (Khuong et al., 14 Oct 2025)	Phase-aware dual DI + fusion	ARP-based, cross-attention	70% (CASME II, +11% over DI-only)

To summarize, micro-expression recognition has transitioned from handcrafted, low-level descriptors to sophisticated deep learning pipelines that combine phase-awareness, attention, prior knowledge, and cross-modal cues. Contemporary best practices employ unified frameworks for amplifying, structuring, and fusing highly transient and subtle facial motion information; future systems are expected to further bridge the gap between controlled laboratory performance and unconstrained real-world affective computing.