AU-LLM: Micro-Expression AU Detection

• AU-LLM is a vision–language framework that detects micro-expressions by translating compact spatio-temporal facial cues into LLM-interpretable representations.
• The framework integrates temporal enhancement, a specialized 3D-CNN backbone, and an Enhanced Fusion Projector to fuse local textures with global semantics.
• AULLM++ extends the approach by reformulating AU prediction into evidence construction, structured modeling, and deduction-based prediction with counterfactual regularization.

Searching arXiv for AU-LLM and related micro-expression AU detection papers. AU-LLM is a vision–language framework for micro-expression Action Unit (AU) detection that uses a parameter-efficiently adapted LLM to reason over a single, compact visual token distilled from multi-level spatio-temporal features of facial video clips (Liu et al., 29 Jul 2025). In this setting, AU detection is treated as a multi-label inference problem over brief, low-intensity facial events whose signal is weak, sparse, and highly sensitive to subject variability and domain shift. The framework couples temporal enhancement, a specialized 3D-CNN backbone, an Enhanced Fusion Projector (EFP), and LLM-based reasoning. A later extension, AULLM++, reformulates AU prediction into evidence construction, structure modeling, and deduction-based prediction, adding structural priors and counterfactual consistency regularization (Liu et al., 9 Mar 2026).

1. Problem setting and conceptual scope

Micro-expression AU detection aims to identify facial muscle activations defined by the Facial Action Coding System (FACS), such as AU2 (Outer Brow Raiser), AU4 (Brow Lowerer), AU7 (Lid Tightener), and AU12 (Lip Corner Puller). Relative to macro-expressions, micro-expressions are brief, low-intensity, and often involuntary; they exhibit extremely low signal-to-noise ratio, strong subject variability, and domain shift across datasets. Benchmark micro-expression datasets contain only hundreds of sequences, so the problem is constrained simultaneously by subtle visual evidence, multi-label ambiguity, and data scarcity (Liu et al., 29 Jul 2025).

The central motivation of AU-LLM is that AU detection in this regime is not only a feature-extraction problem but also a reasoning problem. Conventional CNN pipelines can lose localized, low-amplitude cues when relying on global pooling or coarse feature summarization. AU-LLM therefore attempts to translate visual evidence into a representation that an LLM can interpret and reason over, with the stated goal of modeling the “grammar” of facial muscle movements. The framework addresses three coupled issues: extraction of faint local texture changes, preservation of high-level semantics, and inference under subject and dataset shift.

A common source of ambiguity in the literature is that “LLM-based AU recognition” is not a single methodological family. AU-LLM is specifically a micro-expression AU detector built around compact visual prompting and LLM reasoning, whereas other work such as AU-LLaVA addresses broader unified AU recognition across datasets and output formats (Hu et al., 2024). This suggests that AU-LLM should be understood as a specialized micro-expression variant within a larger multimodal AU-recognition trend.

2. Architectural formulation

AU-LLM comprises four stages: temporal enhancement and spatio-temporal feature extraction, Enhanced Fusion Projector, LLM reasoning, and a linear AU classification head (Liu et al., 29 Jul 2025). The input is a short video clip

$V \in \mathbb{R}^{T \times H \times W},$

with $T=6$ frames and spatial resolution $H=W=64$.

Temporal enhancement is implemented by Learnable Eulerian Dynamics (LED), which re-weights frames to amplify transient onset–apex–offset motion:

$$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$

The LED filter is defined by a learnable upper-triangular matrix with parameters $\alpha$, $r_1$, and $r_2$, and the filtered sequence is obtained via normalized matrix multiplication over time. This stage is intended to amplify subtle temporal variation before visual encoding.

The visual backbone is a specialized 3D-CNN with a Squeeze-and-Excitation block after the second convolution. It extracts mid-level features $F_{\text{mid}}$, which are flattened to

$f_{\text{mid}} \in \mathbb{R}^{d_m},$

and high-level features

$f_{\text{high}} \in \mathbb{R}^{d_h}.$

The mid-level pathway captures local textures and shapes, while the high-level pathway captures global semantics.

The EFP bridges the vision–language semantic gap by concatenating these features and projecting them into the LLM embedding space:

$T=6$0

$T=6$1

In the reported implementation, EFP is a pure MLP-based fusion module without additional normalization, residual connections, gating, or attention inside EFP. The output dimensionality $T=6$2 matches the LLM token embedding size, so $T=6$3 can be injected as a soft visual prompt.

The reasoning stage prepends $T=6$4 to a task prompt and processes the resulting multimodal sequence with a 1.5B-parameter LLM adapted with LoRA:

$T=6$5

with the final fused representation given by the last-token hidden state

$T=6$6

A linear AU head then produces multi-label logits

$T=6$7

where $T=6$8 is the number of AUs, with $T=6$9 for CASME II and $H=W=64$0 for SAMM, and probabilities

$H=W=64$1

3. LLM adaptation, prompting, and optimization

AU-LLM evaluates three 1.5B-parameter backbones: Qwen2-1.5B, Qwen2.5-1.5B, and DeepSeek-R1-Distill-Qwen-1.5B. Adaptation is performed with parameter-efficient fine-tuning via LoRA, with rank $H=W=64$2 and $H=W=64$3, applied to the query and value projections of the self-attention layers. The base LLM weights remain frozen; only the LoRA adapters, the EFP MLP, the AU head, and the visual backbone are trained (Liu et al., 29 Jul 2025).

Visual token injection is implemented by prepending $H=W=64$4 as a special learned visual token to the text prompt. The prompt template reported in the paper is:

“Analyze the facial features to classify action units:”

Dataset-specific AU sets are implicit from training labels. CASME II uses $H=W=64$5, while SAMM uses $H=W=64$6.

Training uses Asymmetric Loss (ASL) to address class imbalance. While the paper also gives the standard binary cross-entropy form,

$H=W=64$7

AU-LLM uses

$H=W=64$8

with $H=W=64$9 and $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$0. Evaluation is based on AU-wise F1-score with macro averaging,

$$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$1

under Leave-One-Subject-Out (LOSO) and cross-domain protocols.

The reported training setup uses PyTorch, Adam with learning rate $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$2, weight decay $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$3, batch size $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$4, and a single NVIDIA H100 GPU. Code is reported as available at the project repository (Liu et al., 29 Jul 2025).

4. Benchmarks, protocols, and reported performance

The two principal evaluation datasets are CASME II and SAMM. CASME II contains 247 micro-expression sequences from 26 subjects at 200 fps, evaluated with LOSO and 8 AUs. SAMM contains 159 sequences from 32 subjects with 4 shared AUs; the paper reports cross-dataset validation and macro F1 (Liu et al., 29 Jul 2025).

The main quantitative results are as follows:

Protocol	AU-LLM result	Comparison reported
CASME II (LOSO, macro F1)	81.4	SSSNet LED 78.4; ResNet18 LED 77.3
SAMM (cross-dataset, macro F1)	61.9	ResNet18 LED 57.3; LBP-TOP 50.2
CASME II $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$5 SAMM	49.9	LED SSSNet 36.5; ResNet-18 37.3
SAMM $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$6 CASME II	52.8	LED SSSNet 46.1; ResNet-18 47.4

On CASME II, AU-DeepSeek R1 (1.5B) achieved an average F1 of 81.4, while AU-Qwen2 and AU-Qwen2.5 achieved 78.8 and 79.5. The DeepSeek variant outperformed SSSNet LED by up to $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$7 points and ResNet18 LED by $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$8 points. On SAMM, AU-DeepSeek R1 achieved 61.9, compared with 57.9 for AU-Qwen2 and 59.7 for AU-Qwen2.5, surpassing ResNet18 LED by $$V' = \mathrm{LED}(V; W \in \mathbb{R}^{T \times T}).$$9 points.

The per-AU CASME II values reported for AU-DeepSeek R1 are AU2 87.0, AU4 89.5, AU7 68.1, and AU12 79.0, with the text noting consistent improvements across several critical AUs, especially brow and lip region micro-movements. On SAMM, the same model achieved AU2 66.9, AU4 71.9, AU7 55.8, and AU12 52.8. The bidirectional cross-domain results suggest better transfer under dataset shift than LED baselines.

5. Ablations, robustness, and empirical interpretation

The ablation results isolate the contribution of multi-level fusion and dynamic visual prompting (Liu et al., 29 Jul 2025). Using only high-level features yielded 80.1 on CASME II and 54.5 on SAMM for the DeepSeek variant, while using only mid-level features yielded 79.3 and 54.7. Both underperformed the full model, indicating that AU-LLM depends on combining local texture cues with global semantics.

Replacing the EFP MLP with a single linear projector reduced performance to 78.6 on CASME II and 58.2 on SAMM. Prompt learning with static textual embeddings also lagged behind EFP-based dynamic visual prompting, with 80.4 versus 81.4 on CASME II and 59.6 versus 61.9 on SAMM. Across all tested LLM backbones, the same trend persisted: EFP and multi-level fusion consistently improved AU detection.

The reported qualitative analysis states that heatmaps show improved localization of AU-specific regions such as AU12 lip corners, AU4 brow, AU15 depressor, and AU17 chin. Subject variability is handled by LOSO, and the compact visual token is described as preserving transferable AU cues under domain shift. The stated failure modes remain subtle intensity confusion for lip-related AUs when motion is extremely faint or occluded. LED temporal enhancement and SE recalibration are reported to reduce diffuse attention relative to baselines.

A plausible implication is that AU-LLM’s gains arise less from generic language generation and more from the LLM’s role as a structured reasoning module operating on a compressed, instance-specific visual premise. The ablations support that interpretation because static prompt learning and single-level features do not match the full model.

6. AULLM++, subsequent developments, and limitations

AULLM++ extends the AU-LLM line by presenting a reasoning-oriented framework that injects visual features into textual prompts as actionable semantic premises and formulates AU prediction into three stages: evidence construction, structure modeling, and deduction-based prediction (Liu et al., 9 Mar 2026). In the abstract and excerpted description, a Multi-Granularity Evidence-Enhanced Fusion Projector (MGE-EFP) distills a compact Content Token (CT), a Relation-Aware AU Graph Neural Network (R-AUGNN) produces an Instruction Token (IT), and Counterfactual Consistency Regularization (CCR) constructs counterfactual samples to improve generalization. The excerpt explicitly notes that the method description, equations, prompt templates, training details, and ablations are not available from the provided text.

Even with those omissions, the reported benchmark gains are clear. On CASME II, “AU-DeepSeek R1++ (1.5B)” achieves an average F1 of 82.4, compared with 78.4 for SSSNet LED and 77.3 for ResNet18 LED, corresponding to gains of $\alpha$0 and $\alpha$1 points. On SAMM, it achieves 62.6, exceeding ResNet18 LED at 57.3 by $\alpha$2 points. The per-AU CASME II values are AU1 93.1, AU2 88.1, AU4 90.5, AU7 67.1, AU12 81.0, AU14 76.1, AU15 80.0, and AU17 82.0; on SAMM they are AU2 70.6, AU4 72.6, AU7 56.5, and AU12 53.5.

The broader literature also includes AU-LLaVA, described as the first unified AU recognition framework based on an LLM, consisting of a visual encoder, a linear projector layer, and a pre-trained LLM (Hu et al., 2024). That line of work emphasizes unified output formatting across AU datasets, whereas AU-LLM and AULLM++ are specialized for micro-expression AU detection.

The main limitations reported for AU-LLM systems are data scarcity, annotation noise, class imbalance, and the computational overhead of using an LLM rather than a pure CNN or Transformer. The 2025 AU-LLM paper notes that EFP is still a relatively simple MLP and identifies gated fusion, cross-attention, multi-token visual interfaces, and richer task prompts as future directions (Liu et al., 29 Jul 2025). The 2026 AULLM++ excerpt further suggests that releasing precise prompt templates, full source, and formal reasoning details would be necessary for complete reproducibility (Liu et al., 9 Mar 2026).