MM-Gesture: Multimodal Micro-Gesture Recognition

• MM-Gesture is a recognition framework that integrates six modalities to accurately classify 32 micro-gesture classes, achieving a top-1 accuracy of 73.213% on the iMiGUE benchmark.
• It employs a dual-branch architecture combining PoseConv3D for skeletal cues and Video Swin Transformer for video modalities enhanced by MA-52 transfer learning.
• A modality-weighted ensemble fuses complementary features from RGB, Taylor-series, optical flow, and depth, effectively capturing subtle, short-duration gestures.

MM-Gesture is a multimodal micro-gesture recognition framework developed by the HFUT-VUT team for the micro-gesture classification track of the 3rd MiGA Challenge at IJCAI 2025. It targets subtle and short-duration micro-gestures by integrating complementary cues from joint, limb, RGB video, Taylor-series video, optical-flow video, and depth video modalities through PoseConv3D and Video Swin Transformer backbones, followed by a modality-weighted ensemble. On the iMiGUE benchmark, MM-Gesture ranked 1st and achieved a top-1 accuracy of 73.213% (Gu et al., 11 Jul 2025).

1. Problem setting and benchmark context

MM-Gesture addresses micro-gesture classification, a setting in which the target events are “spontaneous and fine-grained movements, such as nose touching, hair scratching, or subtle finger rubs” that encode affective and cognitive states. The task is challenging because the motions are unintentional and subtle, often confined to small regions such as the hands and face, and they occur over short temporal spans in which only a few frames are strongly discriminative (Gu et al., 11 Jul 2025).

The framework is built for the iMiGUE benchmark, which contains 31 micro-gesture categories and 1 non-MG class, for a total of 32 classes. The dataset comprises 18,499 labeled MG segments from 359 long videos, amounting to approximately 3.77M frames. The provided modalities are RGB videos and 2D skeletons from OpenPose; 137 keypoints are available, of which a subset of 36 is retained for upper body, hands, and face. The evaluation protocol is cross-subject, with 37 train subjects and 35 test subjects, and the split reported for MM-Gesture is 12,893 train segments, 777 validation segments, and 4,562 test segments (Gu et al., 11 Jul 2025).

A second dataset, Micro-Action 52 (MA-52), is used only for transfer learning. MA-52 contains 22,422 samples with 52 action-level and 7 body-level micro-action classes. In MM-Gesture, it serves as a pre-training source for the RGB Video Swin Transformer, with the goal of injecting broader micro-action priors into the target micro-gesture task (Gu et al., 11 Jul 2025).

2. Modalities and representational design

MM-Gesture is organized around six modalities. Three are used in the PoseConv3D branch—RGB, joint heatmaps, and limb heatmaps—and four are used in the Video Swin Transformer branch—RGB, Taylor-series video, optical-flow video, and depth video. The duplication of RGB across branches is deliberate: one RGB stream participates in cross-modal skeleton fusion, whereas another RGB stream is strengthened by MA-52 transfer learning and then ensembled with the other video modalities (Gu et al., 11 Jul 2025).

The skeletal input starts from 137 OpenPose keypoints, reduced to 36 selected points. Joint coordinates are represented as

$\mathbf{J} \in \mathbb{R}^{T \times V \times 2},$

where $V=36$, and limb coordinates are represented as

$\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$

where $E$ is the number of skeleton edges. These are rasterized into spatiotemporal heatmap volumes,

$$\mathbf{H}_{J} \in \mathbb{R}^{T \times H \times W \times V}, \qquad \mathbf{H}_{L} \in \mathbb{R}^{T \times H \times W \times E},$$

using Gaussian kernels around keypoints for joints and point-to-segment distances for limbs (Gu et al., 11 Jul 2025).

The auxiliary video modalities are all derived from RGB. Taylor-series video is defined as

$$\mathbf{T} \in \mathbb{R}^{(T-\tau)\times H \times W \times 3}, \qquad \mathbf{T}_t = \mathcal{F}_{\text{taylor}}(\mathbf{R}_{t:t+\tau}),$$

and is intended to encode higher-order local temporal dynamics. Optical-flow video is computed with MemFlow,

$$\mathbf{F} \in \mathbb{R}^{(T-1)\times H \times W \times 3}, \qquad \mathbf{F}_t = \mathcal{F}_{\text{flow}}(\mathbf{R}_{t:t+1}),$$

while depth video is produced with Video Depth Anything,

$$\mathbf{D} \in \mathbb{R}^{T \times H \times W \times 3}, \qquad \mathbf{D}_t = \mathcal{F}_{\text{depth}}(\mathbf{R}_t).$$

Taylor emphasizes subtle temporal derivatives, flow encodes explicit motion vectors, and depth provides approximate geometric structure that can disambiguate visually similar 2D gestures (Gu et al., 11 Jul 2025).

Modality	Construction	Backbone role
Joint heatmaps $J$	36 selected OpenPose keypoints rendered as $\mathbf{H}_J$	PoseConv3D
Limb heatmaps $V=36$0	Skeleton edges rendered as $V=36$1	PoseConv3D
RGB $V=36$2	Raw RGB video	PoseConv3D and VideoSwinT
Taylor $V=36$3	$V=36$4	VideoSwinT
Flow $V=36$5	$V=36$6	VideoSwinT
Depth $V=36$7	$V=36$8	VideoSwinT

This representational design treats micro-gesture recognition as a fusion problem across articulated pose, appearance, motion derivatives, explicit motion fields, and monocular geometry. The resulting system is not tied to a single inductive bias, and that multiplicity is central to its empirical performance (Gu et al., 11 Jul 2025).

3. Backbone architecture and fusion mechanisms

The first branch of MM-Gesture uses PoseConv3D for cross-modal fusion of RGB, joint heatmaps, and limb heatmaps. In training, separate PoseConv3D models are optimized for $V=36$9 with cross-entropy losses

$\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$0

The framework then performs paired joint fine-tuning for RGB+J and RGB+L: $\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$1 At inference, probabilities are averaged within each pair: $\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$2 This branch is therefore both cross-modal and late-fused, but the pairing is learned through joint optimization rather than simple post hoc averaging alone (Gu et al., 11 Jul 2025).

The second branch uses Video Swin Transformer as a uni-modal encoder for RGB, Taylor, optical flow, and depth. Video Swin Transformer is a hierarchical 3D Swin Transformer that partitions videos into spatiotemporal patches and applies shifted-window self-attention over space and time. Each modality $\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$3 is trained independently with

$\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$4

A further RGB stream, denoted $\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$5, is first pretrained on MA-52 and then fine-tuned on iMiGUE. This two-stage transfer-learning path is the only branch in the system that explicitly exploits external supervision outside iMiGUE (Gu et al., 11 Jul 2025).

The final decision rule is a probability-level weighted ensemble over six predictors,

$\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$6

with

$\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$7

The weights $\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$8 are selected empirically according to validation performance. This design favors a modular multimodal system over a monolithic end-to-end fusion network. It also means that MM-Gesture combines two distinct fusion logics: paired cross-modal skeleton–RGB fusion inside PoseConv3D and global probability-weighted fusion across all modality-specific experts (Gu et al., 11 Jul 2025).

4. Empirical performance and ablation structure

MM-Gesture was reported as the 1st-ranked system in the MiGA 2025 micro-gesture classification track, achieving 73.213% top-1 accuracy on iMiGUE. The challenge comparison reported in the paper places the method above the MiGA 2025 second-place result of 68.697%, the MiGA 2025 third-place result of 67.010%, the best MiGA 2024 result of 70.254%, and the best MiGA 2023 result of 64.12% (Gu et al., 11 Jul 2025).

The ablation study shows a clear progression from single-modality baselines to the full multimodal ensemble. Single-modality PoseConv3D scores were 65.256% for joints only, 64.686% for limbs only, and 64.511% for RGB only. Multi-modality PoseConv3D improved to 67.229% for $\mathbf{L} \in \mathbb{R}^{T \times E \times 2},$9, and 68.917% for both $E$0 and $E$1. On the Video Swin Transformer side, RGB without pretraining achieved 65.629%, RGB with MA-52 pretraining achieved 66.615%, Taylor achieved 62.845%, optical flow 61.617%, and depth 65.212% (Gu et al., 11 Jul 2025).

A decisive result is that the ensemble improves monotonically as modalities are added. The combination $E$2 reached 70.955%, $E$3 reached 70.802%, and $E$4 reached 71.416%. Adding Taylor yielded 72.095%, adding flow yielded 72.227%, adding depth yielded 72.644%, and the full MM-Gesture configuration $E$5 achieved 73.213% (Gu et al., 11 Jul 2025).

Configuration	Top-1 accuracy (%)	Source
J only	65.256	PoseConv3D
L only	64.686	PoseConv3D
RGB only	64.511	PoseConv3D
RGB (VideoSwinT)	65.629	VideoSwinT
RGB* (MA-52 pretrain)	66.615	VideoSwinT
J + RGB	68.917	PoseConv3D
L + RGB	68.917	PoseConv3D
J + L + RGB*	71.416	Ensemble
J + L + RGB* + Taylor	72.095	Ensemble
J + L + RGB* + Taylor + Flow	72.227	Ensemble
J + L + RGB* + Taylor + Flow + Depth	72.644	Ensemble
Full MM-Gesture	73.213	Final system

Several conclusions follow directly from these results. First, skeleton-only or RGB-only recognition is insufficient for the finest iMiGUE distinctions. Second, transfer learning on MA-52 provides a measurable RGB gain of 0.986 percentage points. Third, Taylor, flow, and depth are weaker as isolated modalities than RGB or skeletons, but they are complementary in the final ensemble. This pattern is consistent with the paper’s central claim that subtle MGs are only partially observable in any single modality (Gu et al., 11 Jul 2025).

5. Position within multimodal gesture research

MM-Gesture belongs to a broader multimodal gesture literature, but its task and representational commitments are narrower than several adjacent lines of work. A useful contrast is the RGB dynamic gesture recognition framework of “Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition,” which uses RGB and depth during training, but only RGB at inference. That system frames multimodality as training-time auxiliary supervision for segmentation and recognition, rather than as persistent inference-time fusion across six modalities (Fan et al., 2021). This suggests a distinction between multimodal training and multimodal inference that MM-Gesture resolves in favor of the latter.

A second contrast is with grounded and situated gesture generation. “Grounded Gesture Generation: Language, Motion, and Space” introduces a framework for linguistically, kinematically, and spatially grounded referential gestures, built around synthetic pointing data and the MM-Conv VR dialogue dataset. There, the central problem is the alignment of speech, full-body motion, and 3D scene context for referential communication; the output is generated motion rather than gesture class labels (Deichler et al., 6 Jul 2025). MM-Conv itself provides full-body motion capture, finger motion, gaze, speech, facial expressions, and AI2-THOR scene graphs for referential dialogues in VR, making it a resource for situated co-speech gesture modeling rather than micro-gesture recognition (Deichler et al., 2024).

MM-Gesture also differs from multimodal co-speech generation frameworks such as MambaGesture and MAG. MambaGesture integrates audio, text, style, and emotion for diffusion-based co-speech gesture generation with MambaAttn and SEAD, while MAG uses a motion-text-audio-aligned variational autoencoder and a multimodal masked autoregressive model without vector quantization. Both aim at full-body gesture synthesis aligned to speech semantics and rhythm, not micro-gesture classification (Fu et al., 2024, Liu et al., 18 Mar 2025). By contrast, MM-Gesture does not model speech, style, or emotion as inputs; its multimodality is visual and pose-centric.

Within the iMiGUE ecosystem itself, “Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition” provides a different multimodal blueprint: RGB video is encoded with MViTv2-S, pose with 2s-AGCN, and the streams are integrated through a Cross-Modal Token Fusion module with memory-powered refinement, reaching 62.87% top-1 accuracy on micro-gesture recognition (Martirosyan et al., 29 Dec 2025). Relative to that architecture, MM-Gesture’s stronger result on the same benchmark family indicates that a broader derived-modality palette—Taylor, flow, depth, plus dual RGB pathways—can outperform token-level RGB-pose fusion alone in the challenge setting (Gu et al., 11 Jul 2025).

This wider comparison suggests that “MM-Gesture” can denote both a specific 2025 iMiGUE system and, more loosely, a research direction centered on multimodal gesture understanding or generation across RGB, depth, pose, audio, language, and 3D scene cues. The precise sense depends on whether the focus is recognition, co-speech generation, or grounded interaction (Gu et al., 11 Jul 2025, Deichler et al., 6 Jul 2025).

6. Limitations and prospective directions

Several limitations are explicit in the design, and some are plausible implications of the reported architecture. A plausible implication is a substantial computational footprint: the full MM-Gesture system combines multiple PoseConv3D streams with four Video Swin Transformer branches, while also requiring preprocessing for MemFlow, Video Depth Anything, Taylor video construction, and OpenPose-based skeleton extraction. This suggests a high offline cost relative to single-backbone recognition systems (Gu et al., 11 Jul 2025).

Another plausible implication is dependency on upstream modality quality. The framework assumes reliable 2D pose extraction, optical flow, and monocular depth estimation. Because the model fuses derived modalities rather than learning them jointly, failures in those upstream estimators can propagate into the classifier. The paper does not present an end-to-end multimodal encoder that jointly optimizes those steps, and this suggests a modular but potentially brittle pipeline under severe domain shift (Gu et al., 11 Jul 2025).

The authors explicitly identify future directions beyond the current system. They propose exploring Multimodal LLMs to introduce semantic understanding, interactive prompts, and context-aware reasoning. They also propose adding further modalities such as gaze, audio, and remote photoplethysmography to improve affective and behavioral interpretation (Gu et al., 11 Jul 2025). In light of related iMiGUE work on Cross-Modal Token Fusion and memory-powered refinement, a plausible research path is to replace probability-level ensembling with deeper cross-modal interaction while preserving the strong complementary signal provided by Taylor, flow, and depth (Martirosyan et al., 29 Dec 2025).

MM-Gesture therefore occupies a specific place in the 2025 micro-gesture literature: it is a challenge-winning, highly engineered ensemble whose main contribution is not a new pretraining objective or a single fusion block, but a carefully staged integration of skeletal, appearance, motion, and geometry cues. Its reported 73.213% top-1 accuracy on iMiGUE establishes a strong reference point for multimodal micro-gesture recognition, while its architecture also exposes the next technical questions: whether the same multimodal breadth can be retained with lower complexity, tighter end-to-end coupling, and richer contextual conditioning (Gu et al., 11 Jul 2025).