---
title: 'MM-Gesture: Multimodal Micro-Gesture Recognition'
url: https://www.emergentmind.com/topics/mm-gesture
type: topic
---

# MM-Gesture: Multimodal Micro-Gesture Recognition

MM-Gesture is a multimodal micro-gesture recognition framework developed by the HFUT-VUT team for the micro-gesture classification track of the 3rd MiGA Challenge at IJCAI 2025. It targets subtle and short-duration micro-gestures by integrating complementary cues from joint, limb, RGB video, Taylor-series video, optical-flow video, and depth video modalities through PoseConv3D and Video Swin Transformer backbones, followed by a modality-weighted ensemble. On the iMiGUE benchmark, MM-Gesture ranked 1st and achieved a top-1 accuracy of 73.213% [2507.08344].

## 1. Problem setting and benchmark context

MM-Gesture addresses micro-gesture classification, a setting in which the target events are “spontaneous and fine-grained movements, such as nose touching, hair scratching, or subtle finger rubs” that encode affective and cognitive states. The task is challenging because the motions are unintentional and subtle, often confined to small regions such as the hands and face, and they occur over short temporal spans in which only a few frames are strongly discriminative [2507.08344].

The framework is built for the iMiGUE benchmark, which contains 31 micro-gesture categories and 1 non-MG class, for a total of 32 classes. The dataset comprises 18,499 labeled MG segments from 359 long videos, amounting to approximately 3.77M frames. The provided modalities are RGB videos and 2D skeletons from OpenPose; 137 keypoints are available, of which a subset of 36 is retained for upper body, hands, and face. The evaluation protocol is cross-subject, with 37 train subjects and 35 test subjects, and the split reported for MM-Gesture is 12,893 train segments, 777 validation segments, and 4,562 test segments [2507.08344].

A second dataset, Micro-Action 52 (MA-52), is used only for transfer learning. MA-52 contains 22,422 samples with 52 action-level and 7 body-level micro-action classes. In MM-Gesture, it serves as a pre-training source for the RGB Video Swin Transformer, with the goal of injecting broader micro-action priors into the target micro-gesture task [2507.08344].

## 2. Modalities and representational design

MM-Gesture is organized around six modalities. Three are used in the PoseConv3D branch—RGB, joint heatmaps, and limb heatmaps—and four are used in the Video Swin Transformer branch—RGB, Taylor-series video, optical-flow video, and depth video. The duplication of RGB across branches is deliberate: one RGB stream participates in cross-modal skeleton fusion, whereas another RGB stream is strengthened by MA-52 transfer learning and then ensembled with the other video modalities [2507.08344].

The skeletal input starts from 137 OpenPose keypoints, reduced to 36 selected points. Joint coordinates are represented as
\[
\mathbf{J} \in \mathbb{R}^{T \times V \times 2},
\]
where \(V=36\), and limb coordinates are represented as
\[
\mathbf{L} \in \mathbb{R}^{T \times E \times 2},
\]
where \(E\) is the number of skeleton edges. These are rasterized into spatiotemporal heatmap volumes,
\[
\mathbf{H}_{J} \in \mathbb{R}^{T \times H \times W \times V}, \qquad
\mathbf{H}_{L} \in \mathbb{R}^{T \times H \times W \times E},
\]
using Gaussian kernels around keypoints for joints and point-to-segment distances for limbs [2507.08344].

The auxiliary video modalities are all derived from RGB. Taylor-series video is defined as
\[
\mathbf{T} \in \mathbb{R}^{(T-\tau)\times H \times W \times 3}, \qquad
\mathbf{T}_t = \mathcal{F}_{\text{taylor}}(\mathbf{R}_{t:t+\tau}),
\]
and is intended to encode higher-order local temporal dynamics. Optical-flow video is computed with MemFlow,
\[
\mathbf{F} \in \mathbb{R}^{(T-1)\times H \times W \times 3}, \qquad
\mathbf{F}_t = \mathcal{F}_{\text{flow}}(\mathbf{R}_{t:t+1}),
\]
while depth video is produced with Video Depth Anything,
\[
\mathbf{D} \in \mathbb{R}^{T \times H \times W \times 3}, \qquad
\mathbf{D}_t = \mathcal{F}_{\text{depth}}(\mathbf{R}_t).
\]
Taylor emphasizes subtle temporal derivatives, flow encodes explicit motion vectors, and depth provides approximate geometric structure that can disambiguate visually similar 2D gestures [2507.08344].

| Modality | Construction | Backbone role |
|---|---|---|
| Joint heatmaps \(J\) | 36 selected OpenPose keypoints rendered as \(\mathbf{H}_J\) | PoseConv3D |
| Limb heatmaps \(L\) | Skeleton edges rendered as \(\mathbf{H}_L\) | PoseConv3D |
| RGB \(R\) | Raw RGB video | PoseConv3D and VideoSwinT |
| Taylor \(T\) | \(\mathbf{T}_t=\mathcal{F}_{\text{taylor}}(\mathbf{R}_{t:t+\tau})\) | VideoSwinT |
| Flow \(F\) | \(\mathbf{F}_t=\mathcal{F}_{\text{flow}}(\mathbf{R}_{t:t+1})\) | VideoSwinT |
| Depth \(D\) | \(\mathbf{D}_t=\mathcal{F}_{\text{depth}}(\mathbf{R}_t)\) | VideoSwinT |

This representational design treats micro-gesture recognition as a fusion problem across articulated pose, appearance, motion derivatives, explicit motion fields, and monocular geometry. The resulting system is not tied to a single inductive bias, and that multiplicity is central to its empirical performance [2507.08344].

## 3. Backbone architecture and fusion mechanisms

The first branch of MM-Gesture uses PoseConv3D for cross-modal fusion of RGB, joint heatmaps, and limb heatmaps. In training, separate PoseConv3D models are optimized for \(m \in \{\text{R}, \text{J}, \text{L}\}\) with cross-entropy losses
\[
\mathcal{L}_{m} = \mathrm{CE}(\hat{\mathbf{y}}_{m}, y).
\]
The framework then performs paired joint fine-tuning for RGB+J and RGB+L:
\[
\mathcal{L}_{\text{R+J}} = \mathcal{L}_{\text{R}} + \mathcal{L}_{\text{J}}, \qquad
\mathcal{L}_{\text{R+L}} = \mathcal{L}_{\text{R}} + \mathcal{L}_{\text{L}}.
\]
At inference, probabilities are averaged within each pair:
\[
\mathbf{P}_{\text{R+J}} = \frac{1}{2}(\mathbf{P}_{\text{R}} + \mathbf{P}_{\text{J}}), \qquad
\mathbf{P}_{\text{R+L}} = \frac{1}{2}(\mathbf{P}_{\text{R}} + \mathbf{P}_{\text{L}}).
\]
This branch is therefore both cross-modal and late-fused, but the pairing is learned through joint optimization rather than simple post hoc averaging alone [2507.08344].

The second branch uses Video Swin Transformer as a uni-modal encoder for RGB, Taylor, optical flow, and depth. Video Swin Transformer is a hierarchical 3D Swin Transformer that partitions videos into spatiotemporal patches and applies shifted-window self-attention over space and time. Each modality \(m \in \{\text{R}, \text{T}, \text{F}, \text{D}\}\) is trained independently with
\[
\mathcal{L}_{m} = \mathrm{CE}(\hat{\mathbf{y}}_{m}, y).
\]
A further RGB stream, denoted \(R^\star\), is first pretrained on MA-52 and then fine-tuned on iMiGUE. This two-stage transfer-learning path is the only branch in the system that explicitly exploits external supervision outside iMiGUE [2507.08344].

The final decision rule is a probability-level weighted ensemble over six predictors,
\[
i \in \{\text{R+J}, \text{R+L}, \text{R}, \text{T}, \text{F}, \text{D}\},
\]
with
\[
\mathbf{P}_{\text{final}} = \sum_i w_i \mathbf{P}_i.
\]
The weights \(w_i\) are selected empirically according to validation performance. This design favors a modular multimodal system over a monolithic end-to-end fusion network. It also means that MM-Gesture combines two distinct fusion logics: paired cross-modal skeleton–RGB fusion inside PoseConv3D and global probability-weighted fusion across all modality-specific experts [2507.08344].

## 4. Empirical performance and ablation structure

MM-Gesture was reported as the 1st-ranked system in the MiGA 2025 micro-gesture classification track, achieving 73.213% top-1 accuracy on iMiGUE. The challenge comparison reported in the paper places the method above the MiGA 2025 second-place result of 68.697%, the MiGA 2025 third-place result of 67.010%, the best MiGA 2024 result of 70.254%, and the best MiGA 2023 result of 64.12% [2507.08344].

The ablation study shows a clear progression from single-modality baselines to the full multimodal ensemble. Single-modality PoseConv3D scores were 65.256% for joints only, 64.686% for limbs only, and 64.511% for RGB only. Multi-modality PoseConv3D improved to 67.229% for \(J+L\), and 68.917% for both \(J+\text{RGB}\) and \(L+\text{RGB}\). On the Video Swin Transformer side, RGB without pretraining achieved 65.629%, RGB with MA-52 pretraining achieved 66.615%, Taylor achieved 62.845%, optical flow 61.617%, and depth 65.212% [2507.08344].

A decisive result is that the ensemble improves monotonically as modalities are added. The combination \(J + \text{RGB}^\star\) reached 70.955%, \(L + \text{RGB}^\star\) reached 70.802%, and \(J + L + \text{RGB}^\star\) reached 71.416%. Adding Taylor yielded 72.095%, adding flow yielded 72.227%, adding depth yielded 72.644%, and the full MM-Gesture configuration \(J + L + R + \text{RGB}^\star + T + F + D\) achieved 73.213% [2507.08344].

| Configuration | Top-1 accuracy (%) | Source |
|---|---:|---|
| J only | 65.256 | PoseConv3D |
| L only | 64.686 | PoseConv3D |
| RGB only | 64.511 | PoseConv3D |
| RGB (VideoSwinT) | 65.629 | VideoSwinT |
| RGB* (MA-52 pretrain) | 66.615 | VideoSwinT |
| J + RGB | 68.917 | PoseConv3D |
| L + RGB | 68.917 | PoseConv3D |
| J + L + RGB* | 71.416 | Ensemble |
| J + L + RGB* + Taylor | 72.095 | Ensemble |
| J + L + RGB* + Taylor + Flow | 72.227 | Ensemble |
| J + L + RGB* + Taylor + Flow + Depth | 72.644 | Ensemble |
| Full MM-Gesture | 73.213 | Final system |

Several conclusions follow directly from these results. First, skeleton-only or RGB-only recognition is insufficient for the finest iMiGUE distinctions. Second, transfer learning on MA-52 provides a measurable RGB gain of 0.986 percentage points. Third, Taylor, flow, and depth are weaker as isolated modalities than RGB or skeletons, but they are complementary in the final ensemble. This pattern is consistent with the paper’s central claim that subtle MGs are only partially observable in any single modality [2507.08344].

## 5. Position within multimodal gesture research

MM-Gesture belongs to a broader multimodal gesture literature, but its task and representational commitments are narrower than several adjacent lines of work. A useful contrast is the RGB dynamic gesture recognition framework of “Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition,” which uses RGB and depth during training, but only RGB at inference. That system frames multimodality as training-time auxiliary supervision for segmentation and recognition, rather than as persistent inference-time fusion across six modalities [2110.15639]. This suggests a distinction between multimodal training and multimodal inference that MM-Gesture resolves in favor of the latter.

A second contrast is with grounded and situated gesture generation. “Grounded Gesture Generation: Language, Motion, and Space” introduces a framework for linguistically, kinematically, and spatially grounded referential gestures, built around synthetic pointing data and the MM-Conv VR dialogue dataset. There, the central problem is the alignment of speech, full-body motion, and 3D scene context for referential communication; the output is generated motion rather than gesture class labels [2507.04522]. MM-Conv itself provides full-body motion capture, finger motion, gaze, speech, facial expressions, and AI2-THOR scene graphs for referential dialogues in VR, making it a resource for situated co-speech gesture modeling rather than micro-gesture recognition [2410.00253].

MM-Gesture also differs from multimodal co-speech generation frameworks such as MambaGesture and MAG. MambaGesture integrates audio, text, style, and emotion for diffusion-based co-speech gesture generation with MambaAttn and SEAD, while MAG uses a motion-text-audio-aligned variational autoencoder and a multimodal masked autoregressive model without vector quantization. Both aim at full-body gesture synthesis aligned to speech semantics and rhythm, not micro-gesture classification [2407.19976] [2503.14040]. By contrast, MM-Gesture does not model speech, style, or emotion as inputs; its multimodality is visual and pose-centric.

Within the iMiGUE ecosystem itself, “Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition” provides a different multimodal blueprint: RGB video is encoded with MViTv2-S, pose with 2s-AGCN, and the streams are integrated through a Cross-Modal Token Fusion module with memory-powered refinement, reaching 62.87% top-1 accuracy on micro-gesture recognition [2512.23291]. Relative to that architecture, MM-Gesture’s stronger result on the same benchmark family indicates that a broader derived-modality palette—Taylor, flow, depth, plus dual RGB pathways—can outperform token-level RGB-pose fusion alone in the challenge setting [2507.08344].

This wider comparison suggests that “MM-Gesture” can denote both a specific 2025 iMiGUE system and, more loosely, a research direction centered on multimodal gesture understanding or generation across RGB, depth, pose, audio, language, and 3D scene cues. The precise sense depends on whether the focus is recognition, co-speech generation, or grounded interaction [2507.08344] [2507.04522].

## 6. Limitations and prospective directions

Several limitations are explicit in the design, and some are plausible implications of the reported architecture. A plausible implication is a substantial computational footprint: the full MM-Gesture system combines multiple PoseConv3D streams with four Video Swin Transformer branches, while also requiring preprocessing for MemFlow, Video Depth Anything, Taylor video construction, and OpenPose-based skeleton extraction. This suggests a high offline cost relative to single-backbone recognition systems [2507.08344].

Another plausible implication is dependency on upstream modality quality. The framework assumes reliable 2D pose extraction, optical flow, and monocular depth estimation. Because the model fuses derived modalities rather than learning them jointly, failures in those upstream estimators can propagate into the classifier. The paper does not present an end-to-end multimodal encoder that jointly optimizes those steps, and this suggests a modular but potentially brittle pipeline under severe domain shift [2507.08344].

The authors explicitly identify future directions beyond the current system. They propose exploring Multimodal Large Language Models to introduce semantic understanding, interactive prompts, and context-aware reasoning. They also propose adding further modalities such as gaze, audio, and remote photoplethysmography to improve affective and behavioral interpretation [2507.08344]. In light of related iMiGUE work on Cross-Modal Token Fusion and memory-powered refinement, a plausible research path is to replace probability-level ensembling with deeper cross-modal interaction while preserving the strong complementary signal provided by Taylor, flow, and depth [2512.23291].

MM-Gesture therefore occupies a specific place in the 2025 micro-gesture literature: it is a challenge-winning, highly engineered ensemble whose main contribution is not a new pretraining objective or a single fusion block, but a carefully staged integration of skeletal, appearance, motion, and geometry cues. Its reported 73.213% top-1 accuracy on iMiGUE establishes a strong reference point for multimodal micro-gesture recognition, while its architecture also exposes the next technical questions: whether the same multimodal breadth can be retained with lower complexity, tighter end-to-end coupling, and richer contextual conditioning [2507.08344].

Source: https://www.emergentmind.com/topics/mm-gesture