Micro-gesture Online Recognition

• Micro-gesture online recognition is the task of localizing and categorizing minute, short-duration gestures in streaming data under strict real-time constraints.
• It leverages techniques such as RGB, skeleton fusion, and non-visual sensor inputs to address challenges like low motion amplitude and ambiguous boundaries.
• Recent benchmarks and models, including query-based and temporal pyramid methods, demonstrate improved F1 scores through adaptive temporal modeling and multimodal fusion.

Searching arXiv for papers on micro-gesture online recognition and closely related benchmarks/methods. Micro-gesture online recognition is the task of temporally localizing and categorizing subtle gesture instances in untrimmed or streaming observations, typically under strict causal or low-latency constraints. In the recent literature, the problem is commonly cast as a micro-scale variant of temporal action detection in which the system must output each gesture’s start time, end time, and category, while coping with extremely short duration, low motion amplitude, ambiguous boundaries, and fine-grained inter-class similarity (Liu et al., 13 Jul 2025, Liu et al., 10 Jun 2026, Shen et al., 5 Jun 2026). The field spans RGB-only detection, RGB-skeleton fusion, skeleton-based online recognition, and non-visual sensing systems such as ultrasound and wearables, with current work divided between true online detection on untrimmed streams and offline clip-level micro-gesture classification that is adjacent but not equivalent to the online setting (Chang et al., 18 Dec 2025, Sang et al., 2017, Lee et al., 15 Aug 2025).

1. Task definition and relation to adjacent problems

A standard formulation treats an untrimmed video $V=\{v_1,\ldots,v_T\}$ or synchronized multimodal stream as input and predicts a set of gesture instances

$$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$

where $t_i^s$ and $t_i^e$ denote temporal boundaries and $c_i$ denotes the gesture category (Liu et al., 13 Jul 2025, Liu et al., 10 Jun 2026). In the stricter causal formulation, the predictor must operate using only past and current frames, without future context, and return proposals of the form $\hat{\varphi}_i=(\hat{t}_s,\hat{t}_e,\hat{c},s)$, where $s$ is confidence (Shen et al., 5 Jun 2026).

Recent papers repeatedly distinguish micro-gesture online recognition from conventional temporal action detection. The main reasons are that micro-gestures are very short, low-amplitude, spontaneous, and visually similar across classes; boundary localization is brittle; and small temporal shifts materially affect the score (Liu et al., 13 Jul 2025, Liu et al., 10 Jun 2026). A related distinction separates online recognition from clip-level classification. Several strong micro-gesture systems on iMiGUE operate on pre-sampled clips and explicitly note that they are not online or streaming methods, even when they use temporal modeling or multimodal fusion (Patapati et al., 19 Jun 2025, Martirosyan et al., 29 Dec 2025). The same distinction holds for earlier skeleton-based micro-gesture classification approaches, which recognize a category for a given clip but do not address streaming boundaries or continual detection (Li et al., 2023).

The broader online-gesture literature contributes two concepts that remain pertinent. One is continuous detection based on temporal evolution of class probabilities, as in SLOTH, where a gesture is recognized from a probability plateau after a derivative peak and can often be detected before execution finishes (Carfi et al., 2018). The other is explicit progression estimation, where a multitask model jointly predicts class and normalized gesture progression $P_t\in[0,1]$, enabling early trigger rules based on progression thresholds rather than fixed frame counts (Gupta et al., 2019). These are not micro-gesture benchmarks, but they formalize the same online requirement: making decisions from incomplete evidence.

2. Datasets, benchmarks, and annotation regimes

Dataset construction is a central bottleneck because micro-gestures require subtle motion capture and precise frame-level boundaries. The recent benchmark literature therefore emphasizes both scale and annotation methodology. OMG-Bench introduces a multi-view self-supervised pipeline to automatically generate skeleton data, complemented by heuristic rules and expert refinement for semi-automatic annotation, and is presented as the first large-scale public benchmark for skeleton-based online micro gesture recognition (Chang et al., 18 Dec 2025). In parallel, challenge-oriented datasets such as SMG and iMiGUE have provided the dominant evaluation settings for RGB, skeleton, and multimodal models (Liu et al., 10 Jun 2026, Zhang et al., 11 Jun 2026).

Benchmark/resource	Setting	Reported characteristics
SMG	Online recognition in untrimmed videos	3,692 annotated micro-gesture instances, 40 subjects, RGB videos + skeleton sequences, 17 categories (Liu et al., 10 Jun 2026)
OMG-Bench	Skeleton-based online micro gesture recognition	40 fine-grained gesture classes, 13,948 instances, 1,272 sequences (Chang et al., 18 Dec 2025)
iMiGUE	Identity-free micro-gesture classification	359 videos, 72 subjects, cross-subject benchmark (Zhang et al., 11 Jun 2026)
HoMG	Offline micro-gesture dataset	960 videos, 30,635 still images, 40 participants, 3 gesture classes (Liu et al., 2017)
Grab-n-Go dataset	Wearable occupied-hand microgesture recognition	18 participants, 35 objects, 20,160 microgesture instances (Lee et al., 15 Aug 2025)

SMG is the main benchmark for challenge-style online recognition in untrimmed videos, but its exact label specification varies across papers. One study defines $\mathcal{C}$ as 17 micro-gesture classes (Liu et al., 10 Jun 2026), whereas another challenge formulation describes 16 micro-gesture classes plus 1 non-micro-gesture class under the official Track 2 setup (Shen et al., 5 Jun 2026). A plausible implication is that benchmark usage and label conventions are partly track-dependent rather than completely uniform across challenge rounds.

iMiGUE occupies a different role. It is repeatedly used for cross-subject clip classification and multimodal micro-gesture recognition, with papers describing it as a 32-way setting, a 31-plus-1 setting, or a 33-class setting depending on the task definition and challenge year (Zhang et al., 11 Jun 2026, Martirosyan et al., 29 Dec 2025, Patapati et al., 19 Jun 2025). This suggests that iMiGUE is better understood as a family of closely related micro-gesture classification protocols than as a single fixed online-detection benchmark.

Earlier resources illustrate the field’s longer trajectory. HoMG used a holoscopic 3D camera to record 3 micro-gesture classes from 40 participants for wearable and AR/VR interaction, but its experiments were offline baselines using LBPTOP or LPQTOP rather than continuous online detection (Liu et al., 2017). Grab-n-Go extends the problem to occupied-hand interaction, organizing 30 microgestures across 5 grasping poses and 35 objects using wrist-worn active acoustic sensing (Lee et al., 15 Aug 2025).

3. Core architectural paradigms in video-based online recognition

Recent online systems mostly inherit the temporal action detection design space, but adapt it for much shorter and subtler events. One family is query-based set prediction. “Micro-gesture Online Recognition using Learnable Query Points” represents each candidate action as a set of learnable temporal points $P=\{t_i\}_{i=1}^{N_s}$ plus query vectors $$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$0, and refines them across decoder layers rather than predicting fixed segments directly (Liu et al., 2024). The update rule

$$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$1

scales displacement by the current span, which the paper argues helps short micro-gesture localization (Liu et al., 2024). The same model combines a Mamba-MHSA block for query semantics with a Multi-Level Interactive Module for point-level local extraction and instance-level mixing, and ranked 2nd in the IJCAI 2024 MiGA Micro-gesture Online Recognition track with an F1 score of 14.34 (Liu et al., 2024).

A second family uses dense temporal pyramids and dynamic heads. The HFUT-VUT 2025 solution enhances DyFADet with category-frequency-based annotation replication and a spatial-temporal attention detection head (Liu et al., 13 Jul 2025). Its augmentation repeats labels according to

$$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$2

thereby increasing the effective frequency of underrepresented classes without direct raw-video duplication (Liu et al., 13 Jul 2025). On SMG, this method achieved an F1 score of 38.03 and ranked first in the Micro-gesture Online Recognition track (Liu et al., 13 Jul 2025).

A third family emphasizes parameter-efficient adaptation of large pretrained video backbones. The Spatial-Temporal Decoupled Adapter inserts lightweight temporal and spatial branches after each Transformer block of a frozen VideoMAEv2-g backbone and feeds the adapted features to ActionFormer (Shen et al., 5 Jun 2026). Its adapter combines two deltas additively,

$$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$3

explicitly separating temporal from spatial adaptation via depthwise convolutions (Shen et al., 5 Jun 2026). The same paper introduces Adaptive Soft Balanced Augmentation, which computes class-dependent augmentation targets from effective sample counts and achieved F1 $$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$4, ranking 1st in Track 2 of the 4th EI-MiGA-IJCAI Challenge (Shen et al., 5 Jun 2026).

A fourth line uses explicit memory. HMATr, introduced together with OMG-Bench, unifies gesture detection and classification through hierarchical memory banks storing frame-level details and window-level semantics, and further employs learnable position-aware queries initialized from memory to encode gesture positions and semantics (Chang et al., 18 Dec 2025). In the abstract-level report, HMATr outperforms state-of-the-art methods by 7.6% in detection rate and serves as a strong baseline for online micro gesture recognition (Chang et al., 18 Dec 2025).

4. Skeletons, multimodality, and the fusion problem

A persistent question in micro-gesture research is whether skeletons or appearance should dominate. The current evidence favors complementarity rather than replacement. DyFADet+ extends DyFADet into a dual-stream RGB-skeleton framework in which both modalities are projected into shared multi-scale temporal embeddings and fused through a gated residual mechanism: $$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$5

$$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$6

with $$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$7 and gate bias initialized to $$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$8 so that training begins near an RGB-only regime (Liu et al., 10 Jun 2026). On SMG, the method reaches F1 $$\Psi=\{\psi_1,\psi_2,\ldots,\psi_N\}, \qquad \psi_i=(t_i^s,t_i^e,c_i),$$9 and ranks 2nd in the Micro-gesture Online Recognition track (Liu et al., 10 Jun 2026).

The accompanying ablation is especially informative. RGB only yields 38.76, skeleton only 22.31, naive RGB+skeleton fusion 38.96, and the gated residual fusion 40.88 (Liu et al., 10 Jun 2026). This directly counters the common assumption that skeleton alone is a sufficient compact representation for micro-gestures. Instead, the results indicate that skeleton motion is useful primarily when it is injected selectively into a stronger RGB representation rather than concatenated indiscriminately (Liu et al., 10 Jun 2026).

Skeleton information also appears in several clip-level methods that are not online detectors but contribute reusable representation ideas. A 3D-CNN skeleton classifier using PoseC3D/SlowOnly plus a joint skeletal and semantic embedding loss reports SMG Top-1 $t_i^s$0 and explicitly notes that it is an offline, non-causal clip-level system rather than an online recognizer (Li et al., 2023). CLIP-MG uses pose-guided semantic query generation and gated multimodal fusion with CLIP ViT-B/16, achieving Top-1 accuracy 61.82% on iMiGUE, but also states that it is not an online recognition method in the strict sense (Patapati et al., 19 Jun 2025). A multimodal iMiGUE framework combining MViTv2-S, 2s-AGCN, Cross-Modal Token Fusion, and Memory-Powered Refinement achieves 62.87% Top-1 accuracy and likewise remains clip-based (Martirosyan et al., 29 Dec 2025).

The latest benchmark work pushes skeletons back into the online regime. OMG-Bench is specifically designed for skeleton-based online micro hand gesture recognition, and its construction pipeline addresses the long-standing difficulty of obtaining precise skeletons and frame-level annotations for subtle motion patterns (Chang et al., 18 Dec 2025). This suggests a partial convergence between earlier skeleton-heavy classification research and newer online-detection formulations.

5. Real-time systems beyond RGB benchmark pipelines

Although current challenge leaders are largely RGB or RGB-skeleton detectors, micro-gesture online recognition also includes alternative sensing systems designed for real-time deployment. The ultrasonic HUG system uses 300 kHz active sensing to generate range-Doppler sequences from subtle finger motions and supports both a state-transition-based HMM and an end-to-end CNN+LSTM (Sang et al., 2017). On a 7-class micro hand gesture dataset with 9 subjects and 5,400 samples, the HMM achieves 89.38% and the end-to-end model 96.34%; in gesture-level throughput, the HMM operates at about 250 Hz and the end-to-end model at about 15 Hz (Sang et al., 2017). This work is notable because it treats micro gestures as fine finger-only motions rather than upper-body cues extracted from video.

Grab-n-Go extends wearable acoustic sensing to occupied-hand use. Its wristband uses two speaker-microphone pairs, 4 acoustic paths, and 4 differential channels, producing an input tensor of $t_i^s$1 over a 1.8 s window (Lee et al., 15 Aug 2025). The classifier, based on ResNet-18, recognizes 30 microgestures across 5 grasping poses and 25 objects with an average accuracy of 92.0%, and a follow-up study reports 92.9% on deformable objects (Lee et al., 15 Aug 2025). The system is described as the first wearable device recognizing subtle hand microgestures while holding various objects (Lee et al., 15 Aug 2025).

Related online gesture-recognition systems contribute operational principles even when they do not target benchmark micro-gestures directly. SLOTH uses a wrist-worn triaxial accelerometer and an LSTM classifier with a continuous gesture-recognition module that monitors $t_i^s$2, then validates a high-confidence plateau to trigger recognition (Carfi et al., 2018). In one online example, 10 out of 12 gestures are correctly classified before the end of the gesture (Carfi et al., 2018). LE-HGR, an RGB-based embedded framework for AR devices, combines lightweight hand detection, keypoint regression, hand-trace mapping, and TraceSeqNN; it reports 94.71% accuracy, 4.4% false positive rate, and 4 ms speed on its own dataset, and 75.3% or 77.2% on the public Nvidia Gesture Dataset at 76 FPS or 34 FPS depending on keypoint source (Xie et al., 2020). A wearable-sensor HRI system for incomplete gesture data further shows that LSTM can classify dynamic gestures from partial trajectories with 97.2% accuracy at 75% completion and 96.5% at 100% completion, albeit with worse training and inference time than simpler feature-based methods (Simão et al., 2023).

These systems are not interchangeable with SMG-style micro-gesture detection, but they establish that the online micro-gesture problem is modality-agnostic at a systems level: all of them must reject background, detect action onset early, and remain stable under incomplete observations.

6. Evaluation, recurring difficulties, and unresolved directions

The main evaluation metric for SMG-style online recognition is the F1 score,

$t_i^s$3

which jointly reflects correct localization and correct classification (Liu et al., 13 Jul 2025, Shen et al., 5 Jun 2026). Some works additionally report mAP at multiple temporal IoU thresholds (Shen et al., 5 Jun 2026), while clip-level iMiGUE research typically reports Top-1 accuracy (Martirosyan et al., 29 Dec 2025, Gu et al., 11 Jul 2025). OMG-Bench emphasizes detection rate as its headline metric (Chang et al., 18 Dec 2025). Because metrics differ across online detection, clip classification, and sensor-based recognition, reported numbers are not directly comparable across the entire literature.

Several difficulties recur across nearly all studies. The first is annotation precision. Progression modeling on the NVIDIA gesture dataset required new tight boundary annotations because weak start/end labels were inadequate for localization and frame-level progression targets (Gupta et al., 2019). OMG-Bench addresses the same issue with multi-view self-supervision, heuristic rules, and expert refinement (Chang et al., 18 Dec 2025). The second is class imbalance. HFUT-VUT’s 2025 online detector uses category-frequency-based adaptive label replication (Liu et al., 13 Jul 2025), while STDA uses Adaptive Soft Balanced Augmentation derived from class rarity and learning difficulty (Shen et al., 5 Jun 2026). The third is cross-subject shift. A multimodal iMiGUE framework introduces Cross-Modal Pseudo-Labeling for unsupervised domain adaptation and shows a strong gain for cross-subject classification, but it is explicitly not an online method (Zhang et al., 11 Jun 2026).

A common misconception is that the strongest micro-gesture papers are automatically online-recognition systems. In fact, many of the highest Top-1 results on iMiGUE are offline clip classifiers or multimodal ensembles rather than streaming detectors. Prototype Learning reaches 70.254% Top-1 on iMiGUE using RGB+pose clip inputs (Chen et al., 2024); MM-Gesture pushes this to 73.213% with six modalities and weighted ensembling (Gu et al., 11 Jul 2025); and several other systems emphasize token fusion, memory refinement, or semantic alignment without providing causal inference or latency analysis (Martirosyan et al., 29 Dec 2025, Zhang et al., 11 Jun 2026). This distinction matters because micro-gesture classification and micro-gesture online recognition solve different problems even when they use related backbones.

The literature also suggests several open directions without yet resolving them. Online methods still struggle with very short segments and boundary precision (Shen et al., 5 Jun 2026). Strong clip-based systems repeatedly point toward multimodal fusion, motion magnification, and better handling of long-tail imbalance (Chen et al., 2024, Patapati et al., 19 Jun 2025). Sensor-based systems highlight the practical importance of background rejection, low false-positive rates, and user-specific adaptation (Lee et al., 15 Aug 2025, Carfi et al., 2018). Taken together, these results indicate that the field is moving toward an overview of three requirements: precise frame-level annotation, robust multimodal or multi-sensor representations, and genuinely causal inference that preserves accuracy under partial observation.