Vision-based Hand Gesture Recognition

• Vision-based Hand Gesture Recognition is the automatic inference of hand pose or gesture class from modalities like RGB, depth, and skeleton data.
• The field employs diverse methodologies—from CNNs and graph networks to few-shot meta-learning—to address both static and dynamic gesture challenges in varied HCI applications.
• Evolving approaches move from handcrafted features to deep learning ensembles, with metrics such as F1-score and accuracy reflecting advances in real-time and continuous gesture recognition.

Vision-based Hand Gesture Recognition (VHGR) is the automatic inference of hand-gesture class or hand pose/structure from visual input, principally RGB images, RGB video, depth images, infrared imagery, optical flow, and pose or skeleton representations derived from cameras. In current literature, the field spans static posture recognition, isolated dynamic gesture recognition, continuous unsegmented sequence recognition, and a related estimation track that predicts hand pose rather than a discrete label. Its principal application domains include human-computer interaction, sign language recognition, robotics, VR/AR, gaming, and broader human-machine interfaces (Linardakis et al., 21 Jan 2025, Foteinos et al., 6 Jul 2025, Shin et al., 2024).

1. Scope, task structure, and field boundaries

Recent surveys formalize a gesture as a temporal sequence of poses,

$$G = \{P_1, P_2, \ldots, P_n\}, \quad P_i \in \mathbb{R}^d,$$

and distinguish two core tasks. Gesture classification maps a static image or sequence

$X = \{x_1, x_2, \ldots, x_t\}$

to a predefined class set

$$f: X \rightarrow \mathcal{C} = \{C_1, C_2, \ldots, C_k\},$$

whereas gesture estimation predicts pose parameters or hand structure

$g: X \rightarrow P = \{p_1, p_2, \ldots, p_n\}.$

The same surveys further separate static VHGR, isolated dynamic VHGR, and continuous dynamic VHGR, and note that dynamic gestures are often described as having preparation, core or nucleus, and retraction phases (Linardakis et al., 21 Jan 2025, Foteinos et al., 6 Jul 2025).

The taxonomic structure of VHGR is now multi-axial rather than merely modal. The 2025 review organizes the literature simultaneously by application domain, task or temporal structure, input modality, information type, camera viewpoint, architecture, and learning strategy. Within this taxonomy, input may be appearance-based, pose-based or skeleton-based, or multimodal; gestures may be semaphoric, language-like, deictic, or manipulative; and viewpoint may be first-person, second-person, or third-person (Foteinos et al., 6 Jul 2025). A related 2024 survey divides HGR more broadly into vision-based and device- or signal-based systems, placing RGB, depth, skeleton, and multimodal visual systems inside the VHGR branch, while treating EEG, EMG, and audio as non-visual modalities (Shin et al., 2024).

A persistent boundary issue is terminological rather than algorithmic. Some papers apply vision architectures to non-camera measurements converted into pseudo-images. The radar-based “Vision Transformer” system of Al-Obaidi et al. operates on spectrograms and angle-of-arrival matrices from a 24 GHz continuous-wave Doppler radar receiver, not on camera imagery, despite the title’s use of “Vision Transformer” (Kehelella et al., 2022). EgoHand occupies a similar boundary: it is positioned explicitly as an alternative to bottom-facing camera systems for VR, replacing visual sensing at inference time with head-mounted mmWave radar and IMUs while keeping a skeleton-based intermediate representation familiar from camera-based pipelines (Lv et al., 23 Jan 2025). This suggests that “hand gesture recognition” is broader than VHGR proper, and that visual pipelines are best understood as the camera-derived branch of that broader field.

2. Visual pipelines and hand representations

Across modalities, the surveys describe a recurrent VHGR workflow: data collection, preprocessing, hand or gesture representation, feature extraction, and classification or sequence decoding. Preprocessing may include segmentation, background suppression, augmentation, frame sampling, pose estimation, ROI extraction, or depth thresholding. Representation may be raw pixels, cropped hand boxes, silhouettes, optical-flow fields, motion summaries, joint coordinates, angles, graph structures, or multimodal feature maps (Shin et al., 2024).

A compact RGB-D instantiation is the “virtual glove marker” system, which uses an Intel RealSense D435i, segments the hand by removing pixels beyond a depth threshold of 500 mm, converts the segmented image to a binary mask, applies a distance transform, and takes the maximum-distance pixel as the palm center. Fingertips and knuckle landmarks are then estimated with MediaPipe, the palm estimate is smoothed with a moving average over the previous 5 frames, and OpenCV renders line segments from palm to knuckles and from knuckles to fingertips to form a synthetic glove-like marker image for CNN classification (McKinnon et al., 2022). The same paper makes explicit the representational trade-off: the virtual glove is a computed visualization of hand structure intended to preserve shape information while avoiding physical glove hardware.

Landmark-centric RGB pipelines form another major representation class. GRLib begins with an RGB camera feed, applies brightness adjustment and rotation augmentation, runs MediaPipe Hands on the augmented frames, selects the first frame in which at least one hand is detected, and classifies the resulting landmark vector with a conventional classifier rather than a fixed neural backbone. For static gestures, the best classifier across the evaluated ASL, HaGRID, and Kenyan datasets is an SVC with RBF kernel; for dynamic gestures, the library uses trajectory quantization, keyframe extraction, and candidate tracking rather than sequence models that require large training sets (Warchocki et al., 2023).

Skeleton-aware formulations generalize this landmark logic into an explicit structured model. One 2024 framework constructs a 3D hand skeleton model from the palm center, the base joints of the five fingers, the intermediate finger joints, and the fingertips, and combines CNN-based spatial feature extraction with LSTM temporal modeling; it also introduces eye tracking as a second modality processed with an attention mechanism and fused through a weighted fusion layer (Shao et al., 2024). At the conceptual level, this reframes hand pose as a graph-like object rather than a texture patch.

Derived modalities can also be computed from RGB streams under real-time constraints. IPN Hand evaluates RGB alone, RGB plus optical flow, and RGB plus semantic segmentation, using SPyNet for flow and HarDNet for hand segmentation. The reported trend is generally

$\text{RGB-Flow} > \text{RGB-Seg} > \text{RGB},$

but semantic segmentation is reported as more than two times faster than optical flow for the extra preprocessing stage, with 8.1 ms for HarDNet segmentation versus 21.9 ms for SPyNet optical flow (Benitez-Garcia et al., 2020). This suggests that the representation choice in VHGR is inseparable from latency budgets.

Classical real-time systems relied on more explicit geometry. An early media-control pipeline used skin-color detection, morphology, Viola–Jones face detection, a particle filter for hand tracking, a distance threshold between face and hand, and hand features derived from contour geometry and Fourier descriptors, followed by a Mixture of Experts classifier (Azad et al., 2014). In this older formulation, the representational emphasis is on stable handcrafted invariants rather than latent embeddings.

3. Learning architectures and methodological evolution

The methodological arc of VHGR moves from hand-crafted appearance and geometry toward learned spatial, temporal, and graph-structured representations. The 2013 survey foregrounded skin-color segmentation, binarization, fingertip localization, contour descriptors, Gabor features, PCA, ANN, HMM, RNN, fuzzy logic, genetic algorithms, Bayesian classifiers, and SVMs, with a strong emphasis on fingertip-based appearance modeling and explicit preprocessing pipelines (Chaudhary et al., 2013). By contrast, later surveys describe a mature deep-learning phase dominated by 2D CNNs for static gestures, 2D CNN plus temporal models and 3D CNNs for dynamic gestures, GCNs for skeleton graphs, and transformer-based or hybrid CNN-transformer systems for longer-range dependencies (Shin et al., 2024).

The contemporary taxonomy is architecture-sensitive. For static VHGR, the 2025 review reports that strong single-stage RGB CNNs and attention-enhanced networks often outperform more elaborate multi-stage pipelines on standard datasets, although multi-stage systems remain useful when the hand is small, far away, or embedded in clutter (Foteinos et al., 6 Jul 2025). For isolated dynamic VHGR, the same review identifies three main families: spatiotemporal backbones such as 3D CNNs and $(2+1)D$ CNNs; spatial-plus-temporal pipelines such as 2D CNN feature extractors followed by LSTM, GRU, TCN, or transformer modules; and short-term spatiotemporal plus long-term temporal designs for longer clips (Foteinos et al., 6 Jul 2025). Skeleton-based systems frequently adopt GCN or spatiotemporal GCN backbones because joints and bones induce an explicit graph.

One influential recent direction is few-shot customization rather than fixed-vocabulary recognition. A 2024 system uses hand skeleton keypoints from Apple Vision or MediaPipe, represents each hand by 21 landmarks and two-handed gestures by 42 keypoints, processes 104 frames per sample, and employs a graph transformer adapted from hypergraph transformer work for skeleton action recognition. The adaptation stage uses MAML with stochastic gradient descent at learning rate 0.025, together with meta-augmentation that synthesizes new two-handed classes by combining one-handed gestures. The paper reports one-demonstration averages of 97.5% for two new gestures, 91% for three new gestures, and 91% for four new gestures in the same-view setting, and an average F1 score of 73% for rejecting out-of-vocabulary gestures (Shahi et al., 2024). A plausible implication is that pose-based meta-learning changes the problem setting from closed-set recognition to user-specific vocabulary acquisition.

Alternative non-neural formulations remain relevant, especially when training data are limited. The $\ell_{1-2}$-regularized sparse representation model treats a test image $b$ as approximately generated by a sparse coefficient vector $x$ over a class dictionary $A$, solving

$X = \{x_1, x_2, \ldots, x_t\}$0

with ADMM. On the reported binary dataset, HOG features reach 0.9667 recognition rate with 250 atoms; on the gray-scale dataset, HOG reaches 0.9972 with 200 atoms (Qin et al., 2021). This suggests that VHGR is not reducible to deep learning alone; sparse coding remains competitive in low-data regimes.

Ensembling is another nontrivial branch of the literature. An ensemble-based CNN system first detects the hand using background separation and binary thresholding, extracts the largest contour, estimates the palm center with a distance transform, resizes the segmented hand to 64 × 64, and trains simplified AlexNet-like, VGGNet-like, and GoogLeNet-like models in parallel, averaging their scores for the final decision. On the reported datasets it achieves 99.80% on Dataset-1, 96.50% on Dataset-2, and 99.70% / 99.76% on the self-constructed dataset (Sen et al., 2022). The result is methodologically significant because it addresses variance and overfitting within a conventional supervised classification setting rather than through more recent self-supervised or transformer-based mechanisms.

4. Datasets, annotation regimes, and evaluation

Dataset design is a decisive variable in VHGR because performance depends strongly on background diversity, user diversity, distance variation, annotation granularity, and whether the task is isolated or continuous. Large contemporary surveys therefore treat datasets not as passive benchmarks but as methodological constraints that shape feasible models and metrics (Linardakis et al., 21 Jan 2025, Foteinos et al., 6 Jul 2025).

Dataset	Regime	Key properties
HaGRID	Static device-control VHGR	554,800 images, 18 gesture classes + 1 extra “no gesture” class, 37,583 subjects, at least 37,583 unique scenes (Kapitanov et al., 2022)
IPN Hand	Continuous RGB VHGR	4,218 gesture instances, 800,491 RGB frames, 50 distinct subjects, 28 diverse scenes, 13 gesture classes plus No-gesture (Benitez-Garcia et al., 2020)
JESTER	Isolated dynamic VHGR	148,092 videos, 27 classes (Linardakis et al., 21 Jan 2025)
EgoGesture	Isolated dynamic VHGR	24,161 videos, 83 classes (Linardakis et al., 21 Jan 2025)
AUTSL	Isolated sign-language VHGR	RGB-D + RGB + skeleton, 226 classes (Linardakis et al., 21 Jan 2025)
PHOENIX-2014-T	Continuous sign-language VHGR	RGB, sentence-level CSLR/SLT benchmark (Foteinos et al., 6 Jul 2025)

HaGRID is representative of the move toward device-control realism. It was created because prior static gesture datasets were too homogeneous in background, subject population, lighting, scene context, or subject-to-camera distance, and because many lacked annotations suitable for full-frame hand detection. HaGRID contains mostly FullHD RGB images, uses COCO-format normalized bounding boxes, permits one or two hand boxes per image, and includes the extra “no gesture” class in 108,056 images. Its 18 static gestures were explicitly selected so that they can be composed into dynamic gestures such as swipe and drag-and-drop through a queue of recent frame-level events with sequence, timing, and positional constraints (Kapitanov et al., 2022).

IPN Hand was constructed to benchmark real-time continuous recognition rather than only isolated clips. It records gestures without transition states, explicitly includes natural hand movements as non-gesture behavior, and exhibits strong variability in gesture duration, with minimum gesture length 9 frames, maximum 650 frames, mean video duration 4,002.5 frames, and mean gesture duration 140 frames. The paper reports that the state-of-the-art ResNeXt-101 model decreases about 30% accuracy when moved from an easier benchmark setting to IPN Hand, using this drop to argue that continuous real-world VHGR is substantially harder than isolated recognition (Benitez-Garcia et al., 2020).

Evaluation metrics vary by task. HaGRID uses F1-score for classification and mAP for detection, reflecting its dual role as a detection and recognition dataset (Kapitanov et al., 2022). The 2025 survey systematizes this further: static and isolated dynamic tasks are commonly evaluated with micro- and macro-averaged accuracy, precision, recall, Jaccard index, and $X = \{x_1, x_2, \ldots, x_t\}$1, whereas continuous recognition uses sequence metrics such as

$X = \{x_1, x_2, \ldots, x_t\}$2

and Edit-Accuracy (Foteinos et al., 6 Jul 2025). IPN Hand instead adopts Levenshtein accuracy for continuous real-time HGR, emphasizing that online evaluation requires sequence consistency rather than only framewise correctness (Benitez-Garcia et al., 2020).

A broader dataset ecology now spans semaphoric command sets, isolated and continuous sign language corpora, egocentric gesture datasets, and special-purpose benchmarks. The 2025 survey lists MS-ASL, WLASL, BSL-1K, AUTSL, BosphorusSign22k, CSL500, PHOENIX-2014, CSL-Daily, How2Sign, BOBSL, OpenASL, and others, while the 2024 survey emphasizes that many of the most reused datasets date from 2016–2020, with ASL-oriented classification benchmarks especially prominent and HO-3D particularly important for estimation (Linardakis et al., 21 Jan 2025, Foteinos et al., 6 Jul 2025). This suggests that VHGR benchmarking is now bifurcated between discrete classification benchmarks and sequence-heavy sign-language corpora.

5. Real-time systems and application domains

The application layer of VHGR is unusually heterogeneous. The literature includes command-and-control systems, sign-language interfaces, telepresence, mediated social touch, assistive rehabilitation, media control, VR/AR interaction, smart-home control, gaming, and UAV piloting (Foteinos et al., 6 Jul 2025, Shao et al., 2024). The same methodological core therefore appears in markedly different latency, robustness, and safety envelopes.

A representative RGB-D real-time system for natural user interfaces reports 95% overall accuracy across five static gestures—One Finger, Two Fingers, Thumbs-up, Shaka, and OK—with per-class accuracies of 96.67%, 90.34%, 90.0%, 93.3%, and 99.3%, respectively. The paper further reports 98.09% training accuracy, 98.42% validation accuracy, 95% accuracy in both static and dynamic backgrounds, and average processing speed or latency of 135 ms. It positions the system specifically for telepresence and rehabilitation, while acknowledging that recognition latency implies the gesture may need to be held for that duration (McKinnon et al., 2022).

UAV control sharpens the real-time requirement. A 2025 system streams video from a UAV front camera, localizes the hand with MediaPipe or segmentation baselines, and ultimately adopts a landmark-based method augmented with YOLOv4 for longer-range detection. The custom dataset contains 6 gesture classes with 1,500 images per class, captured under varied lighting and varying distances. The paper reports 96.14% landmark-based recognition accuracy, about 90% accuracy up to 5 m, false positive rate below 2%, and less than 5% degradation under lighting variation. With YOLOv4, the effective range extends from 1–5 m to 1–10 m. Edge offloading yields average edge processing latencies of 22 ms, 28 ms, and 35 ms on different CPUs, while end-to-end sensing-to-control latency is about 150 ms; reported flight metrics include 92% path tracking accuracy and 96% command completion success rate (Abdalla et al., 22 May 2025).

Social and affective interaction constitutes another branch. The Kinect-based “social gestures” study treats gesture recognition as the perception layer of a mediated social touch system, with target classes including high-five, handshake, hug, shoulder pat, and holding hands. The authors trained right- and left-side variants such as R5, L5, RH, LH, RS, and LS, treated hug as a composite of two simultaneous side-touch gestures, and reported an overall recognition rate of about 89% for the six retained gestures. The paper emphasizes that future work is still needed before practical deployment in videochat-based mediated social touch (Yao et al., 2017).

Earlier HCI deployments targeted local device control. A face-and-hand media-player controller used five command gestures—Stop music, Play music, Next music, Volume up, and Volume down—and reported 99.20% recognition in its best Mixture of Experts configuration, plus 99.40% correctness on a static ASL database and 100% accuracy in the face-recognition stage (Azad et al., 2014). The result is methodologically constrained by skin-based preprocessing and thresholded interaction zones, but it illustrates the longstanding HCI lineage of VHGR.

Practical tooling has also emerged. GRLib is explicitly designed as an open-source Python library that supports static and dynamic gestures from an RGB camera, can be trained on existing datasets, and outperforms MediaPipe Solutions on three diverse real-world datasets. Its dynamic module reports 79% overall recall and 2083 ± 13 FPS for dynamic gesture detection alone on the specified laptop configuration (Warchocki et al., 2023). At the other end of the personalization spectrum, the one-shot customization system was deployed in an iPad app that lets users define gestures, inspect landmarks, delete bad samples, train a model on-device in under a minute, and preview real-time recognition; across the study sessions, it recognized 74 custom gestures with only three failures (Shahi et al., 2024). This suggests that deployment trajectories in VHGR now include both library-based modularity and end-user customization.

6. Persistent challenges, common misconceptions, and future directions

A central misconception in VHGR is that high isolated-clip accuracy implies robustness in continuous or real-world conditions. IPN Hand was designed specifically to challenge that assumption by removing transition states and adding natural non-gesture hand motion, and its benchmark results show that continuous recognition is much harder than isolated classification; the paper’s headline example is the approximately 30% accuracy drop of ResNeXt-101 on the more realistic dataset (Benitez-Garcia et al., 2020). The 2024 and 2025 surveys generalize this point, identifying continuous recognition as one of the main unresolved gaps in the field, primarily because it requires simultaneous segmentation and recognition under co-articulation, ambiguous boundaries, and online constraints (Shin et al., 2024, Foteinos et al., 6 Jul 2025).

A second misconception is that skeleton or landmark representations remove most nuisance factors. The surveys indeed describe skeleton-based VHGR as compact, interpretable, and more robust to clutter, lighting, and partial occlusion than raw RGB. However, they also note recurrent failure modes such as missing keypoints, skipped frames, and zero-valued joints (Shin et al., 2024). The customization paper makes the same dependency explicit: errors from off-the-shelf hand pose estimation propagate through the entire pipeline, and the present system still operates on isolated gestures rather than continuous hand motion (Shahi et al., 2024). A plausible implication is that skeletonization trades appearance variance for upstream estimator brittleness rather than eliminating nuisance factors outright.

Multimodal fusion is often presented as a remedy, and the surveys do report that multimodal vision-centric systems usually outperform single-modality methods because RGB, depth, and skeleton compensate for one another’s weaknesses. Yet the same reviews also emphasize the costs: synchronization problems, higher computational complexity, and more difficult fusion design (Shin et al., 2024, Linardakis et al., 21 Jan 2025). The 2025 survey therefore treats fusion strategy itself—early, late, intermediate, or multi-level—as a first-order modeling decision rather than an implementation detail (Foteinos et al., 6 Jul 2025).

Current research directions follow directly from these bottlenecks. The major surveys converge on the need for larger and more diverse signer-independent and user-independent datasets, stronger temporal modeling for continuous recognition, lightweight architectures for mobile and edge deployment, more reliable pose estimation, better cross-user and cross-domain generalization, few-shot and zero-shot learning, multiview augmentation, and richer multimodal fusion (Shin et al., 2024, Linardakis et al., 21 Jan 2025, Foteinos et al., 6 Jul 2025). The customization literature adds accessibility and personalization concerns: gesture vocabularies should be user-defined when possible, but existing studies still underrepresent people with physical impairments and report some training instability in MAML-style adaptation (Shahi et al., 2024). Taken together, these directions suggest that the next phase of VHGR will be judged less by peak benchmark accuracy alone than by robustness under continuous input, personalization, deployment constraints, and the ability to operate reliably outside controlled scenes.