GesVLA: Gesture-Aware Vision-Language-Action Model

• GesVLA is a gesture-aware VLA system that directly embeds gesture features into a latent control space to resolve spatial ambiguities in cluttered scenes.
• It employs a dual-VLM architecture where gesture-conditioned intent reasoning and cross-attentive perception enable robust action generation.
• Experimental evaluations reveal substantial improvement in robot manipulation success, highlighting the benefits of semi-synthetic gesture pretraining and a two-stage training protocol.

Searching arXiv for GesVLA and closely related gesture-aware VLA work. Gesture-aware Vision-Language-Action Model (GesVLA) denotes a class of VLA systems in which human gesture is treated as a parallel instruction modality rather than as an external heuristic or a post-hoc textual proxy. In the specific formulation introduced in “GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations,” gesture features are encoded directly into the latent space so that they participate in both high-level intention reasoning and low-level action generation, addressing the failure mode of text-only VLA policies in cluttered scenes where instructions such as “pick this and put it there” underspecify the spatial target (Guo et al., 21 May 2026). GesVLA is therefore positioned at the intersection of multimodal grounding, embodied policy learning, and human-robot interaction: it uses gesture-conditioned intent reasoning, cross-attentive perception, and a flow-based action expert to convert deictic human signals into robot manipulation behavior.

1. Problem setting and conceptual basis

The motivating observation behind GesVLA is that existing VLA systems primarily rely on textual instructions and often fail to resolve spatial ambiguity in complex scenes with multiple similar objects. In such cases, VLMs tend to choose the object closest to the fingertip in image space or rely on brittle geometric heuristics, leading to misidentification. GesVLA addresses this by treating deictic gestures, especially pointing and indicating “there,” as a natural complementary modality for spatial disambiguation, and by encoding gesture as a first-class signal directly in the policy latent space (Guo et al., 21 May 2026).

The policy is formalized as

$$\pi(a_{1:K} \mid s_t, O_t, T, G) = F_\theta(x_0, s_t, K_{\text{per}}, V_{\text{per}}),$$

where $O_t$ denotes scene observations, $T$ the language instruction, $G$ the gesture signal, $s_t$ the robot state, and $F_\theta$ a flow-based action expert that denoises an initial noisy action chunk $x_0$ into a smooth trajectory $a_{1:K}$ (Guo et al., 21 May 2026). This formulation makes gesture conditioning intrinsic to the control stack rather than an auxiliary preprocessing step.

Within the broader VLA literature, the gesture modality remains relatively underdeveloped. A survey of VLA models notes that there are few explicit “gesture-aware VLA” systems, even though teleoperation pipelines, egocentric human-video pretraining, and multimodal fusion mechanisms already provide many of the required building blocks (Li et al., 21 Aug 2025). This situates GesVLA as a concrete instantiation of a research direction that had been technically plausible but only sparsely realized.

2. Latent gesture encoding and dual-VLM coupling

GesVLA extracts gesture features from gesture video $G$ by detecting brief motion stagnations typical of pointing and selecting keyframes

$\{g_i\}_{i=1}^F = S(\mathrm{Pose}(G)).$

For each keyframe, MediaPipe extracts four keypoints—the wrist and three joints of the index finger—each represented by $O_t$0, where $O_t$1 is per-pixel depth. These are concatenated into $O_t$2 and projected by an MLP encoder $O_t$3 into gesture tokens

$O_t$4

The resulting tokens are then fed into an intent VLM rather than being discretized into text-only prompts (Guo et al., 21 May 2026).

The architecture is explicitly dual-VLM. The intent VLM, denoted $O_t$5, takes gesture tokens $O_t$6 and language $O_t$7 and produces reasoning outputs $O_t$8 together with cached per-layer KV states: $O_t$9 The perception VLM, $T$0, then consumes scene observations $T$1, language $T$2, and the reasoning outputs $T$3, while cross-attending to the latent cache from $T$4: $T$5 Cross-attention is implemented with standard transformer projections,

$T$6

followed by

$T$7

This mechanism injects gesture-conditioned intent directly into perception features and, via $T$8, into action conditioning (Guo et al., 21 May 2026).

The interaction is asymmetric and computationally structured. Information flows unidirectionally as $T$9 action expert; $G$0 does not attend to downstream modules. KV-cache reuse yields a compute ratio of $G$1, where intent is computed once, perception runs each control step, and the action expert performs multiple denoising iterations per step (Guo et al., 21 May 2026). This design is central to the claim that gesture can remain tightly coupled to action without collapsing latency budgets.

3. Semi-synthetic data engine and training protocol

GesVLA relies on a semi-synthetic gesture data generation pipeline that renders synthetic hands over real RGB-D scenes while preserving visual realism and precise 3D target labels. The pipeline begins with object proposal and target sampling using GroundingDINO; for a selected 2D bounding-box center $G$2, a 3D target point is recovered via camera intrinsics: $G$3 Pointing motion is then generated by sampling a unit approach direction $G$4 and using

$G$5

For multi-target transitions, a parabolic lifting term is added: $G$6 Hand meshes are rendered onto original RGB frames to produce multi-view videos, depth-consistent overlays, and gesture clips, along with language $G$7 and supervision $G$8 (Guo et al., 21 May 2026).

The resulting semi-synthetic corpus contains about 16k samples; 300 RGB-D scenes are used to construct semi-synthetic training for intent reasoning, and 88 held-out scenes are reserved for real-world evaluation. Diversity is introduced through randomized jitter, varied hand assets and appearance, multi-target orders, and diverse object categories and layouts (Guo et al., 21 May 2026).

Training is explicitly two-stage. In Stage 1, $G$9 is pretrained on the semi-synthetic gesture dataset while other modules are frozen. Gesture-conditioned intent pretraining uses a single autoregressive sequence that interleaves text and discretized coordinate tokens, optimized with

$s_t$0

In Stage 2, $s_t$1 is frozen, and $s_t$2 together with the action expert are trained on real robot demonstrations (Guo et al., 21 May 2026).

Optimization uses AdamW with a warmup cosine learning-rate schedule and exponential moving average of parameters. $s_t$3 is initialized from PaliGemma, $s_t$4 from the VLM backbone used in $s_t$5, and the action expert is trained from scratch (Guo et al., 21 May 2026). A key empirical point is that joint training of all modules performs worse than the two-stage strategy with frozen $s_t$6, which suggests that the semi-synthetic gesture pretraining transfers sufficiently well to justify freezing during policy learning.

4. Action generation, control semantics, and operational behavior

The action expert in GesVLA is flow-based. Starting from Gaussian noise $s_t$7, it produces a smooth action chunk according to

$s_t$8

Training uses a flow-matching objective over continuous actions: $s_t$9 where $F_\theta$0 encapsulates $F_\theta$1, and robot state $F_\theta$2 (Guo et al., 21 May 2026). Multi-step denoising yields smooth trajectories, and the predicted action chunk can be executed either in full or as a truncated prefix under closed-loop control.

The gesture modality in the reported system is pointing. The pipeline supports multi-target scenarios and sequential pointing orders, uses 2D image coordinates and depth for wrist plus three index-finger joints per keyframe, and can operate in multi-view settings with a designated right-side “gesture view” (Guo et al., 21 May 2026). The intent head predicts discretized coordinate tokens compatible with autoregressive generation, while visual prompts $F_\theta$3 render target highlights directly on images.

A common simplification would be to describe GesVLA as merely converting gesture into extra text. That description is incomplete. The system does produce textual pointer descriptions and visual prompts from $F_\theta$4, but its central claim is that gesture-conditioned intent also propagates through latent KV states and cross-attention into $F_\theta$5 and onward to the action expert, thereby avoiding a textual-only bottleneck (Guo et al., 21 May 2026).

The deployed hardware emphasizes that the model is a robot policy rather than a perception-only benchmark. Experiments use a 7-DOF ARX5 arm and three cameras at 30 fps—global, right-side, and gripper-mounted—with inference on 1× RTX 4090. $F_\theta$6 is evaluated from the right-side gesture view (Guo et al., 21 May 2026).

5. Experimental evaluation and ablation structure

GesVLA is evaluated on controlled intent reasoning and real-robot manipulation. The real-world task suite comprises Pick-and-Place Block, Select Jelly, and Select Fruit/Vegetable (Guo et al., 21 May 2026). Intent reasoning is measured on 88 real-world samples.

On intent reasoning, the reported baselines are a prompted MLLM and a geometric pipeline. Baseline-1 achieves 38.6% accuracy and 61.4% progress score; Baseline-2 achieves 59.1% accuracy and 78.4% progress score; GesVLA $F_\theta$7 achieves 94.3% accuracy and 97.2% progress score (Guo et al., 21 May 2026). The qualitative interpretation reported with these numbers is that the MLLM often chooses the closest fingertip object, whereas GesVLA infers the target along the pointing direction.

Real-robot manipulation is reported over 20 trials per task, using average success. Text-only VLA and MLLM+VLA each obtain 31.7%; geometric pipeline+VLA obtains 41.7%; GesVLA with decoupled VLMs obtains 61.7%; and full GesVLA obtains 83.3% overall (Guo et al., 21 May 2026). Task-wise, full GesVLA reaches 95.0% on Pick-and-Place Block, 75.0% on Select Jelly, and 80.0% on Select Fruit/Vegetable.

The ablation suite is unusually informative because it separates data, training strategy, and architecture. In the data engine, removing coordinate jitter reduces intent accuracy to 42.0%, removing hand augmentation yields 76.1%, and the full pipeline reaches 94.3%. In the training design, removing the gesture MLP gives 84.1%, removing data augmentation gives 89.8%, and the full design reaches 94.3%. In optimization strategy, joint training of all modules yields 45.0%, two-stage training with $F_\theta$8 unfrozen yields 80.0%, and two-stage training with $F_\theta$9 frozen yields 83.3%, which is the best reported result. In the architectural ablation, removing the visual prompt yields 66.6%, adding a text prompt yields 83.3%, and the full model with cross-attention also yields 83.3%, leading the authors to state that cross-attention already conveys intent effectively (Guo et al., 21 May 2026).

These results support two factual conclusions from the paper. First, gesture-conditioned intent reasoning is not solely a perception add-on; it changes downstream manipulation success materially. Second, synthetic data design and freezing strategy are not minor implementation details but primary determinants of transfer quality.

6. Relation to adjacent frameworks, limitations, and future directions

GesVLA belongs to a broader but still emerging family of gesture-conditioned embodied systems. “SignVLA” formulates gloss-free sign language-driven robotic manipulation, directly mapping visual sign gestures to semantic instructions and then to actions. Its reported system focuses on real-time alphabet-level finger-spelling, using MediaPipe Hands, temporal smoothing, lexical refinement via Levenshtein distance, and a two-rate VLA stack with GR00T N1 and a DiT controller. In sign-conditioned manipulation, temporal smoothing raises success from 79.3% to 84.7% with improved stability (Tan et al., 26 Feb 2026). Relative to GesVLA, SignVLA addresses a different gesture regime: finger-spelled lexical communication rather than deictic spatial disambiguation.

“GIVE” enhances pre-trained VLA models with human gesture understanding without architectural modifications. It injects gesture through a visual pathway that overlays hand skeletons and fingertip rays onto observations and a semantic pathway that appends phase-level descriptions such as pointing or open-palm intent. In its real-world HRI setting, Identify SR increases from 46.7% to 86.7%, and Handover SR from 0.0% to 80.0% (Liu et al., 11 Jun 2026). “GTA-VLA” provides another neighboring formulation by allowing optional spatial priors—points, boxes, and traces—to condition a spatial-visual Chain-of-Thought, achieving 81.2% success on the SimplerEnv WidowX benchmark and showing large gains from single visual interactions in ambiguous scenes (Ling et al., 13 May 2026). A plausible implication is that gesture-aware VLA currently spans at least three integration styles: latent token fusion, pixel-space overlay plus semantic prompting, and serialized spatial priors.

GesVLA also has explicit limitations. The current system focuses on pointing; richer gestures such as sweeps, grasp-shape cues, and bimanual cues are not modeled. Severe occlusions or poor hand visibility degrade keypoint extraction. Keyframe selection reduces temporal redundancy but also drops temporal nuance, and the paper identifies sequence-level temporal models for gestures as a direction for improved disambiguation and intent persistence. Generalization to more objects, tasks, and harder environments is expected to require larger-scale semi-synthetic data and stronger sim-to-real techniques, including physically plausible hand-object occlusions (Guo et al., 21 May 2026).

A broader caution comes from gesture-misrecognition analysis in autonomous-driving scenes, where recurring causes of failure include visibility, occlusion, temporal dynamics, pedestrian behavior variability, and environmental conditions such as glare, shadows, rainfall, and edge-of-frame viewpoints (Tran et al., 9 Aug 2025). Although that study is outside robot manipulation, it suggests that gesture-aware VLA systems will likely need stronger context modeling and uncertainty-aware interpretation if they are to remain reliable beyond controlled tabletop settings.

In this sense, GesVLA is best understood not as a generic replacement for language-conditioned control, but as a specific answer to spatial ambiguity in embodied manipulation: gesture is embedded as a latent control-relevant signal, intent is reasoned separately from online perception, and action is generated by a flow-based expert conditioned on gesture-informed perception features. The current results show that this design is effective for pointing-based target grounding in clutter, while the surrounding literature indicates clear paths toward richer gesture vocabularies, temporal modeling, 3D grounding, and uncertainty-aware interaction (Guo et al., 21 May 2026).