GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations

Abstract: Vision-Language-Action (VLA) models have shown strong potential for general-purpose robot manipulation by unifying perception and action. However, existing VLA systems primarily rely on textual instructions and struggle to resolve spatial ambiguity in complex scenes with multiple similar objects. To address this limitation, we introduce gesture as a parallel instruction modality and propose a Gesture-aware Vision-Language-Action model (GesVLA). Our approach encodes gesture features directly into the latent space, enabling them to participate in both high-level reasoning and low-level action generation, and adopts a dual-VLM architecture to achieve tight coupling between gesture representations and action policies. At the data level, we construct a scalable gesture data generation pipeline by rendering hand models onto real-world scene images. This reduces the sim-to-real visual gap while producing rich data with diverse motion patterns and corresponding pointing annotations. In addition, we employ a two-stage training strategy to equip the model with both gesture perception and action prediction capabilities. We evaluate our approach on multiple real-world robotic tasks, including a controlled block manipulation task for validation and more practical scenarios such as product and produce selection. Experimental results show that incorporating gesture consistently improves target grounding accuracy and human-robot interaction efficiency, especially in complex and cluttered environments. Project page: https://gwxuan.github.io/GesVLA/.

• The paper introduces a dual-VLM architecture that fuses gesture trajectories with language, significantly enhancing spatial intent grounding over text-only methods.
• The method uses a two-stage training pipeline with semi-synthetic data and real-robot demonstrations, achieving an 83.3% task success rate and 94.3% spatial grounding accuracy.
• The integration of latent gesture embeddings and robust data augmentation techniques demonstrates practical improvements in human-robot interaction in cluttered and ambiguous settings.

GesVLA: Gesture-Aware Embodied Manipulation via Latent Multimodal Fusion

Motivation and Problem Statement

GesVLA introduces a paradigm shift in Vision-Language-Action (VLA) modeling by explicitly incorporating gesture as a parallel instruction modality for robotic manipulation. Existing VLA frameworks predominantly rely on language for goal specification. However, linguistic instructions alone are inherently limited in spatial referential precision, especially in scenarios with object ambiguity due to clutter or similarity. GesVLA addresses this spatial grounding challenge by embedding pointing gestures directly into the latent space, thus enabling seamless multimodal intent representation and action generation.

Figure 1: Integrating gestures reduces ambiguity in target specification compared to text-only instructions, enabling more accurate and efficient task execution.

Dual-VLM Architecture and Latent Gesture Embedding

GesVLA leverages a dual-VLM architectural design to decouple intent reasoning from online perception and action generation while maintaining tight latent coupling between modalities. The intent reasoning VLM ($\text{VLM}_{\text{int}}$) fuses gesture trajectories and language to infer target objects/locations. Gestures are processed as sequences of hand keypoints projected into 12-dimensional vector representations via an MLP, capturing spatial directionality and pose semantics. The online perception VLM ($\text{VLM}_{\text{per}}$) receives visual observations and cross-attends to the intent VLM’s latent outputs, ensuring the propagation of inferred intent without discretization bottlenecks.

Action generation is executed via a flow-matching policy, which iteratively denoises a trajectory chunk conditioned on both perception and intention context, producing precise, smooth manipulator trajectories.

Figure 2: The dual-VLM architecture models gesture-conditioned intent reasoning and latent cross-attention-driven action generation.

Data Engine and Two-Stage Training Pipeline

Robust learning of gesture-conditioned intent in real-world settings is impeded by the scarcity of gesture-action-aligned datasets with precise spatial labels. GesVLA’s scalable gesture data engine synthesizes hand trajectories rendered onto real RGB-D backgrounds, producing multimodal samples with language instructions and ground-truth pointing annotations. Jittering and augmentation of hand appearance and pose facilitate sim-to-real transfer, while the pipeline efficiently scales to new scenes, tasks, and motion patterns.

Training proceeds in two stages: (1) semi-synthetic data is used to pre-train the intent reasoning VLM, allowing spatial reasoning with gesture-language fusion; (2) real-robot demonstrations enable joint training of perception and policy modules with $\text{VLM}_{\text{int}}$ frozen, ensuring downstream policy learning does not suffer from synthetic data bias.

Figure 3: Semi-synthetic gesture data engine and staged training pipeline enable spatial supervision and modality specialization.

Experimental Evaluation: Real-World Manipulation and Reasoning

GesVLA is validated in three manipulation tasks: Pick-and-Place Block, Select Jelly, and Select Fruit/Vegetable, each requiring multimodal instruction decoding under object ambiguity. The evaluation setup exploits multiple camera views, with gesture signals provided by experimenters and robot actions executed autonomously.

GesVLA achieves 83.3% average task success rate, significantly outperforming text-only VLA (31.7%), MLLM+VLA (31.7%), and geometric pipeline (41.7%). The intent reasoning VLM obtains 94.3% accuracy in real-world spatial grounding, demonstrating robust sim-to-real transfer and outperforming geometric and native MLLM baselines by large margins.

Figure 4: Real-world manipulation tasks and experimental setup for multimodal instruction-following evaluations.

Ablation and Qualitative Analyses

Ablation studies confirm the critical importance of coordinate jitter, hand visual augmentation, latent gesture projection, and staged optimization strategies for both reasoning and manipulation. Removing gesture embedding or data augmentation leads to pronounced performance degradation, underscoring the necessity of explicit spatial encoding and robustness mechanisms. Freezing the intent VLM during policy training yields optimal downstream performance, demonstrating that semi-synthetic gesture data suffices for intent learning without requiring additional real-robot gesture supervision.

Qualitative comparisons reveal that baseline MLLMs select visually proximal objects to the fingertip, failing to reason about pointing directionality. In contrast, GesVLA consistently grounds target selection along the intended spatial axis, even in cluttered or complex scenes.

Figure 5: On real-world instructions, GesVLA robustly follows pointing direction, outperforming MLLMs that rely on fingertip proximity.

Practical and Theoretical Implications

By embedding gestures at the feature level, GesVLA advances spatial intent specification in embodied systems. The architectural decoupling of intent and perception under cross-modal latent coupling yields inference-efficient policies that generalize in real and synthetic domains. The scalable data pipeline demonstrates that semi-synthetic approaches, augmented with real-scene backgrounds and hand diversity, bridge the sim-to-real gap in spatial referencing.

Practically, GesVLA enables more natural and intuitive human-robot interaction, significantly enhancing manipulation accuracy in ambiguous environments. Theoretically, this framework substantiates the utility of continuous gesture feature fusion for latent multimodal intent representation, opening avenues for explicit spatial reasoning and integrated instruction modalities beyond language.

Future Directions

The current system is evaluated on pointing gestures and simple manipulation tasks. Extending GesVLA to richer gestural vocabularies (e.g., dynamic, referential, multi-step instructions), interactive multi-agent settings, and hierarchical task compositions represents key directions for subsequent research. Incorporating additional modalities (e.g., gaze or touch) and exploring scalable multi-agent datasets may further enhance spatial communication and collaborative manipulation in open-world environments.

Conclusion

GesVLA establishes gesture as a first-class modality in embodied VLA models, leveraging a dual-VLM architecture and scalable data synthesis for robust spatial intent grounding and manipulation. The approach achieves marked improvements in grounding accuracy and manipulation success rates in real-world conditions, substantiating the necessity of multimodal latent fusion for effective AI-driven human-robot interaction.