GazeVLA: Learning Human Intention for Robotic Manipulation

Abstract: Embodied foundation models have achieved significant breakthroughs in robotic manipulation, yet they still depend heavily on large-scale robot demonstrations. Although recent works have explored leveraging human data to alleviate this dependency, effectively extracting transferable knowledge remains a significant challenge due to the inherent embodiment gap between human and robot. We argue that the intention underlying human actions can serve as a powerful intermediate representation for bridging this gap. In this paper, we introduce a novel framework that explicitly learns and transfers human intention to facilitate robotic manipulation. Specifically, we model intention through gaze, as it naturally precedes physical actions and serves as an observable proxy for human intent. Our model is first pretrained on a large-scale egocentric human dataset to capture human intention and its synergy with action, followed by finetuning on a small set of robot and human data. During inference, the model adopts a Chain-of-Thought reasoning paradigm, sequentially predicting intention before executing the action. Extensive evaluations in simulation and real-world settings, across long-horizon and fine-grained tasks, and under few-shot and robustness benchmarks, show that our method consistently outperforms strong baselines, generalizes better, and achieves state-of-the-art performance.

• The paper introduces a novel intention-centric framework that uses human gaze as a bottleneck to transfer manipulation skills across embodiments.
• It employs a two-stage training approach with large-scale human pretraining followed by targeted robot post-training to enhance cross-domain performance.
• Key results include state-of-the-art performance in fine-manipulation and out-of-distribution tasks, with over 85% success in real-world experiments.

Learning Human Intention for Robotic Manipulation: An Analysis of GazeVLA

Introduction

GazeVLA presents a framework for cross-embodiment knowledge transfer in robotic manipulation by leveraging explicit modeling of human intention via gaze as an intermediate representational bottleneck. The motivation is to fundamentally address the data scarcity bottleneck in embodied foundation models, which require large-scale real-robot demonstrations, by exploiting rich, scalable, egocentric human data as a primary source for pretraining. GazeVLA employs a Vision-Language-Intention-Action (VLIA) model, trained first on large-scale multimodal human videos, then post-trained with a small set of robot and human data, achieving intention-informed manipulation without requiring intention annotations from robots. During inference, the intention-action chain reflects a causal behavioral decomposition which manifests in both strong performance and generalization.

Figure 1: GazeVLA learning and inference pipeline: large-scale human pretraining, post-training on robot data, and intention-action causal chain realization.

Dataset Construction and Multimodal Supervision

A core asset of the framework is its diverse, large-scale egocentric dataset, aggregated from 13 established data sources, encompassing RGB, hand, and gaze signals, with consistent spatial registration and multimodal validity masking. The dataset features over 150 million frames, capturing broad variation in backgrounds, tasks, and object interactions. Dedicated post-training data is acquired via Pupil Neon AR glasses for synchronized RGB, high-frequency gaze, and dual-hand motion trajectories using HaWoR.

Figure 2: Multimodal egocentric dataset with hand/gaze annotations, background/action diversity, and robust segmentation.

Data processing unifies coordinate systems, segments videos by atomic action, normalizes linguistic supervision, aligns local reference frames, reconstructs human motion in the MANO parameterization (fingertip, wrist pose, and 6-DoF), and filters intentions (gaze) for quality, resulting in action space $a \in \mathbb{R}^{48}$ and intention $i \in \mathbb{R}^2$ per clip, temporally discretized.

Model Architecture and Chain-of-Thought Paradigm

The core GazeVLA model is built on the PaliGemma vision-language backbone (SigLIP vision encoder + Gemma-2B LM). For each timestep, VLIA accepts language instructions, egocentric sensory input, and agent state. Critically, the forward process sequentially infers intention tokens (via discretized gaze) autoregressively, followed by flow-matched generation of continuous actions, enforcing a strict $$\text{Perception} \rightarrow \text{Intention} \rightarrow \text{Action}$$ chain:

Figure 3: VLIA model: instruction/image conditioning, intention prediction as token, followed by action generation conditioned on inferred intention.

The decoupling enables intention to act as an explicit bottleneck, constraining action generation to the inferred task-relevant target, addressing the conventional "what" (action) versus "why" (intention) dichotomy in imitation learning. Training employs next-token intention loss and conditional flow-matching for action, with intentional synchronization of vision and gaze augmentation to avoid center bias.

Pretraining and Adaptation

A two-stage regime is deployed: (1) Large-scale human egocentric pretraining with partial parameter freezing to preserve foundation model capacity and optimize action experts, (2) Post-training with small human and robot datasets, using 1:1 sampling. Notably, intention annotations are only available for human data; robot data contributes action supervision exclusively. Alignment between modalities is maintained via zero-padding and sequence normalization.

Intention-driven Manipulation and Policy Evaluation

GazeVLA’s effectiveness is validated through synthetic and real-world manipulation tasks. Pretraining results indicate task-conditioned intention and action can be correctly predicted even from visually identical scenes with differing instructions, confirming successful multimodal integration and reasoning.

Figure 4: Model predictions conditioned on counterfactual instructions: intention/inferred action sequence matches specified task, demonstrating non-trivial reasoning capabilities.

Quantitative evaluation in the AV-ALOHA simulation (21-DoF bimanual arm; active vision; in/out-of-distribution distractors) demonstrates state-of-the-art success rates, especially under OOD conditions, surpassing token-based, diffusion, and standard CoT-free VLA baselines.

Figure 5: AV-ALOHA qualitative: intention point (green cross) predicted correctly on OOD manipulations.

GazeVLA’s returns are particularly high for fine-manipulation (e.g., threading, peg insertion) and in presence of OOD distractors/lighting, due to intention bottleneck enabling task grounding beyond direct pixel-action regression.

Real-world Robot Deployment and Generalization

On real hardware (ALOHA bimanual and Unitree G1 dexterous platforms), GazeVLA achieves higher robustness and sample efficiency for gripper and dexterous manipulation. Tasks include pick-and-place, tool use (e.g., screw tightening), bimanual dexterous placement, and long-horizon operations (keyboard typing). The policy attains 85%+ success rates and realizes intention-guided, precise manipulation, outperforming strong baselines in both action precision and error suppression (e.g., object tilting, incorrect keypress).

Figure 6: Real-robot quantitative comparison across manipulation settings; GazeVLA achieves maximal success rates across gripper/dexterous tasks.

Figure 7: Egocentric/exocentric views: intention localization (green cross) aligns with manipulated objects during real-world execution.

Ablation Studies and Policy Analysis

Ablations confirm:

• The causal intention-action chain is essential for high success rates and OOD generalization; disabling intention bottleneck degrades performance.
• Human data for pretraining and joint post-training is vital; naive mixing or finetuning on minimal human-robot data is insufficient and can provoke overfitting or collapse.
• Intention transfer: gaze-derived intention learned from humans transfers to robot policies even without explicit intention labels, and remains robust under novel backgrounds, objects, and configurations.

  Figure 8: OOD generalization: initial object locations vary (orange); intention-action policy adapts robustly.

Theoretical and Practical Implications

Explicit modeling of intention as a causal variable bridges the embodiment gap between human and robot domains in a manner superior to strategies based solely on visual/action alignment or latent action regression as in prior work [egovla, human2robot, Being-H0.5]. Importantly, this approach decomposes the ambiguity of behavioral imitation by anchoring manipulation in "why" (goal) not only "how" (trajectory). For practical robotics, this means intention-augmented VLA models can be pretrained at scale on non-robotic demonstrations, thus side-stepping the limitations of costly, difficult-to-scale robot data acquisition and unlocking rapid adaptation to novel platforms and long-horizon tasks.

Conclusion

GazeVLA introduces a principled intention-centric framework for robot learning from human data. By tightly coupling intention inference (via gaze) and downstream action generation, it achieves robust transfer of high-level knowledge across embodiment gaps with strong generalization profiles and competitive empirical performance. Future work should pair cross-embodiment intention modeling with unified latent action spaces and multi-agent pretraining to further approach general-purpose embodied intelligence.

---

Reference: "GazeVLA: Learning Human Intention for Robotic Manipulation" (2604.22615)