TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

Abstract: Active vision -- where a policy controls its own gaze during manipulation -- has emerged as a key capability for imitation learning, with multiple independent systems demonstrating its benefits in the past year. Yet there is no shared benchmark to compare approaches or quantify what active vision contributes, on which task types, and under what conditions. We introduce TAVIS, evaluation infrastructure for active-vision imitation learning, with two complementary task suites -- TAVIS-Head (5 tasks, global search via pan/tilt necks) and TAVIS-Hands (3 tasks, local occlusion via wrist cameras) -- on two humanoid torso embodiments (GR1T2, Reachy2), built on IsaacLab. TAVIS provides three evaluation primitives: a paired headcam-vs-fixedcam protocol on identical demonstrations; GALT (Gaze-Action Lead Time), a novel metric grounded in cognitive science and HRI that quantifies anticipatory gaze in learned policies; and procedural ID/OOD splits. Baseline experiments with Diffusion Policy and $π_0$ reveal that (i) active-vision generally helps, but benefits are task-conditional rather than uniform; (ii) multi-task policies degrade sharply under controlled distribution shifts on both suites; and (iii) imitation alone yields anticipatory gaze, with median lead times comparable to the human teleoperator reference. Code, evaluation scripts, demonstrations (LeRobot v3.0; ~2200 episodes) and trained baselines are released at https://github.com/spiglerg/tavis and https://huggingface.co/tavis-benchmark.

• The paper introduces TAVIS, a novel benchmark standardizing the evaluation of egocentric active vision in imitation learning.
• It employs dual task suites—TAVIS-Head and TAVIS-Hands—and introduces the GALT metric to quantify anticipatory gaze under both in-distribution and out-of-distribution conditions.
• Results demonstrate that active vision significantly improves manipulation success in task-dependent scenarios, with multi-task policies outperforming single-task ones.

TAVIS: An Egocentric Active Vision Benchmark for Imitation Learning Evaluation

Introduction

This paper introduces TAVIS, a comprehensive, simulation-based benchmark specifically developed to standardize and advance research on egocentric active vision within imitation learning (IL) for robotic manipulation tasks. Despite widespread recognition in recent literature that active gaze control can enhance manipulation, prior studies have lacked a unified testbed for fair, reproducible, and controlled comparison of active vision policies. TAVIS closes this gap by operationalizing two distinct task suites, TAVIS-Head and TAVIS-Hands, evaluated on two humanoid torso robots with simultaneous head-mounted and fixed-camera observations. In addition, the benchmark proposes a novel metric, Gaze-Action Lead Time (GALT), grounded in cognitive science and HRI, to objectively quantify anticipatory gaze in learned policies.

Benchmark Design

Task Suites and Embodiment

TAVIS establishes two controlled and complementary task suites:

• TAVIS-Head focuses on manipulation tasks benefitting from global scene exploration using head-gaze reorientation. Tasks include visual search, intent signaling, clutter resolution, and vertical workspace exploration, instantiated on commodity 3-DoF pan/tilt necks.
• TAVIS-Hands targets scenarios involving local occlusions, requiring fine-grained exploration enabled by wrist-mounted cameras. These tasks decouple perception and manipulation across both arms, stressing local active sensing for occlusion removal.

All tasks operate on a unified 19-DoF canonical action space, abstracting away platform specifics to facilitate cross-embodiment evaluation. The two robots—GR1T2 and Reachy2—feature bimanual manipulation and identical perception pipelines.

Evaluation Protocol

TAVIS introduces three principal controlled axes for evaluation:

1. Camera Mode Isolation: Policies are evaluated on either head-mounted (active vision) or fixed workspace (passive) RGB streams, recorded simultaneously on identical demonstrations, isolating gaze control as the primary variable.
2. ID/OOD Distribution Splits: In-distribution (ID) evaluations use test-time sampling matched to the training regime. Out-of-distribution (OOD) evaluation expands object pose and robot initial state far beyond training distribution, probing extrapolation versus interpolation.
3. Temporal Coordination Metric (GALT): GALT quantifies temporal gaze-to-action relationships, detecting if and by how much a policy’s gaze precedes manipulative action, using proprioceptive signals only.

Each axis is orthogonalized for reproducibility and ablation, enabling direct attribution of policy behavior to design choices.

Empirical Results

Active Vision Benefits and Task Dependency

Paired evaluations demonstrate that active vision confers statistically significant improvements in manipulation success across most tasks, with magnitude highly task-dependent. For example, in conditional-pick, active vision yields +28 to +45 percentage point improvements over fixed cameras, reflecting the importance of visually coordinated cue reading. However, for tasks with fully observable workspaces and no ambiguity (e.g., wait-then-act), head control can occasionally be detrimental, introducing nuisance variance. This underscores that the benefits of active vision are not uniform but conditional on task structure.

Distribution Shift and Generalization

Under both spatial and pose OOD conditions, multi-task policies suffer severe performance degradation on all task types (e.g., TAVIS-Head multi-task mean: 43.0% ID → 25.0% OOD-spatial → 6.6% OOD-init-pose), with degradation more pronounced when active vision is critical. This mirrors memorization vulnerabilities in prior IL benchmarks and confirms that current imitation-based methods have limited inherent robustness to distributional shifts, especially along axes strongly influencing perception-action coupling.

Multi-task and Single-task Policy Behavior

Multi-task policy heads outperform single-task equivalents when trained and evaluated across diverse tasks, in line with scaling effects previously found in large-scale IL. For instance, mean success rates on TAVIS-Hands increase from 53.5% (single-task) to 73.4% (multi-task), suggesting beneficial transfer across related manipulation tasks.

Acquisition of Anticipatory Gaze

A key finding is that imitation alone reliably induces anticipatory gaze behaviors in robot policies—multi-task models trained on human teleoperation data attain GALT distributions (medians ~2–3 s) comparable both in magnitude and distribution to those of the teleoperator. Median GALT differentials are under 200 ms for most tasks, and policies exhibit consistent temporal coupling between gaze and action, making intent more legible both to humans and potentially to collaborative agents.

Theoretical and Practical Implications

TAVIS’s compositional, simulation-based design enables targeted investigation of critical open problems in active vision, including:

• Quantifying the causal impact of gaze control across specific manipulation subproblems.
• Standardized assessment of temporal legibility in robot motion, with direct connections to HRI interpretability metrics.
• Analysis of generalization and robustness to OOD perturbations systemic to real-world deployment.

Methodologically, the GALT metric grounds the evaluation of robot gaze not only in reward-centric optimality, but also in the spatiotemporal structure of human-competent manipulation.

Practically, TAVIS is released with all demonstration datasets, code, and evaluation scripts, substantially lowering reproducibility barriers and facilitating rapid benchmarking for both imitation learning and vision-based robot control communities.

Future Directions

TAVIS’s extensible architecture supports immediate augmentation in several important directions:

• Extending ID/OOD axes: to encompass richer, more realistic visual and semantic distribution shifts such as lighting variation, novel object categories, and complex scene layouts.
• Sim-to-real transfer bridges: enabling benchmarking of policies trained in TAVIS simulation on physical robots.
• Inclusion of higher DoF sensor control: Generalization to more complex eye-gimbal or foveated configurations as explored in recent literature.
• Post-hoc gaze relabeling: to retrofit anticipatory-gaze labels onto legacy demonstration datasets lacking active vision, facilitating much broader comparisons with existing work.
• Integration with vision-LLMs: to explore the interplay between natural-language-conditioned manipulation and active gaze strategies at greater scale.

Conclusion

TAVIS establishes a reproducible and compositional benchmark for egocentric active vision in imitation learning, spanning diverse manipulation tasks across two humanoid robot platforms. Its protocol enables controlled comparative evaluation along critical axes—active versus passive vision, robust generalization, and the emergence of anticipatory gaze timing. Empirical results validate the utility and specificity of active vision, its conditional dependence on task topology, its vulnerability to distribution shift, and its ability to foster human-like gaze coordination through imitation. The benchmark’s flexibility and thorough design make it a central resource for advancing quantitative understanding of active vision’s role in robotic manipulation and HRI, and a solid foundation for future advances in multi-modal perception-action systems.