TAVIS-Head: Benchmark for Global Active Vision

• TAVIS-Head is a benchmark for global active vision using a 3-DoF neck to coordinate gaze and reaching in five manipulation tasks.
• It quantifies performance via task success and anticipatory gaze with the Gaze-Action Lead Time (GALT) metric across headcam versus fixedcam conditions.
• The benchmark supports multi-task policy generalization and embodied gaze control evaluation on humanoid platforms like GR1T2 and Reachy2.

Searching arXiv for the benchmark paper and closely related active-vision imitation-learning context. arXiv search query: "TAVIS benchmark egocentric active vision anticipatory gaze imitation learning" TAVIS-Head is the “global active vision” half of TAVIS, a benchmark for egocentric active vision and anticipatory gaze in imitation learning. It consists of five manipulation tasks in which a policy controls a 3-DoF humanoid neck and receives either a head-mounted egocentric camera stream or a fixed exocentric workspace camera stream. The suite is designed to isolate and measure the contribution of global head motion to visual search over a large workspace, clutter disambiguation, conditional information gathering, and temporal monitoring, while also supporting a quantitative analysis of anticipatory gaze through GALT (Gaze-Action Lead Time) (Spigler, 8 May 2026).

1. Benchmark role and conceptual scope

Within TAVIS, TAVIS-Head is paired with TAVIS-Hands. The distinction is structural rather than cosmetic. TAVIS-Head contains 5 tasks and focuses on global active vision via a pan/tilt neck and a head-mounted camera; TAVIS-Hands contains 3 tasks and focuses on local active vision via wrist cameras under occlusion. In TAVIS-Head, both the headcam and the fixedcam are meaningful sensing modalities, and the tasks are explicitly designed so that head motion can improve performance. In TAVIS-Hands, head/fixedcams are intentionally uninformative, so no fixedcam condition is provided (Spigler, 8 May 2026).

The phrase “global search via pan/tilt necks” denotes using the robot’s head as a low-DoF active sensor to search large regions horizontally across the table and vertically across shelves, and to orient toward cues or status lights. Both GR1T2 and Reachy2 have a 3-DoF neck with roll, pitch, and yaw; in experiments, roll is effectively disabled, so active vision is primarily pitch and yaw. This makes TAVIS-Head a benchmark for embodied gaze control rather than only viewpoint selection.

A plausible implication is that TAVIS-Head targets a regime in which sensing and action are tightly coupled at the policy level: the neck is part of the action space, and the learned controller must coordinate gaze shifts with reaching, grasping, and timing decisions.

2. Embodiments, sensing, and control interface

TAVIS-Head is instantiated on two fixed-base humanoid torsos, GR1T2 and Reachy2. Both embodiments have two 7-DoF arms, a 3-DoF neck, and parallel grippers. The benchmark exposes a canonical 19-D action space: indices 0–6 encode the left arm end-effector target in the canonical hip frame as $(x, y, z, q_w, q_x, q_y, q_z)$; indices 7–13 encode the right arm end-effector target in the same parameterization; indices 14–16 encode head roll, pitch, and yaw in radians as absolute angles; and indices 17–18 encode left and right gripper commands as normalized scalars in $[-1,1]$. A canonical-frame wrapper maps these actions into robot-specific joint commands using damped least-squares inverse kinematics (Spigler, 8 May 2026).

The sensing setup is dual-view by construction. The head camera is rigidly mounted on the head link, has field of view approximately $70^\circ$, and provides $640\times 480$ RGB. The fixed workspace camera is static, positioned at $(0.2, 0, 1.43)\,\mathrm{m}$, pitched approximately $45^\circ$ downward, with field of view $120^\circ$ and the same $640\times 480$ RGB resolution. Both streams are recorded simultaneously for each teleoperated episode. This paired acquisition is the basis of the headcam-vs-fixedcam protocol.

Control and logging are temporally stratified. Teleoperation is recorded at $60\,\mathrm{Hz}$, while policies are trained and evaluated at $20\,\mathrm{Hz}$ after downsampling by 3. For the GALT detector, neck angular speed below $[-1,1]$0 is treated as head stillness, and end-effector linear speed below $[-1,1]$1 is treated as hand stillness.

3. Task suite and environment structure

All TAVIS-Head tasks use a fixed set of 5 YCB objects—soup can, meat can, tuna can, gelatin box, pudding box—uniformly scaled to $[-1,1]$2 and placed on a $[-1,1]$3 high table. Episodes last up to $[-1,1]$4, with randomized object positions and task-specific scene parameters. The suite is engineered so that “where the robot looks and when it looks” materially affects success (Spigler, 8 May 2026).

Task	Core visual demand	Success criterion
conditional-pick	cue reading and side selection	target $[-1,1]$5 and end-effector velocity $[-1,1]$6
wait-then-act	temporal monitoring of signal light	light green and target lifted to $[-1,1]$7
clutter-pick-cube	search for red cube in clutter	red cube lifted to $[-1,1]$8
clutter-pick-lift	language-conditioned object search	correct object lifted to $[-1,1]$9
multi-shelf-scan	vertical search across shelves	target retrieved with $70^\circ$0

In conditional-pick, two objects are placed left and right, and a colored cue card specifies the target: red means the object on the left, green means the object on the right. The robot must reorient its head to look at the card, infer the condition, then shift gaze and arm toward the appropriate object. The task stresses conditional information gathering and gaze-before-reach structure. In the ID split, each object is sampled in $70^\circ$1 strips at $70^\circ$2, and the cue card is centered in front of the robot with size $70^\circ$3.

In wait-then-act, the scene contains one YCB object and a signal light that is initially red and turns green after a random delay. The robot must monitor the light and only then pick the object. This task is less about spatial search than about temporal monitoring and gaze coordination. The ID light delay is sampled in $70^\circ$4, while the OOD-spatial split expands this to $70^\circ$5.

In clutter-pick-cube, the robot must find and pick a red cube among 4 distractor YCB objects. The ID configuration samples objects in a $70^\circ$6 region with minimum separation $70^\circ$7; OOD-spatial expands this to $70^\circ$8 with minimum separation $70^\circ$9. This task stresses long-distance search, clutter disambiguation, and coordinated head–arm motion.

In clutter-pick-lift, one of the 5 YCB objects is named by language and must be located, grasped, and lifted. There are 3 distinct phrasings per object, for 15 prompts total. The object-position ranges are the same as in clutter-pick-cube, but the sensory problem is multimodal: language narrows the search target, while the head must actively scan clutter to identify the named object.

In multi-shelf-scan, the robot is instructed to find a named object on a three-shelf unit with shelf heights $640\times 480$0, $640\times 480$1, and $640\times 480$2, and then retrieve it. The target condition is $640\times 480$3 after retrieval. This is the most explicit vertical-search task in the suite and directly exploits neck pitch. ID randomization uses per-shelf slot jitter $640\times 480$4 and $640\times 480$5.

4. Evaluation protocol and anticipatory gaze

The defining methodological feature of TAVIS-Head is the paired headcam-vs-fixedcam protocol. Each demonstration records the same scene layout, language prompt, arm trajectory, and neck trajectory under two synchronized visual streams: headcam RGB and fixedcam RGB. Policies are then trained separately in the two camera modes. Because the underlying demonstrations are identical, the comparison isolates the effect of egocentric active vision rather than confounding it with different data collection or different task distributions (Spigler, 8 May 2026).

Demonstrations are collected by teleoperation with Meta Quest 3. The robot headcam stream is rendered into the headset, the human operator’s head orientation directly controls the robot neck, and the controllers drive the two end-effectors. A central fixation marker encourages alignment between visual gaze and head orientation, so head motion functions as a proxy for gaze. For TAVIS-Head there are approximately 800 episodes per robot.

Task performance is measured by Task Success Rate (SR) over 96 episodes per condition, evaluated on three splits: id, ood-spatial, and ood-init-pose. In ood-init-pose, end-effector positions are perturbed with $640\times 480$6 and neck yaw and pitch with $640\times 480$7.

Anticipatory gaze is quantified by GALT (Gaze-Action Lead Time). For each successful TAVIS-Head episode,

$640\times 480$8

where $640\times 480$9 is the time when the head-mounted camera reaches its final pre-grasp fixation, and $(0.2, 0, 1.43)\,\mathrm{m}$0 is the time of grasp completion. Positive GALT indicates that gaze arrives before the hand completes the grasp. The detector operates directly on the action trajectory, not on video. It uses a backward lookback horizon $(0.2, 0, 1.43)\,\mathrm{m}$1, forward slack $(0.2, 0, 1.43)\,\mathrm{m}$2, minimum fixation duration $(0.2, 0, 1.43)\,\mathrm{m}$3, head stillness threshold $(0.2, 0, 1.43)\,\mathrm{m}$4, hand stillness threshold $(0.2, 0, 1.43)\,\mathrm{m}$5, and rejects outliers outside $(0.2, 0, 1.43)\,\mathrm{m}$6.

The teleoperator reference on 800 TAVIS-Head episodes per robot yields a detection rate of approximately $(0.2, 0, 1.43)\,\mathrm{m}$7 at $(0.2, 0, 1.43)\,\mathrm{m}$8, with pooled median GALT about $(0.2, 0, 1.43)\,\mathrm{m}$9 for GR1T2 and about $45^\circ$0 for Reachy2. Multi-task headcam policies exhibit median lead times around $45^\circ$1–$45^\circ$2 and match the dataset reference within about $45^\circ$3 on a $45^\circ$4–$45^\circ$5 scale. On four of five tasks the relative error in the median remains within $45^\circ$6–$45^\circ$7; the outlier is multi-shelf-scan, with about $45^\circ$8 deviation, likely due to multiple intermediate shelf fixations.

5. Baselines and empirical findings

TAVIS-Head is accompanied by two baseline policy families implemented in LeRobot: To, a $45^\circ$9-style vision-language-action transformer fine-tuned from lerobot/pi0_base, and Diffusion Policy, a DDPM-style action-diffusion model. Both output the canonical 19-D action vector. Diffusion Policy is trained only on tasks without language, so clutter-pick-lift and multi-shelf-scan are excluded from its TAVIS-Head coverage. Training uses separate checkpoints for headcam and fixedcam, so the headcam version must learn to use neck control as an active sensing mechanism (Spigler, 8 May 2026).

The principal empirical result is that active vision generally helps, but the benefit is task-conditional rather than uniform. For multi-task To on the TAVIS-Head id split, suite-mean success rates are $120^\circ$0 for GR1T2 headcam versus $120^\circ$1 for GR1T2 fixedcam, and $120^\circ$2 for Reachy2 headcam versus $120^\circ$3 for Reachy2 fixedcam. The GR1T2 per-task results make the conditional structure explicit: conditional-pick improves from $120^\circ$4 fixedcam to $120^\circ$5 headcam; clutter-pick-cube improves from $120^\circ$6 to $120^\circ$7; clutter-pick-lift improves from $120^\circ$8 to $120^\circ$9; and multi-shelf-scan improves from $640\times 480$0 to $640\times 480$1. By contrast, wait-then-act is a counterexample: GR1T2 fixedcam reaches $640\times 480$2, while headcam reaches $640\times 480$3.

OOD robustness is limited. For multi-task headcam To, suite-mean TAVIS-Head success rates averaged over robots are $640\times 480$4 on id, $640\times 480$5 on ood-spatial, and $640\times 480$6 on ood-init-pose. This suggests that multi-task scaling helps in-distribution performance, but controlled distribution shifts still produce sharp degradation. The headcam condition is especially sensitive to ood-init-pose, because perturbing the initial head pose changes the egocentric observation more drastically than it changes a static fixed-camera observation.

Single-task versus multi-task training also reveals a scaling effect. On headcam TAVIS-Head id, single-task To attains a suite-mean success rate of $640\times 480$7, while multi-task To attains $640\times 480$8. This indicates that the benchmark is not only a testbed for active sensing, but also for multi-task policy generalization under embodiment-specific perception–action coupling.

6. Infrastructure, practical use, and stated limitations

TAVIS-Head is implemented on IsaacLab and IsaacLab-Arena. The repository is released at https://github.com/spiglerg/tavis, and datasets are released in LeRobotDataset v3.0 format on Hugging Face, including tavis-benchmark/tavis-head-gr1t2 and tavis-benchmark/tavis-head-reachy2. The TAVIS-Head tasks are implemented as Python classes such as ConditionalPick, WaitThenAct, ClutterPickCube, ClutterPickLift, and MultiShelfScan. Episodes contain synchronized RGB streams, canonical 19-D actions, proprioception, language prompts, and task labels, so they support both single-task filtering and multi-task training (Spigler, 8 May 2026).

The benchmark is also designed for extension. A new robot can be added by providing a USD asset, joint indices, gripper interface, and hip-frame offset, then wrapping the embodiment in the canonical 19-D action interface. A new task can be added by implementing an IsaacLab-Arena environment class with procedural scene generation, success criteria, and optional ID/OOD perturbations. This suggests that TAVIS-Head is intended not merely as a frozen evaluation suite but as evaluation infrastructure.

The limitations stated for TAVIS-Head are specific. Only 2–3 DoF necks are considered; high-DoF necks and eye-gimbal configurations are out of scope. Teleoperation uses a single operator and an enforced fixation pattern in which head and gaze are effectively aligned, limiting strategy diversity. GALT is post hoc; directly optimizing it could encourage unnatural head jitter unless additional constraints are imposed. Finally, all tasks share a similar metal-table scene, so richer scene-level variation in textures, lighting, and semantics is not yet explored.

Taken together, TAVIS-Head defines a controlled regime for studying head-controlled egocentric sensing in imitation learning: five tasks that require global pan/tilt search and cue monitoring, two humanoid embodiments, a paired headcam-vs-fixedcam protocol on identical demonstrations, and a quantitative anticipatory-gaze metric that shows imitation alone can yield human-like gaze timing.