TAVIS-Hands: Local Active Vision Benchmark

Updated 5 July 2026

TAVIS-Hands is a benchmark for egocentric active vision that leverages wrist-mounted cameras to reveal critical scene information hidden from head views.
It comprises three tasks—peeking-box, occluded-reach, and blocked-clutter-pick-cube—that assess wrist-driven exploratory policies under local occlusion.
Empirical results show that wrist-only approaches can significantly outperform combined head-wrist setups, emphasizing the need for localized perception.

Searching arXiv for the benchmark paper and closely related context on active vision and wrist-camera manipulation. TAVIS-Hands is the “local active vision” half of TAVIS, a benchmark for egocentric active vision and anticipatory gaze in imitation learning. It is defined as a three-task suite built in IsaacLab on two humanoid torso embodiments, GR1T2 and Reachy2, for scenarios in which critical task information is hidden by local occlusion and cannot be recovered by head motion alone. In this setting, the policy must rely on wrist-mounted cameras for both perception and manipulation, and must often coordinate both arms because the reachable hand is not known in advance (Spigler, 8 May 2026).

1. Benchmark role and conceptual scope

TAVIS introduces evaluation infrastructure for active-vision imitation learning with three primitives: a paired headcam-vs-fixedcam protocol on identical demonstrations, GALT (Gaze-Action Lead Time), and procedural ID/OOD splits. Within that benchmark, TAVIS-Hands is explicitly the suite for local occlusion, complementing TAVIS-Head, which emphasizes global active vision with a pan/tilt neck (Spigler, 8 May 2026).

The defining premise of TAVIS-Hands is that the head camera, and by extension a typical fixed camera, is deliberately made structurally uninformative about the decisive scene information. The benchmark therefore isolates a regime in which local wrist-camera motion functions as the only effective way to peek around or into occlusions. This design separates “active vision helps” from the narrower claim that “head motion helps”: in TAVIS-Hands, head movement alone cannot reveal the target.

A common misunderstanding is to treat TAVIS-Hands as merely a hand-centric variant of a generic manipulation benchmark. Its actual purpose is narrower and more technical: it probes whether imitation-learned policies can move their end-effectors as sensing platforms, acquire missing egocentric evidence through wrist views, and then exploit that evidence for action. The suite is therefore about hands-as-eyes as much as about grasp execution.

2. Local-occlusion task suite

TAVIS-Hands contains three tasks: peeking-box, occluded-reach, and blocked-clutter-pick-cube. All three are designed so that the wrist cameras, not the head camera, provide the decisive observations needed for successful manipulation (Spigler, 8 May 2026).

Task	Prompt	Core local-occlusion mechanism
peeking-box	“Retrieve the object from inside the box.”	The box opening is on one side only and cannot be seen from the head camera
occluded-reach	“Reach around the screen and pick up the object behind it.”	A vertical screen completely blocks the head view of the object
blocked-clutter-pick-cube	“Find the red cube and pick it up.”	The head camera is masked; only wrist cameras can locate and grasp the red cube

In peeking-box, a rectangular box of 20×14×20 cm lies on the table with an opening on either the left or right side, randomized per episode. A single target YCB object is inside the box. Because the head camera cannot see the sides of the box, the policy must use the wrist cameras to determine which side is open, choose the appropriate arm, reach into the box, and lift the object. The technical challenge is not simply grasping an object in a container; it is disambiguating the open side under a viewpoint constraint that is resolvable only by wrist motion.

In occluded-reach, a vertical screen of 16×40 cm is placed at $x=0.27$ m between the robot head and the workspace, and a single YCB object is placed behind it. The head camera’s line of sight is completely blocked. Successful behavior requires exploratory arm motion that carries a wrist camera around the screen edge, acquires object visibility, and then executes grasping. The task therefore measures whether a policy can synthesize an informative viewpoint trajectory rather than merely act on already-visible targets.

In blocked-clutter-pick-cube, the environment inherits the randomization of TAVIS-Head clutter-pick-cube—one red cube plus four distractor YCB objects placed at random positions with minimum separation—but the head camera is masked. The policy can only rely on the wrist cameras to locate and grasp the red cube. This turns a cluttered pick task into a test of local wrist-driven search, making it especially useful for comparing global active vision against wrist-only exploration.

3. Embodiments, observation streams, and control substrate

TAVIS-Hands is instantiated on two simulated humanoid torsos: GR1T2 with Robotiq 2F-85 parallel grippers replacing native hands, and Reachy2 with custom Pollen grippers. Both embodiments provide two 7-DoF arms and a 3-DoF neck, and both operate on a unified 19-D canonical action space defined in a hip-centred frame (Spigler, 8 May 2026).

That 19-D action space is structured as follows: indices 0–6 specify the left-arm IK target $(x, y, z, q_w, q_x, q_y, q_z)$ , indices 7–13 specify the right-arm IK target, indices 14–16 specify head roll, pitch, and yaw, and indices 17–18 specify left and right gripper scalars in $[-1,1]$ . This canonicalization is important because it allows identical task definitions and policy interfaces across both robots.

All on-board cameras—head, left wrist, and right wrist—produce 640 × 480 RGB observations. The head camera field of view is approximately 70° with focal length 15 mm, while the wrist cameras have approximately 53° field of view with focal length 21 mm. Camera-link mounts vary slightly across robots because of geometry differences, but the canonical observation role remains the same: synchronized streams aligned to the agent’s gaze and hand motion (Spigler, 8 May 2026).

The suite is implemented in IsaacLab with IsaacLab-Arena. Teleoperation is recorded at 60 Hz, while policies are evaluated at 20 Hz after downsampling by 3. Arm IK uses damped least-squares null-space solving. These details are not incidental: for TAVIS-Hands, viewpoint acquisition is constrained by reachability and joint limits, so local active vision is inseparable from the manipulator’s kinematics.

4. Demonstrations, policies, and evaluation protocol

The TAVIS dataset contains about 2200 episodes across both suites and both robots. For TAVIS-Hands specifically, demonstrations are collected via Meta Quest 3 VR teleoperation, with 300 episodes per robot for the suite. Recorded streams include synchronized head, fixed, left-wrist, and right-wrist RGB videos, proprioception, 19-D canonical actions, and task identifiers (Spigler, 8 May 2026).

The teleoperation interface shows the head camera feed with optional wrist-camera overlays. Head orientation controls the robot neck; controller poses drive the arms. A central fixation marker is used so that head motion approximates eye gaze. In TAVIS-Hands, this means that human demonstrations already encode a strategy for wrist-based visual exploration under occlusion, rather than only manipulation after target localization.

Baseline learning uses LeRobot with two policy families. The first is To, described as a “ $\pi_0$ -style” vision-language-action flow model fine-tuned from lerobot/pi0_base, with about 3.5B parameters, observation horizon 1, prediction horizon 16 frames, and action chunk 8 frames. The second is Diffusion Policy, with about 272.5M parameters, observation horizon 2, and the same prediction horizon and chunk size. For TAVIS-Hands, the relevant camera-mode configuration is head + wrist, because the suite omits the fixedcam condition by design (Spigler, 8 May 2026).

Evaluation uses three splits. id uses the same randomization ranges as teleoperation. ood-spatial expands initial object placement regions. ood-init-pose perturbs the robot reset with $\sigma = 10$ cm on Cartesian end-effector positions and $\sigma = 10^\circ$ on neck pitch and yaw. For TAVIS-Hands, these shifts are especially informative because they test whether a learned exploratory wrist motion remains valid when the geometry of occlusion or the robot’s starting configuration changes.

5. Empirical performance and benchmark findings

The strongest published TAVIS-Hands baseline results are reported for multi-task To. On GR1T2, the ID split yields 64.6% on peeking-box, 87.5% on occluded-reach, and 58.3% on blocked-clutter-pick-cube, for a suite mean of 70.1%. On Reachy2, the ID split yields 84.4%, 78.1%, and 67.7%, for a suite mean of 76.7% (Spigler, 8 May 2026).

Under ood-spatial, the suite means drop to 49.0% on GR1T2 and 51.0% on Reachy2. Under ood-init-pose, they fall further to 14.6% and 38.2%, respectively. The benchmark therefore shows that local active vision can solve these tasks in distribution at substantial rates, but does not by itself confer strong robustness under controlled distribution shift.

Single-task baselines are materially weaker. For single-task To on the ID split, GR1T2 achieves 47.9% on peeking-box, 68.8% on occluded-reach, and 44.8% on blocked-clutter-pick-cube; Reachy2 achieves 72.9%, 53.1%, and 33.3%. Diffusion Policy on the ID split achieves 69.8%, 83.3%, and 37.5% on GR1T2, and 56.2%, 39.6%, and 28.1% on Reachy2 (Spigler, 8 May 2026).

A particularly informative ablation uses the clutter-cube family to compare three observation regimes: no AV using a fixed-cam TAVIS-Head checkpoint, full AV using TAVIS-Head head + wrist, and wrist-only using TAVIS-Hands blocked-clutter-pick-cube with head masked. The reported numbers are 26.6% for no AV, 45.9% for full AV, and 63.0% for wrist-only. The paper attributes wrist-only outperforming full AV to demonstration design: TAVIS-Hands trains on explicitly exploratory wrist trajectories under occlusion (Spigler, 8 May 2026).

These findings support three benchmark-level conclusions. First, local wrist-camera active vision is a genuine capability rather than a cosmetic observation change. Second, its benefits are task-conditional: occluded-reach is easier than peeking-box or blocked-clutter-pick-cube for the baselines reported. Third, even when active local vision is structurally necessary and effective, learned policies still degrade sharply under OOD perturbations.

6. Metrics, anticipatory behavior, and common misconceptions

TAVIS introduces GALT (Gaze-Action Lead Time) as

$\text{GALT} = t_{\text{hand}} - t_{\text{head}},$

where $t_{\text{head}}$ is the time when the head-mounted camera reaches its final pre-grasp fixation and $t_{\text{hand}}$ is the time of grasp completion. Positive GALT indicates anticipatory gaze. Detection is performed from canonical action trajectories rather than raw images (Spigler, 8 May 2026).

For TAVIS-Hands, however, the published paper does not report explicit GALT histograms or quantitative GALT analyses for the three wrist-camera tasks. GALT remains a head-gaze metric, and the paper treats wrist cameras as perception channels rather than gaze in the strict sense. This matters because TAVIS-Hands is often read as a benchmark of anticipatory wrist vision, but the current metricization of anticipation does not yet quantify wrist-camera lead behavior directly.

Another recurring misconception is that TAVIS-Hands is equivalent to a fixed-camera occlusion benchmark with extra sensors. The benchmark is specifically constructed so that the fixedcam condition is omitted by design: the head camera is structurally uninformative there, and a fixed camera adds no information the wrists do not already provide. The manipulation problem is therefore inseparable from the view-synthesis problem.

The benchmark’s task semantics further reinforce this point. Success is measured by object lift thresholds—1.25 m for peeking-box and occluded-reach, 1.2 m for blocked-clutter-pick-cube—with an end-effector velocity criterion to ensure the object is held rather than mid-flight. That definition makes the benchmark operationally simple, but the underlying challenge is not lifting per se; it is acquiring the missing evidence that makes lifting feasible.

7. Resources, limitations, and relation to adjacent dexterous-hand research

TAVIS releases code, evaluation scripts, demonstrations in LeRobot v3.0 format, and trained baselines. The TAVIS-Hands datasets are published as tavis-benchmark/tavis-hands-gr1t2 and tavis-benchmark/tavis-hands-reachy2, and each episode includes head, fixed, left_wrist, and right_wrist RGB videos, proprioception, canonical 19-D actions, and a task field identifying peeking-box, occluded-reach, or blocked-clutter-pick-cube (Spigler, 8 May 2026).

The main limitations identified for TAVIS-Hands are simulation-only evaluation, demonstrations from a single teleoperator, active-vision support limited to a 3-DoF neck plus wrist cameras, GALT defined only for head gaze, and OOD coverage restricted to object spatial distributions and initial robot pose. The paper accordingly suggests directions such as defining a wrist-camera analogue of GALT, expanding OOD axes to texture, lighting, or geometry changes, and deploying the suite with real wrist cameras on physical robots (Spigler, 8 May 2026).

Within the broader dexterous-manipulation literature, TAVIS-Hands occupies a distinct niche. It studies wrist-camera active vision under local occlusion rather than tactile sensing or in-grasp contact modeling. Recent tactile work instead addresses adjacent issues: TaSA models self-touch and tactile sensory attenuation for in-grasp insertion tasks (Ponnivalavan et al., 5 Feb 2026); SaTA spatially anchors tactile features to the hand’s kinematic frame for sub-millimeter dexterous manipulation (Huang et al., 16 Oct 2025); and TacViT uses Vision Transformers to generalize tactile perception across unseen vision-based tactile sensors on a five-fingered hand (Ford et al., 1 Apr 2026). This suggests a broader integration agenda in which TAVIS-Hands-style local active vision could be combined with spatially grounded tactile sensing and self-touch-aware contact modeling.

In that sense, TAVIS-Hands is best understood not as a general benchmark for “using hands,” but as a controlled experimental substrate for a specific regime of embodied perception: manipulation in which the policy must move its wrists to make the world visible, and must do so before acting. Its empirical value lies precisely in making that regime reproducible, measurable, and separable from global head-based active vision (Spigler, 8 May 2026).