ActiveUMI: Active Egocentric Robot Manipulation

• ActiveUMI is a framework that integrates VR teleoperation with sensorized controllers to capture both hand and head movements for bimanual robotic manipulation.
• It implements embodiment transfer by mapping human demonstration signals to a robot's two manipulation arms and a dedicated camera arm for active viewpoint control.
• Experimental results show policies trained on ActiveUMI data achieve an average 70% success on in-distribution tasks and 56% on novel objects and environments.

ActiveUMI is a framework for robotic manipulation that transfers in-the-wild human demonstrations to robots capable of complex bimanual manipulation while preserving active, egocentric perception. It couples a portable VR teleoperation kit with sensorized controllers that mirror the robot’s end-effectors, records deliberate head movements through a head-mounted display, and maps those signals onto a robot with two manipulation arms and a third camera arm. In the reported experiments, policies trained exclusively on ActiveUMI data achieve an average success rate of 70% on in-distribution tasks and retain a 56% success rate when tested on novel objects and in new environments (Zeng et al., 2 Oct 2025).

1. Definition and conceptual scope

ActiveUMI addresses learning bimanual robot manipulation policies from demonstrations collected directly by humans without requiring a robot during data collection. The target setting is explicitly bimanual, often long-horizon, sometimes precision-sensitive, and may involve deformable or articulated objects. The framework extends Universal Manipulation Interface-style handheld demonstration capture by adding active, egocentric perception: it records not only hand motion but also how an operator moves their head to inspect grasp quality, see around occlusions, align precisely with objects, and monitor task progress (Zeng et al., 2 Oct 2025).

Within this formulation, the central claim is that successful manipulation depends on learning both action and viewpoint behavior. Wrist-mounted cameras are characterized as action-centric because their viewpoints are constrained by arm motion, whereas many tasks require perception that is not colocated with the hand action. ActiveUMI therefore treats head motion as part of the demonstrated behavior and maps it to a robot-mounted camera on a dedicated arm. This makes viewpoint control an explicit part of the learned policy rather than a fixed sensing assumption (Zeng et al., 2 Oct 2025).

The paper frames three specific advances relative to earlier UMI-style systems: active egocentric perception via head tracking, portable wearability using a backpack computer and standalone VR hardware, and flexible embodiment alignment achieved by mounting the target robot’s own end-effector design onto the handheld controller. A plausible implication is that ActiveUMI is best understood not merely as a data-collection device, but as a full demonstration-to-deployment interface in which embodiment transfer and perceptual strategy are co-recorded.

2. System architecture and embodiment transfer

The end-to-end pipeline begins with human demonstration collection. A human operator wears a VR headset and backpack computer, holds two modified VR controllers with gripper-like attachments matching the robot’s end-effectors, and is tracked in 6-DoF for both hand/controller pose and head pose. Wrist fisheye cameras and the head-mounted display front cameras collect visual observations. The system records left and right controller poses, head pose, wrist camera images, egocentric head-camera images, and robot-relevant gripper actions, all aligned in a common world coordinate frame (Zeng et al., 2 Oct 2025).

Robot embodiment transfer is implemented as a one-to-one mapping between left controller tip and left robot gripper tip, right controller tip and right robot gripper tip, and head-mounted display pose and robot head-mounted camera pose. During deployment, the recorded hand trajectories are mapped to two robot arms and the head motion is mapped to a third robot arm carrying a camera. The deployed robot therefore uses two manipulation arms, one camera arm for active viewpoint control, wrist proprioception, and egocentric head camera images, and the policy predicts actions for both manipulation and head movement (Zeng et al., 2 Oct 2025).

The experiments use three 6-DoF ARX R5 robotic arms: two for bimanual manipulation and one for the active head camera. The two manipulation arms each carry a fisheye wrist-mounted camera, and the third arm carries the camera corresponding to the human head-mounted display viewpoint. The handheld controllers are modified Meta Quest 3s controllers whose pose is estimated via the headset’s inside-out tracking. Each controller includes tracked 6-DoF pose, a micro-motor for gripper actuation, an attached copy of the robot gripper, and a fisheye wrist camera. The design is described as non-invasive because the robot gripper is copied onto the controller rather than replaced during data collection (Zeng et al., 2 Oct 2025).

This embodiment strategy reduces the retargeting gap relative to bare-hand capture. The paper does not provide explicit rigid-transform chains, inverse-kinematics derivations, or a formal retargeting optimization. Instead, the contribution is practical: ActiveUMI seeks robot compatibility through physical gripper mirroring and shared coordinate alignment rather than through a mathematically elaborate retargeting layer.

3. Calibration, coordinate consistency, and active egocentric sensing

All data are recorded in absolute coordinates relative to a unified world coordinate system established during initial calibration. The paper describes three calibration methods. First, the operator can press the “B” button to reset the 6-DoF zero point and reposition the base coordinate system, aided by rendered axes in the headset. Second, a physical docking placeholder or jig fixes the controllers in a known relative pose and distance; while docked, pressing a designated button calibrates the virtual coordinate system to that known configuration. Third, when a gripper moves within 3 cm of the zero point, the controller vibrates, providing tactile confirmation that it is near the base position (Zeng et al., 2 Oct 2025).

The only explicit equations given in the paper concern Relative Pose Error, but the typesetting is corrupted in the provided text. As written, the paper includes malformed expressions for $\Delta L$ and $RPE$, and the intended meanings are inferred to involve the absolute difference between replayed and measured distances and a relative error percentage. Because the formulas are corrupted, the calibration contribution is better regarded as system-oriented than formally geometric (Zeng et al., 2 Oct 2025).

Active perception is defined as the robot’s ability to control its camera viewpoint as part of the policy rather than passively receiving images from fixed or arm-tied cameras. “Egocentric” refers to the operator’s head and visual attention, captured through the head-mounted display. The headset tracks real-time 6-DoF head pose and provides front-camera RGB streams, so each demonstration contains paired hand movement, head movement, and corresponding visual observations. During training, the policy receives head-camera observations together with head-motion demonstrations; during deployment, the policy predicts a 6-DoF pose for the robot’s head, which is executed by the dedicated camera arm (Zeng et al., 2 Oct 2025).

The paper does not provide a formal active-sensing objective, policy equation, or architecture specialized for viewpoint control. What is specified is the behavioral interface: head motion is treated as a learned control target. This suggests that ActiveUMI’s novelty lies primarily in demonstration design and sensorimotor coupling rather than in a new formal active-perception learning rule.

4. Demonstration collection and policy learning

In practice, demonstration collection proceeds by calibrating the system in the current environment, having the operator hold the two controller-grippers and wear the head-mounted display, performing the task naturally while moving both hands and head, and recording synchronized motion and image data. The paper also includes immersive 3D robot rendering in VR, with virtual robot arms shown in the headset and aligned with the handheld controllers to provide visual feedback during collection (Zeng et al., 2 Oct 2025).

The demonstrations are described as robot-free during collection, in-the-wild, bimanual, egocentric, and multimodal. The recorded modalities include left and right hand pose trajectories, head pose, wrist camera images, and head-camera images. All sensor and robot data are collected at 30 Hz. The paper does not specify the total number of demonstrations, the number of episodes per task, average episode length, exact train/test split, or a detailed synchronization implementation beyond common system collection. This means the system is operationally characterized in more detail than the resulting dataset is statistically characterized (Zeng et al., 2 Oct 2025).

For policy learning, the authors use $\pi_0$, described as a state-of-the-art vision-language-action model. The model is fine-tuned for 50k iterations using a cosine learning rate scheduler. The deployed observation space includes the egocentric head camera, two wrist cameras, and wrist proprioception or robot state. The action space depends on configuration: ActiveUMI uses two 6-DoF manipulation arms plus one 6-DoF camera arm, together with gripper actions, and is described as 20-DoF; the fixed-head-camera and wrist-only baselines are described as 14-DoF (Zeng et al., 2 Oct 2025).

The paper does not provide explicit loss equations for policy training, and it does not specify whether $\pi_0$ is transformer-only, diffusion-based, recurrent, or otherwise internally structured. Accordingly, the learning method is concretely defined at the level of data interface and fine-tuning regime rather than at the level of architectural internals.

5. Experimental tasks, baselines, and reported results

The evaluation covers six challenging bimanual tasks, although only five are clearly enumerated in the implementation section and the quantitative tables report five named tasks. The named tasks are Block disassembly, Shirt folding, Rope boxing, Toolbox cleaning, Bottle placing, and Take Drink from Bag; there is also a textual mention of PourWater that does not appear in the tables. The paper states that, unless otherwise noted, experiments use 10 trials, while the shirt-folding data-mixing experiment uses 20 trials (Zeng et al., 2 Oct 2025).

Three configurations are compared. The UMI baseline uses wrist-camera-only perception. UMI with Fixed Head Camera adds a top-down static head view but no active head motion. ActiveUMI uses the full method with active head-camera motion. All three configurations use the same $\pi_0$ model, so the principal experimental variable is the viewpoint-control interface rather than the policy backbone (Zeng et al., 2 Oct 2025).

For in-distribution evaluation, the reported average success rates are 26% for UMI, 42% for UMI with Fixed Head Camera, and 70% for ActiveUMI. Task-wise, ActiveUMI achieves 90% on Bottle placing, 70% on Rope boxing, 80% on Shirt folding, 30% on Block disassembly, and 80% on Take Drink from Bag. The corresponding UMI scores are 60%, 20%, 10%, 0%, and 40%, while UMI with Fixed Head Camera achieves 60%, 40%, 40%, 20%, and 50% (Zeng et al., 2 Oct 2025).

For new-environment or novel-object evaluation, the reported average success rates are 6% for UMI, 16% for UMI with Fixed Head Camera, and 56% for ActiveUMI. Task-wise, ActiveUMI achieves 70% on Bottle placing, 50% on Rope boxing, 80% on Shirt folding, 30% on Block disassembly, and 50% on Take Drink from Bag. The fixed-head-camera baseline achieves 30%, 10%, 20%, 0%, and 20%, and the wrist-only UMI baseline achieves 30%, 0%, 0%, 0%, and 0% (Zeng et al., 2 Oct 2025).

These results support the paper’s main experimental conclusion: wrist-only perception is weakest, a fixed third-person view helps, and active head control helps most, especially under occlusion, long-horizon execution, and distribution shift. The evidence is especially strong for generalization, where the gap between ActiveUMI and passive-view baselines becomes largest.

6. Ablations, practical performance, and ecosystem context

The main ablation concerns active versus passive perception. The comparison among wrist-camera-only UMI, fixed head camera, and active movable head camera is used to argue that viewpoint choice itself is a meaningful control variable. The paper does not isolate backpack portability or calibration quality as independent causal factors in dedicated ablations, although it argues that portability is essential for in-the-wild collection and reports throughput and Relative Pose Error observations (Zeng et al., 2 Oct 2025).

For throughput, the paper compares direct human demonstration, ActiveUMI, and teleoperation of a real robot via VR. On Rope boxing, ActiveUMI is 2.06× slower than direct human demonstration, whereas conventional teleoperation is 3.27× slower. On Shirt folding, ActiveUMI is 1.49× slower and teleoperation is 2.63× slower. This positions ActiveUMI as slower than unconstrained human action but substantially faster than conventional robot teleoperation (Zeng et al., 2 Oct 2025).

A data-mixing study on Shirt folding compares 0% teleoperated data, +10% teleoperated data, and +1% teleoperated data, yielding 80%, 90%, and 95% success, respectively. The paper interprets this as evidence that a small amount of real teleoperated data can significantly improve a policy trained mostly on ActiveUMI data. This suggests that ActiveUMI demonstrations can serve as the dominant data source while a small amount of robot-specific grounding improves deployment performance (Zeng et al., 2 Oct 2025).

Several limitations are visible in the reported material. Block disassembly remains difficult even with ActiveUMI, at 30% success. The task list and table contents are not fully consistent. Learning-method details remain sparse because the paper uses $\pi_0$ without fully restating its internals. Calibration mathematics are practical rather than formalized. Deployment also requires a third robot arm for active viewpoint control, which increases hardware complexity (Zeng et al., 2 Oct 2025).

Within the broader UMI ecosystem, ActiveUMI is distinct from UMI-Bench 1.0, which is a real-world benchmark for UMI-style manipulation policies rather than an active-perception framework. UMI-Bench standardizes wrist-view observations, chunked pose-plus-gripper action outputs, scene reset protocols, and audit-ready rollout packaging, but it does not define an “ActiveUMI” benchmark mode or explicit support for active data acquisition or online viewpoint control (Jin et al., 9 Jun 2026). This suggests that ActiveUMI and UMI-Bench are complementary: one contributes a portable demonstration-and-policy framework with active egocentric perception, while the other provides a reproducible evaluation substrate for UMI-style deployment.