AV-ALOHA Robot Simulation Platform

• AV-ALOHA is a simulation platform that adds a dedicated 7-DoF camera arm to a bimanual manipulation setup, decoupling active vision from task execution.
• It extends the ALOHA 2 base with immersive VR teleoperation, synchronized gaze tracking, and a MuJoCo-based model to facilitate precise imitation learning.
• The platform supports varied teleoperation modes and benchmark tasks, demonstrating that active vision boosts performance in visibility-critical scenarios while revealing task-dependent trade-offs.

AV-ALOHA is a robot simulation and teleoperation platform that extends ALOHA 2 with active vision as a controllable subsystem rather than a fixed sensing modality. In its core form, it is a MuJoCo-based extension of the ALOHA 2 MuJoCo Menagerie model that adds a dedicated camera arm to a bimanual manipulation workcell, couples that embodiment to an immersive VR teleoperation interface, and uses the resulting demonstrations for imitation learning and benchmark evaluation on viewpoint-sensitive manipulation tasks (Chuang et al., 2024). Subsequent work further extended the platform into a gaze-aware benchmark by adding eye-tracked VR teleoperation, synchronized gaze logging, distractor-based robustness tests, and foveated Vision Transformer policies (Chuang et al., 21 Jul 2025).

1. Historical basis and conceptual scope

AV-ALOHA inherits its base workcell from ALOHA 2, which was introduced as an enhanced low-cost platform for bimanual teleoperation with two ViperX 6-DoF follower arms, two smaller WidowX leader arms with the same kinematic structure, four Intel RealSense D405 cameras, a 48" × 30" table, and a 20 × 20 mm aluminum extrusion frame. The ALOHA 2 release also included a MuJoCo Menagerie model of the full workcell, matched camera models, imported visual assets for the table and frame, and a system-identification procedure intended to make the simulator “more physically accurate” and useful for “teleoperation and learning in simulation” (Team et al., 2024).

AV-ALOHA is defined by the addition of active vision to that ALOHA 2 substrate. The central design move is the introduction of a third arm whose only role is viewpoint control. The platform therefore separates manipulation from sensing at the embodiment level: two arms execute the task, while the additional arm positions a stereo camera to obtain task-relevant views. This differs from wrist-camera approaches, where viewpoint is constrained by end-effector motion, and from fixed-camera setups, where camera placement is not itself part of the control problem (Chuang et al., 2024).

This design is not merely a hardware modification. In the simulation literature around ALOHA, the ALOHA Unleashed simulation component remained a smaller MuJoCo benchmark for rigid bimanual tasks and ablations, with fixed multi-view RGB observations and no claim of training in simulation and deploying on the real robot. AV-ALOHA instead elevates camera motion to a first-class part of the action stream and the demonstration dataset, making active perception an explicit subject of imitation learning rather than an environmental constant (Zhao et al., 2024).

2. Embodiment, simulated sensors, and software stack

The AV-ALOHA embodiment uses three Interbotix ViperX-300 6-DoF robotic arms, but modifies the camera arm to become 7-DoF by reusing the gripper motor and adding a small 3D-printed bracket that provides an additional pan mechanism. Functionally, the system consists of a left manipulation arm, a right manipulation arm, and an active-vision arm carrying a ZED Mini stereo camera. The two manipulation arms retain grippers; the AV arm is solely tasked with viewpoint control (Chuang et al., 2024).

The full camera system comprises six views: two fixed cameras, two wrist-mounted eye-in-hand cameras, and two stereo cameras on the AV arm. For learning and ablation, the platform evaluates all seven camera combinations among AV, Static, and Wrist: AV; AV + Static; AV + Wrist; AV + Static + Wrist; Static; Static + Wrist; and Wrist. The simulator is implemented in MuJoCo and builds on the ALOHA 2 model from MuJoCo Menagerie, extended with the AV arm to mirror the real robot system (Chuang et al., 2024).

The VR interface is implemented as a Unity application communicating through WebRTC. In the base AV-ALOHA setup, the ZED Mini stereo camera streams two 720p RGB videos at 30 fps, one to each eye in a Meta Quest 2 or 3 headset, producing a first-person stereo view for immersive teleoperation. The paper does not specify MuJoCo version, solver parameters, contact parameters, timestep, integrator, friction settings, XML structure details, camera intrinsics, stereo baseline, or fixed camera poses numerically, so the published description is architectural rather than fully parametric (Chuang et al., 2024).

The later gaze-aware extension preserves the inherited AV-ALOHA structure of two manipulation arms plus a third 7-DOF arm controlling viewpoint, but narrows the learned policy observation to the left-eye camera image $O_{\text{img}}$ and robot proprioception $O_{\text{proprio}}$. The collection pipeline remains stereo and immersive, but the policy itself consumes only the left stereo image together with joint information (Chuang et al., 21 Jul 2025).

3. Teleoperation and demonstration collection

AV-ALOHA supports two teleoperation modes. In VR-only teleoperation, the VR headset controls the AV arm, the VR hand controllers control the two manipulation arms, and controller triggers operate the grippers. In hybrid teleoperation, the VR headset still controls the AV arm, but the two follower manipulation arms are controlled by the original ALOHA 2 leader arms. The base paper uses VR-only control for simulation data collection because it is simpler and requires no extra hardware, while real-world collection uses leader arms plus VR because that arrangement reduces fatigue and improves joint-wise control (Chuang et al., 2024).

The mapping from human motion to robot motion is partly specified. The AV arm receives a target pose from the VR headset, derived from the absolute pose of the user’s head and converted into robot coordinates; after initialization, motion is handled relative to the starting pose. That target is realized by Differential Inverse Kinematics with Damped Least Squares. In VR-only manipulation, tracked hand-controller poses are mapped to the manipulation arms using a Differential IK method with a custom cost function that penalizes joint deviations from their centers and large joint displacement (Chuang et al., 2024).

Demonstrations record all three arms together. For each task in the original AV-ALOHA benchmark, the authors collected 50 episodes of human demonstrations with all three arms while recording from all cameras. The dataset stores and trains on joint position observations and joint position actions, even though teleoperation is performed in Cartesian space. Because the same trajectories are rendered under different camera subsets, the benchmark can compare perception configurations without recollecting demonstrations. In simulation, the platform can also render and record the same trajectories twice to both include and not include the AV arm, which removes a confound that would otherwise appear in camera-based comparisons (Chuang et al., 2024).

The gaze-aware extension modifies this teleoperation pipeline in several concrete ways. It replaces the Meta Quest 3 headset with a Meta Quest Pro to obtain built-in eye tracking, records at 25 FPS, and transmits gaze coordinates for left and right eyes together with the corresponding image ID. To handle latency, each image sent from robot to headset includes a unique frame ID, and returned head pose, hand pose, and gaze data include that same ID so that gaze can be aligned to the correct image. When some images do not receive a gaze label in time, the system interpolates between known eye-tracking labels (Chuang et al., 21 Jul 2025).

4. Benchmark tasks and evaluation protocols

The original AV-ALOHA simulation benchmark contains five simulation tasks and one real-world task. The simulation tasks are Peg Insertion, Slot Insertion, Pour Test Tube, Thread Needle, and Hook Package; the real-world task is Occluded Insertion. The paper explicitly distinguishes tasks that can be done without AV—Peg Insertion, Slot Insertion, and Hook Package—from tasks expected to benefit from AV—Pour Test Tube, Thread Needle, and Occluded Insertion—because the latter depend more strongly on viewpoint planning, occlusion handling, or visibility of small features (Chuang et al., 2024).

Task	Reported role of active vision	Best reported simulation result
Peg Insertion	AV is not strictly necessary	Static: Insert 48
Slot Insertion	Standard camera setup often suffices	Static + Wrist: Insert 78
Pour Test Tube	AV helps because objects are narrow and the marble is small	AV and AV + Wrist: Pour 14
Thread Needle	AV matters because the hole has limited visibility from static cameras	AV and AV + Wrist: Thread 52
Hook Package	Not strongly AV-dependent in the reported setup	Static: Hook 44

The simulation protocol evaluates 12 checkpoints, rolls out each checkpoint 50 times, and reports the best-performing checkpoint. Training uses ACT through LeRobot, with a pretrained ResNet18 visual backbone, action chunk size 50, learning rate $2.5 \times 10^{-5}$, batch size 16, and 15625 training steps; all other parameters are LeRobot defaults (Chuang et al., 2024).

The gaze-aware benchmark reorganizes the task suite. It uses six AV-ALOHA simulation tasks: CubeTransfer, PegInsertion, SlotInsertion, HookPackage, PourTestTube, and ThreadNeedle. It recollects 100 human demonstrations per task, for a total of 600 demonstrations, and adds a distractor setting with three randomly placed distractor objects of varying colors and shapes near task-relevant objects. Policies are trained per task, evaluated every 3,000 training steps for a total of 10 evaluations, and each checkpoint is tested with 50 rollouts under randomized object placement with and without distractors; the best checkpoint success rate is reported. In that benchmark, policy control is evaluated at 8.33 FPS with action chunk size $K = 16$ (Chuang et al., 21 Jul 2025).

5. Learning formulations and empirical findings

In the base AV-ALOHA study, active vision is learned jointly with manipulation rather than by a separate optimizer. Demonstrations contain the manipulation-arm behavior and the AV-arm behavior in one synchronized trajectory, and a single ACT policy is trained end-to-end over the full recorded robot behavior. The principal empirical finding is conditional rather than universal: active vision helps when visibility is the bottleneck, but it is not uniformly superior across tasks (Chuang et al., 2024).

The reported results illustrate that asymmetry. On Peg Insertion, the best insertion score is Static with 48, followed by AV + Static with 46. On Slot Insertion, the best insertion score is Static + Wrist with 78, followed by Static with 66 and AV + Static with 62. By contrast, on Pour Test Tube, AV and AV + Wrist tie for the best pour score at 14, and on Thread Needle, AV and AV + Wrist tie for the best thread score at 52. Hook Package is again a case where fixed views dominate, with Static achieving Hook 44. The paper’s interpretation is that AV helps most on tasks where visual information is the bottleneck, especially Thread Needle and Pour Test Tube, while moving cameras and larger action spaces can hurt when fixed views already provide sufficient information (Chuang et al., 2024).

Several misconceptions are therefore rejected by the benchmark itself. First, AV is not simply “better vision” in all regimes; it can add complexity without improving observability. Second, more cameras are not automatically better: the full camera stack never ranked in the top three on any task. Third, adding a weak or redundant camera can degrade performance; the paper specifically notes that in Thread Needle, adding static cameras to AV drops thread success from 52 to 26. The authors also emphasize that AV alone often performs surprisingly well on the tasks most sensitive to viewpoint selection, motivating the claim that “Active vision might be all you need” in those regimes (Chuang et al., 2024).

The gaze-aware extension replaces ACT with a policy stack built around conditional flow matching. The policy $\pi(A \mid O)$ models an action chunk $A$ of length $K=16$ conditioned on observation $O$. It learns a time-dependent vector field $v_{\theta}(z_t, t, O)$ and is trained with the stated CFM objective

$$\mathcal{L}_{\text{CFM}(\theta)} = \mathbb{E}_{t,(A,O),z_0 \sim p_0} \left[ \left\| v_\theta((1-t)z_0+tA,\ t,\ O) - (A-z_0) \right\|^2 \right].$$

At inference, the method samples $O_{\text{proprio}}$0, integrates

$O_{\text{proprio}}$1

from $O_{\text{proprio}}$2 to $O_{\text{proprio}}$3 using Euler integration with 8 discretization steps, and outputs the generated action chunk. The observation encoder uses a ViT backbone, compresses visual tokens with a Q-Former using 16 learnable queries, encodes proprioception with an MLP and 0.1 dropout, and decodes with a DiT-style transformer conditioned by AdaLN-Zero and cross-attention (Chuang et al., 21 Jul 2025).

6. Gaze-aware extension, efficiency gains, and reproducibility limits

The gaze-aware extension of AV-ALOHA is notable because it turns the platform from an active-camera benchmark into a benchmark for active perception and human attention. It introduces two ways of incorporating gaze. In the two-stage model, Fov-UNet, a UNet with a ResNet18 backbone predicts a gaze heatmap, a spatial softmax converts that heatmap into a 2D keypoint, and the predicted gaze drives foveated tokenization; the gaze predictor is trained separately for 30,000 steps with batch size 64 and learning rate $O_{\text{proprio}}$4 because the foveation operation is non-differentiable. In the end-to-end model, Fov-Act, gaze becomes part of the action space, so the policy jointly predicts future robot actions and future gaze points without major architectural changes (Chuang et al., 21 Jul 2025).

The foveated observation design uses a custom 20-patch pattern with high-resolution small patches near the gaze center and larger patches in concentric peripheral rings. This is contrasted with a fine uniform tokenizer using an $O_{\text{proprio}}$5 grid of $O_{\text{proprio}}$6 patches for 324 tokens and a coarse uniform tokenizer using a $O_{\text{proprio}}$7 grid of $O_{\text{proprio}}$8 patches for 20 tokens. The reported token reduction from 324 to 20 corresponds to about 94% fewer tokens. Table 3 further reports ViT-only latency and FLOPs reductions, and end-to-end policy gains on RTX 3090: Fine training takes 833.2 ms/step and 20949 MiB, whereas Fov-Act takes 108.2 ms/step and 3937 MiB; Fine inference takes 334.7 ms/chunk, whereas Fov-Act takes 87.9 ms/chunk. These figures support the paper’s summary claims of about 7x faster training and about 3x faster inference (Chuang et al., 21 Jul 2025).

Performance gains are concentrated on high-precision and distractor-heavy settings. Without pretraining, Fov-UNet achieves CubeTransfer 100, HookPackage 56, PourTestTube 84, and ThreadNeedle 74, while distractor settings show especially strong foveated robustness on PegInsertion, PourTestTube, ThreadNeedle, and HookPackage. With MAE pretraining, Fov-UNet remains strongest on precise tasks such as PourTestTube and ThreadNeedle. The paper also records an important failure mode: on HookPackage, Fov-Act can struggle when the robot must shift gaze from a package to a tiny hook in the periphery, because peripheral regions are downsampled (Chuang et al., 21 Jul 2025).

Across the AV-ALOHA literature, the main reproducibility limitations are explicit. The base platform papers do not provide full MJCF or XML structure, scene dimensions, camera placements numerically, contact and solver settings, or exact action-vector dimensionality. The original AV-ALOHA paper does not provide calibration and coordinate-frame conventions, hand-eye calibration procedures, or direct repository inventory in the manuscript text. The gaze-aware extension does not clearly specify a train/validation/test demonstration split, and its exact low-level action representation is omitted in the provided text. A plausible implication is that AV-ALOHA should be understood as a well-specified research architecture and benchmark family, but not as a fully self-contained implementation manual (Chuang et al., 2024, Chuang et al., 21 Jul 2025).

What the published record does establish with clarity is the platform’s identity: AV-ALOHA is a MuJoCo-based, ALOHA-derived simulation environment in which bimanual manipulation, active camera placement, immersive teleoperation, and imitation learning are coupled in a single embodied system. Later extensions show that the same platform can also support eye-tracked demonstrations, gaze-conditioned foveation, distractor robustness studies, and policies that jointly reason about where to act and where to look (Team et al., 2024).