ARCADE: AR for Scalable Robot Demonstrations

• ARCADE is an augmented reality framework for scalable robot imitation learning that turns a single human demonstration into many high-quality trajectories.
• It integrates AR visualization, motion planning, and DTW-based automatic validation to overcome traditional challenges in demonstration collection and data diversity.
• The framework boosts sample efficiency and performance in manipulation tasks like reach, push, and pick-and-place, demonstrating practical success in household applications.

ARCADE, short for Augmented Reality for Collection and generAtion of DEmonstrations, is an augmented-reality-driven framework for scaling both the collection and the generation of demonstrations for robot imitation learning, especially Behavioral Cloning (BC), in robot manipulation (Yang et al., 2024). It targets two long-standing bottlenecks in BC: demonstration collection is often cumbersome for non-experts, and BC is data-hungry, typically requiring hundreds of demonstrations to mitigate covariate shift. ARCADE addresses these bottlenecks by making demonstration collection as simple as users performing daily tasks using their hands in AR, and by expanding a single human-derived demonstration into a large set of synthetic demonstrations through AR-in-the-loop validation followed by automatic validation.

1. Conceptual role in imitation learning

Robot Imitation Learning is a technique in which agents learn by mimicking human demonstrations. Within that setting, ARCADE is designed as a framework for robot manipulation tasks that couples augmented reality with scalable data generation. Its central claim is not that policy learning itself is novel, but that the data pipeline can be restructured so that a single natural AR demonstration can be magnified into tens to hundreds of high-quality trajectories that respect user preferences and can be used by plain BC (Yang et al., 2024).

The framework is explicitly motivated by the difficulty of existing collection interfaces such as teleoperation and kinesthetic teaching, and by the large number of demonstrations commonly required by BC. ARCADE therefore treats demonstration acquisition and dataset amplification as a single system problem. This suggests a shift in emphasis from expert-only collection interfaces toward user-facing, AR-mediated interaction, while retaining a standard IL training backend.

2. System architecture and formalization

ARCADE runs on a Microsoft HoloLens 2 augmented reality headset paired with a physical robot, a real Fetch mobile manipulator with a 7-DoF arm (Yang et al., 2024). During demonstration collection, the user wears the HoloLens and sees a digital twin of the robot overlaid in their egocentric view. The robot end effector follows the user’s hand via inverse kinematics, and the gripper is controlled by the user’s finger–thumb distance. Unity Robotics Hub’s ROS-TCP-Endpoint/Connector is used to stream and log trajectories on a separate machine for learning.

A single demonstration is represented as a time-ordered sequence

$\tau^{AR} := \{(p_i, j_i, g_i)\}_{i=1}^N,$

where $p_i$ is the end-effector pose, $j_i$ are arm joint angles, $g_i$ is a binary gripper state, and $i$ indexes time steps. Control is modeled as an MDP

$M = \langle S, A, R, T, \rho_0 \rangle,$

where $S$ is the state space, $A$ the action space, $T$ a deterministic transition function, and $\rho_0$ the initial-state distribution. For Reach and Push, $p_i$0 consists of 7 joint angles; for Pick-And-Place, it has 8 dimensions by adding the gripper angle. Actions are delta joint commands, with an added gripper delta for grasping tasks.

The framework necessarily aligns AR and robot coordinate frames to overlay the digital twin and drive inverse kinematics from hand poses, but the paper does not report an explicit SE(3) calibration procedure or transformation equations. In practice, the alignment is realized by placing and maintaining a consistent digital twin and streaming hand poses to the robot-control backend.

3. Demonstration collection and generation pipeline

ARCADE uses a four-stage workflow (Yang et al., 2024). First, the user performs the task naturally with their hands while wearing the HoloLens, and the system logs $p_i$1. This stage is intended to avoid teleoperation devices and kinesthetic teaching.

Second, ARCADE generates candidate demonstrations by sampling and following waypoints from $p_i$2 while protecting key poses. Let $p_i$3 be the poses extracted from $p_i$4. Waypoints are sampled every $p_i$5 steps, where $p_i$6 is randomly drawn from a range $p_i$7 for each selection. MoveIt’s motion planner and inverse kinematics then generate a trajectory that visits the sampled waypoint set $p_i$8. The resulting random-interval sampling is used to diversify joint-space paths while still reaching critical points.

The Key-Poses Detector protects two kinds of events: grasp and release, obtained directly from the logged gripper states, and sharp turns that coincide with slow motion. The detector computes an angle-based sharp-turn score and a density score over a sliding window, and marks poses as key when both thresholds are exceeded. The protected key set is the union of these poses with the grasp/release indices, and generated candidates must include them.

Third, ARCADE visualizes the generated candidates in AR for rapid human filtering. This produces a user-accepted set $p_i$9, which captures user preferences and safety constraints without requiring re-demonstration. The paper emphasizes that this step is meant to reject candidates with unstable inverse-kinematics segments, potentially hazardous trajectories such as skimming near the table, or behaviors that conflict with user style.

Fourth, after this initial AR-in-the-loop curation, ARCADE continues generating new candidates and validates them automatically with Dynamic Time Warping (DTW). With

$j_i$0

it computes

$j_i$1

For a new candidate $j_i$2, it excludes one random element $j_i$3 from the accepted set and computes

$j_i$4

The candidate is accepted if

$j_i$5

where $j_i$6 controls strictness. Smaller $j_i$7 enforces closer similarity to the accepted set, whereas larger $j_i$8 allows more diversity. Accepted trajectories are added to the scalable set $j_i$9.

4. Policy learning and experimental protocol

Policy learning in ARCADE uses vanilla Behavioral Cloning with a GaussianMLP policy class (Yang et al., 2024). Two multilayer perceptrons predict the mean $g_i$0 and standard deviation $g_i$1 of a Gaussian action distribution, from which actions are sampled as

$g_i$2

Training uses state–action pairs $g_i$3 from the demonstration set. The paper does not specify a particular loss in equation form, training hyperparameters, or compute details.

The experimental platform is a Fetch mobile manipulator. Three archetypal tasks are evaluated. In 3-Waypoints-Reach, the end effector must visit three predefined waypoints $g_i$4, $g_i$5, and $g_i$6. In Push, the robot pushes a tabletop object from a start to a goal point. In Pick-And-Place, the robot grasps an object, moves it to a new location, and releases it.

Evaluation uses Task Completion Error (TCE, meters). For Reach, TCE is the average of the minimum distances reached to the three waypoints. For Push and Pick-And-Place, TCE is the distance between the final object position and its goal. For each task and each policy, 10 trials are run, and mean $g_i$7 standard deviation of TCE are reported.

The comparison includes four training pipelines: ARCADE($g_i$8) trained on a single AR demonstration; ARCADE($g_i$9) trained on 100 ARCADE-generated demonstrations; BL($i$0) trained on a single kinesthetic demonstration; and BL($i$1) trained on 100 demonstrations generated by the same ARCADE procedure but seeded from the kinesthetic demonstration rather than the AR demonstration.

5. Empirical performance and deployment behavior

The reported results show that ARCADE improves sample efficiency and final performance across the three manipulation tasks, especially when scaled to 100 demonstrations (Yang et al., 2024). A two-way ANOVA shows significant main effects of both collection method and dataset size, and a significant interaction, all at $i$2 for all three tasks.

Tukey’s HSD reveals that all four policies differ significantly in Reach and Pick-And-Place at $i$3, ordered best to worst as ARCADE($i$4) > BL($i$5) > ARCADE($i$6) > BL($i$7). For Push, ARCADE($i$8) and BL($i$9) are statistically indistinguishable at $M = \langle S, A, R, T, \rho_0 \rangle,$0, and both significantly outperform BL($M = \langle S, A, R, T, \rho_0 \rangle,$1) at $M = \langle S, A, R, T, \rho_0 \rangle,$2, which in turn outperforms ARCADE($M = \langle S, A, R, T, \rho_0 \rangle,$3) at $M = \langle S, A, R, T, \rho_0 \rangle,$4. These results indicate that datasets generated through ARCADE are at least as good as those generated from kinesthetic-seeded data, and that scaling to around 100 demonstrations substantially boosts BC performance.

The paper also reports deployment on a real household task, Pouring-Water, in which a policy is trained from a single AR demonstration to grasp a bottle and pour water into a cup. The reported success rate is 80%, corresponding to 8/10 trials, with success defined as water being poured into the cup (Yang et al., 2024). Failures were attributed to deformation of the plastic bottle, which perturbed the grasp and pour dynamics.

These results are paired with practical deployment observations. The AR validation stage is described as important for safety and quality because it enables rejection of unnatural motions caused by inverse-kinematics instability, unsafe proximity to surfaces, and behaviors that deviate from user preferences. The automatic validation stage then scales these implicitly specified constraints through DTW similarity filtering.

6. Scope, limitations, and reproducibility

ARCADE focuses on proprioceptive BC for manipulation tasks and does not explore policies with visual observations (Yang et al., 2024). Observations and actions are low-dimensional robot proprioception and their deltas; no RGB or RGB-D inputs are used. A common misunderstanding would therefore be to treat the framework as a vision-based imitation-learning system. The paper instead studies a data-collection and trajectory-generation framework whose learned policies operate on low-dimensional state.

Several limitations are explicit. Success depends on accurate AR hand tracking and a stable AR-to-robot alignment, but the paper does not report a concrete calibration method or transformation equations. Trajectory generation relies on waypoint sampling plus motion planning rather than an explicit optimization objective, so the coverage–similarity trade-off is governed by the waypoint sampling range $M = \langle S, A, R, T, \rho_0 \rangle,$5 and the DTW threshold $M = \langle S, A, R, T, \rho_0 \rangle,$6, and these hyperparameters may require task-specific tuning. The paper also does not report time-per-demo or formal usability metrics, although it notes that user oversight is limited to approving a small initial batch, on the order of 10–15 candidates, after which the process scales automatically.

Reproducibility is supported through an ARCADE homepage and through the use of publicly available components, especially Unity Robotics Hub’s ROS-TCP-Endpoint/Connector and MoveIt. The paper provides pseudocode for the two core algorithmic components, the Key-Poses Detector and Automatic Validation. Within the stated scope, ARCADE demonstrates that natural AR interaction can be used to collect a single demonstration, expand it into a scalable dataset, and train plain Behavioral Cloning policies that perform strongly on multiple real-robot manipulation tasks.