ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback

Abstract: Recent progress in imitation learning from human demonstrations has shown promising results in teaching robots manipulation skills. To further scale up training datasets, recent works start to use portable data collection devices without the need for physical robot hardware. However, due to the absence of on-robot feedback during data collection, the data quality depends heavily on user expertise, and many devices are limited to specific robot embodiments. We propose ARCap, a portable data collection system that provides visual feedback through augmented reality (AR) and haptic warnings to guide users in collecting high-quality demonstrations. Through extensive user studies, we show that ARCap enables novice users to collect robot-executable data that matches robot kinematics and avoids collisions with the scenes. With data collected from ARCap, robots can perform challenging tasks, such as manipulation in cluttered environments and long-horizon cross-embodiment manipulation. ARCap is fully open-source and easy to calibrate; all components are built from off-the-shelf products. More details and results can be found on our website: https://stanford-tml.github.io/ARCap

• The paper introduces ARCap, a portable system that leverages real-time AR feedback to collect robot-executable human demonstrations.
• The paper’s approach employs AR headsets and motion capture gloves to simulate robot kinematics and prevent joint, speed, and collision violations.
• Experimental studies show that robots trained with ARCap data achieve a 35% higher success rate in complex tasks compared to standard methods.

ARCap: Enhancing Robot Learning through Augmented Reality Feedback

The paper "ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback" introduces an innovative approach for enhancing robot imitation learning by leveraging augmented reality (AR) technology. The focus is on addressing the challenges associated with collecting high-quality demonstration data for robotics training, particularly in environments where physical robot hardware is unavailable.

Key Contributions

ARCap emerges as a portable, open-source system designed to improve the quality of human demonstration data through real-time AR feedback. This feedback aids users in providing robot-executable data by simulating robot kinematics and detecting potential issues such as joint and speed limit violations or environmental collisions.

1. Real-time Visual and Haptic Feedback: By providing continuous AR-based feedback, ARCap allows users to adjust their data collection methods. The system visually represents a virtual robot's kinematics overlaid on the user's hand movements, offering instant feedback on movement feasibility relative to robot constraints.
2. Cross-Embodiment Data Collection: The system supports varied robot embodiments, such as parallel-jaw grippers and dexterous multifinger hands. This adaptability is key in providing relevant feedback and guidance for different robotic tasks without needing hardware redesign.
3. Portability and Accessibility: Built using off-the-shelf products, ARCap emphasizes ease of use and deployment, fostering accessibility for users without extensive robotics expertise. Its design allows for in-the-wild data collection, significantly enhancing the scalability of demonstration datasets.

Methodology

ARCap utilizes a suite of hardware, including AR headsets and motion capture gloves, to deliver precise tracking and feedback mechanisms. The system's architecture ensures comprehensive feedback on camera visibility, kinematic compliance, and potential collisions. The AR interface provides a robust framework where users can intuitively adjust actions, reducing the mismatch between human demonstrations and robotic executability.

The paper details the method of capturing and processing data using ARCap. Human actions are recorded via motion capture and transformed into a format suitable for imitation learning. These data, enhanced with AR-based feedback during collection, significantly improve the reproducibility and relevance of the resulting robot training policies.

Experimental Results

Through extensive user studies, ARCap demonstrated a marked improvement in data collection quality over previous systems such as DexCap. In practical tests, robots trained with ARCap-enhanced data performed more successfully in complex tasks. For instance, robots trained to operate in cluttered environments achieved a 35% higher success rate when using data collected through ARCap, compared to other methodologies.

Furthermore, the capability of ARCap to facilitate long-horizon manipulation tasks was evidenced in multi-stage Lego assembly tasks, showcasing the system's effectiveness across different robot embodiments.

Implications and Future Directions

ARCap holds significant implications for the field of robot learning, particularly in democratizing access to high-quality training data. By reducing the need for physical robot systems during data collection, and providing intuitive AR-based feedback, the system empowers a broader range of users to contribute valuable training datasets.

Future research may explore enhancements in AR feedback mechanisms and the integration of advanced visual reasoning systems. Additionally, the potential application of ARCap in mobile and humanoid robotics, with comprehensive tracking of full-body movements, presents a promising direction. Integrating large vision-LLMs could further refine user interactions and improve the efficiency of the demonstration data collection process.

In conclusion, ARCap represents a pivotal advancement in leveraging AR technology for robot imitation learning, offering a versatile, scalable solution for high-quality demonstration data collection.