TWIST: Teleoperated Whole-Body Imitation System (2505.02833v1)

Abstract: Teleoperating humanoid robots in a whole-body manner marks a fundamental step toward developing general-purpose robotic intelligence, with human motion providing an ideal interface for controlling all degrees of freedom. Yet, most current humanoid teleoperation systems fall short of enabling coordinated whole-body behavior, typically limiting themselves to isolated locomotion or manipulation tasks. We present the Teleoperated Whole-Body Imitation System (TWIST), a system for humanoid teleoperation through whole-body motion imitation. We first generate reference motion clips by retargeting human motion capture data to the humanoid robot. We then develop a robust, adaptive, and responsive whole-body controller using a combination of reinforcement learning and behavior cloning (RL+BC). Through systematic analysis, we demonstrate how incorporating privileged future motion frames and real-world motion capture (MoCap) data improves tracking accuracy. TWIST enables real-world humanoid robots to achieve unprecedented, versatile, and coordinated whole-body motor skills--spanning whole-body manipulation, legged manipulation, locomotion, and expressive movement--using a single unified neural network controller. Our project website: https://humanoid-teleop.github.io

• The paper presents TWIST, a system enabling real-time humanoid robot teleoperation through whole-body imitation using a combined RL/BC neural controller and advanced motion retargeting.
• TWIST employs a two-stage teacher-student policy training framework and integrates simulated and real-world motion capture data to achieve robust and accurate real-time performance.
• The system demonstrates successful zero-shot deployment on real robots, performing diverse tasks with improved tracking accuracy, highlighting its potential for dexterous, general-purpose domestic robots.

Overview of TWIST: Teleoperated Whole-Body Imitation System

The paper presents the Teleoperated Whole-Body Imitation System (TWIST), a sophisticated system aimed at enhancing humanoid robot teleoperation capabilities through real-time whole-body motion imitation. TWIST leverages human motion data to control humanoid robots in various tasks, representing a pivotal advancement toward versatile robotic locomotion and manipulation.

Core Contributions

TWIST introduces a paradigm where humanoid robots mimic human-like movements through teleoperation, facilitated by motion capture (MoCap) devices. The primary contributions of this work are:

1. Robust Controller Design: The system employs a single unified neural network controller, trained using a combination of reinforcement learning (RL) and behavior cloning (BC). This approach ensures robust and precise tracking of real-time motions, allowing the robot to perform complex motor skills.
2. Advanced Motion Retargeting: The paper details an innovative motion retargeting process, which successfully bridges the embodiment gap between human movements and humanoid robots. It optimizes both joint positions and orientations to maintain smoothness and tracking accuracy.
3. Two-Stage Policy Training Framework: TWIST utilizes a teacher-student framework, where a privileged expert policy accesses future motion frames to teach a deployable student policy. This methodology overcomes the challenges of real-time motion conversion, mitigating hesitant robotic behaviors.
4. Integration of Real-World and Simulated Data: The research effectively combines large-scale simulated data with small-scale, real-world MoCap captures, addressing the distribution shift and enhancing the generalization capabilities of the trained controller.

Numerical Results and Claims

Throughout rigorous experiments, TWIST demonstrates substantial improvements in tracking accuracy and motion smoothness compared to baseline controllers. The system achieves remarkable zero-shot deployment success on real-world humanoids such as Unitree G1, showcasing diverse tasks from whole-body manipulation to expressive dance motions. The experiments confirm that integrating a small set of real-world MoCap data significantly reduces tracking errors, enhancing the system's applicability in domestic scenarios.

Implications and Future Directions

With TWIST, the scope of humanoid teleoperation extends to complex, coordinated whole-body tasks in real-time. This development has significant implications for general-purpose robotic intelligence, particularly in home environments where adaptability and dexterity are paramount.

Looking forward, potential research could focus on integrating richer sensory feedback systems to provide operators with real-time, immersive robotic perspectives. Additionally, refining teleoperation latency and applying vision-based motion capture methods can enhance the system's robustness and accessibility.

Conclusion

The Teleoperated Whole-Body Imitation System (TWIST) sets a high standard for humanoid robotic teleoperation, overcoming existing limitations in motion coordination and tracking fidelity. By adopting advanced learning-based control mechanisms and strategic data integration, this paper makes significant strides towards more intelligent, flexible, and human-like robotic systems.