OGMP: Oracle Guided Multi-mode Policies for Agile and Versatile Robot Control (2403.04205v3)
Abstract: The efficacy of reinforcement learning for robot control relies on the tailored integration of task-specific priors and heuristics for effective exploration, which challenges their straightforward application to complex tasks and necessitates a unified approach. In this work, we define a general class for priors called oracles that generate state references when queried in a closed-loop manner during training. By bounding the permissible state around the oracle's ansatz, we propose a task-agnostic oracle-guided policy optimization. To enhance modularity, we introduce task-vital modes, showing that a policy mastering a compact set of modes and transitions can handle infinite-horizon tasks. For instance, to perform parkour on an infinitely long track, the policy must learn to jump, leap, pace, and transition between these modes effectively. We validate this approach in challenging bipedal control tasks: parkour and diving using a 16 DoF dynamic bipedal robot, HECTOR. Our method results in a single policy per task, solving parkour across diverse tracks and omnidirectional diving from varied heights up to 2m in simulation, showcasing versatile agility. We demonstrate successful sim-to-real transfer of parkour, including leaping over gaps up to 105 % of the leg length, jumping over blocks up to 20 % of the robot's nominal height, and pacing at speeds of up to 0.6 m/s, along with effective transitions between these modes in the real robot.
- R. McN. Alexander. The gaits of bipedal and quadrupedal animals. IJRR, 1984.
- Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors. arXiv preprint arXiv:2203.1713, 2022.
- Extreme parkour with legged robots. In RoboLetics: Workshop @CoRL 2023, 2023.
- Diffusion policy: Visuomotor policy learning via action diffusion, 2023.
- The mit humanoid robot: Design, motion planning, and control for acrobatic behaviors. In 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), 2021.
- Opt-mimic: Imitation of optimized trajectories for dynamic quadruped behaviors. In ICRA, 2023.
- Repetitive extreme-acceleration (14-g) spatial jumping with salto-1p. In IROS, 2017.
- CoMic: Complementary task learning &; mimicry for reusable skills. In ICML, Proceedings of Machine Learning Research, 2020.
- Dtc: Deep tracking control. Science Robotics, 2024.
- Rl + model-based control: Using on-demand optimal control to learn versatile legged locomotion. IEEE RAL, 2023.
- Learning multimodal bipedal locomotion and implicit transitions: A versatile policy approach. In IEEE IROS, 2023.
- Force-and-moment-based model predictive control for achieving highly dynamic locomotion on bipedal robots. In IEEE CDC, 2021.
- Rapid Locomotion via Reinforcement Learning. In Proceedings of RSS, 2022.
- Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 2022.
- Centroidal trajectory generation and stabilization based on preview control for humanoid multi-contact motion. IEEE RAL, 2022.
- Efficient multicontact pattern generation with sequential convex approximations of the centroidal dynamics. IEEE T-RO, 2021.
- Advanced skills by learning locomotion and local navigation end-to-end. In IEEE IROS, 2022.
- Dynamics-aware unsupervised discovery of skills. In ICLR, 2020.
- Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning. In RSS, 2021.
- Robot parkour learning. In CoRL, 2023.