Adaptive Energy Regularization for Autonomous Gait Transition and Energy-Efficient Quadruped Locomotion (2403.20001v1)
Abstract: In reinforcement learning for legged robot locomotion, crafting effective reward strategies is crucial. Pre-defined gait patterns and complex reward systems are widely used to stabilize policy training. Drawing from the natural locomotion behaviors of humans and animals, which adapt their gaits to minimize energy consumption, we propose a simplified, energy-centric reward strategy to foster the development of energy-efficient locomotion across various speeds in quadruped robots. By implementing an adaptive energy reward function and adjusting the weights based on velocity, we demonstrate that our approach enables ANYmal-C and Unitree Go1 robots to autonomously select appropriate gaits, such as four-beat walking at lower speeds and trotting at higher speeds, resulting in improved energy efficiency and stable velocity tracking compared to previous methods using complex reward designs and prior gait knowledge. The effectiveness of our policy is validated through simulations in the IsaacGym simulation environment and on real robots, demonstrating its potential to facilitate stable and adaptive locomotion.
- W. Xi, Y. Yesilevskiy, and C. D. Remy, “Selecting gaits for economical locomotion of legged robots,” The International Journal of Robotics Research, vol. 35, no. 9, pp. 1140–1154, 2016.
- J. Siekmann, Y. Godse, A. Fern, and J. Hurst, “Sim-to-real learning of all common bipedal gaits via periodic reward composition,” in IEEE International Conference on Robotics and Automation, 2021.
- G. B. Margolis and A. Pulkit, “Walk these ways: Tuning robot control for generalization with multiplicity of behavior,” in Conference on Robot Learning, 2023.
- N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Conference on Robot Learning, 2021.
- G. Feng, H. Zhang, Z. Li, X. B. Peng, B. Basireddy, L. Yue, Z. Song, L. Yang, Y. Liu, K. Sreenath, et al., “Genloco: Generalized locomotion controllers for quadrupedal robots,” in Conference on Robot Learning. PMLR, 2023.
- J. Liang, V. Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox, “Gpu-accelerated robotic simulation for distributed reinforcement learning,” in Conference on Robot Learning, 2018.
- V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance GPU based physics simulation for robot learning,” in Conference on Neural Information Processing Systems, 2021.
- Z. Fu, A. Kumar, J. Malik, and D. Pathak, “Minimizing energy consumption leads to the emergence of gaits in legged robots,” in Conference on Robot Learning, 2021.
- M. Hutter, C. Gehring, A. Lauber, F. Gunther, C. D. Bellicoso, V. Tsounis, P. Fankhauser, R. Diethelm, S. Bachmann, M. Bloesch, H. Kolvenbach, M. Bjelonic, L. Isler, and K. Meyer, “Anymal - toward legged robots for harsh environments,” Advanced Robotics, vol. 31, no. 17, pp. 918–931, 2017.
- “Unitree robotics, go1,” https://www.unitree.com/products/go1, online; accessed Jun. 2022.
- J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics, vol. 4, no. 26, p. eaau5872, 2019.
- G. Ji, J. Mun, H. Kim, and J. Hwangbo, “Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4630–4637, 2022.
- Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath, “Reinforcement learning for robust parameterized locomotion control of bipedal robots,” in IEEE International Conference on Robotics and Automation, 2021.
- S. Chen, B. Zhang, M. W. Mueller, A. Rai, and K. Sreenath, “Learning torque control for quadrupedal locomotion,” in IEEE-RAS International Conference on Humanoid Robots, 2023.
- T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, p. eabk2822, 2022.
- J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science Robotics, vol. 5, no. 47, p. eabc5986, 2020.
- S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo, “Learning quadrupedal locomotion on deformable terrain,” Science Robotics, vol. 8, no. 74, p. eade2256, 2023.
- G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal, “Rapid locomotion via reinforcement learning,” The International Journal of Robotics Research, vol. 43, no. 4, pp. 572–587, 2024.
- M. H. Raibert, “Trotting, pacing and bounding by a quadruped robot,” Journal of Biomechanics, vol. 23, pp. 79–98, 1990.
- X. Da, Z. Xie, D. Hoeller, B. Boots, A. Anandkumar, Y. Zhu, B. Babich, and A. Garg, “Learning a contact-adaptive controller for robust, efficient legged locomotion,” in Conference on Robot Learning, 2020.
- Y. Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots, “Fast and efficient locomotion via learned gait transitions,” in Conference on Robot Learning, 2021.
- H. Duan, A. Malik, J. Dao, A. Saxena, K. Green, J. Siekmann, A. Fern, and J. Hurst, “Sim-to-real learning of footstep-constrained bipedal dynamic walking,” in IEEE International Conference on Robotics and Automation, 2022.
- S. Le Cleac’h, T. A. Howell, S. Yang, C.-Y. Lee, J. Zhang, A. Bishop, M. Schwager, and Z. Manchester, “Fast contact-implicit model predictive control,” IEEE Transactions on Robotics, vol. 40, pp. 1617–1629, 2024.
- G. Gabrielli, “What price speed? specific power required for propulsion of vehicles,” Mechanical Engineering-CIME, vol. 133, no. 10, pp. 4–5, 2011.
- V. Tucker, “The energetic cost of moving about: Walking and running are extremely inefficient forms of locomotion. much greater efficiency is achieved by birds, fish, and bicyclists.” American Scientist, vol. 63, no. 4, pp. 413–419, 1975.
- H. Ralston, “Energy-speed relation and optimal speed during level walking.” Internationale Zeitschrift für angewandte Physiologie einschließlich Arbeitsphysiologie, vol. 17, no. 4, pp. 277–283, 1958.
- B. Umberger and P. Martin, “Mechanical power and efficiency of level walking with different stride rates.” Journal of Experimental Biology, vol. 210, no. 18, pp. 3255–3265, 2007.
- C. Chevallereau and Y. Aoustin, “Optimal reference trajectories for walking and running of a biped robot.” Robotica, vol. 19, no. 5, pp. 557–569, 2001.
- C. D. Remy, “Optimal exploitation of natural dynamics in legged locomotion.” Ph.D. dissertation, Eidgenossische Technische Hochschule., 2011.
- K. Kiguchi, Y. Kusumoto, K. Watanabe, I. K, and F. T, “Energy-optimal gait analysis of quadruped robots.” Artificial Life and Robotics., vol. 6, no. 3, pp. 120–125, 2002.
- A. Muraro, C. Chevallereau, and Y. Aoustin, “Optimal trajectories for a quadruped robot with trot, amble and curvet gaits for two energetic criteria.” Multibody System Dynamics, vol. 9, no. 1, pp. 39–62, 2003.
- Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and D. Pathak, “Robot parkour learning,” in Conference on Robot Learning, 2023.
- L. Sun, H. Zhang, W. Xu, and M. Tomizuka, “Paco: Parameter-compositional multi-task reinforcement learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 21 495–21 507, 2022.
- K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi, “Xirl: Cross-embodiment inverse reinforcement learning,” in Conference on Robot Learning, 2022, pp. 537–546.
- L. Sun, H. Zhang, W. Xu, and M. Tomizuka, “Efficient multi-task and transfer reinforcement learning with parameter-compositional framework,” Robotics and Automation Letters, 2023.
- Z.-H. Yin, L. Sun, H. Ma, M. Tomizuka, and W.-J. Li, “Cross domain robot imitation with invariant representation,” in IEEE International Conference on Robotics and Automation, 2022, pp. 455–461.
- X. Cheng, A. Kumar, and D. Pathak, “Legs as manipulator: Pushing quadrupedal agility beyond locomotion,” in IEEE International Conference on Robotics and Automation, 2023.
- E. Vollenweider, M. Bjelonic, V. Klemm, N. Rudin, J. Lee, and M. Hutter, “Advanced skills through multiple adversarial motion priors in reinforcement learning,” in IEEE International Conference on Robotics and Automation, 2023.
- Boyuan Liang (9 papers)
- Lingfeng Sun (25 papers)
- Xinghao Zhu (26 papers)
- Bike Zhang (17 papers)
- Ziyin Xiong (1 paper)
- Chenran Li (18 papers)
- Koushil Sreenath (90 papers)
- Masayoshi Tomizuka (261 papers)