Sim-to-Real Transfer of Compliant Bipedal Locomotion on Torque Sensor-Less Gear-Driven Humanoid (2204.03897v4)
Abstract: Sim-to-real is a mainstream method to cope with the large number of trials needed by typical deep reinforcement learning methods. However, transferring a policy trained in simulation to actual hardware remains an open challenge due to the reality gap. In particular, the characteristics of actuators in legged robots have a considerable influence on sim-to-real transfer. There are two challenges: 1) High reduction ratio gears are widely used in actuators, and the reality gap issue becomes especially pronounced when backdrivability is considered in controlling joints compliantly. 2) The difficulty in achieving stable bipedal locomotion causes typical system identification methods to fail to sufficiently transfer the policy. For these two challenges, we propose 1) a new simulation model of gears and 2) a method for system identification that can utilize failed attempts. The method's effectiveness is verified using a biped robot, the ROBOTIS-OP3, and the sim-to-real transferred policy can stabilize the robot under severe disturbances and walk on uneven surfaces without using force and torque sensors.
- J. Lee, et al., “Learning quadrupedal locomotion over challenging terrain,” Science Robotics, vol. 5, no. 47, Oct 2020. [Online]. Available: http://dx.doi.org/10.1126/scirobotics.abc5986
- J. Siekmann, et al., “Blind bipedal stair traversal via sim-to-real reinforcement learning,” https://arxiv.org/abs/2105.08328, 2021.
- X. B. Peng, et al., “Learning agile robotic locomotion skills by imitating animals,” in Robotics: Science and Systems, 07 2020.
- J. Tan, et al., “Sim-to-real: Learning agile locomotion for quadruped robots,” 2018. [Online]. Available: https://arxiv.org/abs/1804.10332
- Y. Du, et al., “Auto-tuned sim-to-real transfer,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 1290–1296.
- W. Yu, et al., “Sim-to-real transfer for biped locomotion,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019, pp. 3503–3510.
- J. Tobin, et al., “Domain randomization for transferring deep neural networks from simulation to the real world,” https://arxiv.org/abs/1703.06907, 2017.
- Y. Chebotar, et al., “Closing the sim-to-real loop: Adapting simulation randomization with real world experience,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8973–8979.
- Z. Xie, et al., “Learning locomotion skills for cassie: Iterative design and sim-to-real,” in Proceedings of the Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 100, 2020, pp. 317–329.
- J. Hwangbo, et al., “Learning agile and dynamic motor skills for legged robots,” Science Robotics, vol. 4, no. 26, Jan 2019. [Online]. Available: http://dx.doi.org/10.1126/scirobotics.aau5872
- A. Wang and S. Kim, “Directional efficiency in geared transmissions: Characterization of backdrivability towards improved proprioceptive control,” in 2015 IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 1055–1062.
- H. Matsuki, et al., “Bilateral drive gear—a highly backdrivable reduction gearbox for robotic actuators,” IEEE/ASME Transactions on Mechatronics, vol. 24, no. 6, pp. 2661–2673, 2019.
- S. Hyon and G. Cheng, “Passivity-based full-body force control for humanoids and application to dynamic balancing and locomotion,” in 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2006, pp. 4915–4922.
- G. Mesesan, et al., “Dynamic walking on compliant and uneven terrain using dcm and passivity-based whole-body control,” in 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), 2019, pp. 25–32.
- H. Suzuki, et al., “Torque based stabilization control for torque sensorless humanoid robots,” in 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids), 2017, pp. 425–431.
- J. Siekmann, et al., “Sim-to-real learning of all common bipedal gaits via periodic reward composition,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 7309–7315.
- X. B. Peng and M. van de Panne, “Learning locomotion skills using deeprl: Does the choice of action space matter?” in Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation, ser. SCA ’17, 2017, pp. 12:1–12:13.
- M. Taylor, et al., “Learning bipedal robot locomotion from human movement,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 2797–2803.
- T. Haarnoja, et al., “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” https://arxiv.org/abs/1801.01290, 2018.
- E. Todorov, et al., “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, pp. 5026–5033.
- J. Tan, et al., “Simulation-based design of dynamic controllers for humanoid balancing,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, pp. 2729–2736.
- ROBOTIS CO.,LTD, “ROBOTIS-OP3 INTRODUCTION,” https://emanual.robotis.com/docs/en/platform/op3/introduction/(accessed on 28 Feb 2023).
- T. Akiba, et al., “Optuna: A next-generation hyperparameter optimization framework,” in Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.