Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ViNL: Visual Navigation and Locomotion Over Obstacles (2210.14791v3)

Published 26 Oct 2022 in cs.RO, cs.AI, cs.CV, and cs.LG

Abstract: We present Visual Navigation and Locomotion over obstacles (ViNL), which enables a quadrupedal robot to navigate unseen apartments while stepping over small obstacles that lie in its path (e.g., shoes, toys, cables), similar to how humans and pets lift their feet over objects as they walk. ViNL consists of: (1) a visual navigation policy that outputs linear and angular velocity commands that guides the robot to a goal coordinate in unfamiliar indoor environments; and (2) a visual locomotion policy that controls the robot's joints to avoid stepping on obstacles while following provided velocity commands. Both the policies are entirely "model-free", i.e. sensors-to-actions neural networks trained end-to-end. The two are trained independently in two entirely different simulators and then seamlessly co-deployed by feeding the velocity commands from the navigator to the locomotor, entirely "zero-shot" (without any co-training). While prior works have developed learning methods for visual navigation or visual locomotion, to the best of our knowledge, this is the first fully learned approach that leverages vision to accomplish both (1) intelligent navigation in new environments, and (2) intelligent visual locomotion that aims to traverse cluttered environments without disrupting obstacles. On the task of navigation to distant goals in unknown environments, ViNL using just egocentric vision significantly outperforms prior work on robust locomotion using privileged terrain maps (+32.8% success and -4.42 collisions per meter). Additionally, we ablate our locomotion policy to show that each aspect of our approach helps reduce obstacle collisions. Videos and code at http://www.joannetruong.com/projects/vinl.html

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, et al., “DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames,” in ICLR, 2020.
  2. S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, et al., “Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [Online]. Available: https://openreview.net/forum?id=-v4OuqNs5P
  3. M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, et al., “Habitat: A Platform for Embodied AI Research,” in ICCV, 2019.
  4. A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, et al., “Habitat 2.0: Training home assistants to rearrange their habitat,” in Advances in Neural Information Processing Systems (NeurIPS), 2021.
  5. B. Shen, F. Xia, C. Li, R. Martín-Martín, L. Fan, et al., “igibson 1.0: a simulation environment for interactive tasks in large realistic scenes,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2021, p. accepted.
  6. F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, et al., “SAPIEN: A simulated part-based interactive environment,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  7. M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, et al., “Robothor: An open simulation-to-real embodied AI platform,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3164–3174.
  8. N. Yokoyama, S. Ha, and D. Batra, “Success weighted by completion time: A dynamics-aware evaluation criteria for embodied navigation,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
  9. A. Kadian, J. Truong, A. Gokaslan, A. Clegg, E. Wijmans, et al., “Sim2real predictivity: Does evaluation in simulation predict real-world performance?” 2020.
  10. D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov, “Learning to explore using active neural slam,” in International Conference on Learning Representations (ICLR), 2020.
  11. R. Partsey, E. Wijmans, N. Yokoyama, O. Dobosevych, D. Batra, and O. Maksymets, “Is mapping necessary for realistic pointgoal navigation?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 17 232–17 241.
  12. J. Truong, M. Rudolph, N. Yokoyama, S. Chernova, D. Batra, and A. Rai, “Rethinking sim2real: Lower fidelity simulation leads to higher sim2real transfer in navigation,” in Conference on Robot Learning (CoRL), 2022.
  13. “Robothor challenge @ CVPR 2020,” https://ai2thor.allenai.org/robothor/challenge/.
  14. A. Kumar, Z. Fu, D. Pathak, and J. Malik, “Rma: Rapid motor adaptation for legged robots,” 2021.
  15. A. Iscen, K. Caluwaerts, J. Tan, T. Zhang, E. Coumans, et al., “Policies modulating trajectory generators,” in Conference on Robot Learning.   PMLR, 2018, pp. 916–926.
  16. V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, et al., “Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning.”
  17. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  18. N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Conference on Robot Learning.   PMLR, 2022, pp. 91–100.
  19. M. H. Raibert, “Trotting, pacing and bounding by a quadruped robot,” Journal of biomechanics, vol. 23, pp. 79–98, 1990.
  20. J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim, “Dynamic locomotion in the mit cheetah 3 through convex model-predictive control,” in 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS).   IEEE, 2018, pp. 1–9.
  21. F. Farshidian, E. Jelavic, A. Satapathy, M. Giftthaler, and J. Buchli, “Real-time motion planning of legged robots: A model predictive control approach,” in 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids).   IEEE, 2017, pp. 577–584.
  22. S. Coros, P. Beaudoin, and M. Van de Panne, “Generalized biped walking control,” ACM Transactions On Graphics (TOG), vol. 29, no. 4, pp. 1–9, 2010.
  23. H. Miura and I. Shimoyama, “Dynamic walk of a biped,” The International Journal of Robotics Research, vol. 3, no. 2, pp. 60–74, 1984.
  24. X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” Robotics: Science and Systems (RSS), 2020.
  25. Z. Fu, A. Kumar, J. Malik, and D. Pathak, “Minimizing energy consumption leads to the emergence of gaits in legged robots,” in Conference on Robot Learning (CoRL), 2021.
  26. J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science Robotics, vol. 5, no. 47, p. eabc5986, 2020. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.abc5986
  27. T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, p. eabk2822, 2022.
  28. ——, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, p. eabk2822, Jan. 2022, arXiv: 2201.08117. [Online]. Available: http://arxiv.org/abs/2201.08117
  29. V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter, “DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learning.”
  30. W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, et al., “Visual-locomotion: Learning to walk on complex terrains with vision,” in 5th Annual Conference on Robot Learning, 2021. [Online]. Available: https://openreview.net/forum?id=NDYbXf-DvwZ
  31. G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, et al., “Learning to jump from pixels,” Conference on Robot Learning (CoRL), 2021.
  32. A. Agarwal, A. Kumar, J. Malik, and D. Pathak, “Legged locomotion in challenging terrains using egocentric vision,” in 6th Annual Conference on Robot Learning, 2022.
  33. P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, et al., “On Evaluation of Embodied Navigation Agents,” arXiv preprint arXiv:1807.06757, 2018.
  34. “Gibson challenge @ CVPR 2020,” http://svl.stanford.edu/gibson2/challenge.html.
  35. F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. E. Tchapmi, et al., “Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 713–720, 2020.
  36. J. Truong, D. Yarats, T. Li, F. Meier, S. Chernova, et al., “Learning navigation skills for legged robots with learned robot embeddings,” in International Conference on Intelligent Robots and Systems (IROS), 2020.
  37. M. Sorokin, J. Tan, C. K. Liu, and S. Ha, “Learning to navigate sidewalks in outdoor environments,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3906–3913, 2022.
  38. D. Jain, A. Iscen, and K. Caluwaerts, “From pixels to legs: Hierarchical learning of quadruped locomotion,” Conference on Robot Learning, 2020.
  39. Z. Fu, A. Kumar, A. Agarwal, H. Qi, J. Malik, and D. Pathak, “Coupling vision and proprioception for navigation of legged robots,” in CVPR, 2022.
  40. D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl, “Learning by cheating,” in Conference on Robot Learning (CoRL), 2019.
  41. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv:1707.06347 [cs], Aug. 2017, arXiv: 1707.06347. [Online]. Available: http://arxiv.org/abs/1707.06347
  42. S. Ross, G. J. Gordon, and J. A. Bagnell, “No-regret reductions for imitation learning and structured prediction,” CoRR, vol. abs/1011.0686, 2010. [Online]. Available: http://arxiv.org/abs/1011.0686
  43. F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson env: Real-world perception for embodied agents,” in CVPR, 2018.
  44. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016.
Citations (19)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now