Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Egocentric Visual Self-Modeling for Autonomous Robot Dynamics Prediction and Adaptation (2207.03386v3)

Published 7 Jul 2022 in cs.RO

Abstract: The ability of robots to model their own dynamics is key to autonomous planning and learning, as well as for autonomous damage detection and recovery. Traditionally, dynamic models are pre-programmed or learned from external observations. Here, we demonstrate for the first time how a task-agnostic dynamic self-model can be learned using only a single first-person-view camera in a self-supervised manner, without any prior knowledge of robot morphology, kinematics, or task. Through experiments on a 12-DoF robot, we demonstrate the capabilities of the model in basic locomotion tasks using visual input. Notably, the robot can autonomously detect anomalies, such as damaged components, and adapt its behavior, showcasing resilience in dynamic environments. Furthermore, the model's generalizability was validated across robots with different configurations, emphasizing its potential as a universal tool for diverse robotic systems. The egocentric visual self-model proposed in our work paves the way for more autonomous, adaptable, and resilient robotic systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. K. G. Pearson, “Common principles of motor control in vertebrates and invertebrates,” Annual review of neuroscience, vol. 16, no. 1, pp. 265–297, 1993.
  2. D. S. Marigold and A. E. Patla, “Visual information from the lower visual field is important for walking across multi-surface terrain,” Experimental Brain Research, vol. 188, pp. 23–31, 2008.
  3. J. J. Gibson, “Visually controlled locomotion and visual orientation in animals,” British journal of psychology, vol. 49, no. 3, pp. 182–194, 1958.
  4. D. S. Marigold and A. E. Patla, “Gaze fixation patterns for negotiating complex ground terrain,” Neuroscience, vol. 144, no. 1, pp. 302–313, 2007.
  5. W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang, “Visual-locomotion: Learning to walk on complex terrains with vision,” in 5th Annual Conference on Robot Learning, 2021.
  6. R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang, “Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers,” arXiv preprint arXiv:2107.03996, 2021.
  7. A. Loquercio, A. Kumar, and J. Malik, “Learning visual locomotion with cross-modal supervision,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 7295–7302.
  8. Z. Fu, A. Kumar, A. Agarwal, H. Qi, J. Malik, and D. Pathak, “Coupling vision and proprioception for navigation of legged robots,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 273–17 283.
  9. D. Nguyen-Tuong and J. Peters, “Model learning for robot control: a survey,” Cognitive processing, vol. 12, pp. 319–340, 2011.
  10. J. Bongard, V. Zykov, and H. Lipson, “Resilient machines through continuous self-modeling,” Science, vol. 314, no. 5802, pp. 1118–1121, 2006.
  11. J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” Advances in neural information processing systems, vol. 27, 2014.
  12. R. Kwiatkowski, “Deep self-modeling for robotic systems,” Ph.D. dissertation, Columbia University, 2022.
  13. Y. Hu, R. Kwiatkowski, B. Chen, P. Wyder, and H. Lipson, “Scalable quasi-static self-modeling for physical legged locomotion,” 2023.
  14. J. Carpentier and P.-B. Wieber, “Recent progress in legged robots locomotion control,” Current Robotics Reports, vol. 2, no. 3, pp. 231–238, 2021.
  15. A. Torres-Pardo, D. Pinto-Fernández, M. Garabini, F. Angelini, D. Rodriguez-Cianca, S. Massardi, J. Tornero-López, J. C. Moreno, and D. Torricelli, “Legged locomotion over irregular terrains: State of the art of human and robot performance,” Bioinspiration & Biomimetics, 2022.
  16. I. Maroger, O. Stasse, and B. Watier, “Walking human trajectory models and their application to humanoid robot locomotion,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2020, pp. 3465–3472.
  17. X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” arXiv preprint arXiv:2004.00784, 2020.
  18. S. Gangapurwala, A. Mitchell, and I. Havoutis, “Guided constrained policy optimization for dynamic quadrupedal robot locomotion,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3642–3649, 2020.
  19. G. Bledt, M. J. Powell, B. Katz, J. Di Carlo, P. M. Wensing, and S. Kim, “Mit cheetah 3: Design and control of a robust, dynamic quadruped robot,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2018, pp. 2245–2252.
  20. M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V. Tsounis, J. Hwangbo, K. Bodie, P. Fankhauser, M. Bloesch, et al., “Anymal-a highly mobile and dynamic quadrupedal robot,” in 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS).   IEEE, 2016, pp. 38–44.
  21. A. Bouman, M. F. Ginting, N. Alatur, M. Palieri, D. D. Fan, T. Touma, T. Pailevanian, S.-K. Kim, K. Otsu, J. Burdick, et al., “Autonomous spot: Long-range autonomous exploration of extreme environments with legged locomotion,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2020, pp. 2518–2525.
  22. J. M. Loomis, R. L. Klatzky, R. G. Golledge, J. G. Cicinelli, J. W. Pellegrino, and P. A. Fry, “Nonvisual navigation by blind and sighted: assessment of path integration ability.” Journal of Experimental Psychology: General, vol. 122, no. 1, p. 73, 1993.
  23. C. S. Kallie, P. R. Schrater, and G. E. Legge, “Variability in stepping direction explains the veering behavior of blind walkers.” Journal of Experimental Psychology: Human Perception and Performance, vol. 33, no. 1, p. 183, 2007.
  24. W. H. Warren, B. A. Kay, W. D. Zosh, A. P. Duchon, and S. Sahuc, “Optic flow is used to control human walking,” Nature neuroscience, vol. 4, no. 2, pp. 213–216, 2001.
  25. W. Paulus, A. Straube, and T. Brandt, “Visual stabilization of posture: physiological stimulus characteristics and clinical aspects,” Brain, vol. 107, no. 4, pp. 1143–1163, 1984.
  26. D. M. Wolpert and J. R. Flanagan, “Motor prediction,” Current biology, vol. 11, no. 18, pp. R729–R732, 2001.
  27. R. Shadmehr and J. W. Krakauer, “A computational neuroanatomy for motor control,” Experimental brain research, vol. 185, pp. 359–381, 2008.
  28. B. Katz, J. Di Carlo, and S. Kim, “Mini cheetah: A platform for pushing the limits of dynamic quadruped control,” in 2019 international conference on robotics and automation (ICRA).   IEEE, 2019, pp. 6295–6301.
  29. I. Lenz, R. A. Knepper, and A. Saxena, “Deepmpc: Learning deep latent features for model predictive control.” in Robotics: Science and Systems, vol. 10.   Rome, Italy, 2015, p. 25.
  30. T. Salzmann, E. Kaufmann, J. Arrizabalaga, M. Pavone, D. Scaramuzza, and M. Ryll, “Real-time neural mpc: Deep learning model predictive control for quadrotors and agile robotic platforms,” IEEE Robotics and Automation Letters, vol. 8, no. 4, pp. 2397–2404, 2023.
  31. E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2012, pp. 5026–5033.
  32. E. Coumans and Y. Bai, “2019. pybullet, a python module for physics simulation for games, robotics and machine learning,” 2016.
  33. N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” in 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), vol. 3, 2004, pp. 2149–2154 vol.3.
  34. J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst, “Blind bipedal stair traversal via sim-to-real reinforcement learning,” arXiv preprint arXiv:2105.08328, 2021.
  35. N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Conference on Robot Learning.   PMLR, 2022, pp. 91–100.
  36. B. Chen, R. Kwiatkowski, C. Vondrick, and H. Lipson, “Full-body visual self-modeling of robot morphologies,” arXiv preprint arXiv:2111.06389, 2021.
  37. R. Kwiatkowski and H. Lipson, “Task-agnostic self-modeling machines,” Science Robotics, vol. 4, no. 26, p. eaau9354, 2019.
  38. Y. Hu, J. Lin, and H. Lipson, “Teaching robots to build simulations of themselves,” arXiv preprint arXiv:2311.12151, 2023.
  39. A. Staranowicz and G. L. Mariottini, “A survey and comparison of commercial and open-source robotic simulator software,” in Proceedings of the 4th International Conference on PErvasive Technologies Related to Assistive Environments, 2011, pp. 1–8.
  40. O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al., “Learning dexterous in-hand manipulation,” The International Journal of Robotics Research, vol. 39, no. 1, pp. 3–20, 2020.
  41. I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al., “Solving rubik’s cube with a robot hand,” arXiv preprint arXiv:1910.07113, 2019.
  42. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  43. K. O’Shea and R. Nash, “An introduction to convolutional neural networks,” arXiv preprint arXiv:1511.08458, 2015.
  44. S. Hochreiter and J. Schmidhuber, “Lstm can solve hard long time lag problems,” Advances in neural information processing systems, vol. 9, 1996.
  45. S. L. Oh, E. Y. Ng, R. San Tan, and U. R. Acharya, “Automated diagnosis of arrhythmia using combination of cnn and lstm techniques with variable length heart beats,” Computers in biology and medicine, vol. 102, pp. 278–287, 2018.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube