Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multi-Object Graph Affordance Network: Goal-Oriented Planning through Learned Compound Object Affordances (2309.10426v4)

Published 19 Sep 2023 in cs.RO and cs.AI

Abstract: Learning object affordances is an effective tool in the field of robot learning. While the data-driven models investigate affordances of single or paired objects, there is a gap in the exploration of affordances of compound objects composed of an arbitrary number of objects. We propose the Multi-Object Graph Affordance Network which models complex compound object affordances by learning the outcomes of robot actions that facilitate interactions between an object and a compound. Given the depth images of the objects, the object features are extracted via convolution operations and encoded in the nodes of graph neural networks. Graph convolution operations are used to encode the state of the compounds, which are used as input to decoders to predict the outcome of the object-compound interactions. After learning the compound object affordances, given different tasks, the learned outcome predictors are used to plan sequences of stack actions that involve stacking objects on top of each other, inserting smaller objects into larger containers and passing through ring-like objects through poles. We showed that our system successfully modeled the affordances of compound objects that include concave and convex objects, in both simulated and real-world environments. We benchmarked our system with a baseline model to highlight its advantages.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. J. J. Gibson, “The theory of affordances,” Hilldale, USA, vol. 1, no. 2, pp. 67–82, 1977.
  2. E. Rome, L. Paletta, E. Şahin, G. Dorffner, J. Hertzberg, R. Breithaupt, G. Fritz, J. Irran, F. Kintzler, C. Lörken et al., “The macs project: an approach to affordance-inspired robot control,” in Towards Affordance-Based Robot Control: International Seminar, Dagstuhl Castle, Germany, June 5-9, 2006. Revised Papers.   Springer, 2008, pp. 173–210.
  3. L. Jamone, E. Ugur, A. Cangelosi, L. Fadiga, A. Bernardino, J. Piater, and J. Santos-Victor, “Affordances in psychology, neuroscience, and robotics: A survey,” IEEE Transactions on Cognitive and Developmental Systems, vol. 10, no. 1, pp. 4–25, 2016.
  4. E. Şahin, M. Cakmak, M. R. Doğar, E. Uğur, and G. Üçoluk, “To afford or not to afford: A new formalization of affordances toward affordance-based robot control,” Adaptive Behavior, vol. 15, no. 4, pp. 447–472, 2007.
  5. E. Ugur, E. Oztop, and E. Sahin, “Goal emulation and planning in perceptual space using learned affordances,” Robotics and Autonomous Systems, vol. 59, no. 7-8, pp. 580–595, 2011.
  6. E. Ugur, Y. Nagai, E. Sahin, and E. Oztop, “Staged development of robot skills: Behavior formation, affordance learning and imitation with motionese,” IEEE Transactions on Autonomous Mental Development, vol. 7, no. 2, pp. 119–139, 2015.
  7. E. Ugur and J. Piater, “Emergent structuring of interdependent affordance learning tasks,” in 4th International Conference on Development and Learning and on Epigenetic Robotics.   IEEE, 2014, pp. 489–494.
  8. P. Zech, S. Haller, S. R. Lakani, B. Ridge, E. Ugur, and J. Piater, “Computational models of affordance in robotics: a taxonomy and systematic classification,” Adaptive Behavior, vol. 25, no. 5, pp. 235–271, 2017.
  9. Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,” IEEE transactions on neural networks and learning systems, vol. 32, no. 1, pp. 4–24, 2020.
  10. T. Ahmad, L. Jin, X. Zhang, S. Lai, G. Tang, and L. Lin, “Graph convolutional neural network for human action recognition: A comprehensive survey,” IEEE Transactions on Artificial Intelligence, vol. 2, no. 2, pp. 128–145, 2021.
  11. L. Wu, Y. Chen, K. Shen, X. Guo, H. Gao, S. Li, J. Pei, B. Long et al., “Graph neural networks for natural language processing: A survey,” Foundations and Trends® in Machine Learning, vol. 16, no. 2, pp. 119–328, 2023.
  12. Z. Liu, Y. Zhai, J. Li, G. Wang, Y. Miao, and H. Wang, “Graph relational reinforcement learning for mobile robot navigation in large-scale crowded environments,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 8, pp. 8776–8787, 2023.
  13. Y. Zhou and J. Garcke, “Learning crowd behaviors in navigation with attention-based spatial-temporal graphs,” arXiv preprint arXiv:2401.06226, 2024.
  14. E. Escudie and L. M. J. Saraydaryan, “Attention graph for multi-robot social navigation with deep reinforcement learning,” in International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2024.
  15. P. Battaglia, R. Pascanu, M. Lai, D. J. Rezende et al., “Interaction networks for learning about objects, relations and physics,” in Advances in neural information processing systems, 2016, pp. 4502–4510.
  16. M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum, “A compositional object-based approach to learning physical dynamics,” arXiv preprint arXiv:1612.00341, 2016.
  17. Y. Li, J. Wu, J.-Y. Zhu, J. B. Tenenbaum, A. Torralba, and R. Tedrake, “Propagation networks for model-based control under partial observation,” in International Conference on Robotics and Automation, 2019, pp. 1205–1211.
  18. A. E. Tekden, A. Erdem, E. Erdem, T. Asfour, and E. Ugur, “Object and relation centric representations for push effect prediction,” arXiv preprint arXiv:2102.02100, 2021.
  19. A. Ahmetoglu, M. Y. Seker, J. Piater, E. Oztop, and E. Ugur, “Deepsym: Deep symbol generation and rule learning for planning from unsupervised robot interaction,” Journal of Artificial Intelligence Research, vol. 75, pp. 709–745, 2022.
  20. M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman et al., “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022.
  21. M. Ahn, D. Dwibedi, C. Finn, M. G. Arenas, K. Gopalakrishnan, K. Hausman, B. Ichter, A. Irpan, N. Joshi, R. Julian et al., “Autort: Embodied foundation models for large scale orchestration of robotic agents,” arXiv preprint arXiv:2401.12963, 2024.
  22. M. Hassanin, S. Khan, and M. Tahtali, “Visual affordance and function understanding: A survey,” ACM Computing Surveys (CSUR), vol. 54, no. 3, pp. 1–35, 2021.
  23. S. Qian, W. Chen, M. Bai, X. Zhou, Z. Tu, and L. E. Li, “Affordancellm: Grounding affordance from vision language models,” arXiv preprint arXiv:2401.06341, 2024.
  24. T. Birr, C. Pohl, A. Younes, and T. Asfour, “Autogpt+ p: Affordance-based task planning with large language models,” arXiv preprint arXiv:2402.10778, 2024.
  25. T.-T. Do, A. Nguyen, and I. Reid, “Affordancenet: An end-to-end deep learning approach for object affordance detection,” in 2018 IEEE international conference on robotics and automation (ICRA).   IEEE, 2018, pp. 5882–5889.
  26. E. Ugur, S. Szedmak, and J. Piater, “Bootstrapping paired-object affordance learning with learned single-affordance features,” in 4th International Conference on Development and Learning and on Epigenetic Robotics.   IEEE, 2014, pp. 476–481.
  27. S. Hangl, E. Ugur, S. Szedmak, and J. Piater, “Robotic playing for hierarchical complex skill learning,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2016, pp. 2799–2804.
  28. S. Yang, W. Zhang, R. Song, J. Cheng, and Y. Li, “Learning multi-object dense descriptor for autonomous goal-conditioned grasping,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 4109–4116, 2021.
  29. S. Hart, P. Dinh, and K. Hambuchen, “The affordance template ros package for robot task programming,” in 2015 IEEE international conference on robotics and automation (ICRA).   IEEE, 2015, pp. 6227–6234.
  30. E. Corona, A. Pumarola, G. Alenya, F. Moreno-Noguer, and G. Rogez, “Ganhand: Predicting human grasp affordances in multi-object scenes,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 5031–5041.
  31. P. Mandikal and K. Grauman, “Learning dexterous grasping with object-centric visual affordances,” in 2021 IEEE international conference on robotics and automation (ICRA).   IEEE, 2021, pp. 6169–6176.
  32. A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo et al., “Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching,” The International Journal of Robotics Research, vol. 41, no. 7, pp. 690–705, 2022.
  33. G. Schiavi, P. Wulkop, G. Rizzi, L. Ott, R. Siegwart, and J. J. Chung, “Learning agent-aware affordances for closed-loop interaction with articulated objects,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 5916–5922.
  34. Y. Geng, B. An, H. Geng, Y. Chen, Y. Yang, and H. Dong, “Rlafford: End-to-end affordance learning for robotic manipulation,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 5880–5886.
  35. S. Cheng, K. Mo, and L. Shao, “Learning to regrasp by learning to place,” in Conference on Robot Learning, 8-11 November 2021, London, UK, ser. Proceedings of Machine Learning Research, A. Faust, D. Hsu, and G. Neumann, Eds., vol. 164.   PMLR, 2021, pp. 277–286. [Online]. Available: https://proceedings.mlr.press/v164/cheng22a.html
  36. Y.-C. Lin, P. Florence, A. Zeng, J. T. Barron, Y. Du, W.-C. Ma, A. Simeonov, A. R. Garcia, and P. Isola, “Mira: Mental imagery for robotic affordances,” in Conference on Robot Learning.   PMLR, 2023, pp. 1916–1927.
  37. J. Borja-Diaz, O. Mees, G. Kalweit, L. Hermann, J. Boedecker, and W. Burgard, “Affordance learning from play for sample-efficient policy learning,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 6372–6378.
  38. O. Mees, J. Borja-Diaz, and W. Burgard, “Grounding language with visual affordances over unstructured data,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 11 576–11 582.
  39. B. Moldovan, P. Moreno, M. Van Otterlo, J. Santos-Victor, and L. De Raedt, “Learning relational affordance models for robots in multi-object manipulation tasks,” in 2012 ieee international conference on robotics and automation.   IEEE, 2012, pp. 4373–4378.
  40. B. Moldovan and L. De Raedt, “Occluded object search by relational affordances,” in 2014 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2014, pp. 169–174.
  41. T. Mar, V. Tikhanoff, G. Metta, and L. Natale, “Self-supervised learning of grasp dependent tool affordances on the icub humanoid robot,” in 2015 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2015, pp. 3200–3206.
  42. J. MacQueen et al., “Some methods for classification and analysis of multivariate observations,” in Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, vol. 1, no. 14.   Oakland, CA, USA, 1967, pp. 281–297.
  43. A. Antunes, L. Jamone, G. Saponaro, A. Bernardino, and R. Ventura, “From human instructions to robot actions: Formulation of goals, affordances and probabilistic planning,” in 2016 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2016, pp. 5449–5454.
  44. E. Ugur, E. Şahin, and E. Oztop, “Unsupervised learning of object affordances for planning in a mobile manipulation platform,” in 2011 IEEE International Conference on Robotics and Automation.   IEEE, 2011, pp. 4312–4317.
  45. A. Iriondo, E. Lazkano, and A. Ansuategi, “Affordance-based grasping point detection using graph convolutional networks for industrial bin-picking applications,” Sensors, vol. 21, no. 3, p. 816, 2021.
  46. H. Tan, L. W. 0003, Q. Zhang, Z. Gao, N. Zheng, and G. Hua, “Object affordances graph network for action recognition.” in BMVC, 2019, p. 145.
  47. Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu, “Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 6541–6548.
  48. M. Kulshrestha and A. H. Qureshi, “Structural concept learning via graph attention for multi-level rearrangement planning,” Conference on Robot Learning (CoRL), 2023.
  49. X. Lou, Y. Yang, and C. Choi, “Learning object relations with graph neural networks for target-driven grasping in dense clutter,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 742–748.
  50. M. Wilson and T. Hermans, “Learning to manipulate object collections using grounded state representations,” in Conference on Robot Learning.   PMLR, 2020, pp. 490–502.
  51. Y. Lin, A. S. Wang, E. Undersander, and A. Rai, “Efficient and interpretable robot manipulation with graph neural networks,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 2740–2747, 2022.
  52. Y. Huang, A. Conkey, and T. Hermans, “Planning for multi-object manipulation with graph neural network relational classifiers,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 1822–1829.
  53. S. Tian, Y. Cai, H.-X. Yu, S. Zakharov, K. Liu, A. Gaidon, Y. Li, and J. Wu, “Multi-object manipulation via object-centric neural scattering functions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9021–9031.
  54. D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint, “Learning multi-object dynamics with compositional neural radiance fields,” in Conference on robot learning.   PMLR, 2023, pp. 1755–1768.
  55. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  56. A. E. Tekden, A. Erdem, E. Erdem, M. Imre, M. Y. Seker, and E. Ugur, “Belief regulated dual propagation nets for learning action effects on groups of articulated objects,” in 2020 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2020, pp. 10 556–10 562.
  57. A. E. Tekden, A. Erdem, E. Erdem, T. Asfour, and E. Ugur, “Object and relation centric representations for push effect prediction,” Robotics and Autonomous Systems, vol. 174, p. 104632, 2024.
  58. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Tuba Girgin (3 papers)
  2. Emre Ugur (37 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now