Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Visual Action Planning with Multiple Heterogeneous Agents (2403.16781v1)

Published 25 Mar 2024 in cs.RO

Abstract: Visual planning methods are promising to handle complex settings where extracting the system state is challenging. However, none of the existing works tackles the case of multiple heterogeneous agents which are characterized by different capabilities and/or embodiment. In this work, we propose a method to realize visual action planning in multi-agent settings by exploiting a roadmap built in a low-dimensional structured latent space and used for planning. To enable multi-agent settings, we infer possible parallel actions from a dataset composed of tuples associated with individual actions. Next, we evaluate feasibility and cost of them based on the capabilities of the multi-agent system and endow the roadmap with this information, building a capability latent space roadmap (C-LSR). Additionally, a capability suggestion strategy is designed to inform the human operator about possible missing capabilities when no paths are found. The approach is validated in a simulated burger cooking task and a real-world box packing task.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (17)
  1. A. Wang, T. Kurutach, P. Abbeel, and A. Tamar, “Learning robotic manipulation through visual planning and acting,” in Robotics: Science and Syst., 2019.
  2. C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in IEEE Int. Conf. Robot. Autom., pp. 2786–2793, 2017.
  3. K. Liu, T. Kurutach, C. Tung, P. Abbeel, and A. Tamar, “Hallucinative topological memory for zero-shot visual planning,” in Int. Conf. on Mach. Learn., vol. 119, pp. 6259–6270, 2020.
  4. R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg, “Visuospatial foresight for physical sequential fabric manipulation,” Autonomous Robots, vol. 46, no. 1, pp. 175–199, 2022.
  5. M. Lippi, P. Poklukar, M. C. Welle, A. Varava, H. Yin, A. Marino, and D. Kragic, “Enabling visual action planning for object manipulation through latent space roadmap,” IEEE Trans. Robot., vol. 39, no. 1, pp. 57–75, 2023.
  6. M. Lippi, M. C. Welle, P. Poklukar, A. Marino, and D. Kragic, “Augment-connect-explore: a paradigm for visual action planning with data scarcity,” in IEEE/RSJ Int. Conf. on Intell. Robots and Syst., pp. 754–761, 2022.
  7. J. Sun, D.-A. Huang, B. Lu, Y.-H. Liu, B. Zhou, and A. Garg, “Plate: Visually-grounded planning with transformers in procedural tasks,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4924–4930, 2022.
  8. H. Wu, J. Ye, X. Meng, C. Paxton, and G. S. Chirikjian, “Transporters with visual foresight for solving unseen rearrangement tasks,” in IEEE/RSJ Int. Conf. on Intell. Robots and Syst., pp. 10756–10763, 2022.
  9. A. Torreno, E. Onaindia, A. Komenda, and M. Štolba, “Cooperative multi-agent planning: A survey,” ACM Computing Surveys (CSUR), vol. 50, no. 6, pp. 1–32, 2017.
  10. A. Khamis, A. Hussein, and A. Elmogy, “Multi-robot task allocation: A review of the state-of-the-art,” Cooperative robots and sensor networks 2015, pp. 31–51, 2015.
  11. L. Zhang, J. Zhao, E. Lamon, Y. Wang, and X. Hong, “Energy efficient multi-robot task allocation constrained by time window and precedence,” IEEE Trans. Autom. Sci. Eng., 2023.
  12. S. Liu, J. Shen, W. Tian, J. Lin, P. Li, and B. Li, “Balanced task allocation and collision-free scheduling of multi-robot systems in large spacecraft structure manufacturing,” Robotics and Autonomous Systems, vol. 159, p. 104289, 2023.
  13. M. Lippi, P. Di Lillo, and A. Marino, “A task allocation framework for human multi-robot collaborative settings,” in IEEE Int. Conf. Robot. Autom., pp. 7614–7620, 2023.
  14. C. Chamzas, M. Lippi, M. C. Welle, A. Varava, L. E. Kavraki, and D. Kragic, “Comparing reconstruction-and contrastive-based models for visual task planning,” IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, 2022.
  15. F. Zacharias, C. Borst, and G. Hirzinger, “Capturing robot workspace structure: representing robot capabilities,” in IEEE/RSJ Int. Conf. on Intell. Robots and Syst., pp. 3229–3236, 2007.
  16. A. Makhal and A. K. Goins, “Reuleaux: Robot base placement by reachability analysis,” in IEEE Int. Conf. Robot. Comput., pp. 137–142, 2018.
  17. Unity Technologies, “Unity.” https://unity.com.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now