Papers
Topics
Authors
Recent
Search
2000 character limit reached

Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge

Published 4 Nov 2024 in cs.AI, cs.HC, and cs.RO | (2411.01796v3)

Abstract: We introduce Constrained Human-AI Cooperation (CHAIC), an inclusive embodied social intelligence challenge designed to test social perception and cooperation in embodied agents. In CHAIC, the goal is for an embodied agent equipped with egocentric observations to assist a human who may be operating under physical constraints -- e.g., unable to reach high places or confined to a wheelchair -- in performing common household or outdoor tasks as efficiently as possible. To achieve this, a successful helper must: (1) infer the human's intents and constraints by following the human and observing their behaviors (social perception), and (2) make a cooperative plan tailored to the human partner to solve the task as quickly as possible, working together as a team (cooperative planning). To benchmark this challenge, we create four new agents with real physical constraints and eight long-horizon tasks featuring both indoor and outdoor scenes with various constraints, emergency events, and potential risks. We benchmark planning- and learning-based baselines on the challenge and introduce a new method that leverages LLMs and behavior modeling. Empirical evaluations demonstrate the effectiveness of our benchmark in enabling systematic assessment of key aspects of machine social intelligence. Our benchmark and code are publicly available at https://github.com/UMass-Embodied-AGI/CHAIC.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Modeling and planning with macro-actions in decentralized pomdps. Journal of Artificial Intelligence Research, 64:817–859, 2019.
  2. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2020.
  3. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020.
  4. Design for inclusivity. In Design for Health, pages 305–322. Elsevier, 2020.
  5. Acceptance of healthcare robots for the older population: Review and future directions. International journal of social robotics, 1:319–330, 2009.
  6. Smart help: Strategic opponent modeling for proactive and adaptive robot assistance in households. arXiv preprint arXiv:2404.09001, 2024.
  7. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
  8. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
  9. MMAction2 Contributors. Openmmlab’s next generation video understanding toolbox and benchmark. https://github.com/open-mmlab/mmaction2, 2020.
  10. Dementia and robotics: people with advancing dementia and their carers driving an exploration into an engineering solution to maintaining safe exercise regimes. In Nursing Informatics 2016, pages 545–549. IOS Press, 2016.
  11. Improving inclusivity in robotics design: an exploration of methods for upstream co-creation. Frontiers in Robotics and AI, 9:119, 2022.
  12. Retrospectives on the embodied ai workshop. arXiv preprint arXiv:2210.06849, 2022.
  13. The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied ai. arXiv preprint arXiv:2103.14025, 2021.
  14. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100–107, 1968.
  15. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  16. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  17. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859–865, 2019.
  18. A cordial sync: Going beyond marginal policies for multi-agent embodied tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 471–490. Springer, 2020.
  19. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
  20. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.
  21. Steps toward participatory design of social robots: mutual learning with older adults with depression. In Proceedings of the 2017 ACM/IEEE international conference on human-robot interaction, pages 244–253, 2017.
  22. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80–93. PMLR, 2023.
  23. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  24. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017.
  25. Mapping instructions to actions in 3d environments with visual goal prediction. arXiv preprint arXiv:1809.00786, 2018.
  26. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626, 2022.
  27. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2017–2025, 2022.
  28. Watch-and-help: A challenge for social perception and human-ai collaboration. In International Conference on Learning Representations, 2021.
  29. Nopa: Neurally-guided online probabilistic assistance for building socially intelligent home assistants. arXiv preprint arXiv:2301.05223, 2023a.
  30. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724, 2023b.
  31. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021.
  32. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2186–2188, 2019.
  33. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339–9347, 2019.
  34. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
  35. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740–10749, 2020.
  36. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.00784, 2019.
  37. Learning visuo-haptic skewering strategies for robot-assisted feeding. In 6th Annual Conference on Robot Learning, 2022.
  38. Adaptive coordination in social embodied rearrangement. In International Conference on Machine Learning, pages 33365–33380. PMLR, 2023.
  39. Large language models still can’t plan (a benchmark for llms on planning and reasoning about change). arXiv preprint arXiv:2206.10498, 2022.
  40. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20–36. Springer, 2016.
  41. Altruistic helping in human infants and young chimpanzees. science, 311(5765):1301–1303, 2006.
  42. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35:16509–16521, 2022.
  43. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9068–9079, 2018.
  44. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097–11107, 2020.
  45. Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.02485, 2023.
  46. Combo: Compositional world models for embodied multi-agent cooperation. arXiv preprint arXiv:2404.10775, 2024.
  47. Hazard challenge: Embodied decision making in dynamically changing environments, 2024.
  48. Visual semantic planning using deep successor representations. In Proceedings of the IEEE international conference on computer vision, pages 483–492, 2017.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 0 likes about this paper.

Sign Up for Free