Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots (2310.13724v1)

Abstract: We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed. (2) Human-in-the-loop infrastructure: enabling real human interaction with simulated robots via mouse/keyboard or a VR interface, facilitating evaluation of robot policies with human input. (3) Collaborative tasks: studying two collaborative tasks, Social Navigation and Social Rearrangement. Social Navigation investigates a robot's ability to locate and follow humanoid avatars in unseen environments, whereas Social Rearrangement addresses collaboration between a humanoid and robot while rearranging a scene. These contributions allow us to study end-to-end learned and heuristic baselines for human-robot collaboration in-depth, as well as evaluate them with humans in the loop. Our experiments demonstrate that learned robot policies lead to efficient task completion when collaborating with unseen humanoid agents and human partners that might exhibit behaviors that the robot has not seen before. Additionally, we observe emergent behaviors during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent. Furthermore, our experiments using the human-in-the-loop tool demonstrate that our automated evaluation with humanoids can provide an indication of the relative ordering of different policies when evaluated with real human collaborators. Habitat 3.0 unlocks interesting new features in simulators for Embodied AI, and we hope it paves the way for a new frontier of embodied human-AI interaction capabilities.

• The paper introduces Habitat 3.0, a simulation platform that accurately models humanoid avatars and enables human-in-the-loop evaluations for collaborative home tasks.
• The paper demonstrates a novel integration of realistic humanoid simulation and interactive interfaces, facilitating precise assessment of social navigation and rearrangement tasks.
• The paper’s experiments reveal that learned robot policies adapt effectively to dynamic environments, paving the way for significant advancements in embodied AI.

Habitat 3.0: An Advanced Platform for Human-Robot Collaboration in Simulation

This paper introduces Habitat 3.0, a sophisticated simulation platform aimed at fostering advanced studies of collaborative human-robot tasks within home environments. The platform marks advancements across three primary dimensions: accurate humanoid simulation, human-in-the-loop infrastructure, and collaborative task evaluation. It is specifically designed to address longstanding challenges in modeling complex deformable bodies and diverse appearances and motions, while maintaining high simulation speeds.

Habitat 3.0 is characterized by its innovative humanoid simulation capabilities, which include articulated skeletons with rotational joints and high-fidelity rendering through surface skin mesh integration. The platform employs parameterized body models via SMPL-X to generate realistic body shapes and poses, enhancing humanoid appearance with a diverse library of avatars. Moreover, the platform incorporates a motion and behavior generation policy that supports programmatic control of avatars for navigation and object interaction.

The human-in-the-loop infrastructure uniquely facilitates real human interaction with simulated robots using mouse/keyboard interfaces or VR technology. This feature is crucial for evaluating robot policies with direct human input, thereby mimicking real-world collaborative dynamics that are essential for practical deployments in home environments.

Central to Habitat 3.0 are the social navigation and social rearrangement tasks. These tasks evaluate collaborative dynamics between humanoid avatars and robots. Social Navigation assesses a robot's ability to detect and thus safely follow humanoid avatars within unfamiliar environments. Social Rearrangement focuses on task division and cooperation during object rearrangement activities. Within these tasks, robots are trained using both learned and heuristic policies, gaining insights into human-robot teamwork efficiency.

The experiments conducted within Habitat 3.0 demonstrate significant success in learned robot policies that enhance efficiency in collaborative settings involving unseen humanoids and humans displaying previously unseen behaviors. Emergent behaviors were observed, such as robots yielding space to humanoid agents—a pivotal functionality for effective collaboration in constrained environments.

The results highlight the ability of the automated evaluation with humanoids to predict the relative efficiency of various policies when applied with real human collaborators. While the heuristic expert achieved notable success through privileged access to environment maps, the end-to-end reinforcement learning (RL) policy showcased its ability to adapt through learned behaviors, such as backing up when humanoids are nearby.

A comprehensive paper of human-in-the-loop evaluations with real humans underscores Habitat 3.0's utility in efficiently simulating realistic human-robot interactions. Despite promising results, the work acknowledges certain limitations, such as action scope limitations and challenges in observation space alignment between simulation and real-world interactions. Future directions may enhance simulation tasks beyond strictly non-communicative zero-shot coordination, integrating dialogue systems to further improve collaborative efficiency.

Habitat 3.0 paves the way for significant advancements in embodied AI, promising insights and breakthroughs in real-world applications where robots coexist and collaborate within dynamic human environments. As the platform continues to evolve, it holds the potential to enrich the discourse on embodied AI, offering robust tools for developing the next generation of assistive robotic technologies.