Habitat: A Platform for Embodied AI Research (1904.01201v2)

Abstract: We present Habitat, a platform for research in embodied AI. Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation. Specifically, Habitat consists of: (i) Habitat-Sim: a flexible, high-performance 3D simulator with configurable agents, sensors, and generic 3D dataset handling. Habitat-Sim is fast -- when rendering a scene from Matterport3D, it achieves several thousand frames per second (fps) running single-threaded, and can reach over 10,000 fps multi-process on a single GPU. (ii) Habitat-API: a modular high-level library for end-to-end development of embodied AI algorithms -- defining tasks (e.g., navigation, instruction following, question answering), configuring, training, and benchmarking embodied agents. These large-scale engineering contributions enable us to answer scientific questions requiring experiments that were till now impracticable or 'merely' impractical. Specifically, in the context of point-goal navigation: (1) we revisit the comparison between learning and SLAM approaches from two recent works and find evidence for the opposite conclusion -- that learning outperforms SLAM if scaled to an order of magnitude more experience than previous investigations, and (2) we conduct the first cross-dataset generalization experiments {train, test} x {Matterport3D, Gibson} for multiple sensors {blind, RGB, RGBD, D} and find that only agents with depth (D) sensors generalize across datasets. We hope that our open-source platform and these findings will advance research in embodied AI.

Habitat: A Platform for Embodied AI Research

The paper "Habitat: A Platform for Embodied AI Research" presents a comprehensive system designed to facilitate advancements in embodied AI. This platform comprises two main components: Habitat-Sim and Habitat-API.

Habitat-Sim is a high-performance 3D simulator that supports configurable agents and sensors within photorealistic 3D environments. It leverages datasets like Matterport3D, achieving several thousand frames per second (fps) in single-threaded execution and exceeding 10,000 fps in multi-process configurations on a single GPU, showcasing significant efficiency and scalability.

Habitat-API serves as a modular high-level library for the end-to-end development of embodied AI algorithms. It supports a wide array of tasks, including navigation, instruction following, and question answering, providing robust tools for defining, training, and benchmarking embodied agents. This modularity also allows for easy integration with various datasets and the definition of new tasks.

Key Contributions

• Performance and Flexibility: Habitat-Sim offers an unprecedented speed in simulation, an essential feature for extensive training and experimentation. The simulator's efficiency shifts the bottleneck from simulation to network training optimization, allowing researchers to focus on algorithmic improvements.
• Comprehensive Benchmarking: Using Habitat, the paper revisits comparisons between learning-based navigation approaches and classical SLAM (Simultaneous Localization and Mapping) techniques. It demonstrates that, given sufficient training (up to 75 million steps), learning-based methods can outperform SLAM, particularly when equipped with depth sensors. This finding challenges previous research limited by less extensive training scales.
• Cross-dataset Generalization: The researchers conduct experiments to assess the generalization of navigation agents across different datasets (Matterport3D and Gibson). Results indicate that agents utilizing depth sensors generalize better than those relying solely on RGB inputs. This is a crucial insight for developing robust embodied AI systems capable of operating in diverse environments.

Numerical Results and Implications

• Training Efficiency: The system's high frame rates facilitate the training of agents across millions of steps swiftly. For example, the training of agents to 75 million steps across different dataset configurations took approximately 2267 GPU-hours in total.
• Experimental Findings: Learning-based depth agents achieved superior performance compared to classical SLAM in both Gibson and Matterport3D datasets, with SPL scores of 0.79 and 0.54 respectively. These results indicate significant potential for depth-based perception in embodied AI.
• Generalization: Agents trained on Gibson outperformed those trained on Matterport3D, even when evaluated on Matterport3D, suggesting the benefit of curriculum learning where one starts with simpler environments before progressing to more complex ones.

Implications and Future Directions

The implications of this research are multi-faceted:

• Practical Application: For tasks requiring navigation and interaction within complex environments, leveraging high-performance simulation platforms like Habitat can dramatically enhance the development and deployment of capable AI agents in real-world scenarios.
• Theoretical Advancement: This research contributes to our understanding of embodied AI, particularly the interaction between sensor types and generalization across datasets. It underscores the importance of depth perception for robust navigation.

Prospective Developments

Future initiatives for Habitat include integrating physics simulation to support object manipulation and enabling multi-agent distributed simulations for studying collaborative or competitive scenarios. Additionally, enhancing the realism of sensor and actuation noise models will further bridge the gap between simulated and real-world environments, fostering more applicable AI systems.

Conclusion

The Habitat platform marks a significant step in enabling scalable, efficient research in embodied AI. By providing a flexible, high-performance simulation environment, it allows researchers to explore and benchmark complex AI tasks more effectively. As the community continues to leverage and expand upon this platform, we can expect substantial advancements in the domain of embodied AI.