Necessity of Broad and Rich Simulated Environments for Scaling Embodied Agents

Determine whether obtaining sufficient breadth and richness across many simulated environments and commercial video games is necessary to effectively scale embodied agents that connect language, perception, and action. This investigation should assess the necessity claim by evaluating agents trained on narrow versus broad and rich distributions of environments and tasks, and quantify impacts on generalization, robustness, and capability scaling.

Background

The SIMA project aims to train instructable agents that map natural language instructions and visual observations to keyboard-and-mouse actions across diverse 3D environments, including commercial video games and research simulators. A central hypothesis motivating this design is that broad, rich environmental diversity may be critical for developing general embodied AI capabilities.

In the context of robotics and embodied AI, the authors explicitly conjecture that leveraging many simulated environments and video games provides the breadth and richness necessary for effective scaling. This claim frames an open question about whether such diversity is a necessary condition for scaling embodied agents, as opposed to being merely beneficial or sufficient. Validating this conjecture would inform the design of training regimes and datasets for generalist embodied agents.