I-PHYRE: Interactive Physical Reasoning (2312.03009v2)

Abstract: Current evaluation protocols predominantly assess physical reasoning in stationary scenes, creating a gap in evaluating agents' abilities to interact with dynamic events. While contemporary methods allow agents to modify initial scene configurations and observe consequences, they lack the capability to interact with events in real time. To address this, we introduce I-PHYRE, a framework that challenges agents to simultaneously exhibit intuitive physical reasoning, multi-step planning, and in-situ intervention. Here, intuitive physical reasoning refers to a quick, approximate understanding of physics to address complex problems; multi-step denotes the need for extensive sequence planning in I-PHYRE, considering each intervention can significantly alter subsequent choices; and in-situ implies the necessity for timely object manipulation within a scene, where minor timing deviations can result in task failure. We formulate four game splits to scrutinize agents' learning and generalization of essential principles of interactive physical reasoning, fostering learning through interaction with representative scenarios. Our exploration involves three planning strategies and examines several supervised and reinforcement agents' zero-shot generalization proficiency on I-PHYRE. The outcomes highlight a notable gap between existing learning algorithms and human performance, emphasizing the imperative for more research in enhancing agents with interactive physical reasoning capabilities. The environment and baselines will be made publicly available.

Introduction

Recent advancements in AI have seen the development of agents that can predict the stability of objects and navigate through complex, physics-based puzzles. However, these agents typically deal with scenarios that are static or where only a single intervention is possible. The next step in evaluating and enhancing AI capabilities involves creating scenarios where AI agents can interact with events in real-time, adjust their actions on the go, and manage the precise timing of interventions.

The New Framework: I-PHYRE

Interactive PHysical Reasoning (I-PHYRE) is a new framework aimed at bridging the gap in assessing intuitive physical reasoning in AI. Unlike previous benchmarks that focus on passive observation or one-round intervention, I-PHYRE challenges agents to understand, predict, and interact with dynamic scenarios in real-time. The model involves multi-step planning and immediate responses to a consistently changing environment.

Design and Evaluation

The core task within I-PHYRE is a block elimination game where agents must remove a series of blocks to allow red balls to fall into a hole. The game features four difficulty levels to test the learning and generalization capabilities of agents. In these tasks, the sequence and timing of each action are crucial for success.

When evaluating agents in I-PHYRE, three planning strategies are considered:

• Predefined sequences of actions based on initial scene configurations.
• Adaptive planning where actions are determined according to real-time scenarios.
• A combined approach where pre-planned interventions are dynamically adjusted after each action.

Insights from Experiments

In trials involving both humans and AI agents, human participants consistently outperformed AI models, suggesting that AI requires further advancements in interactive reasoning. Specifically, the AI struggled with generalizing learned behaviors to new scenarios. Detailed analysis revealed that AI agents face significant difficulties in devising multi-step interventions and executing actions with precise timing, hence the need for further research in these areas.

Conclusion

I-PHYRE represents a significant endeavor to push AI agents towards a more sophisticated understanding of physical events and interaction. This includes not just predicting outcomes but also actively planning and revising strategies in the face of changing dynamics. The experiments underscore the complexity of physical reasoning required for AI agents to match human-level performance, establishing a benchmark that will undoubtedly shape the future direction of interactive physical reasoning research.