Learning to Perform Physics Experiments via Deep Reinforcement Learning

Abstract: When encountering novel objects, humans are able to infer a wide range of physical properties such as mass, friction and deformability by interacting with them in a goal driven way. This process of active interaction is in the same spirit as a scientist performing experiments to discover hidden facts. Recent advances in artificial intelligence have yielded machines that can achieve superhuman performance in Go, Atari, natural language processing, and complex control problems; however, it is not clear that these systems can rival the scientific intuition of even a young child. In this work we introduce a basic set of tasks that require agents to estimate properties such as mass and cohesion of objects in an interactive simulated environment where they can manipulate the objects and observe the consequences. We found that state of art deep reinforcement learning methods can learn to perform the experiments necessary to discover such hidden properties. By systematically manipulating the problem difficulty and the cost incurred by the agent for performing experiments, we found that agents learn different strategies that balance the cost of gathering information against the cost of making mistakes in different situations.

• The paper introduces a novel framework where DRL agents conduct interactive physics experiments to infer intrinsic object properties.
• The methodology employs simulated environments like 'Which is Heavier' and 'Towers' to test mass evaluation and object composition.
• Experimental results show agents achieve efficient, strategic learning that rivals targeted algorithmic approaches in physical reasoning.

Overview of "Learning to Perform Physics Experiments via Deep Reinforcement Learning"

The paper "Learning to Perform Physics Experiments via Deep Reinforcement Learning" explores the capabilities of AI agents to infer physical properties of objects in unknown environments through interaction, reminiscent of a scientist conducting experiments. The research is motivated by developmental psychology insights, whereby infants learn about the world by experimenting with objects. The concept of active interaction stands in contrast with purely observational learning, aiming to equip agents with a more profound understanding of physics akin to human scientific intuition.

Problem Definition and Method

The authors propose a novel framework where agents learn to conduct physics experiments in simulated environments, striving to understand intrinsic object properties like mass and cohesion. The study leverages deep reinforcement learning (DRL) to equip agents with the ability to manipulate objects actively to gather data and update beliefs about these objects' properties. The environments designed for these tasks include "Which is Heavier" and "Towers," each posing distinct experimental challenges that demand interaction.

• Which is Heavier: This environment involves selecting the heaviest block from a set by applying interactions to assess each block's mass. The mass assignments are independent of the objects' appearance to ensure reliance on interaction.
• Towers: Designed to determine how many distinct rigid bodies comprise a tower, this task compels agents to knock down towers and infer the correct composition of objects when appearances alone are insufficient.

Through variation in discount factors and problem difficulties, agents learn when to gather more information or risk acting on limited data. This methodology allows for the exploration of strategic trade-offs in decision-making under uncertainty about object properties.

Experimental Results

The agents trained via this framework have shown competencies comparable to targeted algorithmic designs despite the absence of any prior knowledge about physical properties or laws. The systematic experimentation using DRL agents revealed several insights:

• Strategy Differentiation: Agents adapted to varying difficulties, balancing between immediate judgment and comprehensive evaluation.
• Environment-Specific Adaptations: The behaviors and strategies were sensitive to the defined environments, illustrating that learned policies offer significant improvements over random baselines in experimentation efficiency and accuracy.

By demonstrating that numerical and physical property inference can be learned through interactive engagement rather than passive observation, this study illuminates a path toward more robust autonomous understanding in AI systems.

Implications and Future Directions

The practical implications of this work highlight progress toward autonomous systems capable of experimentation, leading to potential applications in robotics, where understanding and manipulation of physical properties are critical. Theoretically, it provides a robust framework for simulating and evaluating scientific exploration patterns in artificial agents.

This work opens avenues for future research in several ways:

• Tool Use and Efficiency: Investigating different forms of interaction, particularly tool use, may enhance inference efficiency and broaden applicability.
• Scalability and Generalization: Developing models that generalize physical understanding across varied tasks could offer insights into more complex problem-solving capabilities.
• Data Efficiency: Improvements to reduce the number of required samples for learning more intricate physical properties could propel real-world deployment.

Overall, this paper marks an incremental advancement in AI’s capability to mimic the scientific exploratory processes, further bridging the gap between artificial systems and human-like reasoning in physical interactions.