ScienceWorld: Is your Agent Smarter than a 5th Grader? (2203.07540v2)

Abstract: We present ScienceWorld, a benchmark to test agents' scientific reasoning abilities in a new interactive text environment at the level of a standard elementary school science curriculum. Despite the transformer-based progress seen in question-answering and scientific text processing, we find that current models cannot reason about or explain learned science concepts in novel contexts. For instance, models can easily answer what the conductivity of a known material is but struggle when asked how they would conduct an experiment in a grounded environment to find the conductivity of an unknown material. This begs the question of whether current models are simply retrieving answers by way of seeing a large number of similar examples or if they have learned to reason about concepts in a reusable manner. We hypothesize that agents need to be grounded in interactive environments to achieve such reasoning capabilities. Our experiments provide empirical evidence supporting this hypothesis -- showing that a 1.5 million parameter agent trained interactively for 100k steps outperforms a 11 billion parameter model statically trained for scientific question-answering and reasoning from millions of expert demonstrations.

• The paper introduces a novel text-based simulation that tests agents’ ability to perform interactive scientific experiments.
• Empirical findings reveal that smaller, interactively trained agents outperform larger static models, highlighting the impact of dynamic learning environments.
• The benchmark integrates diverse fields like thermodynamics, circuitry, and biology to drive advancements in AI’s procedural and declarative reasoning.

SCIENCEWORLD: Evaluating Agents' Scientific Reasoning in Interactive Environments

The paper "SCIENCEWORLD: Is your Agent Smarter than a 5th Grader?" introduces the SCIENCEWORLD benchmark as a novel approach to assess agents' scientific reasoning abilities in a text-based interactive environment. The research focuses on determining whether contemporary models possess genuine reasoning capabilities or if their performance is predominantly reliant on pattern recognition from extensive training data. The benchmark emulates the complexity of standard elementary school science curricula and consists of a series of tasks that require agents to perform virtual scientific experiments rather than just answering questions.

The SCIENCEWORLD Framework

SCIENCEWORLD is a comprehensive simulation environment comprising numerous interconnected locations, tasks, and simulation engines that mimic thermodynamics, electrical circuits, chemistry, and biological processes. The environment encourages agents to apply both declarative and procedural knowledge to perform and explain scientific tasks. By constructing SCIENCEWORLD, the authors aim to transition from simple question answering to tasks that demand an in-depth understanding of scientific principles through interactive learning.

Empirical Findings

The paper evaluates several state-of-the-art models, including Deep Reinforcement Relevance Network (DRRN), KG-A2C, CALM (GPT-2), and two novel transformer-based models adapted from Behavior Cloning and Decision Transformer architectures. The results reveal that smaller, interactively trained agents outperform larger LLMs for these tasks. For instance, a 1.5 million parameter DRRN model consistently achieves higher scores on SCIENCEWORLD tasks than a larger 11 billion parameter T5 model trained with static question-answer sequences. This indicates that interactive environments facilitate the development of more genuine reasoning capabilities.

Implications and Future Directions

The findings from SCIENCEWORLD have profound implications for the design and evaluation of AI systems focused on scientific reasoning. They suggest that grounding agents in interactive simulations is crucial for enabling them to learn reusable and adaptable reasoning skills. Additionally, these results provide insight into model scalability, offering evidence that larger models do not always translate to superior performance in complex reasoning tasks. Future research in AI could benefit from leveraging interactive text environments to refine agents' abilities in navigating procedural tasks and integrating domain-specific knowledge effectively. These advancements could drive enhancements in educational technologies and intelligent systems with applications in diverse scientific and engineering fields.

In summary, SCIENCEWORLD serves as a challenging yet insightful benchmark for AI researchers aiming to cultivate genuine scientific reasoning abilities. The paper not only highlights current model limitations but also pinpoints the potential pathways to more advanced, adaptable AI systems.