Benchmarking World-Model Learning

Abstract: Model-learning agents should gather information to learn world models that support many downstream tasks and inferences, such as predicting unobserved states, estimating near- and far-term consequences of actions, planning action sequences, and detecting changes in dynamics. Current methods for learning and evaluating world models diverge from this goal: training and evaluation are anchored to next-frame prediction, and success is scored by reward maximization in the same environment. We propose WorldTest, a protocol to evaluate model-learning agents that separates reward-free interaction from a scored test phase in a different but related environment. WorldTest is open-ended$\unicode{x2014}$models should support many different tasks unknown ahead of time$\unicode{x2014}$and agnostic to model representation, allowing comparison across approaches. We instantiated WorldTest with AutumnBench, a suite of 43 interactive grid-world environments and 129 tasks across three families: masked-frame prediction, planning, and predicting changes to the causal dynamics. We compared 517 human participants and three frontier models on AutumnBench. We found that humans outperform the models, and scaling compute improves performance only in some environments but not others. WorldTest provides a novel template$\unicode{x2014}$reward-free exploration, derived tests, and behavior-based scoring$\unicode{x2014}$to evaluate what agents learn about environment dynamics, and AutumnBench exposes significant headroom in world-model learning.

• The paper introduces WorldTest, a unified evaluation framework that assesses world-model learning via a two-phase process, separating exploration from testing.
• It utilizes a reward-free interaction phase followed by a derived challenge phase to offer a representation-agnostic, behavior-based metric for generalization.
• Empirical results reveal humans outperform frontier models, underscoring current AI deficits in exploration strategies and belief-updating mechanisms.

Benchmarking World-Model Learning: The WorldTest Framework and AutumnBench

Motivation and Limitations of Existing World-Model Evaluation

World models—internal representations of environment dynamics—are central to flexible, general intelligence. However, the evaluation of world-model learning in artificial agents remains fragmented. Existing approaches fall into four broad categories:

• Non-interactive benchmarks (e.g., ARC, CLEVRER) test generalization from static examples but lack interaction and temporal evolution.
• Representation-based evaluation enforces specific output formats (e.g., next-frame prediction, program synthesis, causal graphs), limiting cross-model and human comparison and often relying on proxy metrics that may not reflect true world-model quality.
• Gym-like benchmarks (e.g., OpenAI Gym, Procgen) focus on reward maximization in fixed environments, conflating world-model quality with policy optimization.
• Unsupervised RL benchmarks separate reward-free exploration from downstream task evaluation, but typically test in the same environment, not in modified or novel settings.

These approaches either restrict the agent’s representational flexibility, fail to test generalization to new tasks or environments, or do not provide a unified, behavior-based metric for world-model quality.

The WorldTest Framework

WorldTest is introduced as a unifying, representation-agnostic, behavior-based protocol for evaluating world-model learning. The framework is defined by two distinct phases:

1. Interaction Phase: The agent interacts with a base environment (a reward-free POMDP) without any explicit objectives or external rewards. The agent may reset the environment arbitrarily, facilitating systematic exploration and hypothesis testing. The agent decides when to proceed to the test phase.
2. Test Phase: The agent is evaluated in a derived challenge environment, constructed by modifying the base environment (e.g., changing dynamics, masking observations, introducing new goals). The agent must use its learned world model to solve a task in this new environment, with performance measured solely by external behavior.

Formally, the protocol is parameterized by a deterministic function $\tau$ that maps the base environment and a sampled task parameter to a derived challenge environment, a reward function, and an evaluation horizon. The agent is scored only on its performance in the test phase, with no access to the challenge environment during exploration.

Key properties:

• Representation-agnostic: No assumptions are made about the agent’s internal model; only behavior is evaluated.
• Goal-free exploration: The agent is not guided by extrinsic rewards during learning.
• Generalization: The test phase can involve tasks or environments not seen during exploration, directly probing the flexibility and robustness of the learned world model.

AutumnBench: Instantiating WorldTest

AutumnBench is a concrete instantiation of WorldTest, comprising 43 grid-world environments specified in the Autumn DSL. Each environment is a reward-free POMDP with partial observability, diverse object types, and a range of deterministic and stochastic dynamics. The environments are designed to be structurally novel, intuitive for humans, and diverse in their underlying rules.

For each environment, three challenge types are defined for the test phase:

• Masked Frame Prediction (MFP): The agent observes a trajectory with masked frames and must infer the missing content in the final observation, selecting from multiple candidates.
• Planning: The agent is given a goal state (specified as a subgrid configuration) and must generate an action sequence to reach it.
• Change Detection (CD): The agent interacts with a modified environment where a rule changes at an unknown time and must identify the earliest timestep at which the change occurs.

This yields 129 distinct tasks, each requiring different aspects of world-model reasoning: prediction, planning, and counterfactual inference.

Empirical Evaluation: Humans vs. Frontier Models

A large-scale empirical study was conducted, evaluating 517 human participants and three state-of-the-art reasoning models (Anthropic Claude, OpenAI o3, Google Gemini 2.5 Pro) on AutumnBench. The evaluation protocol was strictly aligned for both humans (via a web GUI) and models (via a text-based interface), ensuring comparability.

Key findings:

• Humans consistently outperform all models across all environments and task types. Human aggregate scores approach optimality, while models exhibit substantial deficits, especially in deterministic environments and tasks requiring flexible adaptation.
• Scaling compute improves model performance in only a subset of environments. In 25 of 43 environments, increased computational resources yield better scores, but in the remainder, performance plateaus or even degrades, indicating fundamental limitations in current model architectures or training regimes.
• Exploration strategies differ sharply. Humans make extensive use of resets and no-ops, indicative of systematic hypothesis testing and experimental design. Models rarely use these actions, focusing instead on direct manipulation (clicks, directional moves), and fail to leverage resets as a tool for causal inference.
• World-model learning is reflected in action entropy. Human action sequences show rapid reduction in normalized perplexity (entropy), indicating a transition from exploratory to targeted behavior as the world model is refined. Models remain more stochastic and less focused throughout exploration.
• Belief updating and meta-reasoning are key failure points for models. Models often fail to revise their internal hypotheses in light of contradictory evidence, particularly in tasks where the environment changes or where partial observability is critical.

Implementation and Reproducibility

All environments are specified in the Autumn DSL, a functional reactive language for 2D grid POMDPs. The benchmark is fully reproducible, with source code, environment specifications, and evaluation protocols provided. The web-based GUI and text-based interfaces are designed for extensibility and automated evaluation.

Performance metrics include binary success rates for MFP and planning, a graded penalty for late or incorrect change detection, and normalized action entropy for exploration analysis. Random agent baselines are reported for all tasks.

Implications and Future Directions

WorldTest and AutumnBench provide a rigorous, extensible framework for evaluating world-model learning in both artificial and human agents. The empirical results highlight substantial gaps between current frontier models and human-level world-modeling, particularly in generalization, experimental design, and belief revision.

Practical implications:

• Benchmarking: AutumnBench enables systematic, cross-model, and human comparison of world-model learning, decoupled from policy optimization or representational constraints.
• Agent design: The observed deficits in model exploration and belief updating suggest the need for architectures with explicit meta-reasoning, uncertainty quantification, and experimental design capabilities.
• Generalization: The framework is readily extensible to richer domains (e.g., physics, robotics, multi-agent systems), supporting the development and evaluation of more general world-modeling agents.

Theoretical implications:

• Separation of learning and evaluation: By decoupling exploration from downstream task performance, WorldTest provides a clean test of world-model quality, independent of reward shaping or policy learning.
• Behavioral metrics: The use of normalized action entropy and reset frequency as proxies for world-model refinement offers a principled approach to quantifying exploration quality.

Future work should focus on:

• Extending WorldTest to continuous, high-dimensional, and embodied environments.
• Developing agents with explicit mechanisms for hypothesis generation, experimental design, and flexible belief updating.
• Investigating the relationship between exploration strategies, world-model structure, and downstream generalization.

Conclusion

WorldTest and AutumnBench establish a new standard for evaluating world-model learning, emphasizing representation-agnostic, behavior-based, and generalization-focused assessment. The empirical gap between humans and current models underscores the need for advances in agent architectures and training paradigms that support flexible, adaptive, and metacognitive world-model learning. The framework provides a foundation for future research in both the measurement and improvement of world-modeling capabilities in artificial agents.