WorldTest Protocol Overview

• WorldTest is a protocol that defines a two-phase approach, separating unscored exploration from a task-specific scored test phase to evaluate world-model learning.
• Its methodology leverages behavior-based scoring across challenges like masked-frame prediction, planning, and change detection in grid world settings.
• The framework enables direct comparison between human and machine agents, highlighting differences in strategic exploration and hypothesis-driven resets.

WorldTest is a two-phase, black-box evaluation framework for world-model learning that distinctly separates reward-free exploration from a scored test phase comprising one or more derived tasks in a modified but related environment. Its design is representation-agnostic, interactive, and open-ended, supporting rigorous, behavior-based comparison of arbitrary model-learning agents, including humans, classical algorithms, and neural models. WorldTest’s strict separation between unscored exploration and subsequent task evaluation directly measures the agent’s internalization of environment dynamics, beyond reward maximization or next-frame prediction benchmarks (Warrier et al., 22 Oct 2025).

1. Formal Specification and Protocol Structure

WorldTest operates over a reward-free partially observable Markov decision process (POMDP) $\mathcal{M} = \langle S, A, O, T, \Omega \rangle$, with $S$ as state space, $A$ as action set, $O$ as the observation space, $T$ as transition function, and $\Omega$ as the observation kernel. The challenge space is defined by a parameter set $\Xi$ sampled from distribution $P_\Xi$, and a family of transformation functions $$\tau : (\mathcal{M}, \xi) \mapsto (\mathcal{M}', R, H)$$, where $\mathcal{M}'$ is a derived POMDP, $S$0 is a trajectory-based objective, and $S$1 is the evaluation horizon.

Each WorldTest run consists of:

1. Task Type Disclosure: The agent learns the task family (e.g., masked-frame prediction, planning, or change detection) but not the specific challenge instance.
2. Interaction Phase: The agent freely interacts with $S$2 for an unlimited number of steps, choosing $S$3 at each time step, and may reset to the environment’s initial state at will. Rewards are absent in this phase, enforcing a goal-free setting. The agent may end exploration with a special “go-to-test” action.
3. Test Instantiation: The protocol samples $S$4 and generates the challenge $S$5.
4. Policy Generation: The agent, based on its learned internal model $S$6 and knowledge of $S$7, returns a policy $S$8 for the test phase.
5. Test Execution: The agent executes $S$9 in $A$0 for $A$1 steps, collecting a trajectory $A$2.
6. Scoring: The agent’s performance is computed as $A$3 (Warrier et al., 22 Oct 2025).

2. Evaluation Objectives and Scoring Methodologies

WorldTest implements task-specific, behavior-based scoring functions to capture model generalization across masked-frame prediction (MFP), planning (PL), and change detection (CD):

• Masked-Frame Prediction: The agent must choose the correct filling among $A$4 options for masked cells after following a fixed action sequence; binary score: $A$5.
• Planning: The goal is to reach a specified subgrid pattern $A$6 in $A$7 steps; binary score: $A$8.
• Change Detection: The agent must detect the time $A$9 at which an environment event changes. The score depends on the reported $O$0 and is a piecewise function: $O$1 if $O$2, $O$3 if $O$4, and a smoothly decaying penalty $O$5 otherwise, where

$O$6

Agent performance is aggregated as $O$7 across $O$8 i.i.d. tasks per family or globally. To measure exploration focus, normalized perplexity (NP) and its AUC are computed over the action distribution in a sliding window, with

$O$9

Low final NP and AUC indicate convergent, targeted exploration versus random or unfocused behavior (Warrier et al., 22 Oct 2025).

3. AutumnBench: Instantiation in 2D Grid Worlds

AutumnBench operationalizes WorldTest in a suite of 43 grid world environments (3–25 cells per side, diverse object and color sets, 19 stochastic), expressed in the Autumn DSL. Each environment supports all three challenge families—MFP, PL, CD—yielding 129 tasks. Tasks are constructed by:

• MFP: Masking cells in the terminal observation after a fixed action sequence; agent selects among 6 candidates.
• PL: Specified subgrid and color-arrangement goal, to be achieved within a fixed horizon.
• CD: Environment logic altered by a hidden switch at an unknown time $T$0; agent claims "Found change" and selects $T$1 (Warrier et al., 22 Oct 2025).

Environment Attribute	Range/Statistic	Notes
Grid size	$T$2 to $T$3	Majority $T$4
Object types	$T$5–$T$6 per environment
Colors	$T$7–$T$8 per environment
Stochastic dynamics	$T$9 out of $\Omega$0 environments
Code lines per env	$\Omega$1–$\Omega$2

4. Empirical Performance and Behavioral Analysis

The protocol enables systematic benchmarking across human and algorithmic learners. In (Warrier et al., 22 Oct 2025), agents include an aggregate “80th-percentile” human (N=517) and frontier reasoning models (Claude 4 Sonnet, Gemini 2.5 Pro, OpenAI o3):

• Human performance: $\Omega$3–$\Omega$4 average across all challenge types.
• Frontier models: $\Omega$5–$\Omega$6 (change detection lowest, masked-frame highest).
• Scaling trends: Model scaling only improves MFP/PL in $\Omega$7 of environments, and only $\Omega$8 of CD tasks.
• Environment stochasticity: Humans are unaffected by stochasticity; models perform up to $\Omega$9 better on stochastic cases.

Exploration behavior distinctly differs between agents:

• Action mix: Humans: $\Xi$0 clicks, $\Xi$1 directional, $\Xi$2 resets, $\Xi$3 no-ops; models: $\Xi$4 clicks/arrows, $\Xi$57% resets/no-ops.
• Reset frequency: Humans median $\Xi$6 per env; models $\Xi$71.
• Exploration focus: Human NP AUC $\Xi$8, final NP $\Xi$9 vs. models $P_\Xi$0 for both.

Model errors reveal deficiencies in hypothesis testing, belief revision, and targeted resets. In MFP, models often persist with the first consistent rule ignoring new evidence, and rarely employ hypothesis-driven resets (Warrier et al., 22 Oct 2025).

5. Theoretical Motivations and Limitations

WorldTest addresses critical deficiencies of reward-oriented or next-frame prediction evaluations by enforcing reward-free, open-ended learning in the interaction phase. Its design supports representation-agnostic benchmarking and exposes the extent to which learned models can generalize to new challenges. AutumnBench, with its rich grid-world suite, makes these desiderata practical and human-interpretable (Warrier et al., 22 Oct 2025).

Notably, current text-reasoning models lack strategic exploration, robust uncertainty quantification, and dynamic belief updating. The protocol highlights significant headroom between state-of-the-art machine agents and human learners, especially concerning hypothesis-driven resets and adaptive exploration.

However, AutumnBench is limited to symbolic 2D domains. Extensions to continuous control (e.g., physics engines, robotics), multi-agent interaction, and domains with more complex observation and action spaces remain open research problems.

6. Open Problems and Pathways for Extension

Future research directions highlighted in (Warrier et al., 22 Oct 2025) include:

• Adaptation of WorldTest beyond 2D grids, encompassing continuous and real-world problem settings.
• Synthesis of more diverse $P_\Xi$1 functions, including analogy, tool reuse, and multi-step counterfactuals.
• Integration of active learning signals or hybrid reward protocols in the interaction environment.
• Agent architectures driven by intrinsic motivation, hypothesis-driven resets, and full Bayesian belief updating.

A plausible implication is that enhancing agents with these features is essential to bridging the gap to human-like world-model learning and generalization. WorldTest provides a rigorous template for such evaluation and methodological innovation in world-model research (Warrier et al., 22 Oct 2025).