AnyGoal: On World Model Generalization Beyond Expert Rollouts
Abstract: The goal of a world model is to capture the dynamics of an environment. This has the great appeal that it allows agents to plan and simulate a course of actions with the help of a world model. For these uses to be reliable, however, the learned dynamics must behave well beyond the specific state-action combinations observed in expert rollouts. Despite this, the relationship between data coverage, dynamics generalization, and downstream planning performance remains underexplored. We study this relationship through the lens of compositional generalization: can a world model recombine familiar state and action factors into valid dynamics? To answer this, we introduce AnyGoal, a benchmark of goal-shifted task variants built from widely used world-model environments, together with an open-source library. Each task is designed to be within reach of a world model that has learned the underlying dynamics: individual state factors and actions remain within their training support, but their combinations are underrepresented in expert trajectories. Across multiple environments and world model families, we find that current models consistently exhibit failures under these recombinations. By examining dynamics and representation failures across model families, interaction regimes, and evaluation metrics, we study where current world models generalize and where challenges remain for goal-shifted prediction and planning. Together, this framework, benchmark, and analysis provide a basis for evaluating progress toward world models that generalize to new planning tasks.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. What is the paper about?
The paper introduces AnyGoal, a test for checking whether AI systems called world models can understand how the world changes when an agent takes different actions.
A world model is like an internal simulator. For example, a robot might use one to predict:
- “If I move my arm this way, where will the object go?”
- “If I walk around the block, what will I see next?”
- “If I push this object, will it move toward the target?”
The paper asks an important question:
Can a world model handle a new goal using situations and actions it has seen before, but in combinations it has not seen very often?
The authors find that many current world models struggle with this.
2. Main research questions
The researchers focus on several connected questions:
- Can world models recombine familiar information? For example, can a model handle a familiar object, a familiar position, and a familiar action when they appear together in a new way?
- Do models learn real cause-and-effect rules, or do they copy expert behavior? A model might learn that an agent usually moves toward an object in a certain situation. It may fail when the agent is in that same situation but takes a different action.
- Are failures caused by vision or by poor understanding of physics? The model might fail because it cannot recognize a new-looking image. Or it might recognize everything correctly but still not understand what the action will do.
- Are some tasks harder than others? Moving freely through an open area may be easier than pushing, grabbing, or inserting objects, where small mistakes matter a lot.
3. How the researchers studied the problem
Creating the AnyGoal benchmark
The authors created AnyGoal, a collection of new versions of existing tasks. These tasks change the goal but keep the basic world, objects, and possible actions the same.
For example:
- In Push-T, an agent learns to push a T-shaped object toward a target.
- In a new task called Avoid-T, the agent must avoid the object instead.
The model may have already seen:
- the same kinds of agent positions,
- the same object positions,
- the same actions, and
- the same physical environment.
However, it may not have seen those things combined in the new way required by the changed goal.
This is called compositional generalization. A simple analogy is learning the words “red,” “car,” and “fast,” but then being asked to understand the new combination “a fast red car.” The individual pieces are familiar, but the combination is new.
Checking whether the test is fair
The researchers used statistical checks to make sure the new tasks did not introduce completely unfamiliar objects or actions.
They used methods such as:
- Range checks, which ask whether new values fall inside the old minimum and maximum.
- Distance checks, which ask whether new examples are reasonably close to the old examples.
- Density checks, which estimate whether new examples occur in regions similar to the training data.
In everyday terms, these checks make sure the test is not asking the model to do something completely strange. The challenge is mainly the new combination, not a brand-new object or action.
Testing different world models
The study tested several types of world models, including:
- models that predict future images,
- models that predict hidden features instead of pixels,
- models that use learned visual representations, and
- models given the true physical state directly.
The researchers measured two main abilities:
- Prediction: Can the model accurately predict what happens after several actions?
- Planning: Can an agent use the model to choose actions that reach the goal?
They also tested oracle-state predictors. These models were given important information directly, such as the exact positions of objects. This helped the researchers determine whether failures came from seeing the world or from understanding its dynamics.
Finally, the researchers performed action interventions. They kept the same situation but changed the action given to the model. If the model truly understands actions, its prediction should change appropriately.
4. Main findings
World models often fail on new combinations
Across many environments and model types, performance became worse when the goal changed, even though the individual states and actions were familiar.
This suggests that many models are not learning fully reusable rules of physics. Instead, they may partly memorize patterns from the expert demonstrations used for training.
Models may copy behavior instead of responding to actions
Suppose an expert usually walks toward an object whenever it is far away. A model may learn:
“When the agent is far from the object, it will move toward it.”
But the correct rule should be:
“The next state depends on the action that was actually chosen.”
When researchers gave the model unusual actions in familiar situations, some models barely changed their predictions. This showed that they were sometimes relying on the usual expert behavior rather than paying enough attention to the supplied action.
Contact-rich tasks were especially difficult
Tasks involving physical contact—such as pushing, grasping, inserting, or moving objects—usually caused much larger failures than simple movement through open space.
This makes sense because contact tasks are more sensitive. A small error in position or timing can cause an object to miss, fall, or move in the wrong direction.
In open spaces, there may be many successful paths. In contact tasks, there may be only a few actions that work.
Giving models exact state information did not solve everything
Even when the models were given accurate positions and other physical information, they still struggled with many goal changes, especially contact-rich tasks.
This means the problem is not only poor vision. Some models also fail to learn the underlying action-and-physics rules.
Visual representations can create additional problems
End-to-end models must first turn images into useful information, such as object positions and movement. A new combination of familiar factors can produce an image arrangement that was uncommon during training.
As a result, a model may fail in two stages:
- It may misunderstand what it sees.
- It may then make an incorrect prediction about what will happen.
The paper also finds that even when images look familiar, the model’s internal representation—the way it organizes information—may not preserve important details accurately.
Prediction and planning do not always agree
A model can sometimes make imperfect predictions but still find a successful path. This is more likely in simple movement tasks, where many routes may reach the goal.
In contact tasks, inaccurate predictions are more harmful because planning requires precise interactions.
5. Why the findings matter
The results show that doing well on familiar training tasks is not enough to prove that a world model truly understands the environment.
A model might appear successful because it has learned the habits of the expert demonstrations. However, a useful world model should be able to answer:
“What will happen if I take this action here, even if an expert did not usually do that during training?”
This matters for robots, self-driving vehicles, game-playing agents, and other AI systems that need to handle new situations safely.
6. Implications and future impact
The paper suggests that future world models should be designed to:
- pay close attention to the action being taken,
- learn cause-and-effect rules instead of copying behavior patterns,
- represent separate physical factors clearly,
- handle interactions between objects and actions,
- test performance on new combinations, not only familiar tasks, and
- use data that covers important local situations, even if it does not cover every possible state-action combination.
The authors argue that collecting every possible combination of states and actions would be unrealistic. Instead, a model may succeed if it learns smaller, reusable mechanisms—for example, how pushing affects an object, how movement changes position, and how grasping changes control.
Overall, AnyGoal provides a useful warning and a practical test: an AI system should not only perform tasks it has seen before. It should also be able to use familiar knowledge to plan for new goals.
Practical Applications
Immediate Applications
- World-model evaluation and model selection (AI research and robotics).
Use the open-source AnyGoal library to evaluate a world model on both its original expert-goal task and goal-shifted variants such as
Avoid-T,TR-Route, or contact-rich manipulation tasks. A practical workflow is:- train the model on the base dataset;
- run standardized rollout-prediction and MPC evaluations;
- compare in-distribution and goal-shift performance;
- reject or retrain models showing large degradation. This can be integrated into research benchmarks, robotics model cards, and pre-deployment validation pipelines. Dependency: the target environment must be represented by, or adapted to, AnyGoal-style state, action, and goal variants.
Regression testing for model updates (software and robotics engineering). AnyGoal can function as a compositional-generalization test suite in continuous integration. Every change to a visual encoder, action-conditioning mechanism, planning horizon, or training dataset can be tested against fixed base and goal-shifted episodes. This detects models that improve standard benchmark scores while becoming more dependent on expert rollout biases. Dependency: reproducible seeds, stable simulator versions, and consistent evaluation metrics are required.
- Safety screening for model-based controllers (autonomous systems). The paper’s intervention-based action-sensitivity test can be used to check whether a model responds to the supplied action or merely predicts the action typically taken by an expert. Low sensitivity in behavior-biased regions should trigger additional testing before deployment in robots, autonomous vehicles, warehouse systems, or drones. The resulting workflow could include counterfactual action probes, uncertainty thresholds, and a fallback controller. Dependency: safe simulation or offline testing must be available because arbitrary action interventions may be unsafe on physical systems.
- Dataset auditing for state–action coverage (industry and academia). The paper distinguishes marginal coverage from joint state–action coverage. Organizations can use the range, Mahalanobis, KDE, and joint-support analyses to identify datasets in which individual states and actions are familiar but their combinations are rare. This is useful for auditing robotics demonstrations, autonomous-driving logs, industrial-control data, and offline reinforcement-learning datasets. Dependency: meaningful physical factors must be identified, estimated, or approximated, and density estimates become less reliable in high-dimensional spaces.
- Targeted data collection rather than indiscriminate scaling (robotics and industrial automation). Once an under-covered combinatorial region is identified, data-collection tools can be used to generate trajectories specifically for that region—for example, approaching an object from unusual but physically valid directions or applying alternative actions in familiar states. This provides a more efficient alternative to simply collecting more expert demonstrations. Dependency: the simulator or robot must support controlled exploration, and generated trajectories must remain physically valid and sufficiently diverse.
- More informative benchmarking of planning systems (AI evaluation and academia).
- short- and long-horizon prediction;
- free-space and contact-rich tasks;
- oracle-state and pixel-based models;
- latent, pixel, and task-level metrics.
- Dependency: comparisons across models require care because prediction errors in latent, pixel, and oracle-state spaces are not directly equivalent.
- Offline-RL policy validation. Before deploying a policy trained from fixed demonstrations, practitioners can evaluate the learned model and policy on goal changes that preserve familiar state and action marginals but alter their combinations. This reveals whether policy improvement depends on unsupported or poorly modeled transitions. The approach is especially relevant to manipulation, navigation, and industrial control. Dependency: the evaluation goal must be defined without introducing genuinely novel physical factors, since AnyGoal primarily tests recombination rather than broad extrapolation.
- Educational tools for world-model and causal-learning courses.
- behavior prediction and action-conditioned dynamics;
- marginal support and joint support;
- representation failure and dynamics failure;
- prediction accuracy and planning performance.
- Students can train a model on
Push-Tand evaluate it onAvoid-T, then perform action interventions to observe shortcut behavior. Dependency: sufficient computational resources and access to the released environments are needed.
Long-Term Applications
- Robots that adapt to new goals without retraining from scratch (robotics and physical intelligence). A successful successor to the models evaluated in the paper could learn reusable mechanisms for navigation, grasping, pushing, insertion, and obstacle avoidance, then recombine them for new goals. This could support household robots, warehouse manipulators, laboratory automation, and service robots that receive changing instructions. Dependencies: models must learn action-conditioned mechanisms rather than expert-policy shortcuts; contact-rich interactions require substantially better generalization than free-space motion; real-world sensing and actuation noise must also be handled.
- General-purpose model-predictive control for autonomous vehicles and drones. Compositional world models could allow an autonomous system to reuse familiar motion and obstacle dynamics under new routes, targets, traffic patterns, or mission objectives. AnyGoal-style tests could verify whether the model can plan around obstacles rather than merely reproduce common training routes. Dependencies: the relevant state factors must be observable, safety-critical edge cases require very high reliability, and physical dynamics may change with weather, terrain, vehicle load, or actuator condition.
- Contact-aware manipulation models. The paper indicates that contact-rich variants are substantially harder even when oracle state is provided. Long-term systems could address this through regime-dependent models for free motion, contact, grasping, and insertion, combined with sparse or modular mechanisms. Such systems might support robots that transfer between pushing, peg insertion, tool use, and object rearrangement tasks. Dependencies: contact dynamics are highly nonlinear and often partially observed; tactile sensing, regime identification, intervention data, and real-robot validation are likely necessary.
- Mechanism-based or modular world-model architectures. The paper suggests that global coverage of every state–action combination is unrealistic and that local coverage of the inputs relevant to each transition mechanism may be sufficient. This motivates architectures that factorize dynamics into reusable modules—for example, separate mechanisms for agent motion, object motion, contact, grasping, and environment constraints. Such modules could be recombined under new goals or task layouts. Dependencies: the relevant factors and interaction regimes must be identifiable, and incorrect factorization could introduce systematic planning errors.
- Active-learning systems for closing combinatorial coverage gaps. A future data-collection system could identify where action sensitivity is low or joint-support density is poor, select informative interventions, execute them in simulation or on a robot, and retrain the world model. This would create an iterative loop of model diagnosis, targeted exploration, and validation. Potential products include automated robotics-data engines and coverage-aware digital-twin platforms. Dependencies: exploration must be safe and cost-effective; density estimates must guide genuinely useful interventions; and the system must avoid collecting redundant or physically unrealistic data.
- Reliable digital twins for industrial control and maintenance. In manufacturing, energy, and process industries, a compositional world model could simulate alternative operating goals using familiar sensor states and control actions. For example, it might evaluate energy-saving schedules, maintenance procedures, or production changes that were not present in historical operator trajectories. AnyGoal-style evaluation could test whether the digital twin models intervention effects rather than merely replaying normal operations. Dependencies: industrial dynamics may be nonstationary, data may be heavily biased toward nominal operation, and deployment requires calibrated uncertainty, validation against plant data, and fail-safe control.
- Policy and certification standards for learned controllers.
- maintains performance under defined goal shifts;
- responds to counterfactual actions;
- distinguishes free-space from contact-rich failure modes;
- provides uncertainty or abstention in poorly supported regions.
- Dependencies: standardized task definitions, domain-specific thresholds, realistic simulators, and evidence that benchmark performance predicts real-world safety are still required.
- Human-assistance systems that plan under changing objectives. In healthcare, logistics, education, or household automation, a world model could support planning when the user changes priorities—for example, minimizing time instead of energy, avoiding a region instead of reaching it, or rearranging tasks under new constraints. The paper’s findings suggest that such systems should be tested on goal changes rather than evaluated only on the objectives present in demonstrations. Dependencies: human goals must be represented reliably, errors may have asymmetric consequences, and systems should retain human override and transparent failure reporting.
- Compositional generalization theory and diagnostic tooling. The paper provides a basis for future tools that estimate whether a learned model has recovered local transition mechanisms, rather than merely fitting observed trajectories. Possible developments include mechanism-level probes, causal action interventions, factorized uncertainty estimates, and formal guarantees under sparse local coverage. These tools could become standard components of world-model training and evaluation libraries. Dependencies: the underlying physical factors may be unknown in real applications, and theoretical guarantees require assumptions about factorization, regime stability, and the complexity of transition mechanisms.
Glossary
- Action-conditioned dynamics: Environment transitions modeled as dependent on both the current state and supplied action. “aim to learn action-conditioned dynamics of environment”
- Action sensitivity: The degree to which a model’s prediction changes when its input action changes. “We investigate whether this failure mode is reflected in the learned models by measuring their sensitivity to interventional action inputs.”
- AdaLN (Adaptive Layer Normalization): A conditioning mechanism that modulates layer-normalization parameters based on an input. “It uses SIGReg~\citep{lejepa} for anti-collapse regularization and conditions the predictor through AdaLN~\citep{Peebles_2023_ICCV}.”
- Autoregressive prediction: Sequential prediction in which each output can be used to generate later outputs. “We evaluate autoregressive prediction over horizons of 5 and 25 environment steps”
- Behavior policy: The policy that generated the actions in an offline dataset. “We consider the common offline setting in which data are collected by a behavior policy ”
- Behavior-policy shortcut: Reliance on correlations in observed behavior instead of learning the causal effect of an action. “Diagnosing Behavior-Policy Shortcuts Through Action Sensitivity”
- Causal mechanism: A stable process describing how variables produce effects under interventions. “under which reusable mechanisms remain autonomous as their joint occurrence changes”
- Cartesian product: A set containing every possible combination of elements drawn from multiple sets. “We define the combinatorial space as the Cartesian product of the marginal training supports”
- CEM-based MPC: Model-predictive control that uses the Cross-Entropy Method to optimize candidate action sequences. “We evaluate downstream control using the same CEM-based MPC implementation from Stable-WorldModel”
- Combinatorial gap: The set of familiar factor combinations that are absent from the training joint distribution. “AnyGoal targets the combinatorial gap, regions of the combinatorial space that are absent from the training joint support”
- Compositional generalization: The ability to recombine familiar components into previously unseen configurations. “We formalize this criterion as compositional generalization.”
- Compositional distribution shift: A distribution change involving new combinations of familiar variables while preserving the underlying mechanisms. “We study compositional distribution shifts”
- Conditional action support: The set of actions observed for a particular state or state region. “A predictor trained only on such data may degenerate into an action-ignorant model for state regions with restricted conditional action support”
- Contact-rich dynamics: Dynamics involving physical interactions such as touching, grasping, or manipulating objects. “Contact-rich variants consistently degrade, despite using the same privileged state inputs.”
- Counterfactual intervention: Changing an input while holding other conditions fixed to examine its effect. “we intervene on the action input while holding the state fixed.”
- Diffusion Transformer (DiT): A Transformer architecture used to model denoising processes in diffusion-based generative models. “uses a Diffusion Transformer~\citep{Peebles_2023_ICCV} to predict future latents conditioned on action trajectories.”
- Distribution support: The set of values or configurations that can occur with nonzero probability under a distribution. “We also define the marginal support of each factor”
- End-to-end world model: A model that learns perception and dynamics jointly from raw observations. “LeWorldModel (LeWM)~\citep{lewm} is an end-to-end JEPA that jointly learns a visual encoder and an action-conditioned transformer predictor from pixels.”
- Expert rollout: A trajectory generated by a competent or goal-directed policy. “the learned dynamics must behave well beyond the specific state–action combinations observed in expert rollouts.”
- Extrapolation: Prediction beyond the combinations or values represented in the training data. “Such settings are valuable for testing robustness to novel factors or extrapolation”
- Factor-wise support check: A test verifying that each individual state or action factor remains within the training distribution. “we apply three factor-wise support checks to verify that the target distribution remains marginally in-distribution”
- Frozen encoder: A pretrained representation model whose parameters are not updated during subsequent training. “DINO-WM~\citep{dinowm} decouples perception from dynamics by using a frozen DINOv2”
- Generative world model: A world model that generates future observations or latent states, often probabilistically. “including generative world models~\citep{irasim, driftworld}”
- Hallucination: A model-generated prediction that is inconsistent with the actual environment dynamics. “Recent world-model work has studied generalization through object-centric compositional dynamics”
- In-distribution: Belonging to the range or support represented in the training data. “the target distribution remains marginally in-distribution relative to its reference distribution.”
- Interventional action sensitivity: A measure of prediction changes caused by deliberately replacing an observed action with another action. “Interventional action sensitivity.”
- Inverse-dynamics objective: A training objective that predicts the action responsible for a transition between states. “uses the same encoder--predictor family as LeWM with an inverse-dynamics objective”
- JEPA (Joint-Embedding Predictive Architecture): A representation-learning architecture that predicts future embeddings rather than reconstructing raw observations. “LeWorldModel (LeWM)~\citep{lewm} is an end-to-end JEPA”
- Kernel Density Estimation (KDE): A nonparametric method for estimating a probability density from samples. “The KDE check thresholds reference densities at the $0.01$-quantile”
- Latent geometry: The structural arrangement and distances among representations in a learned latent space. “Latent geometry shapes degradation under goal shifts.”
- Mahalanobis distance: A covariance-aware distance measuring how far a point lies from a multivariate distribution’s mean. “The Mahalanobis check uses the reference mean and covariance”
- Marginal support: The set of values taken by one variable or factor, regardless of the values of other variables. “We also define the marginal support of each factor”
- Model-predictive control (MPC): A control strategy that plans with a learned model and repeatedly executes the selected action sequence. “We evaluate rollout prediction error and downstream MPC planning.”
- Oracle-state predictor: A predictor given the true task-relevant state instead of having to infer it from observations. “To separate dynamics learning from visual representation failure, we also include oracle-state predictors”
- Out-of-distribution (OOD): Located outside the support or distribution represented in training data. “Other OOD evaluations introduce states or factors outside the training support”
- Privileged state input: Direct access to information that would normally need to be inferred from observations. “contact-rich variants consistently degrade, despite using the same privileged state inputs.”
- Representation collapse: A failure in which different inputs are mapped to indistinguishable representations. “It uses SIGReg~\citep{lejepa} for anti-collapse regularization”
- Representation geometry: The organization of data points and distances in a learned representation space. “Representation geometry matters under goal shifts.”
- Rollout prediction: Multi-step prediction of future states or observations under a sequence of actions. “We evaluate rollout prediction error and downstream MPC planning.”
- Support mismatch: A discrepancy between the regions represented in a dataset and those encountered during evaluation or policy execution. “This question is closely connected to support mismatch in offline reinforcement learning”
- Temporal U-Net: A U-Net architecture adapted to model information across time in sequential data. “uses an action-conditioned spatial--temporal U-Net to predict future observations directly”
- Transition dynamics: The rules or probability distribution governing how an environment changes from one state to the next. “The goal of a world model is to capture the dynamics of an environment.”
- World model: A learned model of an environment’s dynamics used for simulation, prediction, or planning. “A world model trained on Push-T performs poorly on Avoid-T”
- Zero-quantile threshold: A threshold determined by a specified quantile of a reference distribution. “thresholds distances at the reference $0.99$-quantile”





