Agent World Models: Foundations & Advances

Updated 3 July 2026

Agent World Models are a formal framework where agents simulate environmental dynamics to predict outcomes and plan actions.
They integrate diverse state representations—textual, visual, symbolic, and multimodal—using LLMs, RL controllers, and simulators.
Empirical evaluations show that AWMs improve planning accuracy, task success rates, and coordination in single- and multi-agent scenarios.

An Agent World Model (AWM) is a formal framework and modeling paradigm in which software agents, typically realized as LLMs or reinforcement learning (RL) controllers, possess an explicit simulatable model of their environment’s dynamics. AWMs enable agents to predict the outcome of candidate actions before execution, supporting planning, foresight, counterfactual reasoning, and long-horizon coordination. AWM research now spans textual, symbolic, visual, and multimodal environments, bridging the gap between end-to-end, reactive policies and purpose-built simulators. This article surveys the formalism, methodologies, architectures, and empirical impacts of AWMs in both single-agent and multi-agent contexts, with emphasis on recent advances in structure-aware state representations, planning algorithms, and integration with modern LLM backbones.

1. Formal Definitions and Theoretical Foundations

At the core, an Agent World Model is a parameterized transition model

$(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R})$

where $\mathcal{S}$ is the set of agent-visible states (textual, visual, structured, or latent), $\mathcal{A}$ is the agent’s action space, $\mathcal{T}: \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ is the transition function, and $\mathcal{R}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$ is (optionally) a reward or evaluation function. Agent policy optimization in the presence of an AWM follows the recursions of value iteration or Bellman optimality, with the true environment dynamics $p_\mu$ replaced by the learned or synthesized model $p_f$ :

$\pi^*_f(\hat s_t) = \arg\max_{a'_{t:T-1}} \sum_{\hat s_{t+1:T}} \Bigl[\sum_{k=t}^{T-1}\gamma_k r(g,\hat s_k) + \gamma_T V^g_{\pi, f}(\hat s_T)\Bigr] \prod_{k=t}^{T-1} p_f(\hat s_{k+1} | \hat s_k, a'_k)$

This is realized concretely in frameworks such as SimuRA (Deng et al., 31 Jul 2025), where the predictive kernel $p_f$ is implemented via an autoregressive LLM, and in synthetic, code-driven environments (e.g., (Wang et al., 10 Feb 2026)) by a fully deterministic transition function.

AWMs can also take the symbolic form of an abstract world model defined as a subgoal-transition DAG or a simulator in PDDL/Python, as synthesized in Agent2World (Hu et al., 26 Dec 2025) and DECKARD (Nottingham et al., 2023). In both neural and symbolic cases, the world model is trainable with objectives such as maximum-likelihood next-state prediction, variational objectives, or decision-aware auxiliary losses.

2. State Representation and Model Architectures

AWMs support a variety of state representations and architectural choices, tailored for the task and environment:

Textual states: Natural-language summaries, JSON/DOM trees, or action histories for web and tool environments (Li et al., 8 Jun 2026, Fang et al., 23 Apr 2025, Ganapavarapu et al., 9 May 2026).
Structured sketches: MobileDreamer (Cao et al., 7 Jan 2026) encodes GUI states as sets of UI elements $(\ell_n, \tau_n, b_n)$ , preserving spatial and semantic layout via order-invariant matchings.
Visual and multimodal: ViMo (Luo et al., 15 Apr 2025) and Prisma-World (Sun et al., 8 Jun 2026) generate screenshots or multi-view video streams; states are latent codes or symbolic overlays (STR).
Latent spaces: Many modern frameworks maintain internal latent states $\mathcal{S}$ 0 updated via transformer backbones or VAE-style encoders—see MATWM (Deihim et al., 23 Jun 2025), GAWM (Shi et al., 17 Jan 2025), MCP-Cosmos (Ganapavarapu et al., 9 May 2026).
Executable/Symbolic: PDDL or code-based simulators, where the state is a tuple of predicate values or program variables. Agent2World (Hu et al., 26 Dec 2025) and synthetic code environments (Wang et al., 10 Feb 2026).

Network backbones include encoder-decoders (MobileDreamer), diffusion transformers (Prisma-World), LLMs augmented by LoRA adapters, and adapters for structured multimodal prompts.

3. Learning and Training Objectives

AWMs are typically acquired through either supervised next-state prediction or more sophisticated, decision-aligned training:

Supervised fine-tuning (SFT): Minimize $\mathcal{S}$ 1 over logged trajectories, as in standard model-based RL or for textual world models (Li et al., 8 Jun 2026, Fang et al., 23 Apr 2025).
Order-invariant losses: Structure-aware objectives combining token-level cross-entropy with permutation-invariant set matchings, leveraging optimal transport for aligning predicted sets to ground-truth elements (MobileDreamer (Cao et al., 7 Jan 2026)).
Multi-part objectives: Visual world models employ decomposition (STR + text filling in ViMo (Luo et al., 15 Apr 2025)), latent diffusion, and head-wise auxiliary losses for reward, action masks, continuation (MATWM (Deihim et al., 23 Jun 2025), GAWM (Shi et al., 17 Jan 2025)).
Decision-aligned (agent-authored) targets: Rather than predicting all details of the next observation, AAWM (Cai et al., 24 Jun 2026) synthesizes learning targets based on the agent's explicit knowledge gaps via self-probing, retrieval, and synthesis.
RL-based model learning: Qwen-AgentWorld (Zuo et al., 23 Jun 2026) combines large-scale supervised pre-training, fine-tuning with chain-of-thought rationales, and PPO-style reinforcement learning using judge-based hybrid rubrics. Critic-guided RL with reward augmentation is standard in code-based simulators (Wang et al., 10 Feb 2026, Ganapavarapu et al., 9 May 2026).

AWMs in co-evolutionary frameworks (WebEvolver (Fang et al., 23 Apr 2025)) and self-improving RL loops are trained jointly with agent policies, using synthetic rollouts for continual adaptation.

4. Planning Algorithms and Rollout Imagination

AWM-equipped agents exploit model-driven planning to move beyond reactive behaviors:

Tree-based (rollout) planning: Agents enumerate candidate actions, use the world model to simulate future states (either as structured sketches, text, or images), and select trajectories maximizing goal alignment—typically via a multi-step tree search or Monte Carlo sampling (Cao et al., 7 Jan 2026, Deng et al., 31 Jul 2025, Ritz-Zwilling et al., 15 Apr 2026).
Lookahead and simulation: Beam search, rollout imagination, and MCTS are standard planning paradigms; SimuRA (Deng et al., 31 Jul 2025) and WAC (Shen et al., 17 Feb 2026) explicitly compare one-step vs. deep lookahead, showing substantial gains for the latter.
Multi-agent coordination: Prisma-World (Sun et al., 8 Jun 2026), MATWM (Deihim et al., 23 Jun 2025), and GAWM (Shi et al., 17 Jan 2025) develop architectures to imagine decentralized or globally consistent trajectories, with joint attention and cross-agent latent exchange.
Latent-space speculative planning: MCP-Cosmos (Ganapavarapu et al., 9 May 2026) emphasizes planning in latent representations rather than concrete states, enabling offline simulation and candidate plan extraction prior to execution.

These planning algorithms often integrate learned critics, judge LLMs, or selection LLMs for scoring future states, and ground high-level plans back to concrete action sequences.

5. Empirical Evaluation and Benchmarks

AWM effectiveness is empirically validated across a diverse range of domains with both intrinsic and extrinsic metrics:

World model fidelity: Single- and multi-step prediction accuracy, token-level or structure-level F1, BLEU, Intersection-over-Union (IoU), learned perceptual metrics (LPIPS, SSIM), reprojection error (RPE) for geometric consistency (Sun et al., 8 Jun 2026, Luo et al., 15 Apr 2025).
Task-driven utility: Downstream task success rate, plan success rate, sample efficiency, execution quality (Eq) combining success, call economy, and parameter accuracy (Ganapavarapu et al., 9 May 2026).
Planning improvements: AWM-based planners consistently outperform autoregressive/reactive baselines—in Web browsing, success gains of 4–124% are typical (Deng et al., 31 Jul 2025, Fang et al., 23 Apr 2025, Shen et al., 17 Feb 2026), with sharp reductions in action error and failure loops.
Synthetic environment scaling: The AWM pipeline in (Wang et al., 10 Feb 2026) demonstrates robust out-of-distribution generalization, outperforming LLM-simulated and task-specific RL on tool-use suites.
Fine-grained ablations: Components such as order-invariant learning, overlap-decaying curriculum, and minimap guidance are shown to provide independent gains in spatial and planning fidelity (Cao et al., 7 Jan 2026, Sun et al., 8 Jun 2026).
Symbolic/simulator AWMs: Agent2World (Hu et al., 26 Dec 2025) reports ≥15-point boosts in symbolic model executability and F1 by leveraging simulation feedback; DECKARD (Nottingham et al., 2023) delivers 10–12× sample efficiency improvement for long-horizon item crafting.

6. Architectural Variants and Paradigm Extensions

AWMs now support a rich ecology of modeling paradigms:

Textual and sketch-based models: Token-level LLMs producing structured element sets or free-form future summaries—fast, efficient, and suitable for LLM-native planning (Cao et al., 7 Jan 2026, Deng et al., 31 Jul 2025, Li et al., 8 Jun 2026).
Visual and multimodal world models: Diffusion or transformer-based modules for GUI and video prediction (ViMo, Prisma-World), supporting direct pixel-level planning and multi-agent spatial consistency (Luo et al., 15 Apr 2025, Sun et al., 8 Jun 2026).
Symbolic/simulator world models: Programmatically generated or LLM-emitted simulators in code or PDDL, enabling deterministic evaluation and fine-grained unit/simulation testing (Hu et al., 26 Dec 2025, Nottingham et al., 2023, Wang et al., 10 Feb 2026).
Multi-agent and global-aware world models: MATWM (Deihim et al., 23 Jun 2025) and GAWM (Shi et al., 17 Jan 2025) address non-stationarity, global consistency, and coordination via shared latent worlds, prioritized replay, and transformer-based cross-agent fusion.
Agent-authored world modeling: The AAWM paradigm (Cai et al., 24 Jun 2026) aligns world-model learning signals directly with agent decision needs through self-probing and synthesized, decision-oriented training targets.
Chain-of-thought integration: Qwen-AgentWorld (Zuo et al., 23 Jun 2026) and decision-aware planning architectures (Cai et al., 24 Jun 2026) use intermediate reasoning or decision traces to boost both model fidelity and agent task performance.

7. Open Problems, Limitations, and Future Directions

Several challenges and frontiers remain for Agent World Models:

Scalability and grounding: Automating large-scale, domain-general environment synthesis, and robustly linking AWMs to real-world sensor modalities or knowledge bases (Li et al., 8 Jun 2026).
Robustness to distribution shift: Ensuring simulator and policy fidelity under non-stationarity, drift during exploration, or co-evolutionary adaptation (Fang et al., 23 Apr 2025, Deihim et al., 23 Jun 2025).
Continual and test-time adaptation: Designing AWMs that retain knowledge across domains and adapt at deployment, especially in dynamic or non-stationary multi-agent scenarios.
Integration of reasoning and dynamics: Joint architectures that optimize both explicit reasoning traces and transition accuracy, aligning world models with policy objectives at all levels of abstraction (Zuo et al., 23 Jun 2026, Cai et al., 24 Jun 2026).
Policy–model coupling: Achieving modularity and parameter efficiency while maintaining verifiability and interpretability in world-model queries and usage (Cao et al., 7 Jan 2026, Li et al., 8 Jun 2026).
Decision-alignment: Further closing the gap between what models predict and what agents need for optimal decision making.

AWMs are converging toward being core infrastructure for general agents, catalyzing advances across RL, planning, dialogue, and embodied interaction. As methodologies mature, focus is shifting to scaling, generalization, and integration of structured reasoning with deep world modeling.