Retrieval-Augmented World Model

• R-WoM is a set of architectures that ground world models with external memory at inference time to improve long-horizon procedural simulation.
• It retrieves tutorials, compressed histories, or demonstration chunks to dynamically influence state predictions, action ranking, and reward estimation.
• Empirical results indicate that retrieval augmentation boosts task success rates and stabilizes multi-step prediction in complex digital environments.

Searching arXiv for papers on retrieval-augmented world models and closely related methods. Retrieval-augmented World Model (R-WoM) denotes a class of architectures in which a world model is grounded by external memory at inference time rather than relying only on parametric knowledge. In the computer-use formulation introduced under the same name, R-WoM is a training-free inference-time framework that improves LLM-based agents by grounding imagined future trajectories in retrieved tutorials and documentation; in adjacent lines of work, closely related designs retrieve compressed historical video, demonstration action chunks, domain-specific world models, latent transition memories, prior first-person observations, or scene-graph chunks to support prediction, planning, or grounded reasoning (Mei et al., 13 Oct 2025, Peng et al., 22 Jun 2026, Gao et al., 1 Jun 2026, Yoo et al., 4 Sep 2025, Malato et al., 17 Oct 2025, Monaci et al., 4 Apr 2025, Yu et al., 8 Nov 2025). This suggests that retrieval-augmented world modeling is best understood as an architectural pattern—persistent external memory plus query-conditioned reuse—rather than a single memory format or benchmark family.

1. Definition and conceptual scope

In the computer-use setting, a world model is a model that predicts how a digital environment will change after actions and estimates which actions are likely to lead to success. The trajectory is formalized as

$$(g, (o_1, t_1, a_1), (o_2, t_2, a_2), \ldots, (o_n, t_n, a_n)),$$

where $g$ is the task goal, $o_i$ is the observation at step $i$, $t_i$ is internal reasoning, and $a_i$ is the executed action (Mei et al., 13 Oct 2025). Within that formulation, retrieval augmentation is not merely prompt expansion: retrieval produces an evidence set that conditions future state rollout and reward estimation / action ranking, so the external memory enters the world model itself rather than only the policy prompt (Mei et al., 13 Oct 2025).

The same phrase has broader technical scope in neighboring work. CaR treats the world model as a camera-controlled latent video diffusion model and uses compressed latent representation of the full historical context video as memory, with retrieval implemented by a dedicated geometry-aware attention branch (Peng et al., 22 Jun 2026). IMWM attaches retrieval to planning rather than to transition prediction, retrieving demonstration action chunks to improve proposal coverage in latent MPC (Gao et al., 1 Jun 2026). WorMI retrieves from a repository of world models and composes them with an LLM reasoning policy through world-wise compound attention (Yoo et al., 4 Sep 2025). "Zero-shot World Models via Search in Memory" replaces a trained transition model with similarity search over stored latent transitions (Malato et al., 17 Oct 2025). RANa uses prior first-person views from the same environment as a scene-specific external memory for navigation (Monaci et al., 4 Apr 2025). Open-world 3D scene-graph reasoning stores vectorized graph chunks and retrieves them for question answering, grounding, and planning, but explicitly lacks action-conditioned rollout and is therefore only a partial realization of a world model in the predictive sense (Yu et al., 8 Nov 2025).

A recurring distinction in these papers is between retrieval-augmented reasoning, retrieval-augmented planning, and retrieval-augmented world modeling. Full world-model functionality is strongest where retrieval directly supports transition prediction, rollout, or belief over future states; it is weaker where retrieval only biases action search or supplies grounded context to a downstream reasoner (Mei et al., 13 Oct 2025, Gao et al., 1 Jun 2026, Yu et al., 8 Nov 2025).

2. Core architecture of the computer-use R-WoM

The canonical R-WoM pipeline for computer-use agents begins with a policy model that proposes thought-action pairs,

$$(t_i, a_i) \sim \pi_p\!\left(\cdot \mid g, o_i, \{(o_j, t_j, a_j)\}_{j=v}^{i-1}\right),$$

and a tutorial retrieval stage that constructs an evidence set $\mathcal{E}$ from external documentation (Mei et al., 13 Oct 2025). The knowledge base contains 30k+ chunked tutorial documents drawn from WikiHow, Google Chrome Help, GIMP 3.0 User Manual, Visual Studio Code docs, Ubuntu Help, Thunderbird Support, VLC User Guide, LibreOffice Help, GitLab docs, and Adobe Commerce Admin docs (Mei et al., 13 Oct 2025).

Retrieval is organized as a reasoning-based RAG pipeline. Starting from task goal $g$, the system forms a retrieval query $q$, applies query rewriting to generalize private or overly specific details, performs vector retrieval over tutorial chunks, and then applies LLM reranking to obtain the final evidence set $g$0; the default implementation uses top-5 retrieved chunks (Mei et al., 13 Oct 2025). The world model then performs a LongCoT rollout for each candidate action,

$g$1

producing an imagined trajectory conditioned on the current state, the candidate action, and the retrieved tutorials (Mei et al., 13 Oct 2025). The rollout prompt is instructed to use tutorials when the situation matches standard operations, otherwise fall back to internal world knowledge, while tracking UI details such as visible elements, hidden affordances, cursor or caret position, selection highlights, and global context (Mei et al., 13 Oct 2025).

Action selection uses relative reward estimation rather than absolute scalar scoring. The paper formulates listwise selection as

$g$2

The reward-estimation prompt ranks candidate rollouts comparatively, and the selected action is executed before the loop repeats on the next observation (Mei et al., 13 Oct 2025). The framework is training-free and prompt-based at inference time, and was instantiated with Qwen-2.5-VL-72B, Claude-3.5-Sonnet, and Claude-3.7-Sonnet as both policy and world-model backbones; for retrieval it uses LangChain, a FAISS vector store, and Qwen-3-Embedding-8B, with GTA-1-7B as an auxiliary action-grounding model for OSWorld (Mei et al., 13 Oct 2025).

The architectural point is not simply that retrieval provides more context. The paper’s explicit claim is that retrieval augments the world model rollout itself, grounding future-state simulation and reward estimation with environment-specific procedural evidence. This is the main distinction from a retrieval-augmented policy prompt or a one-shot plan generator (Mei et al., 13 Oct 2025).

3. Diagnostic motivation: why retrieval is introduced into world models

A notable feature of the computer-use R-WoM paper is that it first probes whether LLMs are suitable world models before proposing the remedy. The analysis isolates two core capabilities—future state prediction and reward estimation—through three probing tasks: next-state identification, full-procedure planning alignment, and milestone transition recognition (Mei et al., 13 Oct 2025).

The results are asymmetrical. On next-state identification, which measures short-horizon future-state prediction, performance is relatively strong: Qwen-2.5-VL-72B reaches 77.0% overall, Claude-3.5-Sonnet 81.0%, and Claude-3.7-Sonnet 86.0% (Mei et al., 13 Oct 2025). On milestone transition recognition, which probes local progress estimation, the same models reach 83.7%, 85.7%, and 86.7%, respectively (Mei et al., 13 Oct 2025). By contrast, on full-procedure planning alignment—the long-horizon procedural task—performance drops to 50.0% for Qwen-2.5-VL-72B, 55.0% for Claude-3.5-Sonnet, and 65.0% for Claude-3.7-Sonnet (Mei et al., 13 Oct 2025).

The paper interprets this as evidence that LLM world models are good enough for immediate local transitions and local progress recognition, but degrade sharply on long-horizon procedural simulation because of hallucination, stale training knowledge, and compounding rollout errors (Mei et al., 13 Oct 2025). That diagnosis is consistent with related work in video world models: CaR argues that long-horizon consistency requires both persistent storage of the full history and flexible, query-dependent selection of historical content, because short recent windows and heuristic frame selection both fail under revisitation, hard camera cuts, and large viewpoint changes (Peng et al., 22 Jun 2026).

This suggests that retrieval enters world modeling for a specific reason: not to replace learned priors wholesale, but to anchor long-horizon simulation to external evidence when parametric rollouts become brittle. In the computer-use case, the anchoring evidence is procedural documentation; in video generation, it is compressed historical latent video; in latent planning, it is demonstration action chunks; in nonparametric dynamics, it is stored latent transitions (Mei et al., 13 Oct 2025, Peng et al., 22 Jun 2026, Gao et al., 1 Jun 2026, Malato et al., 17 Oct 2025).

4. Retrieval substrates and memory representations

Across the literature, the retrieved object varies substantially, and that variation largely determines what kind of “world model” is obtained.

System	Retrieved item	Function
R-WoM	tutorial/documentation chunks	grounds rollout and reward estimation
CaR	compressed historical latent video	geometry-aware memory access during generation
IMWM	demonstration action chunk	proposal initialization and hybrid planning cost
WorMI	domain-specific world models	test-time composition with LLM reasoning
Search in Memory	latent transitions	nonparametric next-state prediction
RANa	prior first-person RGB observations	recurrent navigation context
Open-World 3D Scene Graph	scene-graph chunks	grounded QA, grounding, retrieval, planning

The memory schema in "Zero-shot World Models via Search in Memory" is the most direct latent-transition formulation. It stores tuples of the form

$g$3

and predicts the future by retrieving similar latent transitions from memory, taking their successors, and recursively rolling out in latent space (Malato et al., 17 Oct 2025). CaR instead stores no symbolic map or explicit 3D structure; its memory is compressed context tokens derived from historical video, with approximately 97\% token reduction relative to the original latent sequence, and retrieval is implemented as dense differentiable attention over the compressed memory (Peng et al., 22 Jun 2026). WorMI moves the retrieval target up one level of abstraction: the repository contains domain-specific world models $g$4, retrieval uses prototype-based abstract representation matching with Wasserstein distance, and the retrieved models are fused through world-wise compound attention (Yoo et al., 4 Sep 2025).

Other systems are weaker as world models but important as memory architectures. RANa stores raw first-person RGB images $g$5 from prior episodes in the same scene and retrieves them with DINOv2 or OpenCLIP before integrating them into a recurrent policy (Monaci et al., 4 Apr 2025). Open-world 3D scene-graph reasoning stores label-centered chunks derived from an explicit 3D scene graph $g$6, indexed in a vector database for text- or image-conditioned retrieval; this yields a persistent world memory for grounded reasoning, but not an action-conditioned predictive model (Yu et al., 8 Nov 2025). IMWM is even more specific: its bank

$g$7

stores successful demonstration windows and retrieves a top-1 action chunk to initialize CEM, which makes it a retrieval-augmented planner layered on top of a frozen latent world model rather than a retrieval-augmented world model in the narrow architectural sense (Gao et al., 1 Jun 2026).

This variety suggests that “R-WoM” is not defined by a single database design. What unifies the category is the insertion of a persistent external memory into the world-model loop, with retrieval conditioned on the current state, goal, or query.

5. Empirical evidence across domains

On the two computer-use benchmarks used for the named R-WoM framework, the method consistently improves end-to-end task success over Vanilla, RAG, and WoM baselines. For Qwen-2.5-VL-72B, OSWorld rises from 30.84 with RAG and 28.37 with WoM to 38.05 (+23.4%), while WebArena rises from 24.50 with WoM to 28.92 (+18.1%). For Claude-3.7-Sonnet, OSWorld rises from 31.24 with WoM to 39.13 (+25.3%), and WebArena rises from 32.75 with RAG to 35.11 (+7.2%) (Mei et al., 13 Oct 2025). Horizon ablations further show that WoM improves slightly at short horizon but plateaus or degrades after 2 steps, whereas R-WoM remains stronger up to around horizon 3 before tapering, consistent with the claim that grounding helps more as imagination depth increases (Mei et al., 13 Oct 2025).

Related systems report analogous gains in domains where long-horizon consistency or prior experience matters. CaR achieves state-of-the-art results on established benchmarks for long-horizon video world models and is best overall on both SceneFly and SpatialVid scene revisiting; on SceneFly scene revisiting it reports PSNR 21.23, SSIM 0.672, LPIPS 0.209, and FVD 42.1, and on SpatialVid scene revisiting PSNR 20.77, SSIM 0.699, LPIPS 0.231, and FVD 41.2 (Peng et al., 22 Jun 2026). WorMI, evaluated on VirtualHome and ALFWorld, reaches SR 66.12, PS 15.17 on VirtualHome zero-shot unseen tasks plus unseen scenes, compared with SR 45.71, PS 19.04 for SayCanPay, and SR 51.67, PS 20.18 on ALFWorld compared with SR 39.66, PS 23.86 for the same baseline (Yoo et al., 4 Sep 2025). RANa improves navigation when scene memory is available: on ImageNav, RANa-b DINOv2-Graph + Attention achieves SR 90.7, SPL 71.8 versus DEBiT-b SR 83.4, SPL 56.8; on ObjectNav, RANa-t OpenCLIP + Gumbel achieves SR 52.1, SPL 26.9 versus hwRN18+CLIP SR 42.9, SPL 23.0 (Monaci et al., 4 Apr 2025).

The nonparametric dynamics paper provides a complementary kind of evidence. It reports that a search-based world model is comparable to a training-based one on latent reconstruction and perceived similarity, and emphasizes that the best baseline model (30 trajectories) still reports higher average KL than Rollout and Replay-KL with only five encoded trajectories on the reported memory-size ablation (Malato et al., 17 Oct 2025). IMWM shows that retrieval can be critical even when the world model is not the bottleneck: across four pixel-based goal-reaching tasks, IMWM improves mean success on all four, with the largest gains on Two-Room (99.2%, +11.5 percentage points) and OGBench-Cube (94.7%, +28.5 percentage points) (Gao et al., 1 Jun 2026).

Taken together, these results show a repeated empirical pattern: retrieval helps most where the task is long-horizon, revisitation-heavy, search-limited, or procedurally specific, and where a memory item can provide information not available in a short recent context window.

6. Misconceptions, limitations, and open technical directions

A common misconception is that any retrieval-augmented agent automatically constitutes a world model. The literature is more precise. IMWM is best understood as a retrieval-augmented planner built on top of a latent world model, because retrieval does not alter the learned transition model $g$8 (Gao et al., 1 Jun 2026). Open-world 3D scene-graph generation is best described as a retrieval-augmented scene/world memory for grounded reasoning, because it supports explicit current-world representation, semantic retrieval, and grounded planning, but does not support action-conditioned state transition prediction, future rollout or simulation, uncertainty propagation in planning, or temporal relational evolution (Yu et al., 8 Nov 2025). RANa is a retrieval-augmented control architecture with world-model-like external memory, but no explicit learned dynamics, no latent future prediction, and no explicit planning over a learned environment state (Monaci et al., 4 Apr 2025).

The limitations are equally systematic. The computer-use R-WoM depends on tutorial availability, retrieval quality, and nontrivial inference cost: on OSWorld with Claude-3.7-Sonnet, Greedy uses 1,889 calls, ~0.8h, whereas R-WoM uses 9,460 calls, ~3.8h, although it remains much cheaper than WebDreamer: 38,162 calls, ~15.3h (Mei et al., 13 Oct 2025). CaR is strong for static or quasi-static spatial memory but explicitly weak on dynamic objects because retrieved historical observations may be outdated and there is no explicit motion-aware state update mechanism (Peng et al., 22 Jun 2026). WorMI acknowledges computational overhead and dependence on the strengths and weaknesses of the underlying LLM reasoning model, and its scaling with the number of implanted world models is non-monotonic (Yoo et al., 4 Sep 2025). Search in Memory depends critically on latent coverage and struggles in open-ended environments; action-conditioned retrieval can degrade retrieval quality, especially for Replay-L2 and long-horizon Replay-KL (Malato et al., 17 Oct 2025). RANa still lacks online memory updating, uncertainty-aware retrieval, temporal metadata, and explicit planning over the retrieved graph (Monaci et al., 4 Apr 2025).

A broader misconception is that retrieval merely adds facts. In the strongest instances, retrieval changes the computational role of memory inside the model. In R-WoM for computer use, tutorials stabilize multi-step simulation and reward estimation (Mei et al., 13 Oct 2025). In CaR, retrieval is a parallel Retrieval Attention branch that preserves the pretrained self-attention prior while adding geometry-aware memory access (Peng et al., 22 Jun 2026). In Search in Memory, retrieval is the transition mechanism itself (Malato et al., 17 Oct 2025). In WorMI, retrieval selects which pretrained world models will be implanted into the reasoning policy at test time (Yoo et al., 4 Sep 2025).

A plausible implication is that future R-WoM systems will combine several ingredients that appear separately in the current literature: persistent external memory, query-conditioned retrieval, action-conditioned prediction, uncertainty-aware indexing, online memory maintenance, and explicit mechanisms for stale or conflicting evidence. The current body of work already shows that retrieval can augment world models through tutorials, compressed history, demonstrations, latent transitions, model repositories, scene memories, and graph chunks; the remaining challenge is to unify these memory substrates with stronger predictive dynamics and more robust long-horizon control.