Grounded World Model for Semantically Generalizable Planning

Abstract: In Model Predictive Control (MPC), world models predict the future outcomes of various action proposals, which are then scored to guide the selection of the optimal action. For visuomotor MPC, the score function is a distance metric between a predicted image and a goal image, measured in the latent space of a pretrained vision encoder like DINO and JEPA. However, it is challenging to obtain the goal image in advance of the task execution, particularly in new environments. Additionally, conveying the goal through an image offers limited interactivity compared with natural language. In this work, we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space. As a result, each proposed action is scored based on how close its future outcome is to the task instruction, reflected by the similarity of embeddings. This approach transforms the visuomotor MPC to a VLA that surpasses VLM-based VLAs in semantic generalization. On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set comprising 288 tasks that feature unseen visual signals and referring expressions, yet remain solvable with motions demonstrated during training. In contrast, traditional VLAs achieve an average success rate of 22%, even though they overfit the training set with a 90% success rate.

• The paper presents a novel Grounded World Model that transforms traditional MPC by scoring action proposals via cosine similarity in a unified vision-language latent space.
• It employs rendering-based action tokenization and pretrained multi-modal retrieval to enable robust cross-embodiment and compositional planning.
• Empirical results on the WISER benchmark show an 87% test success rate, highlighting superior semantic generalizability compared to SOTA VLA methods.

Grounded World Model for Semantically Generalizable Planning

Motivation and Problem Formulation

Traditional Model Predictive Control (MPC) in visuomotor planning depends on world models that generate predictions of future outcomes given candidate actions, with optimal action selection based on minimizing distance (typically MSE) in latent space between predicted and goal states. Latent world models like DINO-WM and JEPA-WM leverage pretrained vision encoders for efficiency, but critically depend on goal images, which are hard to obtain, especially in novel environments. Moreover, goal specification via images offers limited interactivity relative to natural language, creating a bottleneck for open-ended, semantically rich planning tasks.

To address these limitations, this work introduces a Grounded World Model (GWM) that operates in a vision-language-aligned latent space using pretrained multi-modal retrieval models (specifically Qwen3-VL-Embedding). This design enables direct scoring of action proposals against language task instructions via cosine similarity in the shared embedding space, transforming visuomotor MPC into a Vision Language Action (VLA) system with superior semantic generalization.

Figure 1: Compared to existing World Models like DINO-WM and JEPA-WM, Grounded World Model enables goal specification via natural language, enabling a new approach to build VLA.

Methodology

World Model Architecture and Action Tokenization

The GWM leverages the latent space of Qwen3-VL-Embedding, which encodes images, text, and videos into a unified embedding space, retaining rich multi-modal world knowledge without finetuning the foundation model weights. Candidate trajectories are proposed via KNN retrieval from the demonstration dataset and subsequently scored via predicted future outcome embeddings. Actions are tokenized by rendering robot states as images using URDFs and existing camera parameters (Rendering-based Action Tokenization, RAT), enabling uniform encoding of observations and actions, and supporting cross-embodiment generalization.

Figure 2: The training and inference workflow of GWM-MPC, with rendered images as action tokens and future outcome embeddings directly scored against the instruction embedding.

Training does not require language supervision; supervision is provided via ground-truth future observations projected into the latent space. This decouples semantic grounding from action generation, preserves multi-modal knowledge, and circumvents knowledge forgetting present in VLM-based VLA finetuning.

Planning and Scoring Procedure

At inference, a batch of $N$ candidate action sequences is proposed, each rendered and encoded as a sequence of images. The GWM predicts future outcome embeddings for each candidate. Task instructions are decomposed (when beneficial) into “pick” and “place” subtasks to generate more informative goal embeddings, enhancing both planning reliability and compositional generalization. Final action selection is performed by maximizing normalized cosine similarity between predicted outcome embeddings and decomposed task embeddings.

Semantic Generalization Evaluation: The WISER Benchmark

The paper introduces the World-knowledge Integrated Semantic Embodied Reasoning (WISER) benchmark, comprising 24 categories with 12 training/test tasks per category (288 tasks per split) designed to stress test semantic generalizability. Test tasks feature unseen visual signals and referring expressions; however, required motions are all contained within the training demonstrations (ensuring evaluation focuses solely on semantic grounding, not novel skill acquisition).

Figure 3: Overview of the WISER Benchmark – 24 categories, each with novel test layouts, cube colors, and referring expressions for robust semantic evaluation.

Empirical Results

On WISER, GWM-MPC achieves an 87% success rate on test tasks (unseen visual signals and instructions), outperforming all SOTA VLA baselines, which average only 22% test success. Notably, baselines typically overfit training data (≥90% training success) but fail to generalize semantically: often picking the wrong cube or misplacing objects in test scenes.

Figure 4: Experimental results on WISER for VLAs; the large gap between training and test success rates highlights poor semantic generalizability of current VLA baselines.

GWM-MPC’s strong results demonstrate semantic grounding beyond memorization, contingent only on the knowledge accuracy of the foundation retrieval model (Qwen3-VL-Embedding). Ablation studies confirm RAT’s superiority over naive action encoding; RAT enables cross-embodiment generalization (e.g., zero-shot transfer to xArm6) and higher robustness to hyperparameters. Reduction in training data yields only minor degradation in generalization. GT-MPC, which directly uses ground-truth future embeddings, serves as the upper bound and reveals that GWM predictions add negligible error, occasionally regularizing action selection.

Figure 5: Ablation results on GT-MPC for planning-related hyperparameter selection.

Inference efficiency is slightly lower for GWM-MPC compared to VLA baselines due to sequential embedding generation for each proposal and non-batched Qwen3 encoding; however, training remains light (20 GPU hours).

Figure 6: Inference efficiency comparison (FPS rollout); GWM-MPC demands more compute versus VLA baselines due to sequential rollout proposal evaluation.

Visualizations and Cross-Embodiment Results

Task and environment visualizations illustrate diversity and robustness of the benchmark, confirming the coverage of semantic knowledge categories and the novel test-to-train splits (Figures 9–16).

Related Work and Contrasts

This work advances latent world model research by enabling goal specification via open-vocabulary language instructions, offering a new paradigm for vision-language planning. The WISER benchmark addresses the critical lack of standard evaluation for semantic generalization in VLAs, highlighting widespread deficiencies in semantic generalizability across most SOTA models. The rendering-based tokenization and foundation model alignment uniquely support cross-embodiment and compositional language understanding.

Theoretical and Practical Implications

The results demonstrate that semantic generalization in robotic planning is not achieved by current VLA paradigms due to knowledge forgetting, visual shortcut exploitation, and poor extrapolation to novel signals. By training world models directly in a vision-language-aligned latent space and decoupling action generation from semantic understanding, GWM offers a practical alternative with stronger OOD generalization, compositional task handling, and cross-embodiment transfer—all bottlenecked only by foundations’ retrieval accuracy.

Practically, this suggests that future robotic planning systems should leverage foundation model latent spaces with world model transition functions versus end-to-end action policies, and further improvements should focus on enhancing the multi-modal retrieval backbone for domain adaptation and task coverage.

Conclusion

The paper rigorously evaluates semantic generalization in planning using the WISER benchmark and demonstrates significant limitations in SOTA VLAs. The proposed GWM-MPC system, trained within a vision-language-aligned latent space, achieves substantially higher semantic generalizability, attributed to preserving foundation model knowledge and employing rendering-based action tokenization. These results have immediate implications for robust, compositional, and cross-embodiment robotic planning. Future research should focus on optimization and adaptation of retrieval model backbone and further expansion of standard semantic generalization benchmarks.

Figure 7: Visualization of the difference between GWM and its action-conditioned variant, highlighting superior semantic grounding via rendering-based tokenization.

Figure 8: Libero-goal environment, illustrating zero-shot semantic generalization in compositional planning tasks.

(Figures 9–16)

Figures 9–16: Visualizations of all WISER benchmark configurations, showcasing test/train split diversity and semantic task challenge.