Embodied World Knowledge (EWK) Overview

• Embodied World Knowledge (EWK) is a set of structured, sensorimotor-based representations that integrate causal reasoning, spatial perception, and action-conditioned prediction.
• Recent research employs causal discovery, Bayesian inference, and 3D spatial mapping to continually refine world models for robust decision-making.
• Practical deployments demonstrate that EWK enhances action planning, control, and continual adaptation in complex, real-world 3D environments.

Embodied World Knowledge (EWK) denotes the knowledge by which an embodied agent grounds perception, prediction, reasoning, and action in the physical world. In recent work, the term has been used in several closely related senses: as a continually refined causal world model, as the perceptual, spatial, and object-centric faculties required for purposeful behavior in three-dimensional environments, as the ability of an embodied world model to support both perceptual fidelity and downstream decision-making, and as a curated video-text corpus for language-conditioned world modeling (Yu et al., 21 Jun 2026, Dang et al., 19 Aug 2025, Shang et al., 9 Feb 2026, Zhang et al., 15 Jun 2026). Taken together, these formulations suggest that EWK is best understood as a family of embodied knowledge representations linking sensorimotor interaction, internal state construction, intervention, and control.

1. Conceptual scope

Across the literature, EWK is not restricted to a single representational substrate. Some works treat it as an internal model of causal structure; others operationalize it as region-grounded perception, persistent 3D memory, procedural symbolic knowledge, or benchmarkable embodied competence. The common denominator is that knowledge is not merely linguistic or descriptive: it must support action-conditioned prediction, physically grounded reasoning, and adaptation under interaction.

Lens	Representative formulation	Source
Causal-epistemic	Internal causal world model revised through causal discovery, intervention-driven feedback, and counterfactual reasoning	(Yu et al., 21 Jun 2026)
Perceptual-spatial	“the set of perceptual, spatial and object‐centric faculties” needed to act purposefully in a three-dimensional environment	(Dang et al., 19 Aug 2025)
Functional world modeling	Ability to generate perceptually faithful futures and support policy learning, policy evaluation, and action planning	(Shang et al., 9 Feb 2026)
Bodily-temporal-social grounding	Knowledge grounded in an “active bodily system,” “temporally structured experience,” and “social grounding”	(Incao et al., 2024)
Data-centric foundation	An 8.6M video-text corpus with action-language mapping over 20+ embodiments and 500+ action categories	(Zhang et al., 15 Jun 2026)

This breadth has an important methodological consequence. EWK may reside in weights, external memory, symbolic repositories, voxel maps, scene graphs, or multimodal datasets; what matters is whether those structures encode physically meaningful regularities that can be queried, updated, and exploited during embodied behavior. A plausible implication is that the term names a capability class rather than a single architecture family.

2. Formal representations

A canonical formalization models the embodied environment as a Structural Causal Model,

$$\mathcal{M}=(\mathcal U,\mathcal V,\mathcal F_\theta,P_\theta(\mathcal U)),$$

with endogenous variables $\mathcal V=\{X_i\}_{i=1}^n$, exogenous variables $\mathcal U=\{U_i\}_{i=1}^n$, and structural equations

$X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$

In the embodied-agent setting, one typically writes

$$s_{t+1}=f_\theta(s_t,a_t,u_t),\qquad o_t=g_\theta(s_t,v_t),$$

and the agent maintains a time-indexed internal approximation

$$\mathcal M_t=(\mathcal U,\mathcal V,\mathcal F_{\theta,t},P_{\theta,t}(\mathcal U)).$$

Interventions are expressed with Pearl’s do-operator, forward-causal queries sample from $P_{\theta'}(s_{t+1}\mid s_t,do(a_t))$, and counterfactual inference follows abduction–action–prediction (Yu et al., 21 Jun 2026).

Other frameworks generalize this predictive core by explicitly attaching action utility. AUKAI defines

$$\mathrm{EWK}(\Theta)\equiv\{P_\Theta(s_{t+1}\mid s_t,a_t),\;U_\Theta(s_t,a_t)\},$$

so that embodied knowledge jointly represents next-state distributions and the desirability of interventions. Its multi-scale construction separates micro, meso, and macro state, combines their losses with weights $w_l$, and updates parameters by gradient descent on prediction losses minus utility (Wang, 2 Mar 2025).

A Bayesian interpretation places EWK directly in posterior space. In that view, the agent’s embodied knowledge is its current probabilistic representation of world state or parameters, updated sequentially from sensorimotor data. The resulting Bayes-filter recursion,

$$p(x_t\mid y_{1:t},a_{1:t-1}) = \alpha\,p(y_t\mid x_t)\int p(x_t\mid x_{t-1},a_{t-1})\,p(x_{t-1}\mid y_{1:t-1},a_{1:t-2})\,dx_{t-1},$$

treats perception, action selection, and learning as inference under uncertainty (Liu, 29 Jul 2025).

These formalisms differ in ontology—causal mechanisms, predictive-utility tuples, or posterior distributions—but all move beyond static semantic associations. They construe EWK as structured state about environmental dynamics, action effects, and uncertainty.

3. Perceptual, geometric, and spatio-temporal substrates

One major research line realizes EWK through fine-grained perception. RynnEC defines EWK as perceptual, spatial, and object-centric competence and implements it with a video multimodal LLM built from a vision-language foundation model, a region encoder, and a mask decoder. Its egocentric video based pipeline for generating embodied cognition data yields 1.14 M instance masks, and RynnEC-Bench comprises 22 fine-grained tasks across object cognition and spatial cognition. Quantitatively, RynnEC-7B reaches 56.2 overall and 61.4 on object cognition, and it achieves a 54.5 overall spatial score, a 44.2% relative improvement over Gemini-2.5P (Dang et al., 19 Aug 2025).

A second line emphasizes persistent geometry. “Learning 3D Persistent Embodied World Models” couples a stochastic video-diffusion backbone with an explicit volumetric memory, the “DINO-Map,” and conditions prediction on a 3D spatial map, relative camera pose via Plücker embedding, and the last observed RGB-D frame. The world is discretized into a voxel grid, updated by lifting RGB-D features into 3D and aggregating them voxel-wise. On Habitat–HM3D trajectories, the full model achieves FVD≈92 versus 194 for NWM, PSNR≈22.5 dB with a +5 dB gain, SSIM=0.76, LPIPS=0.157, and SRC=0.817, while long-horizon rollouts remain coherent over 112 frames (Zhou et al., 5 May 2025).

A third line injects physical structure directly into the generative objective. RoboScape jointly learns RGB video generation, temporal depth prediction, and keypoint dynamics. Its dual-branch co-autoregressive Transformer fuses RGB and depth branches and uses SpatialTracker-derived keypoints to impose a self-supervised dynamics loss. The overall objective

$\mathcal V=\{X_i\}_{i=1}^n$0

anchors EWK in 3D geometry and material-sensitive motion. RoboScape achieves best LPIPS, PSNR, AbsRel, $\mathcal V=\{X_i\}_{i=1}^n$1, $\mathcal V=\{X_i\}_{i=1}^n$2, and action-control $\mathcal V=\{X_i\}_{i=1}^n$3PSNR compared to IRASim, iVideoGPT, Genie, and CogVideoX (Shang et al., 29 Jun 2025).

A fourth line encodes EWK as a queryable scene structure rather than a latent video prior. SNOW integrates VLM-derived semantics with point cloud geometry and temporal consistency through HDBSCAN clustering, SAM2-based segmentation, Spatio-Temporal Tokenized Patch Encoding (STEP), SLAM alignment, and a 4D Scene Graph (4DSG). In this formulation, EWK is exactly the contents of the 4DSG plus SLAM-aligned geometry and temporal links. Reported results include 60.1% overall accuracy on NuScenes-QA, 72.3% zero-shot average on RoboSpatial-Home, 73.8% overall on VLM4D, and 38.1 mIoU zero-shot on open-vocabulary LiDAR segmentation (Sohn et al., 18 Dec 2025).

These systems instantiate different answers to the same question: whether embodied knowledge should be stored as region tokens, voxel memory, physics-informed latent dynamics, or graph-structured object history. The literature indicates that all four are viable, and that EWK increasingly depends on explicit spatial persistence rather than frame-local prediction alone.

4. Acquisition, externalization, and continual refinement

The strongest causal account of EWK treats knowledge as self-evolving. In the self-evolving cognitive framework, causal discovery operates over interaction history $\mathcal V=\{X_i\}_{i=1}^n$4 by fitting graph structure $\mathcal V=\{X_i\}_{i=1}^n$5 and mechanism parameters $\mathcal V=\{X_i\}_{i=1}^n$6, while continual refinement is driven by a structural evolution operator

$\mathcal V=\{X_i\}_{i=1}^n$7

The update loop explicitly includes observation, abduction, counterfactual prediction, discrepancy computation with counterfactual invariance, and parameter updates to $\mathcal V=\{X_i\}_{i=1}^n$8, graph $\mathcal V=\{X_i\}_{i=1}^n$9, $\mathcal U=\{U_i\}_{i=1}^n$0, and policy $\mathcal U=\{U_i\}_{i=1}^n$1. In this view, EWK self-evolves through failure signals and counterfactual checks, not just predictive errors (Yu et al., 21 Jun 2026).

Large-scale data curation provides another route. Qwen-RobotWorld uses “Embodied World Knowledge (EWK)” to name the curated corpus that powers language-conditioned video world modeling: 8.6 million video-text pairs, more than 200 million frames, a 70% embodied and 30% general split, over 20 distinct morphologies, and over 500 unique primitives. Its action-language mapping is built with a five-layer hierarchical annotation prompt covering Task Goal, Action Detail, Physical Feedback, Comprehensive Caption, and Concise Caption, followed by automated LLM-based judging and human spot-checks (Zhang et al., 15 Jun 2026).

Several frameworks externalize EWK into structured memory rather than compressing it into a monolithic backbone. Steve-Evolving anchors each subgoal attempt as an experience tuple

$\mathcal U=\{U_i\}_{i=1}^n$2

indexes it in a three-tier experience space, and distills success into reusable skills and failure into executable guardrails. The diagnosis layer exposes state-difference summaries, 11 enumerated failure causes, continuous indicators, and loop detection. On the MCU long-horizon suite, overall success rate rises to 52.5% for Steve-Evolving versus 47.4% for Jarvis-1 + LLM-planner and 45.8% for Optimus-1 + LLM-planner, with especially large gains on Iron, Diamond, and Armor tasks (Xie et al., 13 Mar 2026).

WorldMind likewise externalizes knowledge, but as a symbolic World Knowledge Repository $\mathcal U=\{U_i\}_{i=1}^n$3 composed of Process Experience and Goal Experience. Process Experience verbalizes physical feasibility constraints from prediction errors; Goal Experience distills successful trajectories into meta-instructions. The repository is updated inference-time, without gradient-based retraining. On EB-ALFRED and EB-Habitat, WorldMind improves both Success Rate and Goal-Conditioned Success for GPT-3.5-turbo and GPT-4.1-mini, and the learned repository transfers across models and environments (Ren et al., 19 Jan 2026).

Key-Gram externalizes linguistic priors through deterministic hashed lookup. Instructions are decomposed into key-grams, each maps to layer-wise embeddings via $\mathcal U=\{U_i\}_{i=1}^n$4 table lookup, and the retrieved memory is injected into selected hidden layers through context-adaptive gating and lightweight convolutional fusion. Reported gains include average relative improvements of 29.5% and 9.9% on RoboTwin2.0 for $\mathcal U=\{U_i\}_{i=1}^n$5 and $\mathcal U=\{U_i\}_{i=1}^n$6, 35.8% and 4.5% on LIBERO-Plus zero-shot transfer, and 15.4% and 8.1% on real-world long-horizon tasks (Fan et al., 18 May 2026).

An earlier route teaches EWK to LLMs through simulator experience. “LLMs Meet World Models” uses VirtualHome trajectories for plan generation, activity recognition, counting QA, and object path tracking, then fine-tunes LLMs with EWC regularization and LoRA. Across 18 downstream tasks, the E2WM models achieve an average relative improvement of 64.28%, while held-out Pile perplexity remains nearly unchanged (Xiang et al., 2023).

Collectively, these approaches show that EWK can be acquired from intervention, curated demonstrations, symbolic failure analysis, or simulator-generated experience. They also show that externalization—skills, guardrails, repositories, or memory tables—has become a recurrent alternative to pure parametric storage.

5. Deployment in planning and control

EWK is valuable insofar as it improves downstream action. Primitive Embodied World Models (PEWM) make this explicit by restricting world modeling to short horizons and composing primitive-level predictions. The framework combines a modular Vision-LLM planner, two LoRA modules for planning and grounding, and Start-Goal heatmap Guidance (SGG), where $\mathcal U=\{U_i\}_{i=1}^n$7 conditions diffusion rollouts toward a predicted start–goal motion. The system uses approximately 11 K real teleoperation episodes, approximately 67 K primitives, and causal distillation reduces rollout to approximately 12 FPS on A100 with approximately 11 GB VRAM. The paper reports approximately 29× faster annotation than long-horizon episodes and strong results on RLBench and zero-shot robot evaluations (Sun et al., 28 Aug 2025).

Long-horizon hybrid tasks motivate a different deployment strategy: disentangling persistent scene evolution from robot-centric behavior. World-Ego Modeling factorizes future state into world and ego components,

$\mathcal U=\{U_i\}_{i=1}^n$8

and instantiates this with an implicit separate world-ego planner and a cascade-parallel mixture-of-experts diffusion generator. HTEWorld provides 125K video clips over 4.5M frames and 300 multi-turn evaluation trajectories over 2K instructions. Under identical fine-tuning, WEM reaches EWMScore 61.48 versus 58.40 for the best single-stream baseline, while remaining competitive on standard WorldArena manipulation tasks (Lin et al., 19 May 2026).

Persistent 3D memory also improves planning directly. In “Learning 3D Persistent Embodied World Models,” candidate action chunks are rolled out through the model and ranked by task cost, and MPC searches for

$\mathcal U=\{U_i\}_{i=1}^n$9

This improves final-pose Similarity by approximately 3% over NWM and by approximately 5% over NoMaD alone, while policy learning in unseen scenes becomes approximately 2× faster in SIM after 3 k steps (Zhou et al., 5 May 2025).

Qwen-RobotWorld presents a unified deployment thesis: once language becomes the action interface, the same world model can support synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. With its Double-Stream MMDiT, frozen Qwen2.5-VL semantics, and progressive curriculum, the system ranks 1st overall on EWMBench and DreamGen Bench and outperforms all open-source models on WorldModelBench and PBench (Zhang et al., 15 Jun 2026).

These deployments show a shift in emphasis. EWK is no longer only the substrate of open-loop prediction; it is increasingly engineered to serve as an action prior, a planner, a simulator proxy, or a control-time constraint system.

6. Evaluation, misconceptions, and open problems

A central misconception is that visually realistic prediction implies strong embodied competence. WorldArena directly disputes this. It evaluates embodied world models on 50 diverse tabletop manipulation tasks using 2,500 real videos and 16 metrics spanning visual quality, motion quality, content consistency, physics adherence, 3D accuracy, and controllability, then adds three functional tests: world model as data engine, policy evaluator, and action planner. Its holistic index,

$X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$0

uses equal weights $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$1. The benchmark reports a significant perception-functionality gap: EWMScore versus human evaluation has $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$2, versus data engine $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$3, and versus action planner only $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$4; high perceptual fidelity does not imply high functional utility (Shang et al., 9 Feb 2026).

A second misconception is that EWK can be evaluated only through video generation quality. ENACT instead casts embodied cognition as world modeling from egocentric interaction inside a POMDP and evaluates two sequence-reordering tasks: forward world modeling and inverse world modeling. The benchmark contains 8,972 QA pairs across horizons $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$5 to $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$6. Reported results show a large human–model gap, monotonic degradation with horizon, better inverse than forward performance, a preference for right-handed actions, and sensitivity to non-human-eye-like camera intrinsics and heights (Wang et al., 26 Nov 2025).

The causal-epistemic perspective proposes still another evaluation regime. For self-evolving embodied scientific intelligence, the suggested metrics include Causal Intervention Robustness $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$7, Counterfactual Accuracy $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$8, Failure Utilization $X_i=f_i(\mathrm{Pa}(X_i),U_i;\theta_i).$9, Long-Horizon Causal Consistency $$s_{t+1}=f_\theta(s_t,a_t,u_t),\qquad o_t=g_\theta(s_t,v_t),$$0, Self-Modification Stability, and Cross-Platform Transfer Robustness. These metrics explicitly score whether an agent can discover, refine, and generalize causal world knowledge through embodied experiments, not merely complete tasks (Yu et al., 21 Jun 2026).

Open problems remain broad. The roadmap for embodied and social grounding argues that an agent needs not only an active bodily system and temporally structured experience but also social grounding, including joint attention, perspective-taking, and common ground, while standardized evaluation protocols for these capacities remain largely open (Incao et al., 2024). The Bayesian perspective similarly argues that current systems remain largely confined to closed-physical-world environments and that hierarchical Bayesian architectures, ensembles, multi-fidelity data fusion, and continual online updates may be necessary for truly open physical-world embodied intelligence (Liu, 29 Jul 2025).

The cumulative picture is technically coherent even where the definitions differ. EWK names the representational and procedural machinery through which embodied agents acquire persistent world state, infer causal structure, evaluate interventions, and convert experience into reusable competence. The field’s present trajectory points away from purely predictive, single-stream, frame-local modeling and toward architectures that are causal, persistent, region- or object-centric, externally extensible, benchmarked for functional utility, and explicitly designed to evolve under interaction.