World Engine Frameworks

• World Engine is a framework that constructs and maintains persistent digital world representations using multimodal inputs like text, images, and logs.
• It combines explicit world state management with generative modeling and closed-loop simulation to enable applications in autonomous driving and interactive 3D generation.
• Innovations include dynamic memory management, physics-aware rollout, and high-fidelity rendering techniques that ensure controllable, long-horizon consistency.

Searching arXiv for papers on "World Engine" and closely related world-generation systems. In contemporary research, a world engine denotes a systems-level framework that constructs, maintains, and updates a persistent world representation under conditioning signals such as text, images, videos, geospatial observations, or real-world logs, then renders or simulates observations for downstream interaction, control, or analysis. The term spans several adjacent but distinct lines of work: high-fidelity digital twins for autonomous driving, pose-conditioned generators for controllable video, LLM-orchestrated 3D scene synthesis, editable multiplayer diffusion systems, large-scale terrain and Earth generation, embodied-simulation asset pipelines, and social-simulation world models (Li et al., 18 Jun 2026, Chou et al., 28 Nov 2025, Duan et al., 5 Sep 2025, Po et al., 3 Mar 2026, Sharma et al., 2024, Zhang et al., 14 Apr 2025). This suggests that world engine functions less as a single model class than as a unifying label for architectures that combine explicit world state, generative modeling, and closed-loop interaction.

1. Scope and conceptual boundaries

A recurrent property across the literature is that a world engine is not merely a renderer or a next-frame predictor. It is instead organized around a persistent state that can be queried, edited, extrapolated, or optimized. In autonomous driving, "World Engine: Towards the Era of Post-Training for Autonomous Driving" reconstructs high-fidelity interactive environments from real-world logs and systematically extrapolates them into realistic safety-critical variations for reinforcement-based post-training (Li et al., 18 Jun 2026). In controllable video generation, "Captain Safari: A World Engine" defines a world engine as a generative video model that maintains an explicit, persistent representation of the underlying 3D scene and uses it to synthesize arbitrarily long, 3D-consistent first-person videos under user-specified 6-DoF camera trajectories (Chou et al., 28 Nov 2025). In interactive 3D generation, LatticeWorld accepts textual descriptions and visual instructions as multimodal inputs and creates large-scale 3D interactive worlds with dynamic agents, high-fidelity physics simulation, and real-time rendering (Duan et al., 5 Sep 2025).

System	Core mechanism	Stated domain
World Engine	3D Gaussian Splatting, behavior world model, RL post-training	Autonomous driving
Captain Safari	Persistent world memory, pose-conditioned retriever, DiT	Camera-controlled video generation
LatticeWorld	LLaMA-2-7B, CLIP-ViT, Unreal Engine 5	Interactive 3D world generation
MultiGen	External memory with Memory, Observation, Dynamics modules	Editable multiplayer worlds
EarthGen / ABot-Earth 0.5	Cascaded diffusion or native 3DGS with tiled/LOD deployment	Terrain and geospatial world generation
SocioVerse	Social Environment, User, Scenario, and Behavior engines	Social simulation

The breadth of this usage is further visible in systems that foreground editability, embodiment, or code synthesis. MultiGen introduces an explicit external memory that is continually updated by user actions and queried throughout the generation roll-out, enabling editable experiences and shared multiplayer inference (Po et al., 3 Mar 2026). EmbodiedGen defines a foundational platform for interactive 3D world generation composed of six modules—Image-to-3D, Text-to-3D, Texture Generation, Articulated Object Generation, Scene Generation, and Layout Generation—with simulator-ready URDF outputs (Wang et al., 12 Jun 2025). Code2Worlds formulates 4D generation as language-to-simulation code generation and emphasizes a physics-aware closed loop rather than direct pixel synthesis alone (Zhang et al., 12 Feb 2026).

2. World representations and architectural substrates

World engines differ most sharply in how they represent the underlying world. LatticeWorld uses a symbolic intermediate representation in which each terrain type is mapped to a unique letter and a $32\times 32$ matrix is serialized as a grid string; its generation pipeline is summarized by $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$, $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$, and $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$, with two fine-tuned LLaMA-2-7B models, a CLIP-ViT-based visual encoder, and UE5 as renderer and physics simulator (Duan et al., 5 Sep 2025). WorldGen, by contrast, is organized as a modular pipeline on Blender and Bullet and makes its latent factors explicit through $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$, where $x$ denotes object positions, scales, and classes, $t$ textures and materials, and $m$ motion (Singh et al., 2022).

Another family of systems adopts persistent memory as the primary substrate. Captain Safari stores a memory bank $M=\{m_t\}_{t=0}^T$ of 3D-aware features, samples a dynamic local memory window, and performs pose-conditioned cross-attention retrieval to obtain pose-aligned world tokens for a diffusion transformer (Chou et al., 28 Nov 2025). MultiGen makes memory even more explicit: the world state at time $t$ is $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$0, where $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$1 is a static top-down level map and $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$2 is the set of active player poses; the observation module reads geometry from this memory at every step, while the dynamics module updates poses (Po et al., 3 Mar 2026).

A third substrate is geometric radiance or splat-based representation. In the autonomous-driving World Engine, each Gaussian primitive is parameterized by center, quaternion orientation, anisotropic scale, opacity, and spherical-harmonic color coefficients, enabling free-viewpoint rendering and mutable digital twins reconstructed from logs (Li et al., 18 Jun 2026). ABot-Earth 0.5 also formulates generation directly in the 3D Gaussian Splatting representation; each tile contains $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$3–$\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$4 Gaussians and is deployed through hierarchical LOD structures for real-time visualization on web-based map engines (Qian et al., 8 Jun 2026).

Further variants expose other representational commitments. EarthGen models a continuous RGB function over geo-regions and scales with a cascade of latent diffusion models plus tiled composition, yielding consistent generation across multiple resolutions and up to $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$5 zoom (Sharma et al., 2024). EmbodiedGen centers its outputs on watertight meshes, 3DGS representations, and simulator-ready URDF assets with inertial parameters, collision geometry, friction, mass, semantic tags, and textual descriptions (Wang et al., 12 Jun 2025). Code2Worlds represents the world as executable Blender/Infinigen code, schema documents, and physics-parameter scripts, thereby shifting the semantic-physical interface from direct generation to program synthesis and iterative execution (Zhang et al., 12 Feb 2026).

3. Conditioning, memory, and control

A defining property of world engines is their treatment of conditioning as a first-class systems problem rather than a single prompt-input interface. LatticeWorld accepts textual layout descriptions, optional visual layout instructions such as height maps or sketches, and textual environmental descriptions; CLIP ViT-B/32 features are projected into language-embedding tokens compatible with LLaMA-2, and the resulting multimodal prompt controls both scene layout and environmental configuration (Duan et al., 5 Sep 2025). Text-to-world systems of a more procedural kind, such as the later WorldGen for traversable 3D worlds, use an LLM to map a natural-language prompt into scene parameters, then pass those parameters through procedural blockout, navmesh extraction, diffusion-based reconstruction, object decomposition, and per-object enhancement before export to Unity or Unreal (Wang et al., 20 Nov 2025).

Persistent control over time is handled differently in video world engines. Captain Safari couples user-specified 6-DoF camera paths with a retriever that fetches pose-aligned world tokens from local memory, addressing trajectory following and long-range geometric coherence in the same mechanism (Chou et al., 28 Nov 2025). MultiGen instead allows direct editing of the static map $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$6 before rollout; because every step invokes read_geometry(M,p_t), those edits immediately modify future observations without retraining or changing network structure (Po et al., 3 Mar 2026).

Evaluation work has formalized this multimodal-control perspective. 4DWorldBench maps non-text conditions such as images and videos into a unified textual space via captioning, then applies adaptive dimension selection to decide which alignment or physical-realism sub-dimensions are relevant before generating diagnostic questions (Lu et al., 25 Nov 2025). This design makes control comparable across Image-to-3D, Image-to-4D, Video-to-4D, and Text-to-3D/4D settings. A plausible implication is that future world engines will increasingly expose a common control layer even when their internal representations remain heterogeneous.

At inference time, control can also be mediated by specialized execution infrastructure. Inferix is designed around semi-autoregressive block-diffusion decoding, with global KV-cache management, quantization, and streaming modules; previously generated blocks are never reprocessed once their keys and values are in cache, and the system supports interactive video streaming and prompt updates during generation (Team et al., 25 Nov 2025). This indicates that, in some settings, world engine refers as much to the runtime substrate for long-horizon controllable rollout as to the generative model itself.

4. Dynamics, physics, and closed-loop simulation

Physics and dynamics separate world engines from purely geometric or photometric content generators. In LatticeWorld, the configuration model emits JSON entries of the form { type, quantity, initial_position, behavior_state }, which are translated into UE5 Actor classes; UE5’s PhysX-based solver then handles rigid-body dynamics, gravity, collisions, and cloth or particle effects, while each tick updates agent AI and resolves interactions (Duan et al., 5 Sep 2025). WorldGen uses Blender plus the Bullet physics engine, supporting gravity, collision, wind, drag, procedural fragmentation, and arbitrary camera and lens trajectories, while also producing pixel-perfect annotations such as depth, instance segmentation, semantic segmentation, normals, optical flow, and event-camera frames (Singh et al., 2022).

Code2Worlds treats dynamic fidelity as a failure mode of open-loop code generation and addresses it with a physics-aware closed loop. A PostProcess Agent infers physics constants and scripts dynamics, while a VLM-Motion Critic renders a rollout, evaluates motion alignment, and iteratively revises the actuation script until the output is judged valid (Zhang et al., 12 Feb 2026). The paper explicitly frames this as a remedy for the semantic-physical execution gap.

In autonomous driving, the World Engine framework combines a mutable digital twin with a behavior world model and KL-regularized reinforcement post-training. Safety-critical variants are synthesized by two extrapolation strategies—Scenario Copy and Intent Attack—then filtered through LQR-based trajectory tracking and validity checks before policy optimization. The post-training objective is

$\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$7

with experience drawn from a mixture of real and simulated trajectories (Li et al., 18 Jun 2026). Here the world engine is neither only a simulator nor only a generator; it is an intervention layer for policy alignment under rare, safety-critical interactions.

The same closed-loop logic appears in non-visual domains. SocioVerse organizes social simulation around a Social Environment, User Engine, Scenario Engine, and Behavior Engine, where actions are generated by a hybrid of rule-based ABMs and LLM-powered agents and then compared against real-world quantities through accuracy, RMSE, KL-divergence, diversity, credibility, and representativeness metrics (Zhang et al., 14 Apr 2025). This broadens the meaning of world engine from physical worlds alone to systems that simulate structured interaction between agents and environments.

5. Evaluation regimes and empirical performance

Because world engines target long-horizon structure rather than isolated outputs, their evaluation protocols are correspondingly multi-dimensional. 4DWorldBench organizes assessment into four dimensions—Perceptual Quality, Condition-4D Alignment, Physical Realism, and 4D Consistency—and combines network metrics, MLLM-as-judge, and LLM-as-judge in an adaptive framework. Its human-validation study reports that Physical Realism evaluation with AdaDimen plus an LLM judge achieves approximately $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$8 and $\hat y_l = \mathrm{LLM}_l(x_l,\Phi(v_l))$9 versus VBench2.0’s $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$0, while a style-consistency method improves PLCC from $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$1 to $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$2 (Lu et al., 25 Nov 2025).

Task-specific systems report their own metrics. LatticeWorld evaluates scene layout with mIoU between rendered $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$3 masks and ground-truth matrices, reports a $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$4–$\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$5 absolute mIoU gain over GPT-4o and peers, observes that removing visual embeddings drops variable-height IoU by about $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$6, and states that manual production requires approximately $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$7 artist-days whereas LatticeWorld requires approximately $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$8 days, corresponding to a speed-up of at least $\hat y_c = \mathrm{LLM}_c(x_c,\Phi(v_l),\hat y_l)$9 (Duan et al., 5 Sep 2025). WorldGen evaluates downstream optical-flow transfer by training RAFT on $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$0 synthetic frames and reports Sintel Clean/Final EPE of $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$1, compared with $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$2 for Kubric and $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$3 for AutoFlow (Singh et al., 2022).

Captain Safari targets video quality, 3D consistency, and trajectory following on the OpenSafari benchmark. On the held-out test set it reports FVD $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$4, LPIPS $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$5, $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$6 $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$7, Reconstruction Rate $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$8, AUC@30 $$\mathcal W = \mathrm{Render}(\Psi_l(\hat y_l),\Psi_c(\hat y_c),v_l)$$9, AUC@15 $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$0, and CosSim $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$1; the corresponding human study with $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$2 participants yields an average preference of $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$3 for Captain Safari, $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$4 for the ablation without memory, and $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$5 for all baselines combined (Chou et al., 28 Nov 2025).

Code2Worlds evaluates both object and scene generation on Code4D. Reported figures include O-CLIP $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$6, SGS $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$7, Style-CLIP $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$8, S-CLIP $p(x,t,m)=p(x)\cdot p(t\mid x)\cdot p(m\mid x,t)$9, Richness $x$0, HRS $x$1, Physics Failure Rate $x$2, Motion Smoothness $x$3, and Temporal Flicker $x$4; the abstract summarizes this as a $x$5 SGS gain and $x$6 higher Richness relative to baselines (Zhang et al., 12 Feb 2026). EarthGen, evaluated on extreme $x$7 super-resolution, reports FID/KID of $x$8 on the general set and $x$9 on the urban set, substantially outperforming several super-resolution baselines (Sharma et al., 2024).

The autonomous-driving World Engine uses open-loop and closed-loop driving metrics. On the adapted nuPlan benchmark, a base model pre-trained on $t$0 scenes achieves open-loop $t$1, $t$2, closed-loop Success Rate $t$3, and $t$4; after World Engine post-training, the same model reaches $t$5, $t$6, $t$7, and $t$8, respectively. In a production-scale system, post-training on $t$9 World Engine clips plus $m$0 common clips reduces pedestrian collisions by $m$1, intersection collisions by $m$2, cut-in collisions by $m$3, TTC events by $m$4, common dynamic collisions by $m$5, and static collisions by $m$6; a $m$7 real-road test reports zero disengagements versus one severe cut-in failure for the base model (Li et al., 18 Jun 2026).

6. Applications, limitations, and research trajectory

The application space is correspondingly broad. LatticeWorld explicitly targets embodied AI, autonomous driving, and entertainment through interactive 3D scenes in UE5 (Duan et al., 5 Sep 2025). EmbodiedGen is designed for simulator import into MuJoCo, SAPIEN, Drake, Isaac Lab, and OpenAI Gym through URDF assets with scale and physical properties (Wang et al., 12 Jun 2025). WorldGen and EarthGen emphasize large-scale synthetic-data generation for robotics and computer vision, while ABot-Earth targets digital-earth visualization and closed-loop UAV navigation with under-$m$8-minutes-per-square-kilometer generation and web-map deployment (Singh et al., 2022, Sharma et al., 2024, Qian et al., 8 Jun 2026). SocioVerse extends the paradigm to politics, news, and economics through a user pool of $m$9 million real individuals and scenario-driven LLM-agent interaction (Zhang et al., 14 Apr 2025).

Several limitations recur. Captain Safari notes nontrivial inference time and memory growth with scene length (Chou et al., 28 Nov 2025). LatticeWorld identifies hard-coded agent policies, single-user control, coarse body-part control, and asset-library bounds (Duan et al., 5 Sep 2025). EmbodiedGen states that quality inspection remains below $M=\{m_t\}_{t=0}^T$0, that layout generation is currently an LLM hack, and that dynamic physics such as fluids and deformables are not yet supported (Wang et al., 12 Jun 2025). EarthGen reports data bias toward USA/EU coverage, occasional color inconsistency between zoom levels, collapse on monochrome tiles, and heavy compute demands (Sharma et al., 2024). ABot-Earth reports a remaining quality gap relative to optimized photogrammetry in intricate urban canyons and residual block-boundary artifacts under larger coherence windows (Qian et al., 8 Jun 2026).

The overall research direction is toward tighter coupling of explicit world state, learned retrieval or generation, and closed-loop optimization. MultiGen makes user edits and multiplayer synchronization natural through shared external memory (Po et al., 3 Mar 2026). Inferix treats long-form generation as an inference-engine problem through block diffusion and KV caching (Team et al., 25 Nov 2025). Code2Worlds advocates retrieval-augmented code generation plus physics-aware self-reflection (Zhang et al., 12 Feb 2026). The autonomous-driving World Engine shows that synthetic yet controlled world extrapolation can function as a post-training substrate for safety alignment rather than only as a pre-training data source (Li et al., 18 Jun 2026). Taken together, these systems indicate that the modern world engine is best understood as an integrated apparatus for persistent representation, controllable generation, physically or behaviorally grounded rollout, and measurable long-horizon consistency.