DreamWorld: Controllable Synthetic Worlds

• DreamWorld is a family of research constructs that generate synthetic worlds for reinforcement learning, creative simulation, and interactive authoring.
• Different frameworks focus on learned simulators, unified video modeling, and interactive editing to ensure spatial, temporal, and semantic consistency.
• The approach advances real-to-dream policy transfer, persistent memory, and multimodal interfaces for constructing and navigating controllable 3D environments.

DreamWorld denotes a family of research constructs centered on generated or learned worlds that can be inhabited, navigated, edited, or used for policy learning. In current usage, the term ranges from latent “dream environments” for reinforcement learning, to unified text-to-video world modeling, to interactive text/image-to-video simulators with camera control and persistent memory, to multimodal world-building interfaces and immersive panoramic or 3D environment generation (Wellmer et al., 2021, Tan et al., 28 Feb 2026, Team et al., 15 Jun 2026, Dang et al., 2023). Across these lines of work, the common objective is to replace or augment direct access to reality with a controllable internal world whose dynamics, geometry, semantics, or editability are sufficiently structured to support transfer, long-horizon coherence, or creative authoring.

1. Scope, terminology, and research uses

Across the cited works, DreamWorld is not a single standardized architecture. The term is used for at least five closely related ideas: a learned simulator for Dream-to-Real controller training; a unified world-modeling framework for video generation; an interactive world model with camera navigation, revisits, and promptable events; a generative world-building platform informed by multimodal creative workflows; and a family of immersive panoramic or 3D scene-generation systems (Wellmer et al., 2021, Tan et al., 28 Feb 2026, Team et al., 15 Jun 2026, Dang et al., 2023, Yin et al., 29 Sep 2025, Schnepf et al., 19 May 2026).

Research use	Core object	Representative work
Dream environment	Learned simulator for policy learning and transfer	(Wellmer et al., 2021)
Unified world modeling	Joint pixel-and-world-feature video generator	(Tan et al., 28 Feb 2026)
Interactive world model	Text/image-to-video model with camera and event control	(Team et al., 15 Jun 2026)
World-building platform	Multimodal interface for iterative fictional world creation	(Dang et al., 2023)
Immersive world generation	Panoramic or 3D explorable environment synthesis	(Yin et al., 29 Sep 2025, Schnepf et al., 19 May 2026)

In the reinforcement-learning sense, a DreamWorld is a learned simulator induced by a world model $M$ and encoder $V$, in which an agent can practice “inside its own dream” before transfer to the real environment. In the generative-video sense, DreamWorld is a framework for making video generators behave more like world models by integrating temporal, spatial, semantic, and physical priors. In interactive-video systems, it denotes a controllable simulator with explicit camera navigation, revisit consistency, event composition, and persistent dynamic memory. In creative-tooling work, it denotes a platform for iterative visualization, layered editing, and hierarchical composition of fictional spaces (Wellmer et al., 2021, Tan et al., 28 Feb 2026, Team et al., 15 Jun 2026, Dang et al., 2023).

A common misconception is to treat these usages as interchangeable. The literature instead separates at least three technical targets: latent dynamics for decision making, pixel-level or latent-level world consistency in video generation, and authoring interfaces for human-guided world construction. This suggests that DreamWorld is best understood as an umbrella notion for controllable synthetic worlds rather than as a single method.

2. Dream environments and Dream-to-Real generalization

In the learned-simulator literature, a world model comprises a generative dynamics model $M$, an encoder $V$ such as a VAE, and a controller $C$. In the Ha and Schmidhuber framework summarized in "Dropout's Dream Land: Generalization from Learned Simulators to Reality" (Wellmer et al., 2021), $V$ compresses pixels to $z$, $M$ predicts next latent states, rewards, and termination, and $C$ selects actions from latent and recurrent features. Training $C$ entirely within the learned dynamics is “learning in the dream,” while the central failure mode is the Dream2Real gap: policies can overfit idiosyncrasies of the learned simulator and fail in the true environment.

The distinctive contribution of Dropout’s Dream Land is to convert inference-time dropout into an internal analogue of domain randomization. Each dropout mask $V$0 defines a distinct dynamics instance $V$1, and controller training maximizes expected return across the induced distribution of dream environments:

$V$2

Rather than averaging predictions as in MC dropout, the method treats each mask as its own simulator instance and trains the controller to be robust across them. The paper reports that step randomization outperforms episode randomization when the dynamics model was trained with dropout, with typical settings $V$3 and $V$4 during controller training (Wellmer et al., 2021).

The training pipeline is fully specified: collect trajectories from the real environment, train a VAE on images, train an MDN-RNN dynamics model on $V$5 sequences with per-sequence dropout, train a simple linear controller with CMA-ES inside the masked dream environments, and then transfer to reality without fine-tuning. On DoomTakeCover, the reported real-environment returns over 1000 trials are Random $V$6, Action-LSTM $V$7, GameGAN $V$8, WM $V$9, MC Dropout WM $M$0, and DDL $M$1. On CarRacing, the reported results over 100 trials are Random $M$2, WM $M$3, MC Dropout WM $M$4, and DDL $M$5 (Wellmer et al., 2021).

The DreamWorld idea is extended in two other directions. "DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models" trains an action-conditioned video world model and then fine-tunes a VLM planner entirely inside this DreamWorld using ORPO, reporting average score $M$6 versus best zero-shot $M$7 and decision-time latency of $M$8 s versus $M$9 s or $V$0 s for explicit verification baselines (Jia et al., 17 Mar 2026). "DMWM: Dual-Mind World Model with Long-Term Imagination" augments an RSSM-based System 1 with a logic-integrated System 2 and reports improvements of $V$1 in logical consistency, $V$2 in trial efficiency, $V$3 in data efficiency, and $V$4 gains at long horizons over Dreamer-style baselines on DMControl (Wang et al., 11 Feb 2025). These results do not define DreamWorld identically, but they preserve the same central premise: policy improvement is delegated to an internal model of the world rather than to repeated real interaction.

3. Unified world modeling in video generation

"DreamWorld: Unified World Modeling in Video Generation" defines DreamWorld as a text-to-video framework that jointly models video pixels and heterogeneous world features, rather than aligning a generator to a single external expert (Tan et al., 28 Feb 2026). The motivation is that pixel-level distribution matching alone does not produce a coherent world model: physical commonsense, 3D spatial geometry, temporal consistency, and semantic coherence are heterogeneous dimensions, and optimizing them naively can create instability and flicker.

The core mechanism is the Joint World Modeling Paradigm. DreamWorld extracts temporal dynamics from RAFT optical flow, spatial geometry from VGGT features, and semantic consistency from DINOv2 features. After spatial resampling, temporal pooling, feature standardization, and PCA compression, these are concatenated into

$V$5

with channel sizes $V$6, $V$7, $V$8, and $V$9, giving $C$0 (Tan et al., 28 Feb 2026). The pre-trained Wan2.1 input and output projections are expanded with zero-initialized world channels so that the model initially behaves like Wan2.1 and gradually learns to use the new features.

The training objective combines modality-specific flow-matching losses:

$C$1

with Consistent Constraint Annealing

$C$2

The stated purpose of CCA is to start with strong world priors and decay them to zero near training end, reducing flicker and abnormal highlighting while allowing the backbone to refine visuals (Tan et al., 28 Feb 2026). At inference, Multi-Source Inner-Guidance extends classifier-free guidance to text, temporal, semantic, and spatial conditions using masked-condition predictions.

The reported configuration uses Wan2.1-T2V-1.3B, 81 frames at $C$3, LoRA fine-tuning on 32k WISA videos for 2,000 optimization steps, AdamW with $C$4, $C$5, weight decay $C$6, learning rate $C$7, linear warmup 400 steps, BF16, and gradient checkpointing on $C$8 NVIDIA A100 (Tan et al., 28 Feb 2026). The evaluation reports VBench Overall Score $C$9 versus Wan2.1 $V$0 and Wan2.1-FT $V$1, VBench 2.0 Total Score $V$2, VideoPhy Overall SA $V$3 and PC $V$4, and WorldScore Overall Score $V$5 (Tan et al., 28 Feb 2026).

A notable ablation isolates the contribution of the three priors. VGGT only gives Overall $V$6, VGGT + DINOv2 gives $V$7, and the full model with optical flow reaches $V$8 (Tan et al., 28 Feb 2026). Another ablation reports that extending VideoREPA to multi-source relational alignment reduces the Physics score on VideoPhy from $V$9 to $z$0, which the paper uses to argue that joint synthesis is more stable than alignment-only strategies for multi-source world knowledge (Tan et al., 28 Feb 2026).

4. Interactive world models, persistent memory, and controllable events

"DreamX-World 1.0: A General-Purpose Interactive World Model" presents a general-purpose interactive text/image-to-video world model for controllable long-horizon generation across photorealistic, game-style, and stylized domains (Team et al., 15 Jun 2026). Its stated capabilities are explicit 6-DoF camera control, revisits of previously observed regions, and promptable/composable events. The data engine combines camera-accurate Unreal Engine rendering, action-rich gameplay recordings, and real-world videos with recovered camera geometry, while the model converts a bidirectional generator into a few-step autoregressive world model using causal forcing, DMD-style distillation, and long-rollout training.

A central technical component is E-PRoPE, a lightweight variant of projective positional encoding that retains projective camera geometry while applying camera-aware attention over a spatially reduced token set. The paper reports that E-PRoPE yields approximately $z$1 training time reduction and approximately $z$2 inference latency reduction while maintaining camera-control performance comparable to PRoPE, with Omni-WorldBench camera-control score $z$3 versus $z$4 (Team et al., 15 Jun 2026). The system further introduces Memory-Conditioned Scene Persistence, camera-geometry-based retrieval for revisits, residual recycling for robustness to imperfect memory latents, Event Instruction Tuning for composable event control, and reinforcement learning alignment to recover camera control and visual quality after distillation.

The reported performance places DreamX-World-1.0-5B at camera-control $z$5 and overall $z$6 on the 5-second basic evaluation, outperforming HY-WorldPlay 1.5 and LingBot-World in overall score, which achieve $z$7 and $z$8, respectively (Team et al., 15 Jun 2026). On long-horizon evaluation of approximately 30 seconds, the model reports overall $z$9 versus HY-WorldPlay $M$0 and LingBot-World $M$1. For revisit consistency over 10 seconds, DreamX-World reports gains of APSNR $M$2, ASSIM $M$3, ALPIPS $M$4 lower is better and reported as gain, ADINO-Sim $M$5, and AVPR-Sim $M$6 (Team et al., 15 Jun 2026). With mixed-precision DiT execution, residual reuse, 75\%-pruned VAE decoding, and asynchronous pipeline parallelism, throughput reaches up to $M$7 FPS on eight RTX 5090 GPUs (Team et al., 15 Jun 2026).

"WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory" addresses a related but distinct problem: maintaining dynamic entities that continue to evolve off-camera and re-enter with exact visual identities intact (Wang et al., 2 Jul 2026). Its framework explicitly decouples semantic motion orchestration from visual generation. An LLM plans 3D object trajectories and camera paths; these are projected into identity-colored location masks and appearance-condition videos; a latent video diffusion transformer then generates the visual sequence with persistent context retrieval, asymmetric attention, and a Temporal Drop Mechanism for appearance anchors. The generator input is summarized by

$M$8

The paper reports reconstruction PSNR $M$9, SSIM $C$0, LPIPS $C$1, DSC-DINO $C$2, and DSC-CLIP $C$3 on a held-out set of 100 videos with novel scenes and subjects (Wang et al., 2 Jul 2026). It also reports 3,000 post-training steps, global batch size 64, AdamW learning rate $C$4, BF16 mixed precision, causal chunking at 5-second segments, and training over $C$5 A100 80 GB for approximately 72 hours (Wang et al., 2 Jul 2026). A key claim is that several baselines attain higher static consistency largely by generating less motion, whereas WorldDirector prioritizes identity preservation under complex, extended dynamics (Wang et al., 2 Jul 2026).

Taken together, these systems move DreamWorld from passive video synthesis toward controllable simulation. DreamX-World emphasizes efficient autoregressive world modeling with camera geometry, memory-conditioned revisits, and event tuning; WorldDirector emphasizes explicit planning, identity permanence, and decoupled dynamic control. This suggests two complementary design philosophies within the DreamWorld literature: internal autoregressive simulation from learned priors, and external orchestration of trajectories followed by conditioned rendering.

5. World-building interfaces, panoramic exploration, and navigable 3D worlds

"WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI" frames DreamWorld as a generative world-building platform informed by multimodal interaction design (Dang et al., 2023). WorldSmith supports text input, sketching, and region-based filling; layered edits and hierarchical composition; and a Global Tile View, Detail Tile Editor, Results View, and Tree View. The first-use study reports 229 generation triggers leading to 2748 images, 86 world compositions via blending, and 16 sessions total. Participants created on average 2.05 regions per tile; region descriptions averaged 4.2 words, while scene descriptions averaged 12.4 words (Dang et al., 2023). The reported qualitative findings include preference for multimodal input by 11 of 13 participants, positive assessment of blended results by 11 of 13 participants, and a common request for style and perspective assistance across tiles (Dang et al., 2023).

This authoring perspective differs from simulator-oriented DreamWorld work because it foregrounds human creative workflow rather than autonomous policy learning or world-consistency metrics. Its central principles are D1 multi-modal input, D2 iterative refinement, D3 visual asset generation, and D4 hierarchical composition (Dang et al., 2023). The literature explicitly recommends text bootstrapping followed by region prompts and sketches, drag-and-drop asset reuse, global blending, and stateful branching via Tree View.

For immersive 360-degree worlds, "Unrolling Virtual Worlds for Immersive Experiences" describes a pipeline that maps a panorama to an interior sphere, reprojects local regions to tangent planes to mitigate equirectangular distortion, and restores coherence with Poisson blending or diffusion-based inpainting (Tikhonov et al., 2023). The paper formulates the spherical area distortion as $C$6, motivating local unrolling, latitude-aware LOD, and panorama-aware restoration. Navigation is primarily rotational, with optional multi-plane images for limited parallax (Tikhonov et al., 2023).

"PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusion" pushes panoramic DreamWorld generation toward route-controllable video synthesis (Yin et al., 29 Sep 2025). The PanoExplorer dataset contains 504 high-fidelity 3D environments and 116,759 panoramic video sequences paired with 3D exploration routes. The model uses a Sphere-Aware Diffusion Transformer with geodesic attention on the sphere and an Exploration-Aware controllable branch based on Plücker embeddings. On panoramic evaluation, the reported performance is PSNR $C$7, SSIM $C$8, LPIPS $C$9, FID $C$0, and FVD $C$1, substantially improving on 360DVD, Imagine360, and GenEX. On perspective-crop comparisons for fairness, the model reports $C$2 and $C$3 (Yin et al., 29 Sep 2025).

"SphericalDreamer: Generating Navigable Immersive 3D Worlds with Panorama Fusion" addresses the problem of achieving both omnidirectional coverage and long-range navigability in 3D (Schnepf et al., 19 May 2026). Starting from multiple generated panoramas, it constructs layered depth panoramas, lifts them into 3D spheres, and fuses adjacent spheres through inpainting, depth estimation, and Harmonic Blending under Dirichlet boundary constraints on a k-NN graph Laplacian. The final world is

$C$4

For rotation only, translation only, and rotation plus translation, the method reports Coverage $C$5, $C$6, and $C$7 with BRISQUE $C$8, $C$9, and $V$00, respectively (Schnepf et al., 19 May 2026). Under combined rotation and translation, it also reports CLIP-Score $V$01, C-CLIP $V$02, CLIP-IQA $V$03, and Q-Align $V$04 (Schnepf et al., 19 May 2026).

These systems indicate that DreamWorld can be built either as an editing interface over 2D generative assets or as a geometry-aware pipeline from panoramas to explorable 3D spaces. The former emphasizes interaction modalities and authorial control; the latter emphasizes omnidirectional coverage, trajectory adherence, and spatial consistency.

6. Object-centric activation, embodied interaction, and long-horizon imagination

A recurrent limitation of generated worlds is that they are visually coherent yet operationally inert. "WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes" addresses this by converting static monolithic 3D Gaussian Splatting scenes into editable and physically interactive scenes (Hu et al., 15 May 2026). The pipeline uses a multimodal agent for object discovery, SAM3 for segmentation, mask inverse rendering for 2D-to-3D Gaussian assignment, SAM3D for clean object reconstruction, DiffuEraser and DepthLab for background restoration, ICP and differentiable rendering for alignment, and Poisson reconstruction plus RANSAC plane regularization for collision meshes. On the Marble-World-Model dataset, Interactable Object Recall rises from $V$05 without the agent to $V$06 with the agent, with $V$07 on MWM-easy and $V$08 on MWM-hard (Hu et al., 15 May 2026). In the object-level MOS study, Original scores $V$09 overall for human/GPT-5.5, while New scores $V$10 (Hu et al., 15 May 2026).

DreamWorld also appears in robotic decision making through video-based imagination. DreamPlan treats the video world model itself as a DreamWorld in which a VLM planner is reinforcement fine-tuned without further real-world interaction (Jia et al., 17 Mar 2026). The action-conditioned world model is built on CogVideoX-5B with a ControlNet-style residual branch driven by rendered robot-motion videos, trained on 2,056 exploratory trajectories over rope, cloth, and soft toy tasks. Preference pairs from Best-of-$V$11 imagined futures are used to optimize the planner with

$V$12

The reported average score is $V$13 for DreamPlan versus $V$14 for the best zero-shot baseline, with explicit verification baselines at $V$15 and $V$16 but at far higher computational cost (Jia et al., 17 Mar 2026).

At a more abstract level, DMWM formalizes DreamWorld as a latent simulator for long-term imagination whose failure mode is compounding error under one-step statistical inference (Wang et al., 11 Feb 2025). Its dual-process design combines an RSSM-based System 1 and a logic-integrated neural System 2. The System 2 objective aggregates hierarchical logical reasoning depth and rule regularization:

$V$17

The paper reports average gains of $V$18 in logical consistency, $V$19 in trial efficiency, $V$20 in data efficiency, and $V$21 improvement for horizons $V$22 over Dreamer and related RSSM baselines (Wang et al., 11 Feb 2025).

These works expose a significant conceptual boundary in DreamWorld research. Some systems generate worlds for viewing or editing; others require worlds that are interaction-ready, object-centric, and sufficiently reliable for embodied planning. A plausible implication is that future DreamWorld systems will increasingly be judged not only by visual quality or controllability but also by whether they support object-level manipulation, persistent affordances, and long-horizon decision making under internal simulation.