World4RL: Diffusion Models for Robotic RL

• World4RL is a framework that uses diffusion-based world models as high-fidelity simulators for refining robotic manipulation policies offline.
• It integrates imitation learning pre-training with PPO-based optimization, thereby mitigating the sim-to-real gap and reducing risks in real-robot interactions.
• The framework’s use of innovative two-hot action encoding and diffusion dynamics leads to superior prediction metrics and effective policy improvement.

Searching arXiv for the specified paper and closely related "World4RL" works to ground the article with current citations. World4RL is a framework for robotic manipulation that employs diffusion-based world models as high-fidelity simulators to refine pre-trained policies entirely in imagined environments. It was introduced in "World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation" and is positioned against three longstanding bottlenecks: imitation learning plateaus under scarce and narrow expert data, real-robot reinforcement learning is costly and unsafe, and conventional simulators suffer from the sim-to-real gap. Its central design is to pre-train a diffusion world model on multi-task datasets and then optimize a policy with PPO inside a frozen learned simulator, avoiding online real-world interaction during policy refinement (Jiang et al., 23 Sep 2025).

1. Problem setting and conceptual scope

World4RL targets the setting in which a manipulation policy is first initialized by imitation learning and then improved through reinforcement learning without returning to hardware for exploratory interaction. The motivating claim is that robotic manipulation policies commonly initialized through imitation learning are limited by the scarcity and narrow coverage of expert data, while real-robot RL is both expensive and unsafe, especially in contact-rich settings. Training in classical simulators reduces these costs but introduces sim-to-real discrepancies that degrade transfer (Jiang et al., 23 Sep 2025).

The framework treats modern conditional diffusion models as “learnable simulators.” In this formulation, a world model predicts future observations conditioned on observation history and actions, and a separate reward classifier provides sparse success signals. This differs from prior learned video world-model pipelines that are primarily used for planning at test time by sampling candidate action sequences and selecting among them. World4RL instead performs direct end-to-end policy optimization inside the model, thereby producing a deployable policy rather than a planner coupled to expensive test-time trajectory sampling (Jiang et al., 23 Sep 2025).

A recurring misconception in this area is that generative world models are useful only for model-predictive control or trajectory selection. World4RL explicitly rejects that restriction: the learned world is frozen and used as a simulator for PPO-based policy refinement. Another misconception is that eliminating online interaction removes all modeling concerns. The paper states the opposite in its limitations, noting that with only offline data the world model may become inaccurate for actions far outside the training distribution, constraining policy improvement (Jiang et al., 23 Sep 2025).

2. Core architecture and representation design

The World4RL world model comprises three principal learned components: a diffusion transition model $D_\theta$, a reward classifier $C_\psi$, and a policy-value pair $(\pi_\xi, V_\phi)$. The transition model is a conditional diffusion model predicting the next observation $x^0_{t+1}$ from a finite history of RGB observations $x^0_{t-T:t}$ and actions $a_{t-T:t}$. In practice, the conditioning history is $T=4$ frames, the backbone is a 2D U-Net, and EDM-style preconditioning is used. The reward classifier is a ResNet18-based binary classifier that outputs the success probability $r(s_t,a_t)\in[0,1]$ from the predicted next observation. The policy and value are initialized by imitation learning and later refined by PPO inside the frozen world model (Jiang et al., 23 Sep 2025).

A distinctive element is the two-hot action encoding tailored to robotic manipulation. Rather than discretizing each action dimension into a single bin or compressing it with a tokenizer, World4RL encodes each scalar action $a_i$ by interpolating between two adjacent ordered bins $B=\{b_1,\ldots,b_K\}$. If $C_\psi$0, then

$C_\psi$1

with $C_\psi$2 for $C_\psi$3 and $C_\psi$4. The encoded action $C_\psi$5 concatenates these two-hot vectors across dimensions; in experiments, $C_\psi$6 (Jiang et al., 23 Sep 2025).

The paper characterizes this encoding as lossless, differentiable, and better suited to manipulation because it preserves fine-grained action semantics without the reconstruction errors associated with tokenizers or coarse discretizations. Ablations on Meta-World video prediction report that two-hot achieves the best FVD, FID, and LPIPS across both policy and random rollouts, outperforming one-hot, linear, FAST, and VQ-VAE encodings. This representation choice is therefore not peripheral: it is directly tied to the fidelity of the learned simulator that underwrites policy refinement (Jiang et al., 23 Sep 2025).

3. Diffusion dynamics, reward learning, and RL objective

World4RL adopts EDM preconditioning while remaining interpretable within the standard DDPM formulation. For the data variable $C_\psi$7—here the next observation $C_\psi$8—the forward process is written as

$C_\psi$9

and the reverse process is parameterized by a denoising model conditioned on observations, actions, and temporal context. A common DDPM objective is

$(\pi_\xi, V_\phi)$0

where $(\pi_\xi, V_\phi)$1 and $(\pi_\xi, V_\phi)$2 (Jiang et al., 23 Sep 2025).

In the implementation emphasized by the paper, the EDM denoiser is

$(\pi_\xi, V_\phi)$3

where $(\pi_\xi, V_\phi)$4 is the U-Net backbone and $(\pi_\xi, V_\phi)$5 aggregates temporal conditioning. The corresponding diffusion objective is summarized as

$(\pi_\xi, V_\phi)$6

Temporal conditioning is sequence-based but not recurrent: during training, the model observes the latest $(\pi_\xi, V_\phi)$7 frames and corresponding actions and predicts $(\pi_\xi, V_\phi)$8; during rollouts it autoregressively feeds the last $(\pi_\xi, V_\phi)$9 generated frames and actions back as context (Jiang et al., 23 Sep 2025).

The reward model is a frozen ResNet18 classifier trained with binary cross-entropy:

$x^0_{t+1}$0

After training, imagined rewards are computed as $x^0_{t+1}$1, and these rewards are sparse and binary (Jiang et al., 23 Sep 2025).

Policy refinement uses PPO in the frozen world model $x^0_{t+1}$2. The stated RL objective is

$x^0_{t+1}$3

with PPO updates

$x^0_{t+1}$4

where $x^0_{t+1}$5, and value learning uses

$x^0_{t+1}$6

A crucial detail is that gradients are computed via standard policy-gradient estimators rather than by backpropagating through the frozen diffusion model (Jiang et al., 23 Sep 2025).

4. Training pipeline and algorithmic workflow

World4RL is organized into two stages. In Stage 1, the policy is pre-trained by behavior cloning on demonstrations via

$x^0_{t+1}$7

In parallel, the diffusion transition model is trained on offline multi-task datasets using the observation history, two-hot action encoding, and EDM preconditioning. The reward classifier is trained on success-labeled data and then frozen. In Stage 2, both $x^0_{t+1}$8 and $x^0_{t+1}$9 are frozen, and the initialized policy is refined entirely in imagined rollouts (Jiang et al., 23 Sep 2025).

During imagined rollouts, the policy samples $x^0_{t-T:t}$0, the action is transformed to the two-hot representation $x^0_{t-T:t}$1, the transition model predicts $x^0_{t-T:t}$2, the reward classifier computes $x^0_{t-T:t}$3, and the tuple $x^0_{t-T:t}$4 is stored in a buffer. Once the buffer reaches a threshold, PPO updates are applied to $x^0_{t-T:t}$5 and $x^0_{t-T:t}$6. In experiments, trajectory length is 50 steps, the diffusion model conditions on $x^0_{t-T:t}$7 frames, and two-hot uses $x^0_{t-T:t}$8 bins per dimension (Jiang et al., 23 Sep 2025).

This workflow places World4RL in a specific methodological niche. It is not online real-robot RL, because policy refinement occurs off-robot. It is also not pure planning with a learned world model, because the output is a refined policy learned through repeated RL updates. The frozen-world design is central: it removes instability from simultaneous environment-model and policy updates, while using imagined trajectories to sidestep dangerous exploration on hardware. This suggests a deliberate trade-off between simulator fidelity and policy optimization stability rather than continual simulator adaptation during RL (Jiang et al., 23 Sep 2025).

5. Empirical performance in simulation and on hardware

World4RL is evaluated on both multi-task simulation and real-robot datasets. The simulation benchmark is Meta-World with six tasks; for each task, the dataset contains 50 expert trajectories, 150 trajectories from a pre-trained Gaussian policy, and 30 random rollouts, each with 50 timesteps. The real-robot benchmark uses a Franka Emika Panda across six tasks with 50 human teleoperation demonstrations, 50 Gaussian-policy rollouts, and 50 random rollouts. Tasks include open/close drawer, pick bread in/out, pick apple, and press button (Jiang et al., 23 Sep 2025).

The reported results separate world-model fidelity from downstream control gains. On Meta-World video prediction, World4RL achieves the best FVD, FID, and LPIPS among the compared baselines NWM, iVideoGPT, and DiWA. For policy/random rollouts, the paper reports FVD $x^0_{t-T:t}$9, FID $a_{t-T:t}$0, and LPIPS $a_{t-T:t}$1 for World4RL, compared with $a_{t-T:t}$2, $a_{t-T:t}$3, and $a_{t-T:t}$4 for NWM; $a_{t-T:t}$5, $a_{t-T:t}$6, and $a_{t-T:t}$7 for iVideoGPT; and $a_{t-T:t}$8, $a_{t-T:t}$9, and $T=4$0 for DiWA (Jiang et al., 23 Sep 2025).

The downstream RL results indicate that imagined-environment policy refinement improves substantially over imitation-learning and offline-RL baselines. On six Meta-World tasks with sparse binary rewards, World4RL reaches an average success rate of $T=4$1, compared with $T=4$2 for the Gaussian Policy, $T=4$3 for DP, $T=4$4 for TD3+BC, $T=4$5 for IQL, and $T=4$6 for IRASim-ft. The paper further reports absolute gains over the Gaussian Policy of $T=4$7 on Coffee-Pull-v2 and Lever-Pull-v2, $T=4$8 on Door-Lock-v2, $T=4$9 on Hammer-v2, $r(s_t,a_t)\in[0,1]$0 on Handle-Pull-v2, and $r(s_t,a_t)\in[0,1]$1 on Soccer-v2 (Jiang et al., 23 Sep 2025).

On the Franka platform, policy refinement in the frozen world model transfers to hardware without online exploratory training. Over six real tasks and 20 trials per task, World4RL attains an average success rate of $r(s_t,a_t)\in[0,1]$2, compared with $r(s_t,a_t)\in[0,1]$3 for the Gaussian Policy and $r(s_t,a_t)\in[0,1]$4 for Diffusion Policy. The paper also emphasizes sample efficiency relative to offline-to-online baselines: World4RL reaches comparable or better performance without any online interaction, whereas RLPD needs approximately $r(s_t,a_t)\in[0,1]$5k online steps and Uni-O4 requires approximately $r(s_t,a_t)\in[0,1]$6k online steps (Jiang et al., 23 Sep 2025).

Setting	Metric	Result
Meta-World video prediction	FVD / FID / LPIPS	$r(s_t,a_t)\in[0,1]$7, $r(s_t,a_t)\in[0,1]$8, $r(s_t,a_t)\in[0,1]$9
Meta-World RL	Average success rate	$a_i$0
Franka real robot	Average success rate	$a_i$1
Sample efficiency comparison	Online interaction	None for World4RL; $a_i$2k for RLPD; $a_i$3k for Uni-O4

The ablation evidence is also structurally important. Two-hot action encoding is reported as the most faithful for dynamics modeling, with one-hot, linear, FAST, and VQ-VAE all performing worse on FVD, FID, and LPIPS. The paper further notes that, compared to planning with learned video world models, World4RL avoids expensive test-time sampling and evaluation; IRASim-ft can incur up to $a_i$4 higher computational cost while still underperforming on average (Jiang et al., 23 Sep 2025).

6. Relation to the broader “World4RL” theme, limitations, and open directions

Although World4RL denotes a specific framework for robotic manipulation, subsequent work uses the term more broadly to describe reinforcement learning with world models or reinforcement learning for constructing world simulators. "WorldSample: Closed-loop Real-robot RL with World Modelling" presents a real-robot variant of this theme: it closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement, and introduces Policy-Paced Learning to regulate synthetic data through Q-aware sample selection and uncertainty-guided scheduling. On robot manipulation tasks, it reports an average success rate of $a_i$5 versus $a_i$6 for HIL-SERL, training steps of $a_i$7K versus $a_i$8K, and world-model fidelity gains of $a_i$9 dB PSNR and $B=\{b_1,\ldots,b_K\}$0 SSIM over demonstration-only post-training (Xue et al., 2 Jul 2026).

In generative video modeling, "World-R1: Reinforcing 3D Constraints for Text-to-Video Generation" explicitly frames World4RL as the use of RL to build world simulators from generative video models. That work aligns a pretrained text-to-video policy with 3D constraints using Flow-GRPO, 3D-aware rewards, and a pure-text prompt corpus for world simulation. It reports reconstruction-based 3D consistency gains such as PSNR $B=\{b_1,\ldots,b_K\}$1 and SSIM $B=\{b_1,\ldots,b_K\}$2 for World-R1-Small versus $B=\{b_1,\ldots,b_K\}$3 and $B=\{b_1,\ldots,b_K\}$4 for Wan2.1-T2V-1.3B, while keeping inference unchanged from the base model (Wang et al., 27 Apr 2026).

At the benchmarking level, "Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning" does not mention World4RL explicitly, but it is described as a possible foundational component if World4RL denotes a broad, community benchmark for real-world RL. Gym4ReaL packages six environments spanning water resources, elevators, microgrids, industrial picking, trading, and water distribution systems, with explicit modeling of large state-action spaces, non-stationarity, partial observability, and multi-objective rewards. This suggests that the term “World4RL” now names not only a single robotic-manipulation framework but also an emerging research orientation centered on realistic world models, real-world constraints, and deployable RL evaluation (Salaorni et al., 30 Jun 2025).

For the original World4RL framework, the paper identifies several limitations. With only offline data, the world model may produce inaccurate predictions for actions far outside the training distribution. Sparse rewards can impede exploration and destabilize gradients. Contact-rich scenes and very long horizons remain challenging for generative models. Scaling to broader task sets and multi-robot deployments requires efficient training and careful dataset curation (Jiang et al., 23 Sep 2025).

These limitations clarify the framework’s actual contribution. World4RL does not claim that frozen diffusion world models eliminate simulator error, nor that imagined-environment RL is universally sufficient. Its contribution is more specific: it demonstrates that a diffusion world model trained on diverse manipulation data can be used as a high-fidelity simulator for end-to-end PPO refinement, yielding consistent gains over imitation learning and competitive baselines in both simulation and real-world robotic manipulation. Within the broader “World4RL” landscape, this establishes a concrete recipe for turning generative video prediction into a practical vehicle for safe, off-robot policy improvement (Jiang et al., 23 Sep 2025).