4D Reinforcement Learning (4DRL)

• 4DRL is a framework that leverages spatio-temporal representations to optimize decisions in dynamic 3D environments.
• It applies reinforcement learning to scenarios such as dynamic scene reconstruction, medical imaging, and multimodal reasoning, integrating both discrete and continuous strategies.
• Key reward designs balance geometric fidelity, temporal consistency, and human preference to address challenges of high-dimensional spatio-temporal credit assignment.

4D Reinforcement Learning (4DRL) denotes reinforcement-learning methods whose states, actions, rewards, or optimized policies are defined over $3\mathrm{D}+ \text{time}$ representations, or over tasks that require reasoning about the evolution of 3D structure through time. In the cited literature, “4D” most often means 3D space plus time: dynamic 3D scenes, 4D Gaussian Splatting / Streaming, 4D flow MRI, or visual spatial-temporal intelligence inferred from video (Liang et al., 14 Aug 2025, Dahal et al., 18 Mar 2026, Bisbal et al., 31 May 2025, Yin et al., 28 Feb 2026). A plausible synthesis is that 4DRL is best understood not as a single algorithmic family but as a shared design pattern: reinforcement learning is used whenever the target objective involves spatio-temporal geometry, motion, physical plausibility, preference, or efficiency constraints that are difficult to express as simple differentiable losses (Liang et al., 14 Aug 2025, Lu et al., 3 Mar 2026, Chen et al., 7 May 2026).

1. Meaning of “4D” and scope of the term

In the visual generation literature, 4D is explicitly formalized as a field $f(\mathbf{x}, t)$ with $\mathbf{x}\in\mathbb{R}^3$ and $t\in\mathbb{R}$, and corresponds to 3D geometry evolving in time. The examples given include sequences of 3D human poses in a scene, a dancing character, a person walking through a room, a stereo video with depth consistency, or a 3D scene being progressively generated (Liang et al., 14 Aug 2025). In 4D Gaussian Streaming, the same idea appears as “dynamic scene reconstruction from multi-view videos, where scenes change over time,” represented by time-varying 3D Gaussian primitives (Dahal et al., 18 Mar 2026). In Phys4D, the world is represented as spatio-temporal points $p=(x,y,z,\tau)\in\mathbb{R}^4$, obtained by lifting RGB-D-motion predictions into a 4D point cloud or “worldlines” (Lu et al., 3 Mar 2026). In 4D flow MRI, the source data are a time-resolved 3D velocity field over a volumetric region, with time as the true fourth dimension (Bisbal et al., 31 May 2025).

The terminology is not fully uniform across all cited works. “Learning Robot Exploration Strategy with 4D Point-Clouds-like Information as Observations” uses “4D” in a different sense: each observed point is a 4D feature vector $(x,y,b,d)$ comprising 2D location, a frontier flag, and collision-free distance to the robot, rather than 3D space plus time (Li et al., 2021). By contrast, “4DThinker” and “MLLM-4D” use 4D to mean dynamic spatial understanding from video: 3D space plus time, inferred from 2D RGB inputs, and represented internally through latent imagery or spatiotemporal chain-of-thought structures (Chen et al., 7 May 2026, Yin et al., 28 Feb 2026).

A recurrent misconception is that 4DRL necessarily denotes long-horizon embodied control in a physical 3D simulator. The cited literature is broader. It includes contextual bandits for anchor selection in 4D Gaussian Streaming, outcome-based RL fine-tuning for VLMs and MLLMs, RL over generative world models, and adaptive geometric control in medical imaging (Dahal et al., 18 Mar 2026, Chen et al., 7 May 2026, Lu et al., 3 Mar 2026, Bisbal et al., 31 May 2025). Another recurrent misconception is that every 4DRL method must explicitly optimize temporal actions. In fact, some methods operate per frame or on derived static states: EGS treats each frame as an independent contextual bandit, and the 4D flow MRI method collapses the original 4D acquisition into derived 3D volumes for the RL agent (Dahal et al., 18 Mar 2026, Bisbal et al., 31 May 2025).

2. Formal problem formulations

A general formulation appears in the survey “Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances,” which casts 3D and 4D generation as a Markov Decision Process $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$. In that formulation, the state $s_t$ is the current partial 3D or 4D configuration together with conditioning; the action $a_t$ is a generative edit step such as selecting primitives, choosing a next best view, sampling a motion latent, choosing a denoising step, or placing an object; the transition is the update rule of the generative model; and the reward measures geometry fidelity, temporal smoothness, physical validity, preference score, or multi-view consistency. The objective is $$J(\pi)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t r(s_t,a_t)\right]$$, optimized by PPO, GRPO, A3C, DDPG, or preference-based surrogates such as DPO (Liang et al., 14 Aug 2025).

Several later papers instantiate this template in domain-specific ways. EGS formulates anchor selection for 4D Gaussian Streaming as a contextual bandit. For each frame $f(\mathbf{x}, t)$0, the agent observes a bounded candidate set $f(\mathbf{x}, t)$1, $f(\mathbf{x}, t)$2, encoded as 6-D spatial descriptors $f(\mathbf{x}, t)$3, and chooses an action $f(\mathbf{x}, t)$4 consisting of a discrete budget $f(\mathbf{x}, t)$5 and a subset $f(\mathbf{x}, t)$6 of anchor indices with $f(\mathbf{x}, t)$7. The reward explicitly combines budget penalty, runtime penalty, PSNR target violation, and PSNR gain, and the policy is optimized by a REINFORCE-style objective with an EMA baseline and entropy regularization (Dahal et al., 18 Mar 2026).

Phys4D instead casts the reverse denoising process of a video diffusion world model as a finite-horizon MDP. Its state is $f(\mathbf{x}, t)$8, where $f(\mathbf{x}, t)$9 is the conditioning, $\mathbf{x}\in\mathbb{R}^3$0 is the reverse-time step, and $\mathbf{x}\in\mathbb{R}^3$1 is the current latent; its action is $\mathbf{x}\in\mathbb{R}^3$2; and its reward is terminal only, computed at the end of generation as the negative 4D Chamfer Distance between generated and simulator-derived 4D point sets. Exploration is induced by converting the rectified flow ODE into a stochastic Flow-SDE, and PPO is then used to refine the generative dynamics (Lu et al., 3 Mar 2026).

In 4DThinker, the policy is an autoregressive LM after Dynamic-Imagery Fine-Tuning. The implicit state comprises the encoded video, previously generated text tokens, and internally propagated latent embeddings; the actions are text tokens only; and latent positions are treated as internal computation rather than policy actions. Rewards are sequence-level and outcome-based: answer correctness and correct “think with 4D” format. A key modification over standard GRPO is that policy gradients are restricted to text-token positions, explicitly excluding latent positions to avoid instability caused by the mismatch between continuous latent propagation and discrete log-probabilities (Chen et al., 7 May 2026).

MLLM-4D uses an analogous GRPO-style Reinforcement Fine-Tuning setup, but with a distinct spatiotemporal reward $\mathbf{x}\in\mathbb{R}^3$3 based on predicted 3D camera and object coordinates embedded inside Spatiotemporal Chain of Thought. The total reward is $\mathbf{x}\in\mathbb{R}^3$4, so the policy is rewarded not only for selecting the correct multiple-choice answer but also for producing a structurally valid reasoning trace whose internal 3D states match stereo-derived ground truth (Yin et al., 28 Feb 2026).

Outside generative modeling and MLLM post-training, 4DRL also appears as continuous and discrete control over geometric state spaces. In 4D flow MRI, the agent observes a two-channel local 3D volume $\mathbf{x}\in\mathbb{R}^3$5, parameterized in a plane-centric orthonormal basis $\mathbf{x}\in\mathbb{R}^3$6, and outputs a bounded continuous 5-D action vector of rotations and translations in that local coordinate system. The reward is the decrease in a cost that combines angular error and positional error relative to an expert plane, with an additional reward of 3 in a near-optimal terminal region (Bisbal et al., 31 May 2025). In the exploration paper, the state is the variable-size set $\mathbf{x}\in\mathbb{R}^3$7 with $\mathbf{x}\in\mathbb{R}^3$8, the action space is the frontier set $\mathbf{x}\in\mathbb{R}^3$9, and Double DQN is used to learn per-frontier Q-values (Li et al., 2021).

3. Recurrent algorithmic regimes

Across the cited literature, several recurrent regimes can be identified. The survey explicitly distinguishes “RL Fine-Tuning on Pretrained 3D/4D Generators,” “RL as the Core Training Mechanism / Controller,” “RL as a Controller Over Latent Codes or Generative Primitives (4D Motion),” and “Hierarchical / Multi-Objective RL Structures” (Liang et al., 14 Aug 2025). Later works add outcome-based RL post-training for multimodal reasoning, budget-aware contextual bandits for 4D reconstruction systems, and continuous actor-critic control in 4D medical imaging.

Regime	Representative formulations	Typical optimization
RL fine-tuning on pretrained 3D/4D generators	Carve3D, DreamReward, DreamDPO, Mesh-RFT, Phys-AR, RLVR-World	PPO, GRPO, DPO-like objectives
RL as the core training mechanism / controller	Akizuki et al., Lin et al., point cloud scene completion, QINet, RLSS	A3C, DDPG, PPO
RL over latent motion / generative primitives	Zhao et al., Bailando	PPO, actor-critic fine-tuning
Outcome-based RL for VLM/MLLM reasoning	4DThinker, MLLM-4D	GRPO with KL regularization
Budget-aware resource allocation in 4D representations	EGS	REINFORCE contextual bandit
Continuous geometric control in medical imaging	Adaptive plane reformatting for 4D flow MRI	A3C

The first regime treats RL as an alignment or repair layer applied after maximum-likelihood or reconstruction-based pretraining. Carve3D, HFDream, MVReward, DreamReward, DreamDPO, Mesh-RFT, Phys-AR, and RLVR-World all fit this pattern: a strong base generator remains in place, and RL or preference optimization is added to repair multi-view inconsistency, Janus artifacts, physical inconsistency, or human-preference misalignment (Liang et al., 14 Aug 2025). The survey explicitly characterizes such systems as “RL as last-mile alignment.”

The second and third regimes use RL more structurally. Akizuki et al., Lin et al., point-cloud scene completion, QINet, and RLSS build the generation process itself as a sequential decision problem rather than merely fine-tuning a pretrained generator. Zhao et al. and Bailando are described as canonical 4D RL examples because the agent controls a 3D body over time with rewards encoding physical and semantic constraints such as goal reaching, contact accuracy, penetration avoidance, beat alignment, and half-body consistency (Liang et al., 14 Aug 2025).

The multimodal reasoning papers extend this logic into post-training of language-conditioned systems. 4DThinker introduces DIFT before 4DRL, so that RL refines how the model uses already grounded 4D latent mental imagery; MLLM-4D introduces ST-CoT and ST-reward, keeping the standard MLLM architecture unchanged while shifting post-training toward explicit spatiotemporal structure (Chen et al., 7 May 2026, Yin et al., 28 Feb 2026). EGS shows that RL can also act as a plug-in resource-allocation mechanism rather than a content generator or reasoner: the RL module changes only step 3 of the IGS pipeline, replacing FPS while leaving the reconstruction backbone unchanged (Dahal et al., 18 Mar 2026).

4. Reward design, constraints, and 4D supervision

Reward design is the central technical issue in 4DRL. The survey groups 4D-relevant rewards into geometric and multi-view rewards, temporal and motion rewards, and human preference or AI feedback. The listed geometric terms include Boundary Edge Ratio, Topology Score, Hausdorff distance, IoU, Chamfer distance, and MRC; temporal terms include beat alignment, half-body consistency, goal completion, timeliness, foot-ground contact accuracy, penetration penalties, velocity and acceleration consistency, collision and mass-aware motion heuristics, and identity consistency; preference-based terms extend RLHF and RLAIF into 3D and 4D settings through Reward3D, VLM ranking, and pairwise objectives such as DPO (Liang et al., 14 Aug 2025). The survey also emphasizes multi-objective scalarization for 4D tasks, where no single metric suffices.

In EGS, the reward is explicitly a quality-efficiency-budget trade-off. After a candidate budget and anchor subset are selected, the frozen IGS backbone produces a rendered frame and measured runtime. The reward combines four terms: a penalty on anchor budget $t\in\mathbb{R}$0, a penalty when runtime exceeds the FPS@8192 reference, a penalty when PSNR falls below a mixed target built from FPS@8k and FPS@16k references, and a bonus when PSNR exceeds that target (Dahal et al., 18 Mar 2026). This is a particularly clear instance of RL over computational resource allocation in a 4D representation rather than RL over scene semantics alone.

The multimodal reasoning papers make reward design equally structural. 4DThinker uses $t\in\mathbb{R}$1, where $t\in\mathbb{R}$2 measures answer correctness and $t\in\mathbb{R}$3 enforces the required “think with 4D” format. The implementation uses $t\in\mathbb{R}$4 and $t\in\mathbb{R}$5, and ablations show that removing $t\in\mathbb{R}$6 changes DSR-Bench performance from 34.2 to 32.0, while removing $t\in\mathbb{R}$7 changes it from 34.2 to 33.4 (Chen et al., 7 May 2026). MLLM-4D adds a distinct geometric term: the spatiotemporal reward $t\in\mathbb{R}$8, computed from predicted 3D camera and object centers via an exponentiated Euclidean error, then combined with accuracy and format terms in the total reward. In ablations, GRPO without ST-reward reaches 70.5 on MLLM4D-Bench and 61.4 on VLM4D, whereas full GRPO with ST-reward reaches 72.7 and 63.1 (Yin et al., 28 Feb 2026).

Phys4D provides perhaps the most explicit world-level 4D reward. Generated depth and motion are lifted into a 4D point set $t\in\mathbb{R}$9, simulator trajectories provide $p=(x,y,z,\tau)\in\mathbb{R}^4$0, and the terminal reward is $p=(x,y,z,\tau)\in\mathbb{R}^4$1, where the 4D point-wise distance adds squared spatial error and a time-weighted squared temporal error (Lu et al., 3 Mar 2026). The same paper introduces a “4D world consistency evaluation” suite probing geometric coherence, motion stability, long-horizon physical plausibility, worldline error, drift, failure rate, and novel-time interpolation, thereby shifting evaluation away from appearance-only metrics.

In medical imaging and exploration, rewards remain task-specific but follow the same principle: encode downstream structure rather than pixel reconstruction alone. The 4D flow MRI method rewards reductions in a cost that combines angular and positional plane error and gives an additional reward of 3 once both $p=(x,y,z,\tau)\in\mathbb{R}^4$2 and $p=(x,y,z,\tau)\in\mathbb{R}^4$3 are satisfied (Bisbal et al., 31 May 2025). The exploration paper decomposes reward as $p=(x,y,z,\tau)\in\mathbb{R}^4$4, combining newly discovered area, a reward for reducing the number of frontier groups, and a small action penalty to encourage shorter paths (Li et al., 2021).

5. Representative domains and empirical results

Dynamic scene reconstruction and 4D world modeling provide two of the clearest empirical demonstrations of 4DRL. In EGS, on unseen N3DV scenes in fast rendering at 256 anchors, the RL policy improves PSNR by $p=(x,y,z,\tau)\in\mathbb{R}^4$5–$p=(x,y,z,\tau)\in\mathbb{R}^4$6 dB while running $p=(x,y,z,\tau)\in\mathbb{R}^4$7–$p=(x,y,z,\tau)\in\mathbb{R}^4$8 faster than IGS@8192, and in the same setting uses up to $p=(x,y,z,\tau)\in\mathbb{R}^4$9 fewer anchors than FPS@8192 while improving quality and speed (Dahal et al., 18 Mar 2026). In Phys4D, the three-stage curriculum yields Physics-IQ scores of 16.8% for the base WAN2.2-5B backbone, 22.6% after Stage 2 physics-grounded supervised fine-tuning, and 25.6% after Stage 3 RL, while Tables 4–5 report improvements in 4D Chamfer, worldline metrics, warp-based temporal consistency, and flow EPE (Lu et al., 3 Mar 2026). These results show 4DRL both as resource-aware adaptation and as world-model alignment.

Dynamic spatial reasoning in VLMs and MLLMs is another major application domain. In 4DThinker, Qwen2.5-VL-3B improves on DSR-Bench from 24.6 at base to 31.1 after DIFT and 34.2 after DIFT+4DRL, while Qwen3-VL-32B improves from 28.0 to 45.2 to 62.0 and surpasses the specialized DSR Suite-Model average of 58.9 (Chen et al., 7 May 2026). MLLM-4D reports 73.3 average accuracy for the Qwen2.5-VL-7B backbone and 73.4 overall, 72.7 average over subtasks, for the Qwen3-VL-8B backbone on MLLM4D-Bench; on VLM4D, the Qwen3-VL-8B-based model reaches 61.0%, comparable to Gemini-2.5-Pro at 62.0% and above GPT-4o at 57.5% (Yin et al., 28 Feb 2026). These systems demonstrate that 4DRL can optimize not only physical trajectories or generative fields but also internal reasoning traces that explicitly encode 4D state.

Medical imaging provides a different but equally concrete instantiation. The 4D flow MRI method reports improved accuracy in plane reformatting angular and distance errors, specifically $(x,y,b,d)$0 and $(x,y,b,d)$1 mm, and statistically equivalent flow measurements relative to expert plane reformatting with $(x,y,b,d)$2 (Bisbal et al., 31 May 2025). In the invariance experiment, DQN trained in a fixed global coordinate frame degrades from $(x,y,b,d)$3, $(x,y,b,d)$4 mm on the untransformed scan to $(x,y,b,d)$5, $(x,y,b,d)$6 mm over rigid transforms, whereas the A3C method using a state-dependent local coordinate system changes from $(x,y,b,d)$7, $(x,y,b,d)$8 mm to $(x,y,b,d)$9, $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$0 mm (Bisbal et al., 31 May 2025). The significance lies in the interaction between geometric inductive bias and RL formulation: the coordinate system itself becomes part of the policy design.

The exploration paper shows a fourth empirical pattern. Its proposed 4D point-cloud DQN method “needs shorter average traveling distances to explore whole environments and can be adopted in maps with arbitrarily sizes,” and the same network successfully scales from training maps of at most $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$1 to a larger test map of $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$2 without global-map resizing (Li et al., 2021). Although this paper uses “4D” in a feature-vector sense rather than $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$3, it remains relevant because it demonstrates how RL can be built around structured, variable-size geometric observations rather than fixed images.

6. Limitations, controversies, and future directions

The most consistent limitation across the cited literature is high dimensionality combined with difficult credit assignment. The survey notes that 4D states may include full 3D geometry and multiple time steps, often represented with NeRFs, Gaussians, dense grids, or long pose sequences, and that each RL update can require expensive forward simulation such as running a physics engine, reconstructing a NeRF, or computing multi-view distances. It also emphasizes compounded spatial and temporal credit assignment, brittle physics-based heuristics, noisy human preferences, and instability in policy-gradient fine-tuning of large diffusion or 3D diffusion models (Liang et al., 14 Aug 2025). A plausible implication is that 4DRL research is pushed toward more structured surrogates, offline preference datasets, and hybrid supervised-plus-RL curricula.

The multimodal reasoning papers expose a distinct set of controversies about what exactly RL should optimize. 4DThinker explicitly declines to apply policy gradients to latent imagery positions, restricting gradients to text tokens because continuous latent propagation and discrete log-probabilities mismatch and can destabilize training (Chen et al., 7 May 2026). MLLM-4D similarly operates in a static offline setting, limits videos to about 32 frames, and depends on stereo or strong 4D tracking for data curation (Yin et al., 28 Feb 2026). These design choices suggest an unresolved question: whether future 4DRL systems should continue to optimize only discrete linguistic policies over grounded latent world models, or instead optimize the latent dynamics themselves.

Phys4D and EGS expose further trade-offs. Phys4D depends on simulator-aligned 4D ground truth, performs offline RL rather than online interaction, and treats high-dimensional latent control as the policy space (Lu et al., 3 Mar 2026). EGS treats each frame as an independent contextual bandit and therefore does not perform long-horizon temporal credit assignment, even though it operates over a 4D scene representation (Dahal et al., 18 Mar 2026). These are not contradictions; they indicate that 4DRL spans both full sequential control and one-step resource allocation over time-indexed spatial structures.

The future directions named in the cited works are strikingly convergent. The survey points to unified 4D world-model RL, hierarchical and multi-agent 4D content generation, scaling preference-based RLHF to 4D, physics- and causality-aware policies, and inference-time 4D RL (Liang et al., 14 Aug 2025). EGS points to explicit temporal modeling in the policy, alternative cost models such as bandwidth and memory, and online adaptation to changing streaming conditions (Dahal et al., 18 Mar 2026). 4DThinker highlights richer reward signals, RL that also optimizes latent dynamics directly, and integration with embodied RL agents using 4D latents as world models (Chen et al., 7 May 2026). MLLM-4D points toward interactive AI systems in which the learned 4D reasoning module becomes a perceptual or world-model component inside a closed-loop agent (Yin et al., 28 Feb 2026). Taken together, these directions suggest that the next phase of 4DRL will likely couple internal 4D world representations with action-conditioned environment dynamics, rather than treating generation, reasoning, reconstruction, and control as separate problems.