GEM-4D: Geometry-Grounded Video Model

• The paper introduces GEM-4D, a geometry-grounded video world model that integrates dense 4D correspondence during training to enforce physical continuity in generated videos.
• It employs a dual-branch architecture where a Geometry DiT supervises the video backbone to maintain consistent depth, motion, and rigid-body coherence without added inference cost.
• The model converts geometry-consistent video rollouts into executable robot trajectories using an adaptive inverse dynamics pipeline, significantly improving manipulation success rates.

Searching arXiv for GEM-4D and closely related geometry-grounded video world models. I’ll look up the primary GEM-4D paper and a few directly related references on arXiv to support cross-references. GEM-4D is a geometry-grounded video world model for robot manipulation that augments a video generative backbone with dense 4D correspondence supervision during training, while preserving a single-stream RGB video generator at inference time with no additional runtime cost. Its stated objective is to address a failure mode of video world models in robotics: generated futures can appear visually plausible yet fail to preserve consistent point-level motion, depth structure, and rigid-body coherence, making action extraction unreliable. GEM-4D combines training-time geometry distillation with an inverse dynamics pipeline that converts correspondence-consistent video rollouts into executable robot trajectories for both simulated and real-world manipulation (Zhou et al., 20 May 2026).

1. Terminology and scope

GEM-4D denotes the method introduced in "GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation" (Zhou et al., 20 May 2026). It is specifically a video world model for manipulation, not a generic embodied VLM, occupancy forecaster, or gaseous detector.

The acronym should be distinguished from at least two unrelated 2026 arXiv uses of "GEM". "GEM: Generative Supervision Helps Embodied Intelligence" introduces a generative-supervised embodied vision-LLM and the GEM-4M dataset for embodied pretraining (Zhao et al., 27 May 2026). "GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning" introduces a continuous 4D Gaussian world model for autonomous driving (Chen et al., 17 May 2026). The shared acronym is nominal rather than methodological: GEM-4D is centered on geometry-grounded video generation and action recovery for robot manipulation, whereas those works address embodied VLM pretraining and occupancy forecasting, respectively.

Within the manipulation setting, GEM-4D is framed as a response to a specific limitation of video world models. A single instruction can drive realistic future video generation, but realistic appearance alone does not guarantee physically grounded motion over time. GEM-4D therefore treats geometry supervision as a mechanism for enforcing correspondence consistency, rather than as an auxiliary visualization output (Zhou et al., 20 May 2026).

2. Problem setting and motivating claim

The motivating claim is that photorealism is insufficient for robotics. A planner that extracts actions from predicted frames requires inter-frame correspondences: pixels depicting the same physical 3D point must evolve consistently across time. If generated rollouts exhibit warping, drifting contacts, depth discontinuities, or non-rigid distortions, then they may remain visually plausible to a human observer while being unusable for reliable action extraction (Zhou et al., 20 May 2026).

GEM-4D formalizes this through a correspondence relation between image motion and underlying scene geometry. For a pixel $p_t$ corresponding to a 3D point $X_t$, the next-frame projection is governed by

$$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$

Here $K$ is the camera intrinsic matrix, $R_{t \to t+1}$ and $T_{t \to t+1}$ are relative camera pose, $D(p_t)$ is depth at pixel $p_t$, and $\Delta X_t$ is scene flow or object motion. The paper uses this relation to argue that correspondences are determined by geometry: for static points, depth and camera pose suffice; for dynamic points, object motion is additionally encoded through changes in depth and position across time (Zhou et al., 20 May 2026).

A common misconception in video-based robot planning is that visually convincing rollouts are adequate if an inverse model is sufficiently strong. GEM-4D is explicitly built on the contrary position. Its central claim is that a robot can only act reliably if the generated future preserves physical continuity across frames, including camera motion, depth structure, rigid-body coherence, and object motion consistency (Zhou et al., 20 May 2026).

3. Model architecture and training objective

GEM-4D is implemented as a dual flow-matching system composed of a Video DiT and a Geometry DiT. The Video DiT generates RGB video in latent space. The Geometry DiT predicts geometry representations only during training. The geometry branch reads from the video backbone’s intermediate features, never writes back explicit outputs into the inference graph, and is removed at test time. The result is a training-time multi-branch system with a single-stream inference path (Zhou et al., 20 May 2026).

On the video side, the model uses latent flow matching. Given latent video $z_0$ and noise $X_t$0, the video branch learns a velocity field through

$X_t$1

where $X_t$2 is the language instruction. The video velocity is parameterized as

$X_t$3

with $X_t$4 denoting an intermediate hidden feature (Zhou et al., 20 May 2026).

The geometry side is trained by distillation from a frozen geometry foundation model $X_t$5. Given a training video sequence $X_t$6, the teacher produces a dense geometric representation

$X_t$7

A Geometry DiT then predicts a velocity field in geometry space conditioned only on $X_t$8:

$X_t$9

The total objective is

$$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$0

The paper’s conceptual novelty is that $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$1 acts as a representation-level correspondence regularizer. The model is not trained to emit depth, normals, or flow as explicit inference outputs. Instead, both appearance supervision and geometry-induced supervision act on the same intermediate feature space, forcing the Video DiT to encode structure sufficient for both RGB generation and geometry-consistent correspondences (Zhou et al., 20 May 2026).

This architecture supports the paper’s claim that appearance and geometry can be jointly preserved without increasing inference-time complexity. A plausible implication is that the method is intended to improve actionability without changing the deployment interface of a standard RGB video world model.

4. Dense 4D correspondence supervision

The geometry teacher is motivated by the observation that modern 4D geometry foundation models, including PAGE-4D, Depth Anything V3, VGGT, VGGT4D, and DUSt3R-family models, are trained to infer dense depth and camera pose from video. GEM-4D treats their internal outputs as dense correspondence teachers because those representations already encode the geometric factors that determine pointwise temporal consistency (Zhou et al., 20 May 2026).

The distillation mechanism is asymmetric. The frozen geometry foundation model processes the same training video and supplies $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$2 as dense geometric supervision. The Geometry DiT receives only the video backbone feature $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$3 as conditioning, with no pixels, no explicit depth, and no explicit camera pose. Under this constraint, the only route to minimizing $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$4 is for $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$5 to encode the geometric structure required to predict the teacher representation. The paper characterizes this as injecting correspondence supervision into the video backbone through representation alignment (Zhou et al., 20 May 2026).

This design is important for two reasons. First, it preserves the output space of the original video model: the system remains an RGB video generator rather than a multi-head geometry predictor. Second, it converts geometry from an explicit test-time modality into a training-time supervisory prior. The paper therefore presents geometry grounding not as an additional output burden, but as a means of reorganizing the internal representation so that rollouts become physically and kinematically trustworthy enough for downstream action extraction.

The paper also reports ablations on alternative geometry priors. One variant, GEM-4D(VGGT), uses a different prior and slightly degrades performance, which the authors attribute to VGGT being better matched to static or quasi-static scenes than dynamic manipulation. Another variant, GEM-4D(Dep), uses depth supervision instead of geometry features. Depth supervision remains competitive, but the model still receives only noise for the depth branch and learns depth through the flow-matching objective rather than conditioning on first-frame depth like TesserAct (Zhou et al., 20 May 2026). This suggests that the form of geometry prior matters, not merely the presence of an auxiliary branch.

5. Inverse dynamics and action extraction

GEM-4D couples its video model with an Adaptive Inverse Dynamic System (AIDS) that converts generated rollouts into executable 6-DoF end-effector trajectories. The pipeline is modular and geometry-aware rather than policy-based (Zhou et al., 20 May 2026).

The first stage is 3D scene grounding. Qwen3.5-VL and SAM-2 produce masks for the target object and end-effector. These masks are back-projected using depth and camera intrinsics to obtain point clouds. FoundationPose then aligns the end-effector CAD model to the end-effector point cloud and recovers the initial end-effector pose $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$6 (Zhou et al., 20 May 2026).

The second stage is confidence-gated tracking. Dense keypoints sampled from the end-effector mask are tracked through the generated rollout with CoTracker3. If $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$7 denotes anchor keypoints in the initial frame and $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$8 the subset still tracked at time $$p_{t+1} = K \left[ R_{t \to t+1}\, D(p_t)\, K^{-1} p_t + T_{t \to t+1} + \Delta X_t \right].$$9, the system monitors

$K$0

These quantities separate gradual drift from abrupt collapse. If $K$1 decreases smoothly below a threshold $K$2, the tracker is re-anchored from the latest reliable mask. If $K$3, the system re-grounds using Qwen3.5-VL semantics (Zhou et al., 20 May 2026). The presence of these interventions indicates that even geometry-grounded rollouts can still incur tracking failures, and the action extraction stack is designed accordingly.

The third stage is pose fallback using geometry and kinematics. FoundationPose predicts

$K$4

If confidence is below threshold and the pose jump is too large, the estimate is rejected according to

$K$5

For rejected frames, translation is recovered by back-projecting valid depth pixels in the end-effector mask and taking the 3D centroid, while rotation is recovered by spherical linear interpolation between accepted poses (Zhou et al., 20 May 2026).

The final stage inserts a grasp pose. GraspGen proposes candidates $K$6, and the system selects the candidate nearest a reference pose $K$7:

$K$8

The chosen grasp pose is inserted into the recovered end-effector trajectory, motion is smoothed by interpolation, and inverse kinematics converts the result into executable robot actions (Zhou et al., 20 May 2026).

6. Training data, evaluation protocol, results, and limitations

GEM-4D is trained on ManiSkill3, RLBench, BridgeData v2, and RT-1. Evaluation is conducted in two settings: Droid, a real-world benchmark with 400 unseen samples, and RLBench, a synthetic benchmark with 780 unseen samples. For Droid, depth is estimated using Depth Anything V3; for both settings, point tracking uses CoTracker3, while RLBench provides ground-truth depth (Zhou et al., 20 May 2026).

The reported comparison set includes CogVideoX, Wan 2.2-14B, TesserAct, and Geometry Forcing. The paper explicitly excludes some geometry-heavy methods, such as RoboTransfer, 3DFlowAction, and Liu et al., on the grounds that their input and output modalities differ from GEM-4D’s single-view RGB setting (Zhou et al., 20 May 2026). This is a methodological boundary rather than a generic claim of incomparability.

For 4D scene prediction, the reported metrics are FVD, SSIM, PSNR, AbsRel, $K$9, $R_{t \to t+1}$0, Chamfer distance on reconstructed point clouds, and TAP-Vid-style point tracking accuracy. The paper states that GEM-4D achieves the best or near-best performance across RGB quality, depth quality, and point correspondence quality on both real and synthetic data, highlighting lowest FVD, highest SSIM and PSNR, best depth metrics on real data, and improved Chamfer and tracking metrics (Zhou et al., 20 May 2026).

For manipulation, the central quantitative claim is that GEM-4D improves real-world manipulation success from 61% to 81%. This improvement is specifically reported for real-world Droid manipulation tasks. Task-level gains are also reported: AUTOLab improves from 58% to 75%, CLVR from 65% to 83%, and RAIL from 59% to 87%. On RLBench, GEM-4D achieves 63%–82% success across tasks (Zhou et al., 20 May 2026). The paper attributes these gains to more reliable trajectory extraction from geometry-consistent rollouts and notes that TesserAct often fails to produce executable trajectories.

The ablation study isolates three contrasts: no geometry guidance, depth supervision instead of geometry features, and a different geometry prior. Removing geometry guidance reduces performance. Depth supervision is competitive. Direct VGGT feature supervision slightly degrades results. The paper’s interpretation is that not all geometry priors transfer equally well to dynamic manipulation scenes (Zhou et al., 20 May 2026).

Several limitations are also identified. The inverse dynamics module depends on external components including SAM-2, CoTracker3, FoundationPose, GraspGen, and VLM-based re-grounding. Failures in these subsystems can still limit end-to-end performance. Generated videos can still suffer from tracking drift or abrupt collapse, motivating the gating and fallback logic in AIDS. The evaluation focuses on relatively standard manipulation tasks and rollouts; long-horizon, heavily occluded, or strongly deformable interactions remain challenging (Zhou et al., 20 May 2026). This suggests that GEM-4D should be understood not as a complete replacement for explicit geometric reasoning, but as a training-time mechanism for making video world models substantially more actionable under robotic control constraints.