GEM-4M: Depth-Supervised Multimodal Dataset

• GEM-4M is a large-scale embodied pretraining dataset that integrates semantic instruction with a generative depth objective to embed geometry into visual tokens.
• It comprises about 4 million multimodal samples from heterogeneous sources, supporting tasks like grounding, spatial reasoning, and spatiotemporal planning.
• Empirical evaluations demonstrate that GEM-4M enhances spatial understanding and improves downstream embodied reasoning and robotic action benchmarks.

GEM-4M is the large-scale embodied pretraining dataset introduced with GEM, a Generative-supervised Embodied vision-LLM designed to bridge the gap between the semantic emphasis of instruction- and Q&A-heavy VLM training and the low-level spatial and physical knowledge needed for execution in embodied environments. It comprises approximately 4 million multimodal samples spanning grounding, physical/spatial reasoning, and spatiotemporal planning, and it pairs those modalities with high-quality depth supervision so that the same visual features used for language understanding are also trained to become geometry-aware (Zhao et al., 27 May 2026).

1. Design objective and position within GEM

GEM-4M was curated to make the generative-supervised depth objective in GEM practical and effective. In GEM, the backbone’s visual tokens are trained not only with language objectives but also to condition a DiT-based generator that synthesizes the scene’s depth map. The dataset therefore couples multimodal supervision for instruction following, object reference, layout reasoning, and planning with per-image depth supervision, including cases in which the underlying source data do not provide depth directly.

The design addresses a specific limitation in embodied VLM pretraining. Standard text-guided pretraining paradigms emphasize high-level semantics, whereas embodied execution depends on low-level spatial and physical knowledge. GEM-4M is constructed so that every visual sample contributes simultaneously to linguistic supervision and to a generative geometry objective. A plausible implication is that the dataset is intended less as a conventional instruction-tuning corpus than as a representation-shaping mixture in which geometry is embedded directly into the foundational visual tokens.

The reported downstream behavior is consistent with that design. Adding generative depth supervision improves embodied reasoning benchmarks and transfers to action models, particularly GEM-VLA, in both simulation and real-world robot evaluations.

2. Scale, sources, and composition

GEM-4M comprises approximately 4 million multimodal samples, but the paper does not present a complete per-category accounting that sums to exactly 4 million. It instead gives concrete counts for the largest components and describes additional integrated sources.

Component	Reported scale	Sources or construction
Grounding	1M high-quality Q&A pairs	PACO-LVIS, RoboPoint, RoboAfford++, ShareRobot, Roborefit
Additional grounding annotations	~100k	Generated on RoboMind, DROID, Open X-Embodiment/BridgeData using SAM3
3D spatial perception	100k manually constructed samples	Derived from ScanNet, ScanNet++, ARKitScenes
Spatiotemporal reasoning	1M Q&A pairs	RoboVQA, Robo2VLM, RefSpatial
Spatiotemporal planning	~50k samples	RoboCOIN, RoboMind, Agibot World Colosseo
Integrated open-source spatial data	Exact counts not enumerated	MindCube, ViCA, SPAR, VSI-590K

The grounding portion focuses on spatial grounding, including open-vocabulary detection with boxes, instruction-conditioned localization, and affordances. To further anchor manipulations, approximately 100k additional point and bounding box annotations are generated on top of open robot datasets using SAM3, and the resulting coordinates are normalized in a consistent coordinate frame.

The physical and spatial reasoning mixture includes open-source spatial datasets and a manually constructed set of 100k 3D spatial perception samples derived from ScanNet, ScanNet++, and ARKitScenes, following VSI-Bench methodology for question templates such as absolute and relative distance, sizes, directions, layout attributes, and counts. The spatiotemporal reasoning component aggregates 1M Q&A pairs from RoboVQA, Robo2VLM, and RefSpatial, emphasizing long-horizon robotic reasoning and spatial referring.

The planning subset contributes approximately 50k samples produced from robot datasets with sub-task annotations. These samples construct Q&A pairs in the style of RoboVQA and MolmoACT for sub-task planning and trajectory forecasting. Trajectory traces are created by segmenting and tracking objects with SAM3 and CoTracker3, then smoothing each trajectory with a cubic spline and summarizing it by six uniformly spaced points.

The environment sources are heterogeneous. They include simulated and benchmark environments as well as real-world robot datasets from public corpora such as RoboMind, DROID, and BridgeDataV2/Open X-Embodiment. The 3D scene samples derive from real RGB-D capture datasets, but these are used primarily to generate spatial Q&A pairs; the depth supervision used in training is described as largely pseudo-depth.

3. Modalities, sample structure, and annotation conventions

Each GEM-4M sample minimally includes RGB visual input, natural language supervision, grounding annotations, and depth supervision. The visual input may be a single image or frames extracted from short video segments for planning and trajectory tasks. The natural language component consists of instruction or question text together with answer labels derived from upstream templates and custom generation procedures, including VSI-style spatial questions and RoboVQA/MolmoACT planning templates.

Grounding annotations are either a bounding box or a point denoting the referred or manipulated object. All coordinates are normalized to $[0, 1000]$ to handle resolution variations consistently across datasets. In the practical preprocessing recipe, bounding boxes are axis-aligned minimum rectangles enclosing instance masks, and points are sampled within mask regions. For grounding masks used to derive boxes and points, SAM3 masks must have confidence greater than $0.5$.

For planning and trajectory-bearing samples, the dataset also includes an object instance mask from SAM3 and a tracked trajectory summarized as six points after spline smoothing. The trajectory construction procedure uses object centroid seeding and CoTracker3 tracking in image space; no world-frame projection is described.

The paper does not specify file formats, resolutions, temporal lengths, or directory and split layouts. It also does not provide a manifest schema or concrete storage organization in the main body. The appendix documents prompts used to generate labels, including object label extraction and direct-object extraction, which imply fields such as instruction text, question, answer, normalized box or point coordinates, and optional trajectory points. This suggests that GEM-4M is presented primarily as a multimodal supervision mixture rather than as a rigidly standardized benchmark format.

4. Depth supervision and joint pretraining mechanics

The central technical characteristic of GEM-4M is its depth supervision. When training data lack ground-truth depth maps, GEM uses DepthAnything v3 to generate high-quality pseudo-depth for supervision. The paper does not state that any sensor-based metric depth is included, nor does it specify whether the pseudo-depth is metric or relative. No scale-alignment procedure, stereo or SLAM rendering, camera intrinsics, or camera extrinsics are described.

This depth signal is consumed through a generative objective. The VLM language objective is a cross-entropy next-token loss over the answer conditioned on multimodal tokens from the backbone:

$$\mathcal{L}_{\mathrm{CE}} = - \sum_{i=1}^{T} \log p_\theta \left( y_i \mid y_{< i}, \mathbf{h}_o, \mathbf{h}_l \right),$$

where $\mathbf{h}_o$ and $\mathbf{h}_l$ are final-layer visual and language token features.

The depth generation objective is a flow-matching loss for a DiT-based generator $G_\psi$ conditioned on a connector-projected embedding $\mathbf{c} = C_\phi(\mathbf{h}_o)$:

$$\mathcal{L}_{\mathrm{flow}} = \mathbb{E}_{d,~t \sim \mathcal{U}(0,1),~\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})} \left[ \left\lVert \mathbf{v}_t(\mathbf{x}_t, \mathbf{c}) - \mathbf{u}_t(\mathbf{x}_t \mid d) \right\rVert_2^2 \right].$$

In Stage 3, GEM is trained end to end with the joint objective

$$\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{CE}} + \lambda \,\mathcal{L}_{\mathrm{flow}}, \quad \lambda = 0.1.$$

The architecture tied to GEM-4M uses a Qwen3-VL backbone, a lightweight 2-layer MLP connector, and a Sana DiT-based depth generator head. The pretraining schedule is progressive. Stage 1 freezes the backbone and DiT and trains only the connector with $\mathcal{L}_{\mathrm{flow}}$ for 500 steps. Stage 2 freezes the backbone and trains connector plus DiT with $0.5$0 for 4k steps. Stage 3 unfreezes backbone, connector, and DiT, then trains with $0.5$1 for 1 epoch over GEM-4M.

The reported hyperparameters for VLM pretraining are a global batch size of 128, AdamW, weight decay 0.1, and a cosine learning-rate schedule. The learning rates are $0.5$2 in Stage 1, $0.5$3 in Stage 2, and $0.5$4 in Stage 3; warmup is 0.03 only in Stage 3, and the maximum sequence length is 16384 in Stage 3. Compute is reported as 32 NVIDIA A800 GPUs, and the main text additionally mentions a cosine learning-rate schedule from $0.5$5 to $0.5$6 as an overall scheduler detail.

The paper also specifies what is not used. It does not report auxiliary depth losses such as direct $0.5$7 losses on depth pixels, scale-invariant log losses, or gradient and edge-aware terms, and it does not describe scale-alignment procedures.

5. Empirical effects on embodied reasoning and representation quality

Training on GEM-4M with generative depth supervision improves spatial understanding markedly relative to the same backbone trained with supervised fine-tuning but without the depth objective (Zhao et al., 27 May 2026). On VSI-Bench (All score), Qwen3-VL-2B rises from 50.4 to 62.8 with GEM, and the 8B model rises from 57.9 to 70.6. On MMSI-Bench (All), 8B GEM achieves 35.3 versus 27.7 for Qwen3-VL-8B, and 2B GEM achieves 30.6 versus 23.6 for Qwen3-VL-2B. On CV-Bench (All), GEM-8B reaches 86.6 versus 85.1 for Qwen3-VL-8B, and GEM-2B reaches 81.4 versus 80.0 for Qwen3-VL-2B. On EmbSpatial (All), GEM-8B records 79.4 versus 77.7 for Qwen3-VL-8B.

The paper also reports results on RefSpatial, Where2Place, and RoboSpatial. It states that GEM-8B achieves best overall averages and gives examples including RefSpatial All 44.4 versus Qwen3-VL-8B-SFT 45.8, superior Where2Place All 65.0 with strong seen and unseen scores, and RoboSpatial 66.9. The reported emphasis throughout is that the gains are strongest on tasks requiring distance, layout, and placement reasoning.

The ablation studies isolate the contribution of depth supervision. Replacing depth generation with RGB reconstruction degrades spatial performance: CV-Bench falls from 81.1 to 80.9, VSI All from 63.0 to 60.0, and RoboSpatial from 48.9 to 44.6. Direct end-to-end co-training without the progressive stages also underperforms the default progressive depth setup, with CV-Bench 79.7, VSI All 57.6, and RoboSpatial 44.0. Removing depth supervision entirely, represented by Qwen3VL-{2B,8B}-SFT baselines, consistently lowers accuracy across tables, especially on distance questions in VSI-Bench.

These results support two conclusions stated in the paper. First, depth is a more effective supervisory signal for spatial awareness than RGB reconstruction. Second, progressive training is important to avoid interference between the DiT generator and backbone token spaces. The paper does not report explicit scaling laws or dataset-mixture sensitivity curves, but it shows gains across many benchmarks and at both 2B and 8B model scales.

A common misunderstanding would be to read GEM-4M as a depth-estimation benchmark. The paper does not present it in that way: it does not report RMSE, AbsRel, SqRel, SILog, or $0.5$8 thresholds on GEM-4M, even though it lists those metrics as common reference formulas. Its focus is representational utility for embodied reasoning and downstream action prediction rather than standalone monocular depth evaluation.

6. Role in GEM-VLA, practical usage, and limitations

GEM-4M underpins GEM’s geometry-aware tokens and is directly tied to the downstream action model GEM-VLA. In downstream action finetuning, a flow-matching action loss supervises a DiT-based policy head $0.5$9 conditioned on GEM’s multimodal tokens:

$$\mathcal{L}_{\mathrm{CE}} = - \sum_{i=1}^{T} \log p_\theta \left( y_i \mid y_{< i}, \mathbf{h}_o, \mathbf{h}_l \right),$$0

with the total fine-tuning loss

$$\mathcal{L}_{\mathrm{CE}} = - \sum_{i=1}^{T} \log p_\theta \left( y_i \mid y_{< i}, \mathbf{h}_o, \mathbf{h}_l \right),$$1

The reported downstream results are strong. GEM-VLA achieves a record 96.1% average success on LIBERO across the Spatial, Object, Goal, and Long suites, with suite scores of 99.0, 98.8, 97.1, and 89.3 respectively, surpassing baselines including DepthVLA at 94.9%. On real-world evaluations on a UR5 platform, GEM-VLA averages 43% success against a prior state of the art of 28.7% and shows higher progress scores on long-horizon tasks. On SimplerEnv WidowX, GEM-VLA leads average success at 67.0%, beating Qwen3VL-SFT-VLA at 61.5% and matching or exceeding other spatially enhanced policies. The ablations further report clear drops when depth supervision is removed during fine-tuning, especially on long-horizon and deformable tasks (Zhao et al., 27 May 2026).

For reproduction, the recommended recipe is explicit. One initializes the Qwen3-VL backbone, attaches the 2-layer MLP connector and Sana DiT depth head, and follows the three-stage schedule described above. For GEM-VLA fine-tuning, the paper recommends attaching an RDT2-style action expert and training for 50k steps with $$\mathcal{L}_{\mathrm{CE}} = - \sum_{i=1}^{T} \log p_\theta \left( y_i \mid y_{< i}, \mathbf{h}_o, \mathbf{h}_l \right),$$2, with appendix settings including batch size 256 and action chunk size 32, using observations from three camera views: top, left-wrist, and right-wrist. DepthAnything v3 is run offline to generate pseudo-depth maps for all samples lacking ground-truth depth.

Availability is stated at the project website, with code, models, and datasets available at https://zhaorw02.github.io/GEM/. The paper does not provide a direct download link, license text, or storage footprint in the main body.

The limitations are specific and consequential. Pseudo-depth from monocular estimation may be relative, noisy, or biased by scene textures and lighting, and the paper does not describe metric calibration or quality metrics. Models trained with such supervision could inherit bias in perceived geometry, affecting manipulation accuracy in cases such as specular or deformable objects. GEM-4M also aggregates many public datasets whose distributions may overrepresent household and lab settings relative to industrial or outdoor domains. Although SAM3 confidence thresholding removes low-quality masks, segmentation bias may remain. For real robot deployment, the paper identifies spatial misperception as a safety risk and notes best practices such as collision checking, conservative grasping policies, and human oversight in new environments. It also states that GEM-VLA has not been pretrained on large robot action corpora, leaving further scaling and domain coverage as future work.

Taken together, GEM-4M is defined by the coupling of instruction-centric embodied supervision with a generative depth objective. The dataset’s significance lies not merely in scale but in the requirement that visual samples support both semantic supervision and geometry formation, which the reported experiments link to improved spatial Q&A, grounding, planning, and robot action performance.