---
title: 'GEM-VLA: Geometry-Enhanced VLA Control'
url: https://www.emergentmind.com/topics/gem-vla
type: topic
---

# GEM-VLA: Geometry-Enhanced VLA Control

Searching arXiv for "GEM-VLA" and closely related VLA papers to ground the article in published work.
arxiv_search.query({"search_query":"all:\"GEM-VLA\" OR all:\"Generative Supervision Helps Embodied Intelligence\" OR all:\"G3VLA\"","start":0,"max_results":10,"sort_by":"submittedDate","sort_order":"descending"})
arxiv_search.query({"search_query":"all:\"GEM-VLA\" OR all:\"Generative Supervision Helps Embodied Intelligence\" OR all:\"G^3VLA\"","start":0,"max_results":10,"sort_by":"submittedDate","sort_order":"descending"})
arxiv_search.query({"search_query":"all:\"Generative Supervision Helps Embodied Intelligence\"","start":0,"max_results":5,"sort_by":"submittedDate","sort_order":"descending"})
GEM-VLA is the vision–language–action instantiation of GEM, the “Generative-supervised Embodied vision-language Model” introduced to couple high-level semantic competence with physically grounded scene understanding through depth generation. In its most concrete published usage, GEM-VLA denotes a robot control policy that uses the GEM vision–language backbone as its perception and reasoning core and augments it with a Diffusion Transformer–based action expert for continuous control [2605.28548]. The central claim of this line of work is that making the underlying VLM generate depth maps during pre-training and action fine-tuning yields internal visual representations that are more informative for embodied manipulation than standard text-guided pre-training alone.

## 1. Definition, scope, and nomenclature

The term “GEM-VLA” is used most specifically for the deployed action model in “GEM: Generative Supervision Helps Embodied Intelligence,” where it names the full VLA system built on the GEM backbone [2605.28548]. In that formulation, GEM-VLA is not merely a geometry-aware perception stack; it is a complete control model trained and evaluated in simulation and on real robots.

A common source of ambiguity is that adjacent arXiv literature uses “GEM” for distinct technical programs. G$^3$VLA describes a camera-aware geometric module that injects calibrated structure into pretrained VLAs through intrinsic-conditioned ray embeddings, PRoPE, and bidirectional cross-view fusion, and states that such a module can turn a standard VLA into a “GEM-VLA” in an informal sense [2606.24472]. Separately, the Gaussian Evolution Model for autonomous driving presents GEM as a continuous 4D Gaussian world model and suggests that such a representation could serve as the visual or physical “world core” of a VLA-style system [2605.17682]. This suggests that “GEM-VLA” has both a specific meaning—an embodied control architecture in [2605.28548]—and a broader informal meaning in which “GEM” denotes geometry-enhanced or world-model-based VLA design.

Within the specific GEM paper, however, the defining idea is narrower and more concrete: geometry is introduced through a generative depth objective embedded directly into the VLM, and GEM-VLA is the action-policy instantiation of that geometry-conditioned backbone [2605.28548].

## 2. Core architecture

The base GEM architecture has three components: a VLM backbone \(M_\theta\), a lightweight connector \(C_\phi\), and a depth generative head \(G_\psi\) [2605.28548]. The backbone is based on Qwen3-VL, with 2B and 8B variants. Given visual observation \(o\) and language instruction \(l\), the backbone produces multimodal token representations
\[
\mathbf{h} = (\mathbf{h}_o, \mathbf{h}_l) = M_\theta(o, l),
\]
where \(\mathbf{h}_o\) are visual token features and \(\mathbf{h}_l\) are language token features.

The standard supervised text objective is
\[
\mathcal{L}_{\text{CE}} = -\sum_{i=1}^{T} \log p_\theta(y_i \mid y_{<i}, \mathbf{h}_o, \mathbf{h}_l).
\]
The connector \(C_\phi\) is a 2-layer MLP that projects visual tokens into the conditioning space of the depth generator,
\[
\mathbf{c} = C_\phi(\mathbf{h}_o).
\]

The depth head \(G_\psi\) is a DiT-based generator, implemented using Sana, that predicts the depth map \(d\) of the observation from \(\mathbf{c}\) through a flow-matching objective. During pre-training, GEM is therefore not limited to instruction following, QA, reasoning, and planning; it is also required to generate scene depth from internal visual features.

GEM-VLA extends this architecture with a DiT-based action expert \(A_\omega\) following RDT2 [2605.28548]. The conditioning for the action module is not restricted to the final visual tokens. Instead, key–value tokens from attention blocks in the backbone over the multimodal observation history \(\mathcal{O}=\{o_{1:t},l\}\) are extracted and used as the conditioning representation \(\mathbf{c}_{\text{act}}\). The resulting system couples a multimodal VLM backbone, a depth generator, and a diffusion policy within a single end-to-end trainable stack.

This design clarifies an important point: GEM-VLA does not use depth as an explicit extra sensor channel at policy input. Depth acts as an auxiliary generative target that shapes the backbone’s internal features, and those depth-competent features then condition the action expert [2605.28548].

## 3. Objectives and optimization strategy

The depth generator is trained with a flow-matching loss
\[
\mathcal{L}_{\text{flow}} =
\mathbb{E}_{d,\, t \sim \mathcal{U}(0,1),\, \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})}
\left[
\left\|
\mathbf{v}_t(\mathbf{x}_t, \mathbf{c}) - \mathbf{u}_t(\mathbf{x}_t \mid d)
\right\|^2
\right].
\]
During GEM pre-training, the joint objective is
\[
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{flow}},
\]
with \(\lambda = 0.1\) in the reported experiments [2605.28548].

A distinctive feature of GEM is its progressive three-stage optimization schedule. In Stage 1, the VLM \(M_\theta\) and DiT head \(G_\psi\) are frozen, and only the connector \(C_\phi\) is trained with \(\mathcal{L}_{\text{flow}}\) to align feature spaces. In Stage 2, the VLM remains frozen while \(C_\phi\) and \(G_\psi\) are trained to warm up the depth generator. In Stage 3, all components are unfrozen and optimized jointly under \(\mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{flow}}\). The paper explicitly contrasts this curriculum with direct end-to-end co-training and reports that the progressive scheme is superior [2605.28548].

The same pattern is preserved when moving from GEM to GEM-VLA. For an action chunk \(\mathbf{a}\), the noised action state is
\[
\mathbf{a}_t = (1 - t)\epsilon + t\mathbf{a},
\qquad
t \sim \mathcal{U}(0,1),\ \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I}),
\]
and the action expert minimizes
\[
\mathcal{L}_{\text{action}} =
\mathbb{E}_{\mathcal{O}, \mathbf{a}, \epsilon, t}
\left[
\left\|
\mathbf{v}_t(\mathbf{a}_t, \mathbf{c}_{\text{act}}) -
\mathbf{u}_t(\mathbf{a}_t \mid \mathbf{a})
\right\|_2^2
\right].
\]
During VLA fine-tuning, the reported total loss is
\[
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{action}} + \lambda \mathcal{L}_{\text{flow}},
\]
again with \(\lambda = 0.1\) [2605.28548].

The ablations are central to the interpretation of this design. Against Qwen3-VL-SFT baselines, GEM improves VSI-Bench “All” from 60.0 to 62.8 at 2B and from 68.6 to 70.6 at 8B, MMSI-Bench from 28.9 to 30.6 and from 32.8 to 35.3, and RoboSpatial from 44.6 to 47.4 and from 65.4 to 66.9. The paper further reports that depth is a more effective generative supervision target than RGB reconstruction: RGB supervision gives 80.9 on CV-Bench, 60.0 on VSI-Bench All, and 44.6 on RoboSpatial; direct end-to-end co-training gives 79.7, 57.6, and 44.0; the default depth-based progressive setting gives 81.1, 63.0, and 48.9 [2605.28548]. The authors interpret these differences as evidence that depth particularly enhances distance and spatial relation understanding.

## 4. Data regime and supervision sources

GEM-VLA inherits its perceptual priors from GEM pre-training on GEM-4M, a large-scale embodied QA corpus with depth supervision [2605.28548]. GEM-4M is not itself an action dataset; rather, it is used to pre-train the VLM backbone on grounding, reasoning, planning, and geometry before downstream policy learning.

| Component | Scale | Sources |
|---|---:|---|
| Embodied grounding | about 1M QA pairs + ~100k additional grounding annotations | PACO-LVIS, RoboPoint, RoboAfford++, ShareRobot, Roborefit, RoboMind, DROID, Agibot, Open-X Embodiment |
| Physical and spatial reasoning | 100k manually annotated 3D spatial samples | ScanNet, ScanNet++, ARKitScenes, plus MindCube, ViCA, SPAR, VSI-590K |
| Spatiotemporal planning | about 1M spatiotemporal QA pairs + ~50k additional QA pairs | RoboVQA, Robo2VLM, RefSpatial, RoboCOIN, RoboMind, Agibot World |

For grounding data, annotations include bounding boxes or points, normalized to \([0, 1000]\) in both directions to accommodate varying resolutions [2605.28548]. For planning data, the pipeline described in the paper extracts sub-task frames, identifies manipulated objects via Qwen3, generates masks via SAM3, tracks trajectories using CoTracker3, and then instantiates planning QA templates following RoboVQA and MolmoACT.

Depth supervision comes from both ground-truth depth and pseudo-depth produced by DepthAnythingv3 [2605.28548]. The paper emphasizes that GEM-4M is used to train geometry-aware multimodal representations, whereas action-policy training for GEM-VLA is subsequently performed on robot datasets such as LIBERO, BridgeDataV2 and related data for SimplerEnv, and custom real-robot UR5 trajectories.

## 5. Control formulation and empirical performance

At policy level, GEM-VLA conditions on multimodal observation history rather than a single frame. The backbone processes image observations and language instructions, while the action expert consumes the extracted conditioning representation \(\mathbf{c}_{\text{act}}\) and predicts continuous action chunks through a diffusion policy [2605.28548]. In real-robot experiments the reported setup uses three cameras—top view, left wrist, and right wrist—with action chunk size 32, 50k training steps, global batch size 256, and learning rate \(1\times 10^{-5}\). For LIBERO, GEM-2B plus the action expert is fine-tuned for 20k steps following StarVLA.

The reported benchmark results are summarized below.

| Benchmark | Metric | GEM-VLA |
|---|---|---:|
| LIBERO | Average success | 96.1% |
| SimplerEnv WidowX | Average success | 67.0% |
| Real-world UR5 tasks | Average success | 43% |

On LIBERO, GEM-VLA reports 99.0 on Spatial, 98.8 on Object, 97.1 on Goal, 89.3 on Long, and 96.1 on average, exceeding 94.9 average for both DepthVLA and Qwen3VL-SFT-VLA in the table reproduced by the paper [2605.28548]. On SimplerEnv WidowX, GEM-VLA achieves 58.0 on Put Carrot on Plate, 84.0 on Put Eggplant in Basket, 82.0 on Put Spoon on Towel, 44.0 on Stack Blocks, and 67.0 on average. In real-world UR5 evaluation, the paper reports 43% average success across challenging tasks, compared with 28.7% for the previous state of the art.

These evaluations support a specific interpretation of the geometry signal. The gains do not arise from feeding depth maps directly to the controller; rather, they arise because the same visual tokens that condition the action expert are continuously constrained to remain predictive of depth. The real-robot ablation described in the paper reinforces this: freezing the depth head and training only with \(\mathcal{L}_{\text{action}}\) reduces performance on nearly all tasks, while even the version retaining only pre-trained depth priors still outperforms a VLA based on Qwen3-VL-SFT [2605.28548].

## 6. Relation to other geometry-aware VLAs and open issues

GEM-VLA belongs to a broader movement that seeks stronger geometric inductive bias in VLA systems, but its mechanism differs from other recent proposals. G$^3$VLA introduces geometry through calibrated camera structure—intrinsic-conditioned ray embeddings, PRoPE, and cross-view fusion—without altering the action space or imitation objective [2606.24472]. GEM-VLA instead introduces geometry through a generative depth objective embedded into VLM pre-training and preserved during policy learning [2605.28548]. A plausible implication is that these approaches are complementary rather than mutually exclusive: one injects calibration-aware projective structure into tokens, while the other makes those tokens depth-competent through generation.

The Gaussian Evolution Model offers a different point in the design space. It represents dynamic scenes with explicit 4D Gaussian primitives and suggests this representation as the visual or physical “world core” of a possible VLA [2605.17682]. Relative to that work, GEM-VLA is not a continuous-time world model and does not represent objects as persistent Gaussian primitives. Its emphasis is on improving embodied control by altering the supervision imposed on VLM features rather than on replacing the state representation itself.

Several limitations remain explicit in the GEM paper. GEM-VLA has not yet been pre-trained on truly large-scale robot action datasets; all reported action learning is task-scale or benchmark-scale [2605.28548]. Sim-to-real gaps persist despite the reported gains. Depth supervision depends on ground-truth depth or monocular depth teachers such as DepthAnythingv3, which may introduce bias. The authors also identify future directions: scaling model size and data further, pre-training GEM-VLA on large-scale robot datasets, combining generative world modeling with depth supervision, and enlarging GEM-4M with richer 3D and dynamical annotations.

Taken together, these results position GEM-VLA as a specific answer to a recurrent question in embodied AI: whether generation can improve control. In the formulation of [2605.28548], the answer is affirmative precisely when the generative target is geometrically structured—depth rather than generic RGB reconstruction—and when that target is made intrinsic to both pre-training and downstream policy optimization.

Source: https://www.emergentmind.com/topics/gem-vla