---
title: Vision-Geometry-Action (VGA) Paradigms
url: https://www.emergentmind.com/topics/vision-geometry-action-vga
type: topic
---

# Vision-Geometry-Action (VGA) Paradigms

Vision-Geometry-Action (VGA) denotes a family of architectures and learning paradigms that place explicit or native 3D geometry between visual observation and control, rather than treating policy learning as a direct mapping from 2D images to actions. In manipulation, VGA has been formalized as learning a mapping \(f:v\rightarrow G\), where \(v\) denotes raw visual and proprioceptive inputs and \(G\) denotes native 3D geometric representations that directly condition physical actions [2604.12908]. StemVLA defines the same shift as extending traditional Vision–Language–Action by explicitly introducing a structured 3D spatial-geometry representation between the 2D visual input and the robot’s control commands [2602.23721]. In autonomous driving, DVGT-2 presents VGA as an end-to-end model that jointly outputs dense geometry and trajectory planning for the current frame, advocating dense 3D geometry as the critical cue for decision-making [2604.00813].

## 1. Conceptual basis and scope

The central VGA claim is that physical action is fundamentally geometric. The VGA model for robotic manipulation states that “physical actions are fundamentally defined by geometric properties like 3D positions and spatial relationships,” and argues that the foundation for generalizable robotic control should be a vision-geometry backbone rather than widely adopted vision-language or video models [2604.12908]. In driving, DVGT-2 advances a parallel argument: vehicles operate in a 3D world, and dense 3D geometry provides the most comprehensive information for decision-making [2604.00813]. StemVLA frames the same issue negatively, observing that direct 2D-to-action mappings tend to ignore depth, occlusions, object layouts, and temporal world dynamics, which limits long-horizon planning under dynamic scenes [2602.23721].

This geometric emphasis does not imply a single canonical architecture. Some systems are geometry-first from the backbone upward, as in VGA, DVGT-2, and GAM; others are geometry-enhanced variants of VLA policies, as in GLaD, VEGA, GeoAware-VLA, G\(^3\)VLA, and OBEYED-VLA [2512.09619][2605.10485][2509.14117][2606.24472][2512.22519]. A plausible implication is that VGA is better understood as a design principle—preserve, reconstruct, or distill 3D structure into the action pathway—than as a single model family.

A second conceptual axis is temporal scope. Some VGA systems focus on instantaneous or current-frame geometry, while others explicitly model future or historical structure. StemVLA introduces a 3D Future Spatial-Geometry World Knowledge Predictor and a 4D Historical Spatiotemporal Aggregator; GAM predicts future geometric tokens through a causal future predictor inserted into a pretrained Geometric Foundation Model; DVGT-2 uses temporal causal attention with historical caches for online planning [2602.23721][2606.17046][2604.00813]. This suggests that VGA increasingly overlaps with world modeling when task success depends on anticipation rather than instantaneous localization.

## 2. Architectural patterns in manipulation and driving

One major VGA pattern replaces or augments a standard visual backbone with a pretrained geometric model. GeoAware-VLA replaces a trainable 2D image encoder with a frozen, pretrained geometric encoder, VGGT, and a lightweight projection head that maps geometric features into the policy latent space [2509.14117]. VGA similarly replaces conventional language or video backbones with a pretrained 3D world model, VGGT, and injects geometry into the action decoder through Progressive Volumetric Modulation at every decoder layer [2604.12908]. GEAR-VLA keeps a frozen 2D ViT from Qwen2.5-VL, adds a trainable VGGT 3D spatial encoder through a zero-initialized “3D block,” and connects the VLM to a gradient-decoupled DiT continuous action expert via latent action tokens [2606.08530]. GAM repurposes a pretrained Geometric Foundation Model by splitting it at an intermediate layer, inserting a causal future predictor, and routing predicted tokens through the remaining GFM blocks so that a single backbone produces both future geometry and actions [2606.17046].

A second pattern retains the VLA backbone and injects geometry through alignment. GLaD augments UniVLA with a frozen VGGT teacher and a small feature-alignment MLP, but aligns the LLM hidden states corresponding to visual tokens at the final LLaMA layer rather than altering the vision encoder directly [2512.09619]. VEGA performs alignment earlier: it extracts patch-level tokens from the student DINOv2 branch before FiLM language conditioning and aligns them with frozen DINOv2-FiT3D features using a lightweight projector [2605.10485]. G\(^3\)VLA adds a camera-aware geometric module to pretrained VLAs, combining intrinsic-conditioned ray embeddings, Projective Positional Encoding, and bidirectional cross-view fusion, while leaving the action space and imitation objective unchanged [2606.24472].

A third pattern disentangles perception and control more explicitly. OBEYED-VLA introduces a perception module that grounds multi-view inputs into task-conditioned, object-centric, and geometry-aware observations before passing them to a pretrained VLA policy [2512.22519]. PointACT adopts a dual-system design in which a frozen Qwen2.5-VL provides 2D semantic tokens while a Point Transformer v3 encoder supplies hierarchical 3D point features that are integrated directly into the action decoder through multi-scale point-action interaction [2605.21414]. In autonomous driving, VLGA instantiates a comparable separation through a Mixture-of-Transformers backbone with understanding, perception, geometry, and action experts, where geometry is a dedicated expert supervised by dense pointmap regression against LiDAR [2606.12396].

## 3. Geometry representations and supervision

VGA methods differ sharply in how geometry is represented. DVGT-2 predicts a local point-map representation, with per-camera 3D points
\[
P_t^v(u)=D_t^v(u)\,K_v^{-1}[u_x,u_y,1]^T,
\]
and can also be interpreted as learning a continuous occupancy field \(G_t(\mathbf p)\) conditioned on spatio-temporal image features [2604.00813]. VLGA supervises dense per-pixel pointmaps against LiDAR using a confidence-weighted \(L_1\) loss on predicted 3D points and confidences [2606.12396]. OBEYED-VLA uses masked depth maps \(Z_t^{base,geom}\) and \(Z_t^{wrist,geom}\), optionally back-projectable to point clouds via camera intrinsics \(K\), but in practice keeps depth as a single-channel image for the policy front end [2512.22519].

Other systems keep geometry in latent token space rather than explicit depth or LiDAR at policy time. GLaD uses pooled VGGT features \(g_t\) as a teacher target and minimizes a mean-squared distillation loss together with the latent-action loss,
\[
\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{VLA}}+\lambda\,\mathcal{L}_{\mathrm{distill}},
\]
so that geometry is fused into the final multimodal hidden states that drive action prediction [2512.09619]. VEGA applies a cosine alignment loss,
\[
L_{\mathrm{total}}=L_{\mathrm{action}}+\lambda L_{\mathrm{align}},
\]
with \(\lambda=0.1\), directly on the visual encoder output and removes both teacher and projector at inference [2605.10485]. GeoAware-VLA adds no explicit depth- or point-cloud loss at all; its regularizer is the act of freezing the geometric backbone so that projected features inherit VGGT’s multi-view geometric invariances [2509.14117].

Several VGA systems go further by making geometry itself a structured action-conditioning medium. GST-VLA converts frozen dense depth and semantic patch features into \(N_g=128\) anisotropic 3D Gaussian primitives, each parameterized by a residual mean \(\mu\in\mathbb{R}^3\), log-scale covariance \(\log \sigma\in\mathbb{R}^3\), and opacity \(\alpha\in(0,1)\), then supervises four structured Depth-Aware Chain-of-Thought sequences before action generation [2603.09079]. PointACT uses multi-scale point-cloud features across five Point Transformer v3 hierarchy levels and lets evolving action tokens attend to local and global 3D geometry within bottleneck windows [2605.21414]. G\(^3\)VLA, by contrast, injects calibrated camera structure through ray embeddings and PRoPE biases, and adds a point-head distillation loss from either ground-truth point maps or confidence-gated \(\pi^3\)X teacher predictions [2606.24472].

## 4. Optimization strategies and action coupling

A recurring feature of VGA is multi-objective optimization. VGA jointly trains action, camera, and depth heads on a shared VGGT backbone, with
\[
\mathcal{L}=\mathcal{L}_{\mathrm{action}}+\mathcal{L}_{\mathrm{camera}}+\mathcal{L}_{\mathrm{depth}},
\]
using LoRA over a frozen VGGT and decoupling camera and depth heads at inference so that only the lightweight action head runs at approximately \(10\) Hz [2604.12908]. DVGT-2 uses geometry reconstruction, pose regression, and planning losses together, while its streaming formulation introduces temporal causal attention and a sliding-window cache so that online planning does not require computationally expensive batch processing of multi-frame inputs [2604.00813]. VLGA uses a two-stage schedule: geometry warm-up with \(\lambda_{\mathrm{geom}}=1.0\), followed by joint finetuning with \(L_{\mathrm{joint}}=L_{\mathrm{act}}+\lambda_{\mathrm{geom}}L_{\mathrm{geom}}\) and \(\lambda_{\mathrm{geom}}=0.1\) [2606.12396].

Action generation mechanisms also vary. GEAR-VLA explicitly separates coarse semantic action learning from fine continuous control: an embodied VLM is pretrained autoregressively on language, grounding, FAST-action tokens, and latent action IDs, after which a DiT expert is trained with flow matching on continuous relative end-effector actions \(A_{t,i}=T_{ee,t}^{-1}T_{ee,t+i}\) while stop-gradient shields the VLM from the continuous-control loss [2606.08530]. GAM keeps future geometry and action in a common geometric token manifold by predicting latent tokens at split layer \(L_s=12\), then decoding both future depth and next action chunk through the remaining GFM layers; its total loss is
\[
L_{\mathrm{total}}=\lambda_{\mathrm{act}}L_{\mathrm{act}}+\lambda_{\mathrm{feat}}L_{\mathrm{feat}}+\lambda_{\mathrm{depth}}L_{\mathrm{depth}},
\]
with \(\lambda_{\mathrm{act}}=3\), \(\lambda_{\mathrm{feat}}=1\), and \(\lambda_{\mathrm{depth}}=3\) [2606.17046]. GST-VLA likewise couples geometric reasoning and action synthesis through
\[
L=L_{\mathrm{flow}}+0.5L_{\mathrm{CoT}}+0.1L_{\mathrm{depth}},
\]
trained in three stages to avoid collapse [2603.09079].

These choices indicate a broader methodological point. In VGA, geometry is rarely treated as a passive auxiliary target. Instead, it is used to constrain the latent space, regularize cross-view consistency, supply structured intermediate targets, or directly modulate the action decoder. This suggests that the distinctive property of VGA is not merely additional supervision, but a tighter interface between geometric state and action generation.

## 5. Empirical performance across benchmarks

Reported VGA gains span manipulation and autonomous driving, but the strongest improvements are usually concentrated in spatially sensitive, out-of-distribution, or embodiment-transfer settings.

| System | Setting | Reported result |
|---|---|---|
| GLaD [2512.09619] | LIBERO | 94.1% average success, vs UniVLA 92.5% |
| VGA [2604.12908] | LIBERO / real robot OOD | 98.1% LIBERO average; 58% OOD real-world average vs \(\pi_{0.5}\) 52% |
| GAM [2606.17046] | LIBERO / LIBERO-Plus | 97.6% / 85.5%; 6.9 ms per step |
| GEAR-VLA [2606.08530] | cross-embodiment / grasping | 85.9% on AgileX; 81.0% on pretraining-unseen LDT-01; 90.1% on a 6,360-trial benchmark with 212 unseen objects |
| DVGT-2 [2604.00813] | NAVSIM / nuScenes | PDMS 88.6; EPDMS 88.9; L2 avg(1/2/3 s)=0.78 m |
| VLGA [2606.12396] | nuScenes / Bench2Drive | average L2 0.50 m, 3-second collision 0.18%, Driving Score 79.08 |

Within manipulation, several papers report that the largest gains occur under viewpoint change or explicit spatial challenge. GeoAware-VLA reports zero-shot novel-view LIBERO averages of 82.6 for GeoAware-BAKU versus 37.9 for the BAKU baseline, and 77.9 for GeoAware VQ-BeT versus 41.4 for the VQ-BeT baseline [2509.14117]. VEGA raises the RoboTwin 2.0 six-task average from 56.0% / 22.7% to 67.5% / 30.7% in Easy / Hard settings, and increases real-world average success from 48% to 60% [2605.10485]. PointACT reports 96.0% on LIBERO versus 93.1% for reproduced EO1, and 82.3% on RLBench-10Tasks versus 73.2% for EO1 [2605.21414]. GST-VLA reports 96.4% on LIBERO and 80.2% on SimplerEnv, with the largest gains on precision-demanding tasks such as Close-Drawer [2603.09079].

Robustness analyses refine this picture. GLaD’s LIBERO breakdown shows improvements on OBJECT, GOAL, and LONG—97.4% versus 95.4%, 94.4% versus 91.9%, and 89.4% versus 87.5%—while SPATIAL is 95.0% versus 95.2%; on LIBERO-PRO object perturbations, it reports GOAL 81% versus 62%, LONG 54% versus 47%, and OBJECT 86% versus 82%, but position perturbation remains low [2512.09619]. G\(^3\)VLA improves \(\pi_0\) on LIBERO from 84.6% to 87.0% with \(\pi^3\)X supervision and to 88.1% with ground truth, and in real-robot OOD viewpoint tests reports 70.8→83.3 on Pouring Nut and 25→50 on Test Tube [2606.24472]. In driving, VLGA attributes lower collision rates to dense pointmap supervision, reducing collision average from 0.169% in the baseline to 0.149% with a geometry expert only and to 0.136% with dense pointmap supervision [2606.12396].

## 6. Design debates, limitations, and future directions

A common misconception is that VGA necessarily requires depth sensors or manual 3D annotations at runtime. Multiple papers explicitly reject this. GLaD reports improved spatial reasoning and policy generalization “without requiring explicit depth sensors or 3D annotations” [2512.09619]. VEGA discards its frozen teacher and lightweight projector at inference, introducing zero extra runtime cost [2605.10485]. GeoAware-VLA relies on a frozen geometric encoder rather than explicit 3D labels at policy time [2509.14117]. G\(^3\)VLA can be trained from confidence-gated teacher point predictions and, at inference, keeps only RGB, proprioception, language, and camera calibration [2606.24472].

Another active question is where geometry should enter the policy. GLaD argues for final-layer hidden-state alignment inside the LLM so that geometric priors are “deeply fused” with 2D semantic features and language context [2512.09619]. VEGA argues the opposite: alignment should happen at the visual encoder output, before language fusion, because LLM-level visual tokens are already entangled with semantics and require empirical layer search [2605.10485]. G\(^3\)VLA provides a third perspective, reporting that geometric transfer is most effective when geometry-aware tokens have direct access to the action generation pathway; gains are attenuated on GR00T 1.5 relative to \(\pi_0\) [2606.24472]. Taken together, these results suggest that VGA remains an open design space rather than a settled recipe.

The literature also records persistent limitations. GLaD notes that position-perturbation and compositional generalization remain low (\(<15\%\)), leaving room for explicit layout modeling or dynamic scene inference [2512.09619]. VEGA depends on a strong 3D-aware teacher, and still fails on thin or occluded parts, grasp-pose misalignments, and collisions near workspace boundaries [2605.10485]. GeoAware-VLA notes dependence on a large pretrained geometric model [2509.14117]. StemVLA reports gripper-only manipulation, limited geometric and material diversity with weaker sim-to-real transfer, and occasional jerky motions from the Diffusion Transformer [2602.23721]. G\(^3\)VLA shows that teacher reliability matters: on RoboTwin2.0 handover_block, \(\pi^3\)X supervision yields 41.0% while ground truth yields 49.0%, because teacher failure in clean simulation harms distilled supervision [2606.24472].

Future directions are already explicit in the source papers. GLaD proposes online or self-supervised geometry distillation from depth-prediction models, multi-view or temporal geometry cues such as extending VGGT’s \(T=32\) sequence, integration of 3D scene graphs or explicit object-pose estimators, and real-world deployment using only monocular cameras [2512.09619]. VEGA proposes stronger or more diverse 3D teachers, adaptation to other visual backbones, and coupling spatial alignment with grasp-quality estimation or closed-loop feedback for error recovery [2605.10485]. GEAR-VLA extends VGA toward cross-robot transfer through embodiment canonicalization, using robot-specific state projectors and embodiment-invariant relative end-effector actions to confine embodiment differences to the low-level interface [2606.08530]. In driving, DVGT-2 and VLGA indicate a parallel trajectory in which dense geometry is streamed or reconstructed online and used directly for planning under open-loop and closed-loop evaluation [2604.00813][2606.12396].

Across manipulation and autonomous driving, VGA therefore names a geometry-first reformulation of policy learning: action is conditioned not merely on visual appearance or linguistic semantics, but on dense or structured 3D state, whether represented as point maps, depth fields, Gaussian primitives, point clouds, calibrated token streams, or geometry-aware latent features. Current results associate that reformulation with stronger spatial reasoning, improved robustness to viewpoint and appearance shifts, and competitive or state-of-the-art performance on several benchmarks, while leaving unresolved the most effective locus of geometric grounding, the role of teacher supervision, and the balance between explicit reconstruction and latent alignment [2604.12908][2604.00813].

Source: https://www.emergentmind.com/topics/vision-geometry-action-vga