Vision-Geometry-Action (VGA) Paradigms

• VGA is a design principle that inserts explicit 3D geometric representations between sensory inputs and control actions, enhancing spatial reasoning and robustness.
• It applies to robotic manipulation and autonomous driving, replacing direct 2D-to-action mappings with dense geometry cues for improved planning.
• VGA systems optimize through multi-objective training with pretrained geometric encoders, improving performance in spatially challenging and out-of-distribution scenarios.

Vision-Geometry-Action (VGA) denotes a family of architectures and learning paradigms that place explicit or native 3D geometry between visual observation and control, rather than treating policy learning as a direct mapping from 2D images to actions. In manipulation, VGA has been formalized as learning a mapping $f:v\rightarrow G$, where $v$ denotes raw visual and proprioceptive inputs and $G$ denotes native 3D geometric representations that directly condition physical actions (Song et al., 14 Apr 2026). StemVLA defines the same shift as extending traditional Vision–Language–Action by explicitly introducing a structured 3D spatial-geometry representation between the 2D visual input and the robot’s control commands (Xiao et al., 27 Feb 2026). In autonomous driving, DVGT-2 presents VGA as an end-to-end model that jointly outputs dense geometry and trajectory planning for the current frame, advocating dense 3D geometry as the critical cue for decision-making (Zuo et al., 1 Apr 2026).

1. Conceptual basis and scope

The central VGA claim is that physical action is fundamentally geometric. The VGA model for robotic manipulation states that “physical actions are fundamentally defined by geometric properties like 3D positions and spatial relationships,” and argues that the foundation for generalizable robotic control should be a vision-geometry backbone rather than widely adopted vision-language or video models (Song et al., 14 Apr 2026). In driving, DVGT-2 advances a parallel argument: vehicles operate in a 3D world, and dense 3D geometry provides the most comprehensive information for decision-making (Zuo et al., 1 Apr 2026). StemVLA frames the same issue negatively, observing that direct 2D-to-action mappings tend to ignore depth, occlusions, object layouts, and temporal world dynamics, which limits long-horizon planning under dynamic scenes (Xiao et al., 27 Feb 2026).

This geometric emphasis does not imply a single canonical architecture. Some systems are geometry-first from the backbone upward, as in VGA, DVGT-2, and GAM; others are geometry-enhanced variants of VLA policies, as in GLaD, VEGA, GeoAware-VLA, G$^3$VLA, and OBEYED-VLA (Guo et al., 10 Dec 2025, Wang et al., 11 May 2026, Abouzeid et al., 17 Sep 2025, Peng et al., 23 Jun 2026, Vo et al., 27 Dec 2025). A plausible implication is that VGA is better understood as a design principle—preserve, reconstruct, or distill 3D structure into the action pathway—than as a single model family.

A second conceptual axis is temporal scope. Some VGA systems focus on instantaneous or current-frame geometry, while others explicitly model future or historical structure. StemVLA introduces a 3D Future Spatial-Geometry World Knowledge Predictor and a 4D Historical Spatiotemporal Aggregator; GAM predicts future geometric tokens through a causal future predictor inserted into a pretrained Geometric Foundation Model; DVGT-2 uses temporal causal attention with historical caches for online planning (Xiao et al., 27 Feb 2026, Han et al., 15 Jun 2026, Zuo et al., 1 Apr 2026). This suggests that VGA increasingly overlaps with world modeling when task success depends on anticipation rather than instantaneous localization.

2. Architectural patterns in manipulation and driving

One major VGA pattern replaces or augments a standard visual backbone with a pretrained geometric model. GeoAware-VLA replaces a trainable 2D image encoder with a frozen, pretrained geometric encoder, VGGT, and a lightweight projection head that maps geometric features into the policy latent space (Abouzeid et al., 17 Sep 2025). VGA similarly replaces conventional language or video backbones with a pretrained 3D world model, VGGT, and injects geometry into the action decoder through Progressive Volumetric Modulation at every decoder layer (Song et al., 14 Apr 2026). GEAR-VLA keeps a frozen 2D ViT from Qwen2.5-VL, adds a trainable VGGT 3D spatial encoder through a zero-initialized “3D block,” and connects the VLM to a gradient-decoupled DiT continuous action expert via latent action tokens (Zhang et al., 7 Jun 2026). GAM repurposes a pretrained Geometric Foundation Model by splitting it at an intermediate layer, inserting a causal future predictor, and routing predicted tokens through the remaining GFM blocks so that a single backbone produces both future geometry and actions (Han et al., 15 Jun 2026).

A second pattern retains the VLA backbone and injects geometry through alignment. GLaD augments UniVLA with a frozen VGGT teacher and a small feature-alignment MLP, but aligns the LLM hidden states corresponding to visual tokens at the final LLaMA layer rather than altering the vision encoder directly (Guo et al., 10 Dec 2025). VEGA performs alignment earlier: it extracts patch-level tokens from the student DINOv2 branch before FiLM language conditioning and aligns them with frozen DINOv2-FiT3D features using a lightweight projector (Wang et al., 11 May 2026). G$^3$VLA adds a camera-aware geometric module to pretrained VLAs, combining intrinsic-conditioned ray embeddings, Projective Positional Encoding, and bidirectional cross-view fusion, while leaving the action space and imitation objective unchanged (Peng et al., 23 Jun 2026).

A third pattern disentangles perception and control more explicitly. OBEYED-VLA introduces a perception module that grounds multi-view inputs into task-conditioned, object-centric, and geometry-aware observations before passing them to a pretrained VLA policy (Vo et al., 27 Dec 2025). PointACT adopts a dual-system design in which a frozen Qwen2.5-VL provides 2D semantic tokens while a Point Transformer v3 encoder supplies hierarchical 3D point features that are integrated directly into the action decoder through multi-scale point-action interaction (Chen et al., 20 May 2026). In autonomous driving, VLGA instantiates a comparable separation through a Mixture-of-Transformers backbone with understanding, perception, geometry, and action experts, where geometry is a dedicated expert supervised by dense pointmap regression against LiDAR (Yao et al., 10 Jun 2026).

3. Geometry representations and supervision

VGA methods differ sharply in how geometry is represented. DVGT-2 predicts a local point-map representation, with per-camera 3D points

$P_t^v(u)=D_t^v(u)\,K_v^{-1}[u_x,u_y,1]^T,$

and can also be interpreted as learning a continuous occupancy field $G_t(\mathbf p)$ conditioned on spatio-temporal image features (Zuo et al., 1 Apr 2026). VLGA supervises dense per-pixel pointmaps against LiDAR using a confidence-weighted $L_1$ loss on predicted 3D points and confidences (Yao et al., 10 Jun 2026). OBEYED-VLA uses masked depth maps $Z_t^{base,geom}$ and $Z_t^{wrist,geom}$, optionally back-projectable to point clouds via camera intrinsics $v$0, but in practice keeps depth as a single-channel image for the policy front end (Vo et al., 27 Dec 2025).

Other systems keep geometry in latent token space rather than explicit depth or LiDAR at policy time. GLaD uses pooled VGGT features $v$1 as a teacher target and minimizes a mean-squared distillation loss together with the latent-action loss,

$v$2

so that geometry is fused into the final multimodal hidden states that drive action prediction (Guo et al., 10 Dec 2025). VEGA applies a cosine alignment loss,

$v$3

with $v$4, directly on the visual encoder output and removes both teacher and projector at inference (Wang et al., 11 May 2026). GeoAware-VLA adds no explicit depth- or point-cloud loss at all; its regularizer is the act of freezing the geometric backbone so that projected features inherit VGGT’s multi-view geometric invariances (Abouzeid et al., 17 Sep 2025).

Several VGA systems go further by making geometry itself a structured action-conditioning medium. GST-VLA converts frozen dense depth and semantic patch features into $v$5 anisotropic 3D Gaussian primitives, each parameterized by a residual mean $v$6, log-scale covariance $v$7, and opacity $v$8, then supervises four structured Depth-Aware Chain-of-Thought sequences before action generation (Sarowar et al., 10 Mar 2026). PointACT uses multi-scale point-cloud features across five Point Transformer v3 hierarchy levels and lets evolving action tokens attend to local and global 3D geometry within bottleneck windows (Chen et al., 20 May 2026). G$v$9VLA, by contrast, injects calibrated camera structure through ray embeddings and PRoPE biases, and adds a point-head distillation loss from either ground-truth point maps or confidence-gated $G$0X teacher predictions (Peng et al., 23 Jun 2026).

4. Optimization strategies and action coupling

A recurring feature of VGA is multi-objective optimization. VGA jointly trains action, camera, and depth heads on a shared VGGT backbone, with

$G$1

using LoRA over a frozen VGGT and decoupling camera and depth heads at inference so that only the lightweight action head runs at approximately $G$2 Hz (Song et al., 14 Apr 2026). DVGT-2 uses geometry reconstruction, pose regression, and planning losses together, while its streaming formulation introduces temporal causal attention and a sliding-window cache so that online planning does not require computationally expensive batch processing of multi-frame inputs (Zuo et al., 1 Apr 2026). VLGA uses a two-stage schedule: geometry warm-up with $G$3, followed by joint finetuning with $G$4 and $G$5 (Yao et al., 10 Jun 2026).

Action generation mechanisms also vary. GEAR-VLA explicitly separates coarse semantic action learning from fine continuous control: an embodied VLM is pretrained autoregressively on language, grounding, FAST-action tokens, and latent action IDs, after which a DiT expert is trained with flow matching on continuous relative end-effector actions $G$6 while stop-gradient shields the VLM from the continuous-control loss (Zhang et al., 7 Jun 2026). GAM keeps future geometry and action in a common geometric token manifold by predicting latent tokens at split layer $G$7, then decoding both future depth and next action chunk through the remaining GFM layers; its total loss is

$G$8

with $G$9, $^3$0, and $^3$1 (Han et al., 15 Jun 2026). GST-VLA likewise couples geometric reasoning and action synthesis through

$^3$2

trained in three stages to avoid collapse (Sarowar et al., 10 Mar 2026).

These choices indicate a broader methodological point. In VGA, geometry is rarely treated as a passive auxiliary target. Instead, it is used to constrain the latent space, regularize cross-view consistency, supply structured intermediate targets, or directly modulate the action decoder. This suggests that the distinctive property of VGA is not merely additional supervision, but a tighter interface between geometric state and action generation.

5. Empirical performance across benchmarks

Reported VGA gains span manipulation and autonomous driving, but the strongest improvements are usually concentrated in spatially sensitive, out-of-distribution, or embodiment-transfer settings.

System	Setting	Reported result
GLaD (Guo et al., 10 Dec 2025)	LIBERO	94.1% average success, vs UniVLA 92.5%
VGA (Song et al., 14 Apr 2026)	LIBERO / real robot OOD	98.1% LIBERO average; 58% OOD real-world average vs $^3$3 52%
GAM (Han et al., 15 Jun 2026)	LIBERO / LIBERO-Plus	97.6% / 85.5%; 6.9 ms per step
GEAR-VLA (Zhang et al., 7 Jun 2026)	cross-embodiment / grasping	85.9% on AgileX; 81.0% on pretraining-unseen LDT-01; 90.1% on a 6,360-trial benchmark with 212 unseen objects
DVGT-2 (Zuo et al., 1 Apr 2026)	NAVSIM / nuScenes	PDMS 88.6; EPDMS 88.9; L2 avg(1/2/3 s)=0.78 m
VLGA (Yao et al., 10 Jun 2026)	nuScenes / Bench2Drive	average L2 0.50 m, 3-second collision 0.18%, Driving Score 79.08

Within manipulation, several papers report that the largest gains occur under viewpoint change or explicit spatial challenge. GeoAware-VLA reports zero-shot novel-view LIBERO averages of 82.6 for GeoAware-BAKU versus 37.9 for the BAKU baseline, and 77.9 for GeoAware VQ-BeT versus 41.4 for the VQ-BeT baseline (Abouzeid et al., 17 Sep 2025). VEGA raises the RoboTwin 2.0 six-task average from 56.0% / 22.7% to 67.5% / 30.7% in Easy / Hard settings, and increases real-world average success from 48% to 60% (Wang et al., 11 May 2026). PointACT reports 96.0% on LIBERO versus 93.1% for reproduced EO1, and 82.3% on RLBench-10Tasks versus 73.2% for EO1 (Chen et al., 20 May 2026). GST-VLA reports 96.4% on LIBERO and 80.2% on SimplerEnv, with the largest gains on precision-demanding tasks such as Close-Drawer (Sarowar et al., 10 Mar 2026).

Robustness analyses refine this picture. GLaD’s LIBERO breakdown shows improvements on OBJECT, GOAL, and LONG—97.4% versus 95.4%, 94.4% versus 91.9%, and 89.4% versus 87.5%—while SPATIAL is 95.0% versus 95.2%; on LIBERO-PRO object perturbations, it reports GOAL 81% versus 62%, LONG 54% versus 47%, and OBJECT 86% versus 82%, but position perturbation remains low (Guo et al., 10 Dec 2025). G$^3$4VLA improves $^3$5 on LIBERO from 84.6% to 87.0% with $^3$6X supervision and to 88.1% with ground truth, and in real-robot OOD viewpoint tests reports 70.8→83.3 on Pouring Nut and 25→50 on Test Tube (Peng et al., 23 Jun 2026). In driving, VLGA attributes lower collision rates to dense pointmap supervision, reducing collision average from 0.169% in the baseline to 0.149% with a geometry expert only and to 0.136% with dense pointmap supervision (Yao et al., 10 Jun 2026).

6. Design debates, limitations, and future directions

A common misconception is that VGA necessarily requires depth sensors or manual 3D annotations at runtime. Multiple papers explicitly reject this. GLaD reports improved spatial reasoning and policy generalization “without requiring explicit depth sensors or 3D annotations” (Guo et al., 10 Dec 2025). VEGA discards its frozen teacher and lightweight projector at inference, introducing zero extra runtime cost (Wang et al., 11 May 2026). GeoAware-VLA relies on a frozen geometric encoder rather than explicit 3D labels at policy time (Abouzeid et al., 17 Sep 2025). G$^3$7VLA can be trained from confidence-gated teacher point predictions and, at inference, keeps only RGB, proprioception, language, and camera calibration (Peng et al., 23 Jun 2026).

Another active question is where geometry should enter the policy. GLaD argues for final-layer hidden-state alignment inside the LLM so that geometric priors are “deeply fused” with 2D semantic features and language context (Guo et al., 10 Dec 2025). VEGA argues the opposite: alignment should happen at the visual encoder output, before language fusion, because LLM-level visual tokens are already entangled with semantics and require empirical layer search (Wang et al., 11 May 2026). G$^3$8VLA provides a third perspective, reporting that geometric transfer is most effective when geometry-aware tokens have direct access to the action generation pathway; gains are attenuated on GR00T 1.5 relative to $^3$9 (Peng et al., 23 Jun 2026). Taken together, these results suggest that VGA remains an open design space rather than a settled recipe.

The literature also records persistent limitations. GLaD notes that position-perturbation and compositional generalization remain low ($^3$0), leaving room for explicit layout modeling or dynamic scene inference (Guo et al., 10 Dec 2025). VEGA depends on a strong 3D-aware teacher, and still fails on thin or occluded parts, grasp-pose misalignments, and collisions near workspace boundaries (Wang et al., 11 May 2026). GeoAware-VLA notes dependence on a large pretrained geometric model (Abouzeid et al., 17 Sep 2025). StemVLA reports gripper-only manipulation, limited geometric and material diversity with weaker sim-to-real transfer, and occasional jerky motions from the Diffusion Transformer (Xiao et al., 27 Feb 2026). G$^3$1VLA shows that teacher reliability matters: on RoboTwin2.0 handover_block, $^3$2X supervision yields 41.0% while ground truth yields 49.0%, because teacher failure in clean simulation harms distilled supervision (Peng et al., 23 Jun 2026).

Future directions are already explicit in the source papers. GLaD proposes online or self-supervised geometry distillation from depth-prediction models, multi-view or temporal geometry cues such as extending VGGT’s $^3$3 sequence, integration of 3D scene graphs or explicit object-pose estimators, and real-world deployment using only monocular cameras (Guo et al., 10 Dec 2025). VEGA proposes stronger or more diverse 3D teachers, adaptation to other visual backbones, and coupling spatial alignment with grasp-quality estimation or closed-loop feedback for error recovery (Wang et al., 11 May 2026). GEAR-VLA extends VGA toward cross-robot transfer through embodiment canonicalization, using robot-specific state projectors and embodiment-invariant relative end-effector actions to confine embodiment differences to the low-level interface (Zhang et al., 7 Jun 2026). In driving, DVGT-2 and VLGA indicate a parallel trajectory in which dense geometry is streamed or reconstructed online and used directly for planning under open-loop and closed-loop evaluation (Zuo et al., 1 Apr 2026, Yao et al., 10 Jun 2026).

Across manipulation and autonomous driving, VGA therefore names a geometry-first reformulation of policy learning: action is conditioned not merely on visual appearance or linguistic semantics, but on dense or structured 3D state, whether represented as point maps, depth fields, Gaussian primitives, point clouds, calibrated token streams, or geometry-aware latent features. Current results associate that reformulation with stronger spatial reasoning, improved robustness to viewpoint and appearance shifts, and competitive or state-of-the-art performance on several benchmarks, while leaving unresolved the most effective locus of geometric grounding, the role of teacher supervision, and the balance between explicit reconstruction and latent alignment (Song et al., 14 Apr 2026, Zuo et al., 1 Apr 2026).