Driving Visual Geometry Transformer (DVGT)

Updated 4 July 2026

The paper introduces DVGT, which recovers dense metric 3D point maps and ego-poses from unposed multi-view inputs to support autonomous driving.
It employs a cascaded transformer architecture with intra-view, cross-view, and temporal attention to fuse multi-camera data without explicit 3D geometric priors.
DVGT-2 extends the framework to a streaming Vision-Geometry-Action paradigm that jointly outputs dense geometry and trajectory planning for real-time driving applications.

Driving Visual Geometry Transformer (DVGT) denotes a line of driving-targeted visual geometry models that treat dense 3D reconstruction as a primary representation for autonomous driving. In its original form, DVGT reconstructs a global dense 3D point map from a sequence of unposed multi-view visual inputs and predicts ego poses, while remaining free of explicit 3D geometric priors and directly predicting metric-scaled geometry (Zuo et al., 18 Dec 2025). DVGT-2 extends this line into a streaming Vision-Geometry-Action (VGA) model that jointly outputs dense geometry and trajectory planning for the current frame, arguing that dense 3D geometry is the critical cue for autonomous driving because vehicles operate in a 3D world (Zuo et al., 1 Apr 2026).

1. Problem setting and conceptual scope

DVGT addresses the problem of recovering dense scene geometry from surround-view imagery in a form usable for driving. The original formulation takes a short sequence of unposed, surround-view images

$\mathcal{I}=\{I_{t,n}\}_{t=1\ldots T,\;n=1\ldots N}$

and predicts a continuous, metric-scaled, dense 3D point map

$\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$

for each pixel in image $I_{t,n}$ , expressed in the ego-vehicle coordinate of frame $t=1$ , together with ego-vehicle poses $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ (Zuo et al., 18 Dec 2025).

The motivation given for this formulation is that dense 3D geometry is the foundation for safe motion planning, collision avoidance, and scene understanding in autonomous driving. The original DVGT explicitly targets deficiencies of methods that either predict only per-view depth or occupancy, often requiring known camera poses, or reconstruct at a relative scale and then align to metric scale. DVGT instead aims to be fully metric, end-to-end, and camera-agnostic (Zuo et al., 18 Dec 2025).

DVGT-2 repositions this geometry-first view within end-to-end driving. Its stated context is that end-to-end autonomous driving has evolved from sparse perception to vision-language-action models, whereas DVGT-2 proposes an alternative Vision-Geometry-Action paradigm. In that formulation, dense 3D geometry is treated as the most comprehensive information for decision-making, and the model jointly performs geometry reconstruction and planning in an online streaming setting (Zuo et al., 1 Apr 2026).

A recurrent point in this literature is that “DVGT” can refer both to the specific 2025 model and, more broadly, to a family of driving visual geometry transformers. This suggests a representation-centered research direction rather than a single fixed architecture.

2. Original DVGT architecture

The original DVGT consists of three stages: image feature extraction via a DINO-pretrained ViT-L backbone, a cascaded Spatial-Temporal Geometry Transformer, and prediction heads for dense point-map and ego-pose (Zuo et al., 18 Dec 2025).

For feature extraction, each image is tokenized by a frozen DINOv3-ViT-L into patch tokens, and a learned “ego token” is appended to each view. The concatenated sequence

$z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$

is augmented with a learnable temporal embedding $\mathrm{Pos}_t$ (Zuo et al., 18 Dec 2025).

The transformer backbone stacks $L$ identical blocks, each containing three specialized attentions executed sequentially:

Intra-View Local Attention refines local structure within each image using a fixed local window mask.
Cross-View Spatial Attention fuses information across the $N$ camera views of the same frame.
Cross-Frame Temporal Attention captures temporal consistency for each camera across frames (Zuo et al., 18 Dec 2025).

This factorization is central to DVGT’s design: local image structure, multi-camera fusion, and temporal consistency are separated rather than collapsed into a single global attention layer. No camera intrinsics or explicit 3D projection is used. Only temporal embeddings are added at the input of each block to signal frame order (Zuo et al., 18 Dec 2025).

The decoder has two principal heads. The refined patch tokens are fed into a 3D-point-map head $\mathcal H_{\mathrm{point}}$ to predict

$\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 0

while the refined ego tokens are summed over views,

$\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 1

and passed through a pose head $\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 2 to regress a 7-D representation of the $\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 3 transform $\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 4 (Zuo et al., 18 Dec 2025).

The learning objective combines a metric-scaled point-map loss with uncertainty prediction, a spatial gradient term, and an ego-pose loss. The total loss is

$\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 5

with $\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 6 in the point-map uncertainty term (Zuo et al., 18 Dec 2025).

3. Metric reconstruction, camera agnosticism, and empirical profile

A defining claim of DVGT is that it is free of explicit 3D geometric priors. The network uses no 2D–3D projection layers or explicit intrinsics/extrinsics, and because the ground-truth point-map is metric, the model directly learns metric reconstruction. The reported evaluation does not rely on post-alignment such as Umeyama or ICP (Zuo et al., 18 Dec 2025).

This camera agnosticism is not merely architectural rhetoric; it is reflected in training. DVGT is trained on a mixture of nuScenes, OpenScene, Waymo, KITTI, and DDAD, with sampling ratio nuScenes : KITTI : OpenScene : Waymo : DDAD $\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 7. Each training batch samples a random scene, random frames $\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 8, and random views $\hat P_{t,n}(u,v)=(x,y,z)\in\mathbb R^3$ 9 up to 48 images/GPU. Optimization uses 160 k iterations on 64 A100/H20 GPUs, AdamW, peak learning rate $I_{t,n}$ 0, cosine schedule, 8 k-step linear warmup, gradient clipping with norm $I_{t,n}$ 1, bfloat16, and gradient checkpointing (Zuo et al., 18 Dec 2025).

Empirically, the original model is positioned as a dense 3D reconstruction system first. Its principal metrics are Accuracy, Completeness, Abs Rel, and $I_{t,n}$ 2. On five driving benchmarks, DVGT is reported to outperform CUT3R*, VGGT, MapAnything, StreamVGGT, and Driv3R* without any post-alignment (Zuo et al., 18 Dec 2025).

Benchmark example	DVGT	Comparison reported
nuScenes Abs Rel	0.069	VGGT* 0.243
nuScenes $I_{t,n}$ 3	0.953	VGGT* 0.729
nuScenes Accuracy	0.457 m	VGGT* 1.300 m
nuScenes Completeness	0.494 m	VGGT* 1.498 m

The same paper reports that for ego-pose estimation, AUC@30° for rotation+translation is on par with or better than general models on most datasets, with the example of DDAD at 95.1% versus VGGT 92.8%. For comparison to driving-specific depth on nuScenes LiDAR-GT depth, DVGT reports Abs Rel 0.13 and $I_{t,n}$ 4, with no scale post-processing (Zuo et al., 18 Dec 2025).

A common misconception is that dense geometry models for driving must depend on calibration and explicit projective structure. DVGT is an explicit counterexample: its stated design goal is camera-agnostic metric reconstruction without geometric priors.

4. DVGT-2 and the streaming Vision-Geometry-Action reformulation

DVGT-2 addresses a limitation of earlier geometry reconstruction systems, including DVGT: most rely on computationally expensive batch processing of multi-frame inputs and therefore cannot be applied to online planning. DVGT-2 is a streaming transformer that processes inputs in an online manner and jointly outputs dense geometry and trajectory planning for the current frame (Zuo et al., 1 Apr 2026).

Its input embeddings combine three token types per view. Given $I_{t,n}$ 5 images

$I_{t,n}$ 6

a frozen ViT-L backbone pre-trained by DINOv3 produces per-view visual tokens

$I_{t,n}$ 7

Two learnable global tokens are added per view: pose tokens

$I_{t,n}$ 8

and trajectory tokens

$I_{t,n}$ 9

and all tokens are concatenated into

$t=1$ 0

The core geometry transformer again uses stacked blocks with Intra-View Local Attention and Cross-View Spatial Attention, but its temporal component becomes Temporal Causal Attention over past $t=1$ 1 frames (Zuo et al., 1 Apr 2026).

Causal masking is implemented as

$t=1$ 2

where $t=1$ 3 if key $t=1$ 4 is in the “future” relative to query $t=1$ 5. To support unbounded streaming, DVGT-2 uses a relative temporal positional encoding, MRoPE-I, so that cached features remain reusable (Zuo et al., 1 Apr 2026).

The model outputs four quantities at each step:

$t=1$ 6

where $t=1$ 7 holds cached intermediate features from the past $t=1$ 8 frames. The prediction heads are a DPT-style decoder for a dense pointmap

$t=1$ 9

an anchor-based diffusion pose head producing a 7D relative pose $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 0, and an anchor-based diffusion trajectory head producing the future $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 1-step trajectory

$\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 2

in $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 3 (Zuo et al., 1 Apr 2026).

The streaming mechanism uses a fixed-size sliding-window cache with FIFO updates:

$\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 4

The resulting per-frame complexity comparison is reported as batch $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 5 over $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 6 frames, full-history streaming $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 7, and sliding-window DVGT-2 $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 8 constant (Zuo et al., 1 Apr 2026).

This reformulation is not merely computational. It changes DVGT from a reconstruction model into a joint geometry-planning system, while preserving dense 3D geometry as the primary internal signal.

5. Planning, efficiency, and cross-rig generalization

In DVGT-2, once geometry tokens $\{\hat T_t\in\mathrm{SE}(3)\}_{t=1\ldots T}$ 9 have been updated, trajectory tokens $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 0 attend over them and over ego-status embeddings to diffusion-decode the future trajectory. The planning loss includes

$z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 1

with the diffusion head further imposing a denoising objective on anchor offsets. The stated information flow is Dense geometry $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 2 trajectory tokens $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 3 future waypoints, which in practice allows the model to leverage pixel-aligned 3D structure when reasoning about drivable corridors and obstacle avoidance (Zuo et al., 1 Apr 2026).

A central empirical claim is that the same trained DVGT-2 can be directly applied to planning across diverse camera configurations without fine-tuning, including closed-loop NAVSIM and open-loop nuScenes benchmarks. At training time, the model randomly samples 2–8 cameras and aspect ratios [1.6–3.3]; at test time, the same weights handle 6×80° rigs or 360° rigs without finetuning (Zuo et al., 1 Apr 2026).

Its efficiency profile is reported as follows: batch DVGT costs approximately 1.9 s/frame on 8-view, 16-frame input; StreamVGGT grows linearly to approximately 2 s/frame by frame 16; DVGT-2 holds steady at approximately 0.27 s/frame, approximately 3.7 FPS, and $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 4 with respect to sequence length. With $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 5, DVGT-2 stays under 16 GB even at 200 frames, whereas batch and full-history streaming blow out memory after 10–30 frames (Zuo et al., 1 Apr 2026).

OpenScene streaming methods	Acc $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 6	Time
CUT3R*	1.858	0.35 s
StreamVGGT*	2.209	1.94 s
Driv3R*	0.884	0.56 s
DVGT-2	0.440	0.27 s

On the same OpenScene comparison, DVGT-2 reports Comp 0.450, AbsRel 0.040, $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 7 0.977, and AUC 70.3. On nuScenes, Waymo and DDAD, it is reported to similarly match or exceed prior streaming methods in AbsRel and $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 8, at approximately 3.7 FPS (Zuo et al., 1 Apr 2026).

For planning, closed-loop NAVSIM v1 reports PDMS 88.6 for DVGT-2 (no RL) and 90.3 for DVGT-2-NAVSIM (fine-tuned), identified as a new SOTA among camera-only methods. On NAVSIM v2, EPDMS is 88.9 for DVGT-2 and 89.6 for DVGT-2-NAVSIM. On open-loop nuScenes, DVGT-2 reports average L2 0.78 and average collisions 0.19, versus UniAD at 1.03 and 0.31, GenAD at 0.91 and 0.43, and OmniDrive at 0.84 and 0.94 (Zuo et al., 1 Apr 2026).

The window-size ablation reports $z^{(0)}_{t,n} = [\,f^{(0)}_{t,n,1}\,\Vert\,\dots\Vert\,f^{(0)}_{t,n,M}\,\Vert\,e^{(0)}_{t,n}]$ 9– $\mathrm{Pos}_t$ 0 as the best balance between temporal context and pose-drift accumulation. This is one of the clearest places where the model’s geometry and streaming assumptions interact directly with systems-level deployment constraints (Zuo et al., 1 Apr 2026).

The broader literature around DVGT exposes several major design alternatives. “DriveVGGT: Visual Geometry Transformer for Autonomous Driving” argues that autonomous driving introduces priors absent in generic feed-forward reconstruction, namely minimal overlap between camera views, known intrinsics and extrinsics, and fixed relative camera positions. It therefore injects camera-rig priors via relative-pose tokens, uses a Temporal Video Attention module per camera, introduces Multi-camera Consistency Attention with normalized relative pose embeddings, and adds an absolute scale head and an ego vehicle pose head (Jia et al., 27 Nov 2025). This stands in direct contrast to DVGT’s decision to avoid explicit geometric priors.

A second alternative appears in “Visual Implicit Geometry Transformer for Autonomous Driving,” which estimates a continuous 3D occupancy field from surround-view camera rigs using a calibration-free architecture and a common metric bird’s-eye-view coordinate frame. Rather than predicting pixel-aligned pointmaps as the primary representation, it predicts a continuous function $\mathrm{Pos}_t$ 1 for occupancy, trained self-supervised from synchronized image-LiDAR pairs. The model reports lowest Average Rank 1.8 across NuScenes, AV2, Waymo, ONCE, and NuPlan for point-map estimation, and on Occ3D-nuScenes reports F1 0.7115 and IoU 0.5658, ranking 3rd overall but 1st among methods without 3D or calibration labels (Shirokov et al., 5 Feb 2026). This shows that the geometry-first philosophy can be instantiated either as pixel-aligned pointmaps or as implicit BEV occupancy.

A third direction emphasizes deployment efficiency. “LiAuto-GeoX: Efficient Grounded Driving Transformer” trains a high-capacity geometry model with sparse LiDAR priors and distills it into a compact 155M-parameter student using mask-guided depth-aware distillation and relative-pose relational distillation. It reports 220 FPS on KITTI while maintaining transfer to downstream planning, occupancy prediction, and future-frame prediction, including 90.6 PDMS in trajectory prediction, 24.63 mIoU in occupancy prediction, and 47.67 IoU in future-frame prediction (Lian et al., 4 Jun 2026). This suggests that dense 3D reconstruction can be reframed from a heavy perception objective into a deployable driving representation.

Transfer beyond reconstruction and planning is also documented. “VGGT-MPR: VGGT-Enhanced Multimodal Place Recognition in Autonomous Driving Environments” adopts a frozen geometry-centric transformer as an off-the-shelf backbone for place recognition, using geometry-rich visual embeddings, LiDAR densification with predicted depth maps, and a training-free re-ranking mechanism based on cross-view keypoint tracking. Reported results include AR@1 98.28% on nuScenes Boston-Seaport and AR@1 76.05% in zero-shot nuScenes-to-UGV transfer (Xu et al., 23 Feb 2026). Although this work instantiates VGGT rather than DVGT specifically, it is evidence that geometry transformers developed for driving can function as reusable backbones in adjacent autonomy tasks.

These comparisons reveal two recurring debates in the field: whether dense geometry should be encoded as pixel-aligned pointmaps or continuous occupancy, and whether driving models should remain calibration-free or explicitly exploit known rig geometry. The literature does not settle these questions uniformly; instead, it demonstrates that each choice produces different trade-offs in scale recovery, cross-camera consistency, and deployment efficiency.

7. Limitations and prospective extensions

The main limitations explicitly identified for DVGT-2 are global ego-pose drift accumulation when only local relative poses are predicted, the trade-off between temporal context and drift as window size changes, and the absence of explicit modeling of dynamic agents beyond ego trajectory (Zuo et al., 1 Apr 2026). These are not minor implementation details; they indicate where a geometry-centric driving representation remains incomplete relative to full-world dynamic modeling.

The proposed extensions in the same work are correspondingly targeted: incorporating a lightweight global world-coordinate correction loop or loop-closure module, fusing LiDAR or radar features in the same streaming framework, extending the diffusion heads to multimodal or probabilistic planning, and jointly training a dynamic-object token branch for richer agent prediction (Zuo et al., 1 Apr 2026). A plausible implication is that future DVGT-family systems will become less purely reconstructive and more explicitly state-space-oriented, while still using dense geometry as the organizing representation.

At a higher level, the DVGT literature establishes a consistent proposition: dense, metric 3D geometry can serve as the central latent for autonomous driving, not merely as an auxiliary perception product. The original DVGT formulates this proposition as end-to-end, camera-agnostic dense reconstruction (Zuo et al., 18 Dec 2025); DVGT-2 reformulates it as streaming geometry-conditioned planning (Zuo et al., 1 Apr 2026); related systems test whether calibration, BEV occupancy, LiDAR grounding, or distillation provide a more favorable operating point (Jia et al., 27 Nov 2025, Shirokov et al., 5 Feb 2026, Lian et al., 4 Jun 2026). The resulting research area is therefore best understood not as a single model family with one canonical architecture, but as a geometry-first program for autonomous driving.