LiDAR-VGGT: Dense 3D Point Cloud Fusion

• LiDAR-VGGT is a framework that merges LiDAR inertial odometry with VGGT to achieve large-scale, dense colored point cloud reconstruction.
• It employs a two-stage coarse-to-fine fusion pipeline with pre-fusion metric initialization and post-fusion regularized cross-modal registration to overcome calibration sensitivity and sparsity issues.
• Global pose graph optimization combined with session-based alignment ensures robust and visually detailed maps ideal for robotics, autonomous driving, and multi-robot collaboration.

LiDAR-VGGT is a framework for large-scale colored point cloud reconstruction that tightly couples LiDAR inertial odometry with the Visual Geometry Grounded Transformer (VGGT) through a two-stage coarse-to-fine fusion pipeline. It was introduced to address a complementary failure mode in existing systems: LiDAR inertial visual odometry provides accurate metric geometry and can support global consistency, but remains sensitive to extrinsic calibration and typically yields sparse maps, while VGGT can infer camera poses, depth, and dense 3D point maps from multi-view images in a feed-forward manner, yet lacks metric scale and is limited in large environments. LiDAR-VGGT combines coarse metric initialization, regularized cross-modal $Sim(3)$ registration, and global pose graph optimization to produce dense, globally consistent colored point clouds (Wang et al., 3 Nov 2025).

1. Motivation and problem formulation

The target problem is large-scale colored point cloud reconstruction for robotics. In the LiDAR-VGGT formulation, this map quality matters for perception, navigation, scene understanding, multi-robot collaboration, and autonomous driving localization and planning. The motivating observation is that the two main source paradigms have complementary weaknesses. Classical LIVO systems such as FAST-LIVO2 and R3LIVE tightly fuse LiDAR, IMU, and camera data, offering accurate metric geometry from LiDAR and globally consistent mapping when loop closure and PGO are used, but they depend heavily on accurate extrinsic calibration and tight timestamp synchronization, and sparse LiDAR scans produce holes and low-density maps. By contrast, VGGT directly infers camera poses, depth, and dense 3D point maps from multi-view images, yielding dense visually clean reconstructions without explicit optimization at inference time, but it is scale-ambiguous and difficult to scale to large scenes (Wang et al., 3 Nov 2025).

The scalability issue is not incidental. In the broader VGGT literature, dense global attention is identified as the principal runtime bottleneck because its cost scales quadratically with the number of input images and tokens; for 100 frames, the global attention matrix would take roughly 35 GB at half precision, and the dense attention computation rather than the patch encoder or MLP heads becomes dominant (Wang et al., 8 Sep 2025). A separate scalability study reports that vanilla VGGT cannot exceed around 300 frames on an 80 GB GPU, while memory-efficient variants still encounter high latency or out-of-memory failure as frame count reaches 500–1000 (Park et al., 2 Jul 2026). This context explains why LiDAR-VGGT adopts session-wise decomposition rather than attempting a single monolithic inference pass.

Formally, the method starts from a long RGB sequence

$S_N = \{I_1, I_2, \dots, I_N\},$

where $I_i \in \mathbb{R}^{3 \times H \times W}$. VGGT produces

$f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$

where $\mathbf{g}_i \in \mathbb{R}^9$, $D_i \in \mathbb{R}^{H \times W}$, $P_i \in \mathbb{R}^{3 \times H \times W}$, and $Tr_i \in \mathbb{R}^{C \times H \times W}$. For large scenes, the sequence is split into $K$ overlapping sessions,

$\mathbb{S}_K = \{S_1, S_2, \dots, S_K\},$

each with $S_N = \{I_1, I_2, \dots, I_N\},$0 frames, and each session produces a VGGT local map

$S_N = \{I_1, I_2, \dots, I_N\},$1

The central task is to align each VGGT session $S_N = \{I_1, I_2, \dots, I_N\},$2 to the corresponding LiDAR session $S_N = \{I_1, I_2, \dots, I_N\},$3 by estimating a similarity transform

$S_N = \{I_1, I_2, \dots, I_N\},$4

with rotation, translation, and scale (Wang et al., 3 Nov 2025).

2. Coarse-to-fine fusion architecture

LiDAR-VGGT is organized into three stages: a pre-fusion module, a post-fusion module, and global pose graph optimization. The paper characterizes this as the first LiDAR + VGGT framework for dense, metric-scale, globally consistent colored mapping, with a robust pre-fusion initialization stage, an enhanced cross-modal $S_N = \{I_1, I_2, \dots, I_N\},$5 registration stage, and a new colored-map evaluation toolkit (Wang et al., 3 Nov 2025).

The architectural logic is asymmetric but deliberate. LiDAR inertial odometry supplies the metric anchor and global consistency reference, while VGGT supplies dense local geometry and visual completeness. The pre-fusion module uses LiDAR inertial odometry to robustly initialize and refine VGGT session poses and coarse scale. The post-fusion module then performs a stronger cross-modal $S_N = \{I_1, I_2, \dots, I_N\},$6 registration between VGGT and LiDAR point clouds, with a bounding-box-based scale regularizer designed to reduce distortions caused by inconsistent fields of view between LiDAR and camera sensors. Finally, global PGO merges all sessions into a globally consistent map (Wang et al., 3 Nov 2025).

This division of labor reflects an important property of the VGGT backbone. Mechanistic analysis of VGGT reports that its middle-layer global attention heads appear to perform correspondence matching and that epipolar geometry becomes decodable from intermediate representations, indicating that VGGT internalizes a usable geometric structure while also relying on learned priors (Bratulić et al., 12 Dec 2025). In the LiDAR-VGGT setting, that makes VGGT particularly suitable for dense local reconstruction, even though it remains insufficient by itself for metric-scale mapping in large environments.

3. Pre-fusion module: metric initialization and robust pose-scale refinement

The pre-fusion module has three components: LIO mapping with loop closure, pose registration, and linearity validation with scale RANSAC. On the LiDAR side, LiDAR and IMU are fused by FAST-LIO2 to generate LiDAR odometry and a LiDAR map. RING++ then detects loop closures, loop constraints are inserted into a pose graph, and PGO refines the LiDAR trajectory, producing a globally consistent metric reference (Wang et al., 3 Nov 2025).

Pose registration is performed session by session. LiDAR poses are transformed into camera poses in the world frame using the known extrinsics, and VGGT poses are matched to the nearest camera poses in time: $S_N = \{I_1, I_2, \dots, I_N\},$7 Let $S_N = \{I_1, I_2, \dots, I_N\},$8 denote the translation of the VGGT pose and $S_N = \{I_1, I_2, \dots, I_N\},$9 the translation of the camera pose. The initial similarity transform $I_i \in \mathbb{R}^{3 \times H \times W}$0 is estimated by minimizing

$I_i \in \mathbb{R}^{3 \times H \times W}$1

using the Umeyama method (Wang et al., 3 Nov 2025).

The paper identifies a specific failure mode in this step: Umeyama may estimate scale and translation well while producing unreliable rotation when the trajectory is close to a line. To detect this, it computes a linearity score from PCA eigenvalues $I_i \in \mathbb{R}^{3 \times H \times W}$2: $I_i \in \mathbb{R}^{3 \times H \times W}$3 High $I_i \in \mathbb{R}^{3 \times H \times W}$4 indicates near-linear motion, low rotational diversity, and greater risk of poor rotation estimation. Rotation is then refined by aggregating relative rotations. If $I_i \in \mathbb{R}^{3 \times H \times W}$5 are VGGT rotations and $I_i \in \mathbb{R}^{3 \times H \times W}$6 are camera rotations, the method defines

$I_i \in \mathbb{R}^{3 \times H \times W}$7

projects $I_i \in \mathbb{R}^{3 \times H \times W}$8 to $I_i \in \mathbb{R}^{3 \times H \times W}$9 by SVD,

$f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$0

and obtains the corrected rotation

$f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$1

This stage is intended to stabilize alignment precisely in the geometric regime where trajectory structure underconstrains rotation (Wang et al., 3 Nov 2025).

Scale is further refined by a scale RANSAC with linearity-aware sampling. Given initial scale estimates $f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$2 and linearity scores $f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$3, the inlier threshold is

$f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$4

with $f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$5. Sampling uses

$f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$6

For a candidate scale $f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$7, inliers are

$f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$8

Outlier scales are corrected using the nearest inlier and the scale from overlapping-pose $f(S_N) = (\mathbf{g}_i, D_i, P_i, Tr_i)_{i=1}^N,$9. The output of pre-fusion is therefore not the final alignment, but a coarse and robust metric-scale initialization for each session (Wang et al., 3 Nov 2025).

4. Post-fusion registration and global consistency

The post-fusion module refines the coarse alignment at the point-cloud level. Let $\mathbf{g}_i \in \mathbb{R}^9$0 be the VGGT session point cloud and $\mathbf{g}_i \in \mathbb{R}^9$1 the LiDAR reference session. After applying the pre-fusion transform to obtain an initialized cloud $\mathbf{g}_i \in \mathbb{R}^9$2, LiDAR-VGGT solves for a refined similarity transform $\mathbf{g}_i \in \mathbb{R}^9$3. Without regularization, the alignment objective is

$\mathbf{g}_i \in \mathbb{R}^9$4

where $\mathbf{g}_i \in \mathbb{R}^9$5 are source VGGT points and $\mathbf{g}_i \in \mathbb{R}^9$6 are target LiDAR points (Wang et al., 3 Nov 2025).

The distinguishing feature of LiDAR-VGGT is the regularized version of this objective: $\mathbf{g}_i \in \mathbb{R}^9$7 Here $\mathbf{g}_i \in \mathbb{R}^9$8 is the coarse scale from pre-fusion, and

$\mathbf{g}_i \in \mathbb{R}^9$9

where $D_i \in \mathbb{R}^{H \times W}$0, $D_i \in \mathbb{R}^{H \times W}$1 is the number of source points, and $D_i \in \mathbb{R}^{H \times W}$2 is the diagonal length of the source bounding box. With $D_i \in \mathbb{R}^{H \times W}$3 and $D_i \in \mathbb{R}^{H \times W}$4 fixed, the scale update has closed form: $D_i \in \mathbb{R}^{H \times W}$5 The stated purpose of this regularizer is to reduce scale drift caused by camera/LiDAR field-of-view mismatch. The paper’s interpretation is that unconstrained similarity optimization may overfit only the overlapping region, stretching or compressing non-overlapping regions; the bounding-box-based prior anchors the optimized scale to the LiDAR-derived estimate and stabilizes alignment (Wang et al., 3 Nov 2025).

After session-level registration, global consistency is enforced with pose graph optimization. The graph includes intra-session constraints from VGGT poses and inter-session constraints from ICP alignment of overlapping regions, and optimization is implemented in GTSAM. The first frame of the first session is fixed as the world reference frame. The refined poses are then propagated to all colored point clouds, yielding globally aligned sessions, loop-consistent reconstruction, and a dense metric-scale colored map (Wang et al., 3 Nov 2025).

5. Evaluation protocol, datasets, and reported performance

LiDAR-VGGT is evaluated on four datasets: the public MARS-LVIG dataset with sequences HKAirport01, HKisland01, AMvalley01, AMtown01, and HKairport03; the FAST-LIVO2 dataset with Brightwall and CBDBuilding; the MUN_FRL dataset with LightHouse and Flight; and a self-collected dataset with Company, Street, and TechnologyPark. The paper compares against VGGT-Long, VGGT-SLAM in $D_i \in \mathbb{R}^{H \times W}$6 and $D_i \in \mathbb{R}^{H \times W}$7 variants, SLAM3R, and FAST-LIVO2 in robustness tests (Wang et al., 3 Nov 2025).

A notable feature of the evaluation is a colored-map assessment toolkit introducing four metrics.

Metric	Definition	Direction
CD	Bidirectional average RGB distance	Lower is better
CF	$D_i \in \mathbb{R}^{H \times W}$8	Higher is better
LCR	Local color recall within threshold and geometric radius	Higher is better
CCS	Voxel-wise color covariance trace	Lower is better

The paper also reports mapping geometry metrics including CD, AC, ICP overlap, AWD, and Fitness. On the mapping geometry benchmark, LiDAR-VGGT is described as best overall. Representative examples include AMtown01 with CD $D_i \in \mathbb{R}^{H \times W}$9, ICP overlap $P_i \in \mathbb{R}^{3 \times H \times W}$0, AWD $P_i \in \mathbb{R}^{3 \times H \times W}$1, and Fitness $P_i \in \mathbb{R}^{3 \times H \times W}$2; HKisland01 with CD $P_i \in \mathbb{R}^{3 \times H \times W}$3, ICP overlap $P_i \in \mathbb{R}^{3 \times H \times W}$4, and AWD $P_i \in \mathbb{R}^{3 \times H \times W}$5; and LightHouse with CD $P_i \in \mathbb{R}^{3 \times H \times W}$6, ICP overlap $P_i \in \mathbb{R}^{3 \times H \times W}$7, and AWD $P_i \in \mathbb{R}^{3 \times H \times W}$8. For colored map quality, the method is reported as usually best or near-best, including AMtown01 with CD* $P_i \in \mathbb{R}^{3 \times H \times W}$9, CF $Tr_i \in \mathbb{R}^{C \times H \times W}$0, CCS $Tr_i \in \mathbb{R}^{C \times H \times W}$1, and LCR $Tr_i \in \mathbb{R}^{C \times H \times W}$2, HKisland01 with CD* $Tr_i \in \mathbb{R}^{C \times H \times W}$3, CF $Tr_i \in \mathbb{R}^{C \times H \times W}$4, CCS $Tr_i \in \mathbb{R}^{C \times H \times W}$5, and LCR $Tr_i \in \mathbb{R}^{C \times H \times W}$6, and LightHouse with LCR $Tr_i \in \mathbb{R}^{C \times H \times W}$7, the strongest among reported values (Wang et al., 3 Nov 2025).

The ablation study directly targets the fusion design. Pre-fusion only is worst because it lacks direct point-cloud fusion. Normal $Tr_i \in \mathbb{R}^{C \times H \times W}$8 is strong but can suffer severe scale distortion. Regularized $Tr_i \in \mathbb{R}^{C \times H \times W}$9 improves or matches normal $K$0 while being more stable. On HKairport03, normal $K$1 gives RMSE $K$2, AWD $K$3, and CD $K$4, while regularized $K$5 gives RMSE $K$6, AWD $K$7, and CD $K$8. On AMvalley01, the difference is larger: normal $K$9 gives RMSE $\mathbb{S}_K = \{S_1, S_2, \dots, S_K\},$0, AWD $\mathbb{S}_K = \{S_1, S_2, \dots, S_K\},$1, and CD $\mathbb{S}_K = \{S_1, S_2, \dots, S_K\},$2, whereas regularized $\mathbb{S}_K = \{S_1, S_2, \dots, S_K\},$3 gives RMSE $\mathbb{S}_K = \{S_1, S_2, \dots, S_K\},$4, AWD $\mathbb{S}_K = \{S_1, S_2, \dots, S_K\},$5, and CD $\mathbb{S}_K = \{S_1, S_2, \dots, S_K\},$6. The paper attributes these gains to suppression of the scale blow-up that appears in unconstrained alignment (Wang et al., 3 Nov 2025).

Robustness tests compare LiDAR-VGGT against FAST-LIVO2 under inaccurate extrinsics and missing time synchronization. FAST-LIVO2 is reported to degrade heavily and produce blurred maps, whereas LiDAR-VGGT remains much more robust and produces sharper, denser maps. This is a central empirical argument for the coarse-to-fine design: session-level cross-modal alignment is less fragile than a classical tightly calibrated LIVO pipeline when calibration or synchronization is imperfect (Wang et al., 3 Nov 2025).

6. Relation to other VGGT systems, misconceptions, and limitations

LiDAR-VGGT occupies a specific position in the emerging VGGT literature. It is not a generic label for any system that combines VGGT with LiDAR-related signals. VGGT-MPR, for example, uses VGGT as a unified geometric engine for multimodal place recognition, with a global retrieval module and a training-free re-ranking mechanism based on cross-view point tracking, but its task is retrieval rather than dense metric-scale mapping (Xu et al., 23 Feb 2026). SceneVGGT is an online 3D semantic SLAM framework built on VGGT for indoor RGB video streams, and its own description explicitly states that it is not a LiDAR-centric method: LiDAR or depth is used only as metric-scale grounding during alignment, not as the core mapping input (Gelencsér-Horváth et al., 12 Feb 2026). LiDAR-VGGT is therefore best understood as a dense mapping framework whose defining contribution is cross-modal coarse-to-fine fusion between LIO and session-wise VGGT reconstructions (Wang et al., 3 Nov 2025).

A second common misconception is that LiDAR-VGGT is an end-to-end multimodal transformer. The paper explicitly identifies the opposite as a limitation: LiDAR is not directly inserted into the VGGT transformer backbone, so the cross-modal coupling is not end-to-end tight. Future work is stated as integrating LiDAR cues directly into VGGT (Wang et al., 3 Nov 2025). The present system instead couples modalities through pose alignment, point-cloud registration, and global optimization.

A third misconception is that the main problem is only scale. In the broader VGGT literature, dense global attention creates a severe runtime bottleneck, and viewpoint redundancy can actively dilute useful geometric signals (Wang et al., 8 Sep 2025, Park et al., 2 Jul 2026). LiDAR-VGGT addresses large scenes by chunking them into overlapping sessions and aligning them with LiDAR-derived metric references, but it does not modify the underlying attention mechanism. A plausible implication is that block-sparse global attention or diversity-aware view partitioning could complement LiDAR-VGGT’s session-based fusion, although that combination is not reported.

The method’s practical limitations are also explicit. Chunk-based alignment methods can fail or distort geometry when overlap is small; holes in color recall are largely due to geometric mismatch in low-overlap areas; and the coupling remains indirect because the transformer itself does not ingest LiDAR tokens (Wang et al., 3 Nov 2025). Those caveats define the current scope of the framework. Within that scope, LiDAR-VGGT is designed to bridge a precise gap: LiDAR injects metric scale and global geometric stability into VGGT, while VGGT injects density and visual completeness into the LiDAR map, yielding globally consistent, metric-scale, dense colored point clouds for robotics (Wang et al., 3 Nov 2025).