GPA-VGGT:Adapting VGGT to Large scale Localization by self-Supervised learning with Geometry and Physics Aware loss

Abstract: Transformer-based general visual geometry frameworks have shown promising performance in camera pose estimation and 3D scene understanding. Recent advancements in Visual Geometry Grounded Transformer (VGGT) models have shown great promise in camera pose estimation and 3D reconstruction. However, these models typically rely on ground truth labels for training, posing challenges when adapting to unlabeled and unseen scenes. In this paper, we propose a self-supervised framework to train VGGT with unlabeled data, thereby enhancing its localization capability in large-scale environments. To achieve this, we extend conventional pair-wise relations to sequence-wise geometric constraints for self-supervised learning. Specifically, in each sequence, we sample multiple source frames and geometrically project them onto different target frames, which improves temporal feature consistency. We formulate physical photometric consistency and geometric constraints as a joint optimization loss to circumvent the requirement for hard labels. By training the model with this proposed method, not only the local and global cross-view attention layers but also the camera and depth heads can effectively capture the underlying multi-view geometry. Experiments demonstrate that the model converges within hundreds of iterations and achieves significant improvements in large-scale localization. Our code will be released at https://github.com/X-yangfan/GPA-VGGT.

• The paper presents a novel loss design that integrates physics and geometry constraints into VGGT without modifying its architecture.
• It employs a multi-sequence self-supervision framework with sliding windows to improve trajectory consistency and reduce drift.
• Empirical results on KITTI demonstrate reduced Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) compared to state-of-the-art methods.

Multi-Sequence Physics-Aware Self-Supervision for Transformer-Based Large-Scale Localization

Introduction and Motivation

Transformer-based frameworks such as the Visual Geometry Grounded Transformer (VGGT) have demonstrated increased capacity for modeling global visual and geometric dependencies in 3D scene understanding and camera pose estimation. However, unlocking their potential for large-scale localization without ground-truth supervision remains elusive. Existing methods either restrict themselves to local, pairwise constraints—limiting global consistency and long-range reasoning—or rely heavily on supervised training, constraining deployment in unlabeled, real-world environments. The discussed paper proposes GPA-VGGT, an adaptation of VGGT via a novel multi-sequence, physics- and geometry-aware self-supervised training framework, targeting scalable and physically plausible localization using only unlabeled image sequences.

The methodological core is a structured loss design enforcing 3D geometric consistency and robust photometric alignment across multiple frames and temporal windows, explicitly addressing the limitations of existing pairwise or locally supervised paradigms. This approach is intended to align model optimization with underlying physical constraints, enabling the model to learn reliable geometry and camera trajectories robust to dynamic content and environmental variability.

Figure 1: Comparison between traditional self-supervised methods, standard VGGT, and the proposed multi-sequence 3D self-supervised framework, which introduces geometric reasoning into VGGT solely via loss design.

Methodology

Multi-Frame Self-Supervised Framework

The framework organizes input sequences into sliding windows of $S$ frames, where every frame can serve as a geometric anchor (keyframe), and the remaining frames act as source views. This multi-anchor, multi-source arrangement maximizes the geometric constraint density, thus reducing baseline degeneracy and improving observability over extended sequences.

The network retains the vanilla VGGT architecture, incorporating a split prediction head for depth and relative camera pose.

Figure 2: (a) VGGT-based network with depth and camera heads; (b) multi-frame projections with validity masking; (c) per-pixel hard source selection via minimum photometric–geometric cost.

All transformations are parameterized by the shared camera intrinsic calibration, with augmentation-induced updates explicitly propagated to maintain strict photogeometric alignment.

Physics- and Geometry-Aware Loss Formulation

Two coupled loss terms form the supervision backbone:

• Photometric Consistency: For physically valid static points, the predicted geometry should enable image-to-image warping with minimal photometric discrepancy, measured via a hybrid of SSIM and $L_1$ distance.
• Geometric Consistency: The discrepancy between the predicted depth of a point reprojected into a source frame and the depth predicted directly for that location is penalized via a scale-invariant geometric loss.

To robustly handle dynamic objects, occlusions, and lighting changes, a minimum-cost selection mechanism is deployed. For each pixel, supervision is backpropagated only along the source frame yielding minimum combined photometric plus geometric cost, automatically ignoring unreliable or outlier correspondences.

Further, auto-masking discards regions with ambiguous or unobservable geometry (e.g., moving at camera speed or textureless), by comparing the residual with a zero-motion hypothesis.

Trajectory Propagation and Inference

For long sequence inference, camera poses are predicted in overlapping windows and aligned via shared anchor frames, enabling the chaining of pose predictions into globally consistent long-range trajectories.

Figure 3: Overlapping sliding window inference strategy with pose propagation across shared anchor frames for constructing long-range trajectories.

Experimental Results and Analysis

Quantitative and Qualitative Evaluation

Experiments are conducted on the KITTI Odometry Benchmark, focusing on sequences featuring extended trajectories and strong dynamic and illumination challenges. The model converges rapidly (hundreds of iterations), in both settings with full fine-tuning and aggregator- plus-heads fine-tuning.

Critically, GPA-VGGT demonstrates lower Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) across sequences compared to both conventional self-supervised pipelines and supervised transformer baselines. For Sequence 07, GPA-VGGT achieves an ATE of 12.541m, surpassing all state-of-the-art baselines. Notably, it remains robust even when compared to models such as MapAnything and DUSt3R, which leverage full supervision or dense pairwise regressive matching.

Trajectory comparison on challenging sequences reveals that GPA-VGGT maintains physical plausibility, with reduced drift and improved alignment to ground-truth over long distances.

Figure 4: Predicted trajectories for KITTI Sequences 07 and 09, demonstrating close alignment of GPA-VGGT results (blue) with the ground truth (black) and visibly less drift than competing baselines.

Depth predictions generated by GPA-VGGT are observed to be significantly smoother and temporally consistent. Discontinuities align with object boundaries, and planar surfaces are accurately reconstructed.

Figure 5: Comparison of depth estimation outputs. GPA-VGGT exhibits coherent, artifact-free depth maps with clear object and structure delineation.

Claims and Implications

The paper makes the bold claim that loss design alone, without architectural modification, is sufficient to induce robust large-scale geometric reasoning in high-capacity visual transformer models. Empirical results contradict the commonly held notion that large transformer models require heavy supervised pre-training or explicit architectural enhancements for strong generalization in unseen, wild environments.

Practical implications include more feasible deployment of geometry foundation models in real-world SLAM, AR/VR, and autonomous driving pipelines operating over large, unlabeled datasets. Theoretically, these results suggest that self-supervised physical consistency constraints are critical for scalable 3D perception, even for models capable of global dependency modeling, and that architectural capacity must be synergistically paired with appropriately structured objectives.

Future Directions

Several research opportunities emerge: (1) extending multi-sequence self-supervision to joint segmentation and object-level reconstruction; (2) combining explicit bundle adjustment and gradient-based transformer optimization for further improvement in global consistency; (3) scaling to other sensor modalities and multimodal 3D scene understanding; and (4) adapting the proposed robust view selection paradigms to more complex, dynamic environments.

Conclusion

The GPA-VGGT framework provides strong evidence that global, physically grounded self-supervised loss strategies can equip transformer-based visual geometry models with robustness and scalability previously obtainable only by supervised or highly specialized systems. By relying on physical consistency and dense, multi-sequence constraints rather than architectural changes, GPA-VGGT closes the gap between foundation geometric modeling capacity and practical deployment in large-scale, unlabeled environments. This loss-driven paradigm shift marks a critical advancement in self-supervised 3D perception, highlighting the necessity of aligning learning objectives with the underlying structure of visual reality (2601.16885).