G3T-Long: Gravity-Aligned 3D Reconstruction

• G3T-Long is a submap-based incremental 3D reconstruction pipeline that uses gravity-aligned coordinate frames to reduce rotational alignment complexity.
• It employs GA-Procrustes analysis and global pose-graph optimization within a 5 DoF space to enhance pose accuracy and structural fidelity.
• The pipeline achieves improved drift resilience and computational efficiency by transforming local submaps into a unified upright frame.

G3T-Long is a submap-based, incremental 3D reconstruction pipeline based on the Gravity-Grounded Geometry Transformer (G3T), designed to process arbitrarily long RGB sequences by leveraging gravity-aligned coordinate frames. By predicting upright pointmaps whose local frames share a common physical vertical (gravity) axis, G3T-Long achieves substantial reductions in rotational alignment complexity compared to camera-centric baselines—enabling more stable, accurate, and drift-resilient reconstruction of large-scale scenes. The G3T-Long system has been shown to deliver improved pose and structure accuracy on standard benchmarks, with a streamlined architecture that requires only minor modifications from conventional transformer-based 3D predictors (Kani et al., 26 May 2026).

1. Motivation and Conceptual Advances

Traditional feed-forward 3D reconstruction pipelines, such as VGGT and its derivatives, predict pixel-aligned pointmaps in the coordinate system of the first camera frame. This choice imposes arbitrary pitch and roll orientations relative to world gravity, so each new pointmap must be related to others via a 7 DoF similarity ($\mathrm{Sim}(3)$) transform encompassing 3D rotation, 3D translation, and scale. Such arbitrary alignment introduces the need for full 3D rotation estimation at every pairwise submap alignment, which is a key driver of drift and instability.

G3T-Long departs fundamentally from this approach by predicting pointmaps directly in an upright, gravity-aligned world frame, where the "up" axis corresponds to the physical gravity direction. Between submaps, only the yaw (rotation about the vertical axis), translation, and scale remain free, reducing all chunk-chunk alignments to a 5 DoF similarity ($\mathrm{Sim}_y(3)$) transform—1 DoF yaw, 3 DoF translation, and 1 DoF scale. This structurally-constrained alignment search space supports significantly more robust incremental and global optimization, particularly over long sequences (Kani et al., 26 May 2026).

2. Pipeline Structure and Workflow

G3T-Long implements a chunk-and-merge methodology, extending the VGGT-Long framework to operate entirely within a gravity-aligned regime. Its operational stages are:

1. Chunk Generation: The full input sequence of $N$ RGB frames is partitioned into $K$ chunks via a sliding window, each containing $M$ frames with $O$ frame overlap (default $M=25$, $O=7$). G3T is applied to each chunk’s subset, producing upright, pixel-aligned pointmaps $$\mathcal{X}_k^G = \{ X_i^G \in \mathbb{R}^{H \times W \times 3} \}$$, each in its own local gravity frame.
2. Pairwise Submap Alignment: Overlapping chunks are aligned using gravity-aligned Procrustes (GA-Procrustes) analysis, which restricts rotations to yaw-only ($\mathrm{Rot}_y$) transformations. The optimization objective is

$\mathrm{Sim}_y(3)$0

Alignment is efficiently solved by 2D SVD in the $\mathrm{Sim}_y(3)$1-plane and lifting the solution to a 3D yaw.

1. Global Pose-Graph Optimization: All relative chunk constraints—including loop-closures—are jointly optimized in the 5D Lie algebra $\mathrm{Sim}_y(3)$2 of gravity-aligned similarity transforms. The non-linear least squares objective is

$\mathrm{Sim}_y(3)$3

where $\mathrm{Sim}_y(3)$4 indexes loop-closure pairs, and optimization proceeds via Levenberg–Marquardt.

2. Chunk Fusion: Following pose-graph optimization, each submap is transformed by its estimated $\mathrm{Sim}_y(3)$5 into the global upright frame and merged, typically via weighted averaging with per-point confidences.

This procedure achieves frame-to-frame alignment and global consistency with reduced computational cost, improved memory efficiency, and enhanced robustness to drift as only one chunk is loaded at any moment.

3. Mathematical Foundations

Camera-Gravity Transform and Similarity Constraints

The mapping between camera-centric and gravity-aligned frames is given by:

$\mathrm{Sim}_y(3)$6

where $\mathrm{Sim}_y(3)$7 is the roll about the optical axis and $\mathrm{Sim}_y(3)$8 is the pitch about the camera’s $\mathrm{Sim}_y(3)$9-axis.

Full similarity ($N$0) pose estimation involves:

• Scale $N$1
• Rotation $N$2 (3 DoF)
• Translation $N$3

Gravity-aligned similarity ($N$4) restricts $N$5 to yaw-only, for a total of 5 DoF.

Loss Functions

• GA-Procrustes Alignment: Enforced via Eq. (2), which minimizes squared distances under yaw-only rotation and similarity.
• Global Optimization: Eq. (3) operates in the Lie algebra of $N$6, summing alignment residuals over all chunk associations.
• Geodesic Rotation Error: Used to evaluate camera-to-gravity prediction quality:

$N$7

4. Architectural Specifics

G3T-Long inherits almost all modules from VGGT, with two principal modifications concerning the point and camera heads:

• Point Head: Continues to output pixel-aligned 3D pointmaps but in the gravity frame of the first frame ($N$8), not arbitrary camera-centric coordinates.
• Camera Head:
  • Local head outputs $N$9, comprising a gravity-to-camera quaternion $K$0 to invert roll and pitch, and focal length intrinsics $K$1.
  • Relative head outputs $K$2, concatenating a yaw quaternion ($K$3) and translation ($K$4).

All other modules, including backbone, feature aggregation, and depth estimation, are unaltered from VGGT. Inputs are $K$5 images, token dimension 256, and the final point head tensor is $K$6.

5. Implementation Details and Computational Aspects

• Training: G3T-Long is fine-tuned from open-source VGGT for 40 epochs using 8 A100 GPUs, with each batch sampling 2–12 views (up to 96 images/GPU), gradient accumulation 4, and a cosine learning rate starting at $K$7 after one warmup epoch.
• Chunking: Default parameters are max 25 frames per chunk, 7 overlap, loop-chunk size 3.
• Alignment Optimization: GA-Procrustes alignment per chunk overlap requires a single 2×2 SVD, with computational complexity $K$8 in the number of points.
• Global Pose Graph: Optimizes $K$9 parameters via PyPose’s unconstrained solver.
• GPU Memory Efficiency: Peak usage is greatly reduced compared to processing the full sequence, because only one chunk is active at any time.

6. Benchmarking, Experiments, and Ablation Studies

Datasets and Evaluation Metrics

• Uprightness and multiview structure are measured on 7 Scenes, NRGBD, and ETH3D, reporting camera-to-gravity rotation error ($M$0), accuracy at $M$1, $M$2, $M$3, and structure metrics: ACCuracy (ACC), Completeness (COMP), and Normal Consistency (NC) after Procrustes alignment.
• Incremental reconstruction is evaluated on ten TUM RGBD sequences, with pose metrics—absolute pose error rotation (APE$M$4 in degrees), translation (APE$M$5 in meters), and vertical drift ($M$6). Structure metrics (ACC/COMP/NC) are frame-independent.

Comparative Results

Performance of G3T(-Long) is compared to:

• VGGT with GeoCalib post-processing (camera$M$7gravity)
• VGGT-Long (7 DoF Procrustes + $M$8 global LM)
• G3T-Long (GA-Procrustes + $M$9 global LM)

Table: Representative Pose and Structure Improvements

Metric	VGGT-Long	G3T-Long	Notes (Scene)
APE$O$0 (degrees)	2.963	1.429	fr1/plant, TUM RGBD
APE$O$1 (meters)	0.444	0.255	fr2/ps, TUM RGBD
Vertical drift $O$2	---	$O$330–50% $O$4	TUM RGBD
ACC	0.032	0.024	fr1/plant, structure
COMP	0.041	0.033	fr1/plant, structure
NC	$O$5unchanged	$O$6improved	Structure quality

G3T-Long reduces pose error and vertical drift while maintaining or improving per-chunk structure quality and normal consistency. Overlap ablations confirm that the margin of improvement is largest under limited chunk overlap (Kani et al., 26 May 2026).

7. Context, Limitations, and Broader Implications

By constraining all chunk alignments to gravity-aligned, yaw-only rotations in the upright world frame, G3T-Long realizes an empirically and theoretically significant reduction in the search space and instability sources of incremental 3D scene reconstruction. This enables more precise pose estimation, higher structure fidelity, and lower drift in long, multi-view sequences.

A plausible implication is that exploiting gravity-aligned representations benefits not only transformer-based 3D predictors but also future large-scale scene understanding systems incorporating SLAM, neural field mapping, or autonomous navigation, especially in built environments with strong vertical regularities. Potential limitations may arise outdoors or in scenes without clear global vertical, where defining a unique "up" axis is infeasible.

Overall, G3T-Long exemplifies the value of leveraging physical scene priors (such as uprightness) and restrictive mathematical symmetry groups to streamline both network design and large-scale optimization in geometric computer vision (Kani et al., 26 May 2026).