Gravity Grounded Geometry Transformer (G3T)

• The paper introduces a gravity-aligned framework that regresses upright pointmaps and decomposed camera poses to reduce rotational ambiguity in multi-view 3D reconstruction.
• It leverages a CNN backbone with a transformer-based cross-view feature aggregator that aligns images using globally consistent vertical cues.
• The approach achieves superior pose accuracy (e.g., ≈1.78° error with 8 views) and enhances drift resistance in long-horizon mapping through incremental submap fusion.

The Gravity Grounded Geometry Transformer (G3T) is a feed-forward multi-view 3D reconstruction architecture that replaces traditional camera-centric prediction with a gravity-aligned paradigm. By exploiting the globally consistent vertical cues found in real-world scenes, G3T directly outputs upright pointmaps and corresponding camera pose estimates in a shared, axis-aligned coordinate frame. This alignment reduces the rotational ambiguity across views, facilitating robust geometric attention for dense 3D scene understanding, and underpins an incremental mapping pipeline—G3T-Long—with improved drift resistance and submap fusion accuracy (Kani et al., 26 May 2026).

1. Model Architecture

G3T is constructed atop the VGGT (Vision Geometry Grounded Transformer) framework—a pixel-aligned, feed-forward geometry transformer for multi-view 3D reasoning. The architecture comprises three principal streams: a CNN backbone to extract per-image feature maps, a cross-view feature aggregator transformer leveraging pixel-aligned attention, and prediction heads for depth, pointmap, and camera pose.

The central innovations in G3T relative to VGGT are:

• Gravity-Aligned Output: The point head directly regresses 3D pointmaps in the gravity-aligned frame of reference tied to view 1, denoted $X_i^{G_1} \in \mathbb{R}^{H\times W\times 3}$.
• Decomposed Camera Heads: Camera prediction is split into a local head, which outputs gravity-to-camera rotation (as a quaternion $q^l_i$) and focal length $f^l_i$, and a relative head, which regresses the yaw angle ($q^r_i$), and translation $t^r_i$, each relative to the anchor view.

Inputs are sets of RGB images $\{I_i\}_{i=1}^N$ of spatial dimensions $H \times W$ and, optionally, camera intrinsics or FoV guesses. The network predicts upright scene-centric pointmaps, depthmaps, camera-to-gravity rotations ($R_i^l$), relative yaws ($R_{y,i}$), and translations ($t^r_i$).

2. Gravity-Aligned Coordinate Frames

G3T defines the coordinate frame transformation between the camera-centric view $q^l_i$0 and the gravity frame $q^l_i$1, where $q^l_i$2's $q^l_i$3-axis encodes the global vertical (gravitational “up”). Given the gravity vector $q^l_i$4 in $q^l_i$5, the transformation $q^l_i$6 is found via

$q^l_i$7

such that

$q^l_i$8

This transformation applies roll and pitch (but not yaw), resulting in a unique alignment per image that positions each camera in a consistent upright frame. G3T’s local head directly regresses the requisite quaternion, and alignment can also be established by minimizing mean squared distances between corresponding pointmaps via Procrustes analysis. The frame transformation is then

$q^l_i$9

where $f^l_i$0.

3. Transformer-Based Feature Aggregation

At inference, for each pixel in the target view, G3T unprojects 3D query coordinates into the shared gravity frame, enabling cross-attention with features in auxiliary views. Since all pointmaps reside in an upright, gravity-aligned frame, inter-view alignment reduces from a general $f^l_i$1 rotation to a single yaw DOF ($f^l_i$2) about the vertical axis.

This specialization simplifies the matching of geometric and appearance features—attention mechanisms in the transformer only need to accommodate yaw misalignment. Accurate yaw estimation becomes central, as roll and pitch are consistent due to gravity normalization across views.

4. Loss Functions and Training Objectives

G3T adopts (with modifications) the standard losses used in VGGT. All losses compare predictions against scale-normalized, gravity-aligned ground truth targets. The training objective aggregates three principal loss terms:

1. Upright Pointmap Regression:

$f^l_i$3

where $f^l_i$4 is a confidence weight and $f^l_i$5 are normalized predictions.

2. Depth Regression:

$f^l_i$6

3. Camera-to-Gravity Pose Supervision:

$f^l_i$7

The overall training loss is

$f^l_i$8

where $f^l_i$9 are weighting coefficients.

5. Submap-Based Incremental 3D Reconstruction (G3T-Long)

G3T-Long adapts the VGGT-Long pipeline to the gravity-aligned regime, enforcing a 5-DoF alignment (1-DoF yaw, 3-DoF translation, 1-DoF scale) for submap fusion. RGB sequences are partitioned into overlapping chunks; each chunk’s images are jointly processed to yield upright pointmaps. Chunk alignment utilizes "GA-Procrustes"—a closed-form procedure operating on the x–z plane to estimate yaw via SVD.

Inter-chunk global optimization minimizes the deviation of yaw poses across adjacent (and loop-closure) chunk pairs, operating in the 5D Lie algebra of the reduced similarity group, using Levenberg–Marquardt optimization. Only yaw and translation drift are present, yielding substantial improvements in global consistency compared to general 7-DoF alignment.

6. Empirical Evaluation

G3T exhibits substantial improvements in camera pose estimation, structural accuracy, and submap fusion consistency:

• Camera-to-Gravity Rotation Accuracy: On the 7Scenes dataset, mean geodesic error for gravity prediction is 1.92° (one view, G3T local head) versus 6.78° for GeoCalib and 2.00° for Procrustes. With 8 views, error drops to ≈1.78°, with >26% of samples within 1°.
• Multi-View Structure Metrics: Completeness (ACC), accuracy (COMP), and normal consistency (NC) on G3T_P closely match VGGT_P (e.g., ACC ≈ 0.026 m, COMP ≈ 0.029 m, NC ≈ 0.78 on 7Scenes).
• Submap Reconstruction: On the TUM RGBD dataset, G3T-Long reduces average vertical drift $q^r_i$0 per chunk by ≈50% and improves structure metrics (e.g., ACC decreases from ~0.05 m to ~0.04 m on "fr1/360," NC improves).
• Qualitative Performance: G3T output point clouds are upright under extreme camera poses; VGGT+GeoCalib frequently retains residual misalignment. Failure cases appear with images lacking clear vertical cues.

7. Significance and Core Insights

G3T demonstrates that anchoring scene geometry prediction and inter-view alignment to the gravity axis provides a substantial advantage in real-world environments, where vertical structure is ubiquitous. By collapsing rotational ambiguity from 3D ($q^r_i$1) to 1D (yaw), G3T enables more accurate pose estimation, facilitates robust multi-view feature attention, and dramatically improves the stability and drift-resilience of long-horizon mapping pipelines. The approach is most effective in settings with unambiguous gravity cues; performance may degrade in cases lacking vertical structure, such as close-up floor captures. The G3T methodology establishes a template for future multi-view 3D models seeking to exploit global scene priors (Kani et al., 26 May 2026).