R^3: Relative 3D Reconstruction

• R^3 is a methodology for monocular 3D reconstruction that reframes pose estimation as a relative regression task.
• It leverages a lightweight MLP atop a DINOv2 transformer to predict pairwise relative transforms with learned confidence weights.
• The approach consistently achieves state-of-the-art results in streaming and offline modes with bounded-memory efficiency and robust performance.

$R^3$ is a methodology for monocular 3D reconstruction that reframes pose estimation as a relative regression task, enabling feed-forward geometry foundation models to avoid the structural limitations of global coordinate frame dependence. By regressing pairwise relative transforms and learned confidence weights via a lightweight MLP atop a DINOv2-based transformer, $R^3$ supports both causal, bounded-memory streaming and offline full-context 3D reconstruction. The approach yields consistent performance for long video sequences without relying on bundle-adjustment or global-absolute pose regression, and achieves state-of-the-art results on pose estimation and point-map reconstruction benchmarks with competitive efficiency (Xu et al., 26 May 2026).

1. Motivation and Problem Formulation

Traditional geometry foundation models—including VGGT, $\pi^3$, and DA3—directly regress the pose $\mathbf T_i \in SE(3)$ of each camera relative to a fixed global world frame. While effective for short offline sequences, these models encounter several fundamental obstacles in long or streaming contexts:

• The necessity to pick and maintain an arbitrary temporal origin (e.g., fixing $\mathbf T_1 = I$),
• Increasing translation magnitudes $\|\mathbf t_i\|$ that grow with sequence length, pushing pose predictions out of the network's training distribution,
• The complexity of maintaining a consistent global frame in a feed-forward context.

$R^3$ formulates 3D reconstruction as a relative regression problem in $SE(3)$. Given a sequence of camera frames with ground-truth poses $\mathbf T_i = (\mathbf q_i, \mathbf t_i)$, $R^3$ learns to predict relative transforms

$R^3$0

directly for arbitrary frame pairs $R^3$1. This strategy ensures that:

• Relative displacement magnitudes scale with inter-frame baselines and remain well-behaved over arbitrarily long sequences,
• Supervision from even a short $R^3$2-frame window scales as $R^3$3 via shared pairwise constraints.

2. Model Architecture and Inputs

The $R^3$4 pipeline employs the DA3 backbone, a DINOv2-based vision transformer, which produces a single $R^3$5-dimensional "camera token" $R^3$6 per input image $R^3$7. For each frame pair $R^3$8, a dedicated pairwise pose head—a lightweight MLP—receives the concatenated token $R^3$9 and outputs:

• $\pi^3$0 predicted unit quaternion for relative rotation,
• $\pi^3$1 predicted translation,
• $\pi^3$2 scalar confidences for rotation and translation.

Activation schemes include ReLU-type activations for pose components and Softplus for confidences. A per-frame head also regresses intrinsic focal length $\pi^3$3 from $\pi^3$4.

3. Confidence-Weighted Relative Training Loss

Supervision over all ordered frame pairs is administered through confidence-weighted $\pi^3$5 residuals: $\pi^3$6 where $\pi^3$7 are ground-truth relative transformations, and $\pi^3$8 denotes the quaternion-to-axis-angle map.

Each loss is weighted by its associated confidence: $\pi^3$9

$\mathbf T_i \in SE(3)$0

with $\mathbf T_i \in SE(3)$1. The $\mathbf T_i \in SE(3)$2 term prevents the trivial solution $\mathbf T_i \in SE(3)$3. In effect, higher confidences correlate with lower residual errors, enabling the model to trust high-quality predictions while downweighting difficult ones. During inference, these confidence scores serve as aggregation weights for pose fusion.

4. Pose Aggregation and Inference Procedure

To produce absolute camera trajectories for downstream 3D tasks, $\mathbf T_i \in SE(3)$4 aggregates relative predictions using a confidence-weighted fusion mechanism. Two principal inference modes are supported:

• Causal Streaming (Bounded-Memory)
  1. Anchor $\mathbf T_i \in SE(3)$5 for the first frame.
  2. Maintain a keyframe bank $\mathbf T_i \in SE(3)$6 of up to $\mathbf T_i \in SE(3)$7 context frames.
  3. For each new frame $\mathbf T_i \in SE(3)$8, compute relative pose predictions for all $\mathbf T_i \in SE(3)$9.
  4. Generate candidate absolute pose hypotheses $\mathbf T_1 = I$0, $\mathbf T_1 = I$1 and fuse them via softmax-weighted averages using the confidences.
  5. Restrict $\mathbf T_1 = I$2 size and composition via learned confidence and token-similarity gating.

• Full-Context Offline

  1. Remove causal attention masking to enable all-pair predictions.
  2. Run the fusion/aggregation as above, or apply optional confidence-weighted pose-graph optimization with Huber losses.

This duality enables a single $\mathbf T_1 = I$3 checkpoint to perform both causal streaming and non-causal offline inference, with or without additional refinement steps.

5. Comparison: Offline vs. Streaming Reconstruction

Mode	Context Size	Pair Computation	Memory Scaling
Streaming	$\mathbf T_1 = I$4	$\mathbf T_1 = I$5 per new frame	Bounded
Offline (Full-Context)	$\mathbf T_1 = I$6	$\mathbf T_1 = I$7 for $\mathbf T_1 = I$8 frames	$\mathbf T_1 = I$9 (if resources permit)

In streaming, memory and compute remain bounded, enabling real-time operation ($\|\mathbf t_i\|$0 FPS at 372M parameters), while offline mode leverages the full pairwise topology for additional accuracy with minimal extra cost. Crucially, $\|\mathbf t_i\|$1 supports thousand-frame sequences under a 48GB memory budget, with global-absolute regression models either drifting substantially or running out-of-memory in equivalent regimes.

6. Empirical Performance and Ablation

On canonical benchmarks, $\|\mathbf t_i\|$2 (372M parameters, streaming mode) achieves the following mean Absolute Trajectory Errors (ATE):

• Sintel (50 frames): $\|\mathbf t_i\|$3

• TUM-dynamics (90 frames): $\|\mathbf t_i\|$4
• ScanNet (90 frames): $\|\mathbf t_i\|$5

This performance matches or surpasses streaming models with $\|\mathbf t_i\|$6 parameters (StreamVGGT, SpanN3R, CUT3R, TTT3R), highlighting the efficacy of relative regression with a shared MLP head. $\|\mathbf t_i\|$7 also demonstrates robustness to distractor frames (SR $\|\mathbf t_i\|$8 on RobustNeRF and ETH3D protocols), stable long-sequence metrics (Acc/Comp/NC on 7-Scenes up to 1000 frames remain flat), and effective keyframe bank management using token-similarity thresholds ($\|\mathbf t_i\|$9).

Ablations confirm that replacing absolute-pose regression with the $R^3$0 relative-objective reduces RPE and ATE by $R^3$1–$R^3$2\% in both streaming and offline. Accuracy benefits from increased aggregation neighborhood size, peaking at all-reference averaging, with additional gains ($R^3$3–$R^3$4\%) from optional pose-graph optimization.

7. Limitations and Prospective Directions

$R^3$5 shifts the challenge of global frame consistency to a lightweight $R^3$6 aggregation step and maintains relative supervision in-distribution across arbitrarily long sequences. Learned confidences unify loss weighting, pose-fusion, keyframe management, and outlier rejection. However, the approach requires supervision heuristics (thresholds, bank size) to be tuned for new domains and moderately benefits from periodic resets or bridging on very long streams to mitigate residual drift. Absence of end-to-end bundle-adjustment limits fine-grained correction across highly disconnected or occluded trajectories.

Future efforts center on scaling to larger foundation backbones and datasets, learning adaptive keyframe management and reset policies, incorporating light bundle-adjustment or joint depth–pose optimizers, and exploring broader context windows with partially bidirectional streaming (Xu et al., 26 May 2026). $R^3$7 establishes relative regression with confidence-weighted $R^3$8 fusion as a scalable and efficient paradigm for unified streaming and offline 3D reconstruction.