Speed3R: Fast Sparse 3D Reconstruction

• Speed3R is a sparse feed-forward 3D reconstruction model that leverages dual-branch global sparse attention to efficiently perform camera pose and dense geometry inference.
• It replaces computationally expensive dense attention with a compression branch for global context and a selection branch for fine-grained details, achieving up to 12.4× speedup.
• The model maintains competitive accuracy on benchmarks like ScanNet, RE10K, and Tanks & Temples, striking an effective efficiency–accuracy trade-off.

Speed3R is a sparse feed-forward 3D reconstruction model that achieves substantial acceleration in multi-view 3D scene modeling by replacing computationally expensive dense attention with a dual-branch global sparse attention mechanism. Drawing inspiration from the structure-from-motion principle that a sparse set of keypoints suffices for robust pose estimation, Speed3R executes both camera pose and dense geometry inference in a single end-to-end pipeline while maintaining accuracy on standard benchmarks at a fraction of the computational cost (Ren et al., 9 Mar 2026).

1. Architecture and Pipeline Overview

Speed3R follows an “encode–transform–predict” paradigm common to recent feed-forward 3D reconstruction models such as VGGT and $\pi^3$. Given a sequence of $N$ input images $\{I_i\}$, each image is encoded by a shared patch-based feature extractor, e.g., DINOv2, to produce per-view image tokens $X_{\text{img}}$ and a smaller set of special tokens $X_{\text{spec}}$ (such as camera or reference tokens). These tokens are processed through $T$ alternated transformer blocks featuring:

• Local Frame Attention: self-attention within each image/view
• Global Sparse Attention (GSA): efficient cross-view information propagation

The GSA module is the principal innovation, enforcing structured sparsity to avoid the $O(M^2)$ computational bottleneck of standard dense attention, where $M$ is the total number of tokens aggregated from all views. The architecture culminates in regression heads that jointly estimate per-view camera poses $\hat{C}_i$ and dense depth maps $\hat{D}_i$ (with pixel-wise uncertainty $N$0).

2. Global Sparse Attention Mechanism

The GSA module splits the input sequence of tokens $N$1 into special tokens $N$2 and image tokens $N$3, proceeds with projection into $N$4, $N$5, $N$6 representations, and orchestrates information flow through two specialized branches:

1. Full Attention on Special Tokens: Special tokens (e.g., those encoding camera or reference information) attend to all tokens, computed via:

$N$7

1. Compression Branch (Coarse Prior):
   • Downsamples $N$8 using non-overlapping average pooling with window $N$9 to reduce spatial resolution.
   • Performs dense attention in the compressed space, yielding $\{I_i\}$0 and guide scores $\{I_i\}$1.
   • Upsamples coarse outputs $\{I_i\}$2 by nearest neighbor to match the original token count, producing $\{I_i\}$3.
2. Selection Branch (Sparse Fine-Grained Attention):
   • Uses guide scores to select the top-$\{I_i\}$4 most informative regions for each compressed query index.
   • Gathers original high-resolution keys/values for those regions.
   • Computes attention only over the selected top-$\{I_i\}$5 keys/values, outputting $\{I_i\}$6.
3. Gated Aggregation:
   • Each image token learns a per-token gate to combine coarse (compression branch) and fine (selection branch) outputs:

$\{I_i\}$7

where $\{I_i\}$8.

1. Concatenation:
   • Final GSA output is concatenated as $\{I_i\}$9.

This arrangement mimics keypoint matching and propagates global context efficiently with aggressive sparsity.

Branch	Input Region	Operation Type
Compression	Downsampled	Dense attention
Selection	Top-k originals	Sparse attention
Gated Aggregation	All	Weighted fusion

3. Computational Complexity and Inference Acceleration

The dense attention baseline has $X_{\text{img}}$0 per-head complexity, where $X_{\text{img}}$1 scales with number of views and patches per view. GSA splits the global attention as:

• Compression: $X_{\text{img}}$2
• Selection: $X_{\text{img}}$3

With $X_{\text{img}}$4, $X_{\text{img}}$5, and $X_{\text{img}}$6, Speed3R achieves a substantial reduction in dominant $X_{\text{img}}$7 compute. In empirical evaluation with 1024 images and 256 patches per view ($X_{\text{img}}$8), Speed3R reduces end-to-end inference time by $X_{\text{img}}$9 (from $X_{\text{spec}}$0 s to $X_{\text{spec}}$1 s using $X_{\text{spec}}$2 backbone). The practical speedup is moderated by memory, special-token handling, and kernel-level implementation.

4. Training Objectives and Implementation Details

Speed3R supports both VGGT and $X_{\text{spec}}$3 backbones, each with tailored training protocols:

• Speed3R-VGGT employs knowledge distillation from dense VGGT:

$X_{\text{spec}}$4

• Camera loss: Huber error on camera poses.
• Aleatoric depth loss incorporating per-pixel uncertainty and depth gradients.
  • Speed3R-π³ follows the original multi-term loss of $X_{\text{spec}}$5:
• Aligned depth L1 loss (mean-normalized)
• Surface-normal loss (using MoGe)
• Relative pose loss across all predicted pairs

Implementation specifics include: compression window $X_{\text{spec}}$6, top-$X_{\text{spec}}$7 selection $X_{\text{spec}}$8, 80 training epochs (800 steps/epoch), base learning rate $X_{\text{spec}}$9, gradient accumulation (batch size 32 on 8 H20 GPUs). For large datasets, input view sampling is 2–24 views per sequence.

5. Quantitative Performance and Benchmarking

Speed3R attains near-parity with dense reconstruction models across principal evaluation metrics, even under 80–95% token sparsity. Key reported results:

Pose Estimation (AUC%)

• Two-View (ScanNet-1500):
  • VGGT dense: 37.45/59.24/75.69 (@5/10/20)
  • Speed3R-VGGT: 37.02/59.11/75.62
  • $T$0 dense: 38.76/61.57/77.61
  • Speed3R-π³: 36.97/59.83/76.38
• Multi-View (RE10K, CO3Dv2, 10 views):
  • VGGT dense: 74.17 (AUC@30)
  • Speed3R-VGGT (84% sparsity): 74.81
  • $T$1 dense: 87.37
  • Speed3R-π³ (94% sparsity): 87.17
• Long-Sequence (Tanks & Temples, $T$2300 views):
  • VGGT dense: 34.5 s, AUC@30=77.67
  • Speed3R-VGGT: 6.55 s (5.2×), AUC=76.57
  • $T$3 dense: 22.3 s, AUC=79.63
  • Speed3R-π³: 4.19 s (5.3×), AUC=79.77

Dense Point-Cloud Metrics (DTU & ETH3D)

• Speed3R variants match dense backbones within $T$4 on accuracy, completeness, and normal-consistency.

Efficiency–Accuracy Trade-off

Speed3R sets a new efficiency–accuracy Pareto frontier relative to training-free sparse baselines (Block-Sparse, FastVGGT), with >80% sparsity and competitive pose accuracy.

6. Ablation and Design Analysis

Ablation studies underscore the necessity of both the compression and selection branches. Removal or drastic reduction of either branch yields notable drops in pose AUC (e.g., selection branch removal on RE10K: $T$5, −2.91), or lesser gains at the expense of accuracy when varying $T$6 or window size. Reference-frame attention and register tokens yield marginal but consistent improvements. Distillation (teacher–student) is also crucial for peak performance, with uniform $T$7 drops in the absence of such supervision.

7. Kernel Implementation and Differentiability

The dual-branch attention is implemented as a fused Triton+FlashAttention2 kernel, leveraging on-chip SRAM to efficiently stream top-$T$8 selection and avoid materializing full score matrices. This approach enhances data locality and practical runtime. Both branches are constructed to be fully differentiable, preserving compatibility with end-to-end backpropagation and large-scale scene reconstruction workflows.

Speed3R’s dual-branch GSA mechanism enables feed-forward 3D models to concentrate computation on a sparse set of informative regions, yielding an up to $T$9 inference speedup on long input sequences with minimal losses in pose and geometry accuracy relative to their dense-attention counterparts (Ren et al., 9 Mar 2026).