SaLon3R: Structure-aware Long-term Generalizable 3D Reconstruction from Unposed Images (2510.15072v1)

Abstract: Recent advances in 3D Gaussian Splatting (3DGS) have enabled generalizable, on-the-fly reconstruction of sequential input views. However, existing methods often predict per-pixel Gaussians and combine Gaussians from all views as the scene representation, leading to substantial redundancies and geometric inconsistencies in long-duration video sequences. To address this, we propose SaLon3R, a novel framework for Structure-aware, Long-term 3DGS Reconstruction. To our best knowledge, SaLon3R is the first online generalizable GS method capable of reconstructing over 50 views in over 10 FPS, with 50% to 90% redundancy removal. Our method introduces compact anchor primitives to eliminate redundancy through differentiable saliency-aware Gaussian quantization, coupled with a 3D Point Transformer that refines anchor attributes and saliency to resolve cross-frame geometric and photometric inconsistencies. Specifically, we first leverage a 3D reconstruction backbone to predict dense per-pixel Gaussians and a saliency map encoding regional geometric complexity. Redundant Gaussians are compressed into compact anchors by prioritizing high-complexity regions. The 3D Point Transformer then learns spatial structural priors in 3D space from training data to refine anchor attributes and saliency, enabling regionally adaptive Gaussian decoding for geometric fidelity. Without known camera parameters or test-time optimization, our approach effectively resolves artifacts and prunes the redundant 3DGS in a single feed-forward pass. Experiments on multiple datasets demonstrate our state-of-the-art performance on both novel view synthesis and depth estimation, demonstrating superior efficiency, robustness, and generalization ability for long-term generalizable 3D reconstruction. Project Page: https://wrld.github.io/SaLon3R/.

• The paper presents an online method that uses saliency-aware quantization to drastically reduce Gaussian redundancy.
• It achieves high-quality rendering and accurate depth estimation over 50+ views at more than 10 FPS without relying on camera poses.
• The approach incorporates a point transformer for anchor refinement, ensuring strong generalization and zero-shot performance across datasets.

Structure-aware Long-term Generalizable 3D Reconstruction from Unposed Images: SaLon3R

Introduction and Motivation

SaLon3R introduces a structure-aware, long-term generalizable framework for 3D Gaussian Splatting (3DGS) reconstruction from unposed and uncalibrated image sequences. The method addresses the limitations of prior generalizable 3DGS approaches, which suffer from excessive redundancy and geometric inconsistencies when fusing per-pixel Gaussians across many views, especially in long-duration video streams. SaLon3R is the first online generalizable GS method capable of reconstructing over 50 views at more than 10 FPS, achieving 50–90% redundancy removal without requiring camera parameters or test-time optimization.

Figure 1: Comparison between FreeSplat and SaLon3R, demonstrating superior online generalizable Gaussian reconstruction, high-quality rendering, accurate depth estimation, and significant redundancy removal.

Methodology

Incremental Structure-aware Reconstruction

SaLon3R leverages a feed-forward network for online 3DGS reconstruction. The backbone (CUT3R or VGGT) predicts global pointmaps, per-pixel Gaussian latents, and a saliency map encoding regional geometric complexity for each input frame. The network incrementally updates the global Gaussian set $\mathcal{G}_t$ and camera parameters $\{\bK_t, \bP_t\}$ for each frame, enabling efficient online scene representation.

Figure 2: SaLon3R pipeline: incremental prediction of camera parameters, pointmaps, and pixel-wise Gaussian latents, followed by saliency-aware quantization and structure-aware refinement.

Saliency-aware Gaussian Quantization

To address redundancy, SaLon3R introduces a saliency-aware quantization mechanism. The predicted Gaussian centers are voxelized, and points within each voxel are aggregated into compact anchors using a normalized weighted sum based on learned saliency scores. This prioritizes high-complexity regions, preserving geometric fidelity while aggressively pruning redundant Gaussians.

Structure-aware Refinement via Point Transformer

A lightweight Point Transformer V3 refines anchor attributes and saliency, capturing spatial relationships and resolving cross-frame geometric and photometric inconsistencies. The transformer operates on serialized anchors, enabling local attention and structure-aware learning. Adaptive Gaussian growing is employed, where the refined saliency controls the density of decoded Gaussians, allocating more primitives to complex regions and pruning in low-complexity areas.

Figure 3: Adaptive Gaussian growing: increasing $\beta$ threshold removes redundant Gaussians, improving performance with minimal degradation at high thresholds.

Training and Loss Functions

The reconstruction backbone is pretrained and fixed during training. The photometric loss combines perceptual (LPIPS) and L1 losses between rendered and ground-truth images. Edge-aware depth regularization enforces surface smoothness. The total loss is:

$$\mathcal{L}_{total} = \lambda_1\mathcal{L}_1 + \lambda_2\mathcal{L}_{\text{LPIPS}} + \lambda_3 \mathcal{L}_{\text{smooth}}$$

with $\lambda_1=1.0$, $\lambda_2=0.05$, $\lambda_3=0.0005$.

Experimental Results

Novel View Synthesis and Depth Estimation

SaLon3R achieves competitive or superior performance compared to both pose-required and pose-free baselines on ScanNet, ScanNet++, and Replica datasets. It consistently outperforms prior methods in depth estimation, even without pose or intrinsic input, and maintains high rendering quality with aggressive Gaussian pruning.

Figure 4: Qualitative comparison on ScanNet: SaLon3R yields sharper RGB and more accurate depth than baselines, especially in challenging regions.

Figure 5: Extrapolation view synthesis: SaLon3R maintains consistent 3D structure and rendering quality in out-of-distribution views, outperforming FreeSplat.

Generalization and Zero-shot Performance

SaLon3R demonstrates strong generalization in zero-shot settings, outperforming previous methods on cross-dataset evaluations. The structure-aware refinement enables robust extrapolation and interpolation, critical for scalable online reconstruction.

Ablation Studies

Component-wise ablation reveals that saliency-aware quantization and point transformer refinement are essential for balancing compactness and rendering fidelity. Adaptive Gaussian growing further optimizes density control. Increasing context views improves depth and perceptual metrics up to a saturation point, with stable FPS and Gaussian count, confirming scalability.

Figure 6: Visualization of rendered 3D saliency: high values concentrate on object boundaries and textured regions, validating the saliency mechanism.

Figure 7: Additional qualitative results: SaLon3R maintains high-quality RGB and depth rendering across diverse scenes and view configurations.

Figure 8: Global Gaussian splatting and extrapolation: SaLon3R reconstructs more consistent 3D structure than FreeSplat, especially in challenging extrapolated views.

Implementation Considerations

• Computational Requirements: SaLon3R achieves >10 FPS on A100 GPUs for 50+ views, with aggressive Gaussian pruning reducing memory footprint by up to 90%.
• Scalability: The feed-forward architecture and anchor-based quantization enable online processing of long video streams without test-time optimization or pose estimation.
• Limitations: Performance may degrade in dynamic scenes due to static reconstruction assumptions. Pose drift in the backbone can introduce artifacts in very long sequences; keyframe selection or bundle adjustment may mitigate this.
• Deployment: The method is suitable for real-time applications in robotics, AR/VR, and large-scale scene capture, where pose information is unavailable or unreliable.

Theoretical and Practical Implications

SaLon3R advances the state-of-the-art in generalizable 3DGS by introducing structure-aware inductive bias and saliency-driven quantization, enabling efficient, robust, and scalable online reconstruction from unposed images. The integration of point cloud transformers for anchor refinement sets a precedent for structure-aware scene representation learning. The method's strong generalization and redundancy removal capabilities have direct implications for real-world deployment in unconstrained environments.

Future work should address dynamic scene reconstruction, incorporating temporal modeling and motion tracking to extend SaLon3R to 4D globally consistent reconstruction. Enhancing backbone alignment and integrating global optimization strategies will further improve robustness in large-scale, multi-room scenarios.

Conclusion

SaLon3R presents a comprehensive solution for structure-aware, long-term generalizable 3D reconstruction from unposed images, combining saliency-aware quantization and transformer-based refinement to achieve state-of-the-art efficiency, fidelity, and generalization. Its design enables scalable online reconstruction, robust novel view synthesis, and accurate depth estimation, with significant implications for practical deployment and future research in dynamic scene understanding.