StreamGS: Online Generalizable Gaussian Splatting Reconstruction for Unposed Image Streams (2503.06235v2)

Abstract: The advent of 3D Gaussian Splatting (3DGS) has advanced 3D scene reconstruction and novel view synthesis. With the growing interest of interactive applications that need immediate feedback, online 3DGS reconstruction in real-time is in high demand. However, none of existing methods yet meet the demand due to three main challenges: the absence of predetermined camera parameters, the need for generalizable 3DGS optimization, and the necessity of reducing redundancy. We propose StreamGS, an online generalizable 3DGS reconstruction method for unposed image streams, which progressively transform image streams to 3D Gaussian streams by predicting and aggregating per-frame Gaussians. Our method overcomes the limitation of the initial point reconstruction \cite{dust3r} in tackling out-of-domain (OOD) issues by introducing a content adaptive refinement. The refinement enhances cross-frame consistency by establishing reliable pixel correspondences between adjacent frames. Such correspondences further aid in merging redundant Gaussians through cross-frame feature aggregation. The density of Gaussians is thereby reduced, empowering online reconstruction by significantly lowering computational and memory costs. Extensive experiments on diverse datasets have demonstrated that StreamGS achieves quality on par with optimization-based approaches but does so 150 times faster, and exhibits superior generalizability in handling OOD scenes.

StreamGS: Online Generalizable Gaussian Splatting Reconstruction for Unposed Image Streams

The paper, titled "StreamGS: Online Generalizable Gaussian Splatting Reconstruction for Unposed Image Streams," introduces a novel methodology aimed at enhancing the reconstruction of 3D Gaussian Splatting (3DGS) for image streams without requiring pre-known camera poses. Addressing the deficiencies in both efficiency and generalizability encountered in existing 3DGS methods, the authors propose an innovative pipeline named StreamGS that performs online 3D reconstruction in real-time.

Key Contributions

The primary achievements of the research are as follows:

1. StreamGS showcases an online 3D reconstruction pipeline that processes streams of unposed images without relying on preset camera parameters, marking a significant development in the domain.
2. The adaptive refinement component enhances cross-frame consistency by using content-adaptive descriptors, reducing the redundancy inherent in adjacent frames and concurrently cutting down computational costs.
3. The method boasts an impressive performance in novel view synthesis, with reconstructions performed 150 times faster than existing optimization-based methods such as CF-3DGS, maintaining comparable quality and demonstrating superior handling of out-of-domain scenes.

Methodology

The methodology leverages Gaussian Splatting to transform image streams into a cohesive 3D representation. StreamGS progressively constructs the 3D representation by integrating each frame with a feed-forward approach and employs a pre-trained DUSt3R model for initial 3D point prediction. The model then refines this using content-adaptive descriptors, improving the robustness of pixel correspondences across frames to better aggregate 3D Gaussian features and control density adaptively.

This research involves multiple processes:

• Initialization of two-view reconstruction where both 3D points and coarse camera parameters are estimated.
• Adaptive refinement to enhance camera poses and 3D points using newly established matches between consecutive frames.
• A feed-forward Adaptive Density Control (ADC) technique, which efficiently merges redundant Gaussian elements to reduce memory usage while maintaining accuracy.

Numerical Results and Implications

The reported results are compelling, showing that StreamGS processes frames 150 times faster than optimization-centric approaches like CF-3DGS while sustaining a close quality of view synthesis. Evaluation on a variety of datasets, including RE10K, ACID, ScanNet, DL3DV, and MVImgNet, indicates that StreamGS excels in generalizability, able to tackle out-of-domain scenes with proficiency superior to conventional methods that either depend on exhaustive optimization or known camera parameters.

Practical and Theoretical Implications

StreamGS's real-time constraint adaptation and its fast processing capabilities make it particularly suitable for applications in real-time scene reconstruction, augmented reality (AR), and virtual reality (VR) where users require immediate feedback.

Theoretically, this research emphasizes the utility of feed-forward networks for complex 3D tasks, paving the way for further exploration of lightweight architectures that balance efficiency and accuracy.

Future Directions

Future research could extend StreamGS by incorporating deeper neural architectures to better handle reflections and textures lacking in significant detail. Additionally, there is potential for fusing semantic information with geometric data to enhance the semantic-aware reconstruction. Finally, adapting the proposed methodology to leverage temporal coherence beyond immediate frame pairs could further improve accuracy and robustness in dynamic or rapidly changing scenes.

In summary, StreamGS represents a significant advancement in real-time 3D scene reconstruction from image streams, offering a robust, generalizable solution adaptable across new and unforeseen domains while markedly improving processing efficiency.