Epipolar-Free 3D Gaussian Splatting for Generalizable Novel View Synthesis (2410.22817v2)

Abstract: Generalizable 3D Gaussian splitting (3DGS) can reconstruct new scenes from sparse-view observations in a feed-forward inference manner, eliminating the need for scene-specific retraining required in conventional 3DGS. However, existing methods rely heavily on epipolar priors, which can be unreliable in complex realworld scenes, particularly in non-overlapping and occluded regions. In this paper, we propose eFreeSplat, an efficient feed-forward 3DGS-based model for generalizable novel view synthesis that operates independently of epipolar line constraints. To enhance multiview feature extraction with 3D perception, we employ a selfsupervised Vision Transformer (ViT) with cross-view completion pre-training on large-scale datasets. Additionally, we introduce an Iterative Cross-view Gaussians Alignment method to ensure consistent depth scales across different views. Our eFreeSplat represents an innovative approach for generalizable novel view synthesis. Different from the existing pure geometry-free methods, eFreeSplat focuses more on achieving epipolar-free feature matching and encoding by providing 3D priors through cross-view pretraining. We evaluate eFreeSplat on wide-baseline novel view synthesis tasks using the RealEstate10K and ACID datasets. Extensive experiments demonstrate that eFreeSplat surpasses state-of-the-art baselines that rely on epipolar priors, achieving superior geometry reconstruction and novel view synthesis quality. Project page: https://tatakai1.github.io/efreesplat/.

• The paper presents eFreeSplat, an epipolar‐free model that leverages a self-supervised Vision Transformer to bypass traditional epipolar geometry for robust novel view synthesis.
• It employs an iterative cross-view Gaussians alignment technique to harmonize depth scales, reducing artifacts and improving rendering quality in sparse, non-overlapping views.
• Evaluations on datasets such as RealEstate10K and ACID show that eFreeSplat outperforms conventional epipolar-based methods in PSNR, SSIM, and computational efficiency.

Overview of Epipolar-Free 3D Gaussian Splatting for Generalizable Novel View Synthesis

The paper "Epipolar-Free 3D Gaussian Splatting for Generalizable Novel View Synthesis" presents an innovative approach to improving the quality and generalizability of novel view synthesis (NVS) by leveraging a novel method that eschews reliance on epipolar geometry. The proposed model, named eFreeSplat, addresses critical shortcomings in existing 3D Gaussian Splatting (3DGS) techniques that depend on epipolar priors. This reliance often fails under real-world conditions with sparse views, occluded regions, and non-overlapping images. By integrating a self-supervised learned Vision Transformer (ViT) for multiview feature extraction, eFreeSplat provides an efficient and effective solution for generalizable NVS tasks.

The research addresses several theoretical and practical gaps in the field of 3D vision and rendering. Notably, it challenges the conventional dependency on epipolar geometry for determining pixel correspondences across images—a method that proves fragile under non-ideal conditions. By utilizing a ViT backbone pre-trained on a large dataset for cross-view completion, eFreeSplat synthesizes novel views without degrading quality in challenging scenarios, such as when images have minimal overlap or are taken from vastly different angles.

Methodological Advancements

1. Epipolar-Free Approach: The core advancement is the departure from epipolar line constraints which are traditionally used to establish geometric correspondences between images. Instead, the use of a ViT for feature extraction enables the model to infer consistent 3D structural information by analyzing cross-view features at a high level. This makes the method robust to scenarios that render epipolar priors ineffective.
2. Iterative Cross-View Gaussians Alignment Method: An innovative component of the methodology is the Iterative Cross-view Gaussians Alignment (ICGA) technique, designed to harmonize depth scales across multiple views. By iteratively refining Gaussians' attributes through a feedback loop using warped view features, the method reduces discrepancies in the scale of predicted depth maps. This ensures more accurate rendering, reducing artifacts commonly introduced by scale-inconsistencies when aggregating multiview information.
3. Self-Supervised Pre-training: By implementing a self-supervised vision transformer model pre-trained on extensive cross-view datasets, eFreeSplat naturally incorporates 3D priors without explicit geometric constraints. The pre-training provides a robust understanding of global spatial relations across views, crucial for overcoming the challenges posed by overlapping and occluded regions.

Numerical Results and Evaluation

eFreeSplat was evaluated against leading state-of-the-art approaches like pixelSplat and MVSplat, especially in wide-baseline NVS tasks utilizing the RealEstate10K and ACID datasets. The results demonstrated superior performance, achieving higher geometric fidelity and rendering quality:

• The model achieved significant improvements over epipolar-based models, reflected in metrics such as PSNR and SSIM.
• It reduced artifacts and inaccuracies in both depth and color reconstructions, particularly effective in scenes with minimal reference input overlap.
• In terms of computational efficiency, eFreeSplat provided competitive rendering times, showing promise for real-time applications.

Implications and Future Directions

The implications of eFreeSplat are far-reaching. Practically, it simplifies deploying NVS systems in unconstrained environments, such as augmented reality or autonomous driving, where input images often have irregular overlaps and obstructions. Theoretically, it proposes a shift towards data-driven geometric understanding, emphasizing feature-based correspondences over hard-coded geometric rules.

Future research could explore expanding the training datasets to enhance the model's robustness further, possibly integrating multi-modal data inputs such as LiDAR alongside visual data to enrich scene understanding. Additionally, exploring the interplay between this model's architecture and generative capabilities of diffusion models could unlock new paradigms in 3D scene synthesis and manipulation.

In conclusion, eFreeSplat marks a promising advance in the field of 3D novel view synthesis with its epipolar-free methodology, opening avenues for more robust and versatile applications in AI and computer vision.