How to Use Diffusion Priors under Sparse Views? (2412.02225v1)

Abstract: Novel view synthesis under sparse views has been a long-term important challenge in 3D reconstruction. Existing works mainly rely on introducing external semantic or depth priors to supervise the optimization of 3D representations. However, the diffusion model, as an external prior that can directly provide visual supervision, has always underperformed in sparse-view 3D reconstruction using Score Distillation Sampling (SDS) due to the low information entropy of sparse views compared to text, leading to optimization challenges caused by mode deviation. To this end, we present a thorough analysis of SDS from the mode-seeking perspective and propose Inline Prior Guided Score Matching (IPSM), which leverages visual inline priors provided by pose relationships between viewpoints to rectify the rendered image distribution and decomposes the original optimization objective of SDS, thereby offering effective diffusion visual guidance without any fine-tuning or pre-training. Furthermore, we propose the IPSM-Gaussian pipeline, which adopts 3D Gaussian Splatting as the backbone and supplements depth and geometry consistency regularization based on IPSM to further improve inline priors and rectified distribution. Experimental results on different public datasets show that our method achieves state-of-the-art reconstruction quality. The code is released at https://github.com/iCVTEAM/IPSM.

• The paper introduces Inline Prior Guided Score Matching (IPSM) to overcome the limitations of diffusion priors in sparse-view 3D reconstruction.
• It integrates 3D Gaussian Splatting with depth and geometric consistency regularization, achieving significant gains in SSIM, LPIPS, PSNR, and AVGE metrics.
• The approach streamlines novel view synthesis by eliminating the need for fine-tuning and extensive supervisory data while enhancing reconstruction quality.

An Examination of Diffusion Priors in Sparse View 3D Reconstruction

The paper "How to Use Diffusion Priors under Sparse Views?" investigates an advanced approach for 3D reconstruction from sparse-view inputs, leveraging diffusion models as priors. Novel View Synthesis (NVS) remains a challenging problem in computer vision, particularly when limited input views are provided. Prevalent methods predominantly rely on semantic or depth priors, which although effective, introduce constraints regarding the supervisory data requirements. The diffusion models, although promising as visual priors, have been found wanting in performance due to intrinsic challenges in entropy and mode deviation when dealing with sparse views.

Inline Prior Guided Score Matching (IPSM)

The authors present a novel method, Inline Prior Guided Score Matching (IPSM), developed to address the diffusion prior limitations. IPSM intends to correct rendered image distribution by utilizing inline priors, leveraging the geometric consistency inherent in the pose relationships between input views. This rectification splits the original SDS optimization objective into sub-objectives, effectively controlling for mode deviation without necessitating fine-tuning or pre-training of diffusion models.

Methodology and Approach

The paper explores the intersection of rendering techniques and diffusion models, utilizing 3D Gaussian Splatting (3DGS) as the reconstruction backbone. The authors integrate IPSM with this framework, enhancing the scene's three-dimensional geometric fidelity by supplementing a depth and geometric consistency regularization alongside the inline priors. This technique aims to mitigate the poor reconstruction performance typically encountered with sparse views, by leveraging visual-rectification guidance to stabilize the optimization process toward the desired mode.

A key component of this approach is the inline prior's ability to use rendered depth maps to accurately transform view-dependent images and establish consistency in unseen views. This is particularly important in rectifying distributions from sparse views, a contribution that has proved effective in stabilizing the mode-seeking process of score distillation.

Empirical Evaluation and Results

The authors demonstrate their method's prowess through extensive experiments conducted on benchmark datasets like LLFF and DTU. Their results illustrated that the proposed IPSM-Gaussian pipeline significantly outperformed state-of-the-art methods across multiple metrics. They showcased notable improvements in SSIM, LPIPS, PSNR, and AVGE scores, underlining the robustness of their method against traditional approaches.

An experimental analysis further revealed that conventional SDS approaches inadvertently lead to performance degradation, attributed to optimization failures induced by insufficient mode-seeking in sparse view conditions. Conversely, the IPSM not only remedied these issues but also achieved a marked improvement in qualitative and quantitative reconstruction quality, lending credence to its utility in complex view reconstruction tasks.

Broader Implications and Future Directions

The advancements introduced by IPSM reveal important theoretical implications for sparse view 3D reconstruction—demonstrating the effective use of rectified score matching to harness the potential of diffusion models without additional resource overheads like fine-tuning or pre-training. Practically, this approach simplifies the requirements for integrating diffusion priors into NVS systems, potentially broadening the applicability of such methods in real-time systems and applications with constraints on computational resources.

Looking forward, an encouraging direction is the exploration of enhanced depth regularizations and more sophisticated inline prior models, potentially through neural architectures adept at synthesizing even more constrained datasets. Furthermore, investigating the scalability of such diffusion model integrations in other realms of computer vision, such as augmented reality and virtual reality real-time rendering, could offer fascinating new research opportunities. As these diffusion-based methods mature, their impact on NVS and wider machine learning fields will likely grow, paving new pathways for efficient, high-quality 3D scene reconstructions.