CoCoGaussian: Leveraging Circle of Confusion for Gaussian Splatting from Defocused Images (2412.16028v2)

Abstract: 3D Gaussian Splatting (3DGS) has attracted significant attention for its high-quality novel view rendering, inspiring research to address real-world challenges. While conventional methods depend on sharp images for accurate scene reconstruction, real-world scenarios are often affected by defocus blur due to finite depth of field, making it essential to account for realistic 3D scene representation. In this study, we propose CoCoGaussian, a Circle of Confusion-aware Gaussian Splatting that enables precise 3D scene representation using only defocused images. CoCoGaussian addresses the challenge of defocus blur by modeling the Circle of Confusion (CoC) through a physically grounded approach based on the principles of photographic defocus. Exploiting 3D Gaussians, we compute the CoC diameter from depth and learnable aperture information, generating multiple Gaussians to precisely capture the CoC shape. Furthermore, we introduce a learnable scaling factor to enhance robustness and provide more flexibility in handling unreliable depth in scenes with reflective or refractive surfaces. Experiments on both synthetic and real-world datasets demonstrate that CoCoGaussian achieves state-of-the-art performance across multiple benchmarks.

• The paper presents a novel framework that integrates the Circle of Confusion into Gaussian Splatting to reconstruct 3D scenes from defocused images.
• It employs a physically grounded model with learnable aperture and depth parameters to accurately estimate defocus blur and CoC diameters.
• Experimental results on synthetic and real-world datasets demonstrate state-of-the-art performance using metrics like PSNR, SSIM, and LPIPS.

Overview of CoCoGaussian: Leveraging Circle of Confusion for Gaussian Splatting from Defocused Images

The paper "CoCoGaussian: Leveraging Circle of Confusion for Gaussian Splatting from Defocused Images" introduces a novel framework for reconstructing 3D scenes using defocused images, advancing the methodology of 3D Gaussian Splatting (3DGS) in the context of defocus blur. The primary objective of this research is to address the realistic challenges posed by acquiring 3D scene representations from images affected by defocus blur, by leveraging a concept known as the Circle of Confusion (CoC).

Technical Approach

The paper proposes CoCoGaussian, a model that integrates the Circle of Confusion into Gaussian Splatting, to effectively recreate 3D scenes using defocused images. The CoC refers to the optical effect where the radiance from a point in a scene that is not located on the focus plane forms a circular pattern on the image sensor, leading to defocus blur. CoCoGaussian models the CoC in a physically grounded manner by employing 3D Gaussians and calculating the CoC diameters using depth and a learnable aperture parameter, denoted as $K$.

The framework utilizes several Gaussian sets to approximate the CoC shape and introduces a learnable scaling factor to increase flexibility, particularly in scenarios involving reflective or refractive surfaces where depth may be uncertain. Moreover, CoCoGaussian extends its applicability by learning aperture and focus plane information, thus allowing for customizable depth-of-field configurations during rendering.

Experimental Results

The research demonstrates that CoCoGaussian achieves superior performance on both synthetic and real-world datasets when compared with other state-of-the-art methods. These datasets include the Deblur-NeRF synthetic and real-world datasets and the DoF-NeRF dataset. The framework's efficacy is measured using PSNR, SSIM, and LPIPS, showing state-of-the-art results in each case. Notably, CoCoGaussian's robust handling of depth uncertainties and its flexibility in adapting CoC parameters were crucial to achieving such performance gains.

Implications and Future Work

This work holds significant implications for areas requiring high-quality 3D scene renderings from imperfect image data, like virtual reality, augmented reality, and robotics. By overcoming the limitation of defocus blur, CoCoGaussian enhances the potential of 3DGS for more versatile and robust applications in these fields. This research also opens opportunities for further exploration into the dynamic adaptation of rendering parameters in response to changing scene configurations or environmental conditions.

Future directions could include refining the adaptive capabilities of CoCoGaussian, particularly by enhancing the model's capacity to deal with underestimations of CoC diameter. Furthermore, exploring the integration of this model with newer neural architectures for improved scalability and performance would be a promising extension of this work.