Differentiable Point-based Inverse Rendering (2312.02480v2)

Abstract: We present differentiable point-based inverse rendering, DPIR, an analysis-by-synthesis method that processes images captured under diverse illuminations to estimate shape and spatially-varying BRDF. To this end, we adopt point-based rendering, eliminating the need for multiple samplings per ray, typical of volumetric rendering, thus significantly enhancing the speed of inverse rendering. To realize this idea, we devise a hybrid point-volumetric representation for geometry and a regularized basis-BRDF representation for reflectance. The hybrid geometric representation enables fast rendering through point-based splatting while retaining the geometric details and stability inherent to SDF-based representations. The regularized basis-BRDF mitigates the ill-posedness of inverse rendering stemming from limited light-view angular samples. We also propose an efficient shadow detection method using point-based shadow map rendering. Our extensive evaluations demonstrate that DPIR outperforms prior works in terms of reconstruction accuracy, computational efficiency, and memory footprint. Furthermore, our explicit point-based representation and rendering enables intuitive geometry and reflectance editing.

• The paper introduces a hybrid technique that combines point-based splatting with volumetric methods to tackle inverse rendering challenges.
• The paper reduces computational overhead by eliminating multiple ray samples and integrating an efficient shadow detection mechanism.
• The evaluations reveal enhanced accuracy and intuitive editing capabilities for geometry and reflectance in digital object reconstruction.

Differentiable point-based inverse rendering (DPIR) introduces an innovative approach to the longstanding challenge of inverse rendering—estimating the shape and reflective properties of real-world objects from images. This technology is vital for applications that involve relighting, virtual and augmented reality, and digital object reconstruction.

DPIR picks up where previous techniques have left gaps, particularly in dealing with the discontinuous nature of objects and the limited samples of light and view angles in the inverse rendering process. Traditional methods using mesh-based and volumetric rendering have faced difficulties with these challenges, leading to inaccuracies and inefficiencies.

At the heart of DPIR is a hybrid strategy that combines point-based rendering with volumetric techniques to create a novel forward-rendering engine that doesn't require multiple samplings per ray. This is contrasted with volumetric rendering that typically calls for multiple samples per ray, leading to higher computational demand.

The DPIR employs a two-fold representation: a hybrid geometric representation and a regularized basis BRDF (Bidirectional Reflectance Distribution Function) for reflectance. The hybrid geometric representation leverages point-based splatting (a form of rendering) for efficient rendering, while also drawing on volumetric properties to provide detailed geometry and stability in inverse rendering tasks. The second part of the representation, the regularized basis BRDF, is designed to overcome the ill-posed nature of inverse rendering caused by limited angular light and view samples.

An additional innovation in DPIR is the efficient shadow detection method that cuts down the computational heaviness often seen in existing learning-based inverse rendering methods. This is incorporated within DPIR's framework, streamlining the entire process for inverse rendering.

Extensive evaluations demonstrate that DPIR outperforms prior methods in terms of accuracy, computational efficiency, and memory requirements. In addition, the explicit point representation allows for intuitive editing of the geometry and reflectance of rendered scenes, providing flexibility for different visualization and editing purposes.

In summary, DPIR is a significant step forward for inverse rendering, delivering higher accuracy and efficiency while reducing the computational resources required. It also offers potential applications in editing and refining digital object representations, marking a substantial advancement in computer vision and graphics.