MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors (2409.15273v2)

Abstract: Recent works in inverse rendering have shown promise in using multi-view images of an object to recover shape, albedo, and materials. However, the recovered components often fail to render accurately under new lighting conditions due to the intrinsic challenge of disentangling albedo and material properties from input images. To address this challenge, we introduce MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that incorporates a 2D prior on texture and material properties. We present StableMaterial, a 2D diffusion model prior that refines multi-lit data to estimate the most likely albedo and material from given input appearances. This model is trained on albedo, material, and relit image data derived from a curated dataset of approximately ~12K artist-designed synthetic Blender objects called BlenderVault. we incorporate this diffusion prior with an inverse rendering framework where we use score distillation sampling (SDS) to guide the optimization of the albedo and materials, improving relighting performance in comparison with previous work. We validate MaterialFusion's relighting performance on 4 datasets of synthetic and real objects under diverse illumination conditions, showing our diffusion-aided approach significantly improves the appearance of reconstructed objects under novel lighting conditions. We intend to publicly release our BlenderVault dataset to support further research in this field.

• The paper introduces a novel pipeline that integrates a learned 2D diffusion prior to significantly improve 3D inverse rendering accuracy in challenging lighting scenarios.
• It employs the StableMaterial model, trained on a synthetic BlenderVault dataset, using conditional diffusion and Score Distillation Sampling for robust albedo and ORM estimation.
• Experimental results show superior PSNR, SSIM, and LPIPS metrics, underscoring its potential impact on VR, ecommerce, and digital content creation applications.

MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors

The recent advancements in inverse rendering methodologies have showcased significant potential in recovering shape, albedo, and materials from multi-view images. Yet, the accuracy of rendering under novel lighting conditions remains challenging. This problem primarily arises from the intrinsic difficulty associated with disentangling albedo and material properties from the input images. The paper "MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors" introduces an innovative approach that leverages a 2D material diffusion prior to improve the fidelity of recovered 3D representations under diverse illumination.

MaterialFusion builds upon the limitations of conventional 3D inverse rendering techniques that struggle with the disentanglement of various surface properties from lighting information. The key contribution of the paper is introducing StableMaterial, a 2D diffusion model trained on artist-crafted datasets to predict albedo and material properties from images. This model, when integrated with a Score Distillation Sampling (SDS) approach, significantly enhances the performance of inverse rendering systems in novel lighting scenarios.

Methodology

The authors propose a pipeline that incorporates a learned 2D prior from a dataset of synthetic objects rendered in Blender, termed BlenderVault, comprising approximately 12K high-quality, diverse synthetic objects. The dataset significantly extends the training capabilities by providing high-quality Physically Based Rendering (PBR) assets. The paper integrates this dataset with the existing capabilities of Neural Radiance Fields (NeRF) and follows an approach to explicitly represent geometry as meshes with a simplified Disney principled BRDF model for material textures.

StableMaterial employs a conditional diffusion model applied to albedo and ORM (occlusion, roughness, metallicness) 2D maps from multi-view images to estimate materials under varying illumination. The training setup utilizes an encoder from Stable Diffusion 2.1 for the initial image representation, and a unique UNet structure adapted for contextual material understanding. The authors deploy this model in conjunction with an inverse rendering framework where SDS is instrumental in guiding the optimization of 3D properties.

Experiments and Results

The paper's empirical assessments span multiple datasets, including NeRF Synthetic, NeRFactor, BlenderVault, and Stanford-ORB. MaterialFusion's performance was gauged against current state-of-the-art methods, notably nvdiffrecmc, Relightable 3D Gaussian, and TensoIR. Across these datasets, MaterialFusion consistently outperforms these techniques in metrics such as PSNR, SSIM, and LPIPS, demonstrating superior quality in material and relighting estimation.

Specifically, in the Fidelity of Albedo and ORM predictions, MaterialFusion achieved substantial improvements in capturing high-frequency details and maintaining consistency across different viewpoints. The paper also involved ablation experiments to verify the impact of various components in the loss function, confirming the robustness of the proposed SDS+ loss formulation in enhancing overall performance.

Implications and Future Work

The implications of this work extend both practically and theoretically within the AI and Computer Vision communities. Practically, the improved accuracy in material reconstruction under diverse lighting conditions holds potential applications in virtual reality, ecommerce, and digital content creation, where realistic object representation is crucial. Theoretically, the integration of 2D diffusion priors in 3D tasks opens new research avenues in leveraging large-scale 2D model capabilities for enhancing 3D inverse rendering accuracy.

Future research directions could explore refining the network architectures and training procedures to further mitigate the information loss in encoding and decoding stages. Additionally, leveraging multi-view attention mechanisms during training, rather than just during inference, could be beneficial in obtaining more consistent material predictions. Another promising direction is adapting the approach for real-world scenarios where lighting conditions are even more unpredictable and complex.

Conclusion

"MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors" marks a substantial advance in the field of inverse rendering by demonstrating the efficacy of a learned 2D material diffusion prior in refining the accuracy of 3D reconstruction properties. This approach not only addresses fundamental limitations of earlier methods but also sets a new benchmark in the quest for more reliable and nuanced 3D representations under novel illumination conditions. The introduction of BlenderVault and the novel application of StableMaterial create a compelling case for the broader applicability and potential future developments in this domain.