DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models (2405.17176v1)

Abstract: 2D diffusion model, which often contains unwanted baked-in shading effects and results in unrealistic rendering effects in the downstream applications. Generating Physically Based Rendering (PBR) materials instead of just RGB textures would be a promising solution. However, directly distilling the PBR material parameters from 2D diffusion models still suffers from incorrect material decomposition, such as baked-in shading effects in albedo. We introduce DreamMat, an innovative approach to resolve the aforementioned problem, to generate high-quality PBR materials from text descriptions. We find out that the main reason for the incorrect material distillation is that large-scale 2D diffusion models are only trained to generate final shading colors, resulting in insufficient constraints on material decomposition during distillation. To tackle this problem, we first finetune a new light-aware 2D diffusion model to condition on a given lighting environment and generate the shading results on this specific lighting condition. Then, by applying the same environment lights in the material distillation, DreamMat can generate high-quality PBR materials that are not only consistent with the given geometry but also free from any baked-in shading effects in albedo. Extensive experiments demonstrate that the materials produced through our methods exhibit greater visual appeal to users and achieve significantly superior rendering quality compared to baseline methods, which are preferable for downstream tasks such as game and film production.

• The paper introduces a novel geometry- and light-aware diffusion framework that generates high-quality PBR materials from text by eliminating baked shading effects.
• It employs a modified Classifier Score Distillation loss and a hash-grid-based material representation to ensure consistency with object geometry and realistic lighting.
• Quantitative comparisons and user studies demonstrate DreamMat’s superior performance in overall quality, fidelity to text prompts, and effective material-light disentanglement.

DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models

Introduction

In computer graphics, the creation of high-quality object appearances significantly enhances rendering realism, especially in applications such as movies, games, and AR/VR. The generation of physically-based rendering (PBR) materials from text descriptions represents a progressive direction in addressing the labor-intensive and expertise-demanding process of object appearance creation.

The paper "DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models" introduces a novel approach to ameliorate the challenges associated with generating PBR materials. The method, dubbed DreamMat, leverages geometry- and light-aware diffusion models to refine material generation, thereby eliminating baked-in shading effects within albedo maps and enhancing the overall rendering quality.

Methodology

The DreamMat framework addresses several fundamental challenges inherent in distilling PBR materials from 2D diffusion models. Traditional 2D diffusion models, while powerful in generating final shading colors, often fail at accurate material parameter decomposition, leading to unrealistic rendering effects caused by baked-in shading highlights and shadows. DreamMat’s key innovations lie in two critical stages: the introduction of a geometry- and light-aware 2D diffusion model and the randomized application of environment lights during the material distillation process.

Material Representation and Rendering

The PBR materials are represented using a hash-grid-based approach, parameterized by albedo, roughness, and metallic values. The rendering equation, split into diffuse and specular components, employs Monte Carlo sampling techniques to render the object’s appearance under varying lighting conditions. By fixing the environmental lights to known HDR images, the ill-posed nature of the material decomposition task is mitigated, ensuring that the material generation aligns with realistic lighting contexts.

Distillation with Geometry- and Light-aware Diffusion Models

The distillation process employs a modified Score-Distillation Sampling (SDS) loss, termed Classifier Score Distillation (CSD) loss, to refine the generated materials. The CSD loss leverages both positive and negative text prompts, enhancing the fidelity of the generated appearances relative to the given descriptions. The incorporation of geometric conditions (depth and normal maps) and light conditions (predefined materials under a specified environment light) in the diffusion model’s training ensures that the generated images remain consistent with both the object’s geometry and the lighting environment.

Training and Implementation

The geometry- and light-aware diffusion model is finetuned using a substantial dataset of rendered images from the Objaverse, enriched with conditional images based on different geometry and lighting conditions. The material generation is performed on a mesh by distilling the finetuned diffusion model in conjunction with rendered images under various environment lights.

Results

Extensive qualitative and quantitative comparisons with state-of-the-art methods, including TANGO, TEXTure, Text2Tex, and Fantasia3D, demonstrate DreamMat’s superiority in generating high-quality, light-consistent materials. The user paper, involving 42 respondents, highlighted DreamMat’s excellence across several metrics such as overall quality, fidelity to text prompts, and the efficacy of material-light disentanglement.

Implications and Future Work

DreamMat sets a precedence for integrating robust diffusion models with physically-informed rendering constraints, thus paving the way for more sophisticated and intuitive tools for material generation in computer graphics. Future directions may explore enhancing the generalizability of this approach to more complex scenes and diverse material types. Further research may also investigate optimizing the efficacy and computational efficiency of the distillation process.

Conclusion

The DreamMat methodology introduces a significant advancement in the domain of text-guided PBR material generation, adeptly handling the nuanced challenges of lighting and geometric consistency. The framework not only demonstrates superior performance over existing methods but also significantly broadens the spectrum of practical applications, ensuring high-quality material generation suitable for modern rendering engines and robust enough for detailed scene compositions.