MaPa: Text-driven Photorealistic Material Painting for 3D Shapes (2404.17569v2)

Abstract: This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa

• The paper proposes the MaPa framework, which generates photorealistic, editable 3D materials from text by fusing segment-controlled diffusion with procedural material graphs.
• It employs a novel segmentation of 3D shapes onto 2D planes to guide material graph initialization and iterative optimization via differentiable rendering.
• Experimental results show MaPa outperforms baselines in metrics like FID and KID, demonstrating superior resolution, editability, and visual fidelity.

Enhanced 3D Material Generation from Text Descriptions Using MaPa Framework

Introduction to the MaPa Framework

The presented paper introduces the MaPa framework, which stands for Material Painting. This is a system designed to address the challenge of generating photorealistic and editable materials for 3D models from textual descriptions. It leverages segment-wise procedural material graphs as a novel representation approach. Unlike traditional methods which commonly use extensive paired datasets, MaPa utilizes a pre-trained 2D diffusion model to establish a creative and effective bridge that links textual information and material graph outputs.

Methodological Insights

• Segment-controlled Diffusion Model: The framework adopts a method where the 3D shape is segmented, and these segments are projected onto a 2D plane. This segmentation serves to guide the synthesis of aligned 2D images which are later used to infer and optimize the material graphs for each segment.
• Material Graph Initialization and Optimization: Material graphs are initialized by fetching similar existing graphs from a pre-constructed library. These graphs are then optimized in relation to the synthesized 2D image via a differentiable rendering module, ensuring that the final material is reflective of the textual description provided.
• Iterative Enhancement: The process iteratively refines the material representation, where subsequent optimization steps focus on segments lacking sufficient detail, enhancing overall consistency and photorealism.

Experimental Results and Evaluations

The tests demonstrate that MaPa outperforms existing baseline methods across multiple metrics, including photorealism, resolution, and editability. For quantitative validation, metrics such as FID (Frechet Inception Distance) and KID (Kernel Inception Distance) were utilized, highlighting MaPa's superior performance. Additionally, a user paper affirmed the qualitative advantage of MaPa, with users rating it highly in terms of visual quality and fidelity to text descriptions.

Theoretical and Practical Implications

From a theoretical standpoint, the integration of procedural material graphs with learning-based methods for generating 3D materials proposes a significant shift towards more automated and refined content creation processes in computer graphics and related fields. Practically, the ability to edit the materials after generation without substantial degradation in quality potentially reduces the time and effort required in digital content creation, particularly in entertainment and VR/AR applications.

Future Directions

Future work might focus on expanding the material graph library to cover a broader range of materials or enhancing the diffusion model to handle more complex segmentations and descriptions. Further efficiency improvements could also be achieved by developing methods for direct inference of material properties without needing iterative optimization procedures.

Conclusion

The MaPa framework introduces a compelling advancement in the field of 3D graphic content rendering, particularly in the automated generation of realistic and editable materials from textual descriptions. Its methodological novelties provide a robust foundation for future research and practical applications in the domain of 3D content creation.