TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion (2401.09416v1)

Abstract: We present TextureDreamer, a novel image-guided texture synthesis method to transfer relightable textures from a small number of input images (3 to 5) to target 3D shapes across arbitrary categories. Texture creation is a pivotal challenge in vision and graphics. Industrial companies hire experienced artists to manually craft textures for 3D assets. Classical methods require densely sampled views and accurately aligned geometry, while learning-based methods are confined to category-specific shapes within the dataset. In contrast, TextureDreamer can transfer highly detailed, intricate textures from real-world environments to arbitrary objects with only a few casually captured images, potentially significantly democratizing texture creation. Our core idea, personalized geometry-aware score distillation (PGSD), draws inspiration from recent advancements in diffuse models, including personalized modeling for texture information extraction, variational score distillation for detailed appearance synthesis, and explicit geometry guidance with ControlNet. Our integration and several essential modifications substantially improve the texture quality. Experiments on real images spanning different categories show that TextureDreamer can successfully transfer highly realistic, semantic meaningful texture to arbitrary objects, surpassing the visual quality of previous state-of-the-art.

• The paper introduces TextureDreamer, a framework that automates 3D texture synthesis using geometry-aware score distillation for realistic outputs.
• It leverages personalized variational score distillation and ControlNet architectures to transfer textures onto complex shapes using few input images.
• The method overcomes traditional limitations by enhancing 3D consistency and photorealism, thereby democratizing detailed texture creation.

Introduction to TextureDreamer

The creation of realistic, detailed textures for 3D content holds a paramount place in various applications, such as augmented and virtual reality, robotics, and the entertainment industry. Traditional methods for crafting textures for 3D assets can be labor-intensive, expensive, and generally reliant on professional artists. Recent years have seen strides towards automating this process, yet the challenges of needing large sets of images or being constrained to specific object categories have remained obstacles. A new framework, known as TextureDreamer, aims to break down these barriers by transferring textures from a minimal number of images (typically 3 to 5) onto any target 3D shape.

Key Innovations of the Framework

TextureDreamer is not just another step but a leap forward in automation for texture creation. This method leverages the concept of personalized geometry-aware score distillation (PGSD), drawing on the strength of diffusion-based generative models. These models, originally trained on massive text-image pair datasets, have held the spotlight for their capacity to produce high-quality, diverse images from text prompts. TextureDreamer twists the narrative by using these models to extrapolate texture details from a handful of images.

The method outshines previous ones by not requiring densely sampled views or category-specific data to produce highly detailed, relightable textures. It combines personalized modeling for texture extraction, variational score distillation for finer texture representation, and ControlNet architecture, explicitly guiding the generative process using geometry information. This fusion creates textures that are semantically meaningful and visually richer compared to what presently exists.

Overcoming Challenges in Texture Synthesis

TextureDreamer addresses two primary limitations that have troubled past attempts at texture synthesis. It utilizes Variational Score Distillation (VSD) rather than the traditional Score Distillation Sampling, which is known to cause images to appear overly smooth or saturated. By treating the full 3D representation as a variable and aligning it with the pre-trained diffusion model, VSD allows the system to produce more photorealistic outputs without heavy reliance on guidance weight, crucial for achieving lifelike textures.

To solve 3D consistency issues, a common pitfall where textures lack alignment with the object geometry, TextureDreamer introduces a geometry-aware approach. It incorporates normal maps from the 3D mesh into the distillation process, which vastly improves how textures wrap around and adhere to the complexities of the 3D object's shape. Experiments conducted across various real-world images and object categories have illustrated that this method significantly surpasses existing techniques in transferring textures that maintain fidelity to the original images while conforming seamlessly to the 3D models.

Potential Impact and Future Directions

TextureDreamer stands poised to democratize the process of texture creation significantly. Its ability to produce high-quality textures from a small set of uncorrelated images could make detailed and realistic 3D modeling more accessible to a wider audience, beyond the domain of trained professionals, potentially sparking a transformation in the fields of 3D graphics and content generation.

As it often goes with innovation, TextureDreamer is not without its limitations. Special textures that are non-repetitive or unique may challenge the framework, and input images with a limited range of viewpoints may result in inconsistencies. Nonetheless, these challenges open avenues for future research and refinements that could further enhance the framework's capabilities.

TextureDreamer marks an empowering step toward more efficient, intelligent, and inclusive methods for 3D texture generation, offering exciting prospects for creators and technologists alike in the pursuit of ever-more immersive and realistic digital worlds.