ZeST: Zero-Shot Material Transfer from a Single Image

Abstract: We propose ZeST, a method for zero-shot material transfer to an object in the input image given a material exemplar image. ZeST leverages existing diffusion adapters to extract implicit material representation from the exemplar image. This representation is used to transfer the material using pre-trained inpainting diffusion model on the object in the input image using depth estimates as geometry cue and grayscale object shading as illumination cues. The method works on real images without any training resulting a zero-shot approach. Both qualitative and quantitative results on real and synthetic datasets demonstrate that ZeST outputs photorealistic images with transferred materials. We also show the application of ZeST to perform multiple edits and robust material assignment under different illuminations. Project Page: https://ttchengab.github.io/zest

• The paper presents a novel zero-shot method that transfers material textures using only a single exemplar image without explicit 3D modeling.
• It leverages diffusion models and adapters to implicitly manage geometry, illumination, and material properties for photorealistic results.
• The approach supports versatile applications in digital art, game design, and e-commerce while outlining avenues for further refinement.

ZeST: A Novel Approach for Zero-Shot Material Transfer in Single Images

Introduction

The creation of ZeST, a pioneering approach for zero-shot material transfer directly onto objects in single images, addresses significant challenges in the fields of computer graphics and design. Distinct from existing methods, ZeST exploits diffusion models and adapters to transfer material appearances from an exemplar image to a target object in an input image without necessitating explicit 3D modeling, illumination analysis, or material property specification. This development is particularly relevant for applications in game design and e-commerce, where altering object materials can be both complex and labor-intensive.

Key Contributions

ZeST introduces a streamlined, zero-shot methodology for material transfer, leveraging the capabilities of pre-existing inpainting diffusion models and diffusion adapters. Key aspects of its innovation include:

• Zero-shot, Training-Free Material Transfer: Unlike prior works that depend on the aggregation of multiple exemplars and finetuning, ZeST requires only a single exemplar image for material transfer, effectively working in a zero-shot, training-free manner.
• Implicit Handling of 3D, Illumination, and Material Properties: By employing 2D depth estimation and segmentation alongside latent material transfer, ZeST circumvents the need for explicit 3D modeling, material property definitions, and illumination adjustments.
• Versatile Application Potential: The method supports a variety of graphics applications, including real-world image editing, multi-object material editing within a single image, and lighting-aware material transfers for textured mesh rendering.

Methodological Insights

ZeST's pipeline comprises several carefully chosen components to tackle the challenges of material transfer:

• Material Encoding: Utilizes IP-Adapter for encoding a material exemplar into a latent representation, isolating material aspects for transfer.
• Geometry and Illumination Guidance: Employs ControlNet for depth-based geometry preservation and a novel latent-space illumination guidance strategy that uses a combination of inpainting diffusion and foreground grayscaling to maintain illumination cues.
• Synthetic and Real-world Evaluation Datasets: For quantitative and qualitative analyses, the paper introduces new datasets encompassing a wide range of materials and objects under varying illumination conditions, demonstrating ZeST's adaptability and effectiveness.

Evaluation and Applications

Both qualitative and quantitative results underscore ZeST's proficiency in generating photorealistic images with accurately transferred materials, excelling in comparison to various baselines. Key applications highlighted include:

• Multiple Material Edits in a Single Image: ZeST can iteratively impart different materials to multiple objects within a single image, enriching the scope for complex image editing tasks.
• Lighting-Aware Material Transfer: Adjustments in the processing of input images allow ZeST to perform material transfers that are sensitive to changes in illumination, proving particularly useful when dealing with untextured mesh renderings.

Future Directions and Limitations

While ZeST represents a significant stride towards practical, zero-shot material editing in images, it exhibits certain limitations, such as partial material transfers and challenges in handling multiple materials within an exemplar image. These issues predominantly arise due to the latent space operations and models' prior knowledge. Addressing these limitations opens avenues for future research, potentially incorporating more nuanced region and material-specific encodings, or advanced models capable of disentangling complex material properties more effectively.

Conclusion

ZeST marks a notable advancement in the field of material transfer, rendering the process both accessible and efficient across varied applications. By avoiding the traditional requirements of explicit 3D modeling and material property adjustments, it paves the way for enhanced creativity and efficiency in digital art, game design, and beyond.