LightIt: Illumination Modeling and Control for Diffusion Models (2403.10615v2)

Abstract: We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limitations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes. Additionally, we use our generated dataset to train an identity-preserving relighting model, conditioned on an image and a target shading. Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.

• The paper introduces LightIt, a novel framework that enables explicit illumination control in diffusion models.
• It employs a three-stage pipeline involving image pairs and shading maps to achieve precise lighting direction and quality.
• Experimental results show significant improvements in lighting prediction and perceptual quality over baseline methods.

LightIt: Illumination Modeling and Control for Diffusion Models

The paper presents LightIt, a novel method to achieve explicit illumination control in image generation processes, particularly focusing on diffusion models. This framework addresses the current limitations of generative methods, which often lack the capability to control lighting explicitly—an essential aspect for artistic expressions like mood setting or cinematic design.

Methodology

The method centralizes around generating image pairs along with normal and shading maps to serve as training data for controlling illumination through diffusion models. LightIt employs a three-stage pipeline to estimate direct shading, conditioned on geometry and specific lighting directions. First, the method predicts image features, projects those into a 3D feature grid, and estimates a density field. Utilizing this spatial information, the model traces rays to compute shade and shadow maps, which are further refined to achieve direct shading.

The paper utilizes a residual control encoder and decoder framework to enhance the feature representation required for lighting control, thereby ensuring consistent and controllable outputs without compromising the initially learned image textures from diffusion models like Stable Diffusion.

Datasets and Experiments

The authors highlight their use of the Outdoor Laval dataset, extending it with rendered shading maps to create a comprehensive dataset supporting geometry and lighting control in image synthesis. Experiments are conducted in various domains including relighting and text-driven image generation to test the efficacy of the system.

In addition to their train-test regimes using paired images and shading maps, a perceptual user paper evaluates the model's fidelity in maintaining consistent lighting across generated images, suggesting that LightIt's outputs are preferred over uncontrolled methods regarding lighting fidelity and image perceptual quality.

Numerical Results

The paper delineates several notable quantitative assessments. In perceptual image generation quality evaluations, LightIt demonstrates superior lighting prediction quality (L-PQ) with a score of 95.57 in comparison to the baseline (4.43). Moreover, the paper reflects better integration with text alignment (T-PQ) and improved perceptual image quality (I-PQ), indicating that improved lighting inherently enhances image fidelity.

Implications and Future Work

LightIt sets a foundational step in controlled generative modeling with explicit lighting in mind, broadening the potential application scope in both artistic and practical domains. By providing a framework for training diffusion models with precise lighting controls, the paper offers a pathway toward more semantically coherent and realist image generation technologies.

Looking forward, the researchers hint at expanding this model to accommodate complex lighting scenarios, such as point and area light sources. Integrating a robust lighting estimation method with this framework may further reduce the dataset burden and improve model applicability.

Conclusion

This paper contributes a substantial advancement in the field of controlled image synthesis, demonstrating that diffusion models can be augmented with lighting controls through strategic data augmentation and feature conditioning pipelines. The method potentially broadens the use case for generative models in creative industries requiring fine-tuned image aesthetics.