LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation

Abstract: Diffusion models have demonstrated high-quality performance in conditional text-to-image generation, particularly with structural cues such as edges, layouts, and depth. However, lighting conditions have received limited attention and remain difficult to control within the generative process. Existing methods handle lighting through a two-stage pipeline that relights images after generation, which is inefficient. Moreover, they rely on fine-tuning with large datasets and heavy computation, limiting their adaptability to new models and tasks. To address this, we propose a novel Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation (LGTM), which manipulates the initial latent noise of the diffusion process to guide image generation with text prompts and user-specified light directions. Through a channel-wise analysis of the latent space, we find that selectively manipulating latent channels enables fine-grained lighting control without fine-tuning or modifying the pre-trained model. Extensive experiments show that our method surpasses prompt-based baselines in lighting consistency, while preserving image quality and text alignment. This approach introduces new possibilities for dynamic, user-guided light control. Furthermore, it integrates seamlessly with models like ControlNet, demonstrating adaptability across diverse scenarios.

• The paper introduces a training-free method that manipulates channel 1 of the latent noise to control global lighting direction and intensity in diffusion models.
• It integrates a light mask with initial noise modulation to achieve photometric realism without retraining, outperforming traditional prompt-based methods in light accuracy.
• The technique is composable with ControlNet for joint structural and illumination guidance, though it may adjust subject pose to ensure consistent lighting.

Training-Free Light-Guided Control in Text-to-Image Diffusion via Initial Noise Manipulation

Introduction

Text-to-image diffusion models have exhibited exceptional performance in conditional image generation, predominantly in controlling high-level structural or semantic attributes such as object layout, edge guidance, or segmentation. Despite this progress, prior work has neglected fine-grained photometric properties, especially lighting, which play a pivotal role in realism, aesthetics, and the affective impact of generated images. "LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation" (2603.24086) introduces a novel approach that enables direct, precise, and real-time user control over lighting direction during image synthesis, all without expensive re-training or model modification. The key insight is the identification and manipulation of specific latent channels within the VAE-based latent space of Stable Diffusion, revealing a disentangled structure where global illumination is localized to a single latent channel.

Existing Methods and Their Limitations

Previous approaches to controllable lighting in image generation either rely on two-stage relighting pipelines—such as DelightNET and IC-Light—that require training on illumination-annotated datasets, or utilize prompt-based methods, which are unreliable for consistent photometric rendering. Two-stage relighting methods are computationally intensive, dependent on significant GPU resources and large-scale curated datasets, limiting their scalability and adaptability to new generative backbones.

Prompt engineering in Stable Diffusion, while model-agnostic and training-free, falls short in consistently realizing directional illumination as specified in the text prompt (Figure 1).

Figure 1: Existing prompt-based methods do not yield consistent lighting control; in contrast, LGTM ensures the generated image follows the user-specified direction.

Methodology

Channel-wise Sensitivity Analysis

The core discovery is that the VAE latent space channels encode specific visual factors in a highly disentangled manner. Channel-wise sensitivity analyses demonstrate that scaling channel 1 of the latent tensor $z_T \in \mathbb{R}^{H/8 \times W/8 \times 4}$ directly influences global brightness and alters the inferred illumination direction. In contrast, channels 2--4 have minor, predominantly chromatic effects (Figure 2).

Figure 2: Only manipulation of channel 1 significantly alters the scene's overall brightness and light direction; other channels mainly affect color tone.

Light Conditional Generation (LCG)

User control is operationalized via an interactive interface where the user specifies a light source position or direction. A corresponding light mask $m_l$—a linear gradient from the light source—defines the spatial attenuation of light. This mask is then mapped to the spatial dimensions of the latent representation and solely applied to channel 1.

Latent Space Light Guidance (LSLG)

Rather than modifying the denoising networks, LGTM perturbs only the initial noise sample:

$\hat{z}_T^1 = z_T^1 \odot (1 + m_l),$

where $z_T^1$ is the initial noise for channel 1, and $m_l$ is the spatial mask. This technique leaves the base diffusion model untouched and is compatible with arbitrary model architectures.

Figure 3: User specifying a text prompt and lighting condition; channel 1 of the latent noise is spatially modulated to guide illumination throughout the generative process.

Empirical Evaluation

Experiments utilize SDXL as a backbone and benchmarks against conventional prompt-based methods on the Asirra Cat and Dog dataset. Metrics include FID (distributional similarity), NIMA (aesthetics), CLIP-I/T (text-image consistency), and a custom light accuracy score derived from YOLOv8-based object detection and shadow direction estimation.

Qualitative inspection demonstrates that LGTM consistently adheres to the user-defined light direction and intensity, producing coherent shadows and lighting gradients that complement the semantic content dictated by the prompt (Figure 4).

Figure 4: LGTM is able to faithfully align both the global light direction and local shadow geometry as requested, unlike the prompt-based baseline.

Quantitative evaluations reinforce this finding: in cat and dog images with left or right lighting, prompt-based baseline models produce correct shadow alignment only 51–53% of the time (random guess), while LGTM achieves 77–79% alignment across categories. NIMA and CLIP consistency are well-preserved; the FID increases—consistent with a shift away from the dataset's marginal lighting distribution—are an expected artifact of strong illumination guidance.

Joint Structural+Lighting Control

LGTM's training-free design allows seamless integration with ControlNet, enabling simultaneous conditioning on edge/canny maps (geometry) and lighting cues. It outperforms standard ControlNet, which is incapable of photometric control, both qualitatively and quantitatively (Figure 5).

Figure 5: LGTM with ControlNet achieves precise alignment of both explicit edge structure and illumination direction, demonstrating composable conditional control.

Analysis and Limitations

A notable behavioral property emerges when using LGTM with structural conditions: the model sometimes alters subject orientation to enforce physically plausible lighting, even overriding explicit ControlNet constraints. While this enhances photorealism in some cases, it can counteract precise pose controls required in other applications (Figure 6). This exposes a non-trivial entanglement between geometry and photometry in current generative models.

Figure 6: Subject's pose is adjusted to maintain global light consistency, occasionally overriding explicit ControlNet conditioning.

Implications and Future Directions

The introduction of direct latent-space lighting control reconfigures the methodology of visual prompt engineering for diffusion models. It establishes that high-level photometric attributes can be manipulated without explicit dataset annotation or retraining and that the latent channels of current pre-trained VAEs possess highly disentangled structure relevant to photorealistic visual control. These findings raise the prospect of unsupervised discovery and manipulation of other perceptual attributes, potentially facilitating more granular and interpretable controllability in large diffusion models.

The tendency for lighting control to override geometry-conditional signals warrants further investigation into architectural and loss-function interventions that could disentangle photometric and geometric controls. Future research could extend this approach to real-time video diffusion, higher-resolution harmonization, and broader classes of machine-guided photo-editing tasks.

Conclusion

This work demonstrates precise, training-free control of lighting in text-to-image diffusion by leveraging the latent space structure of Stable Diffusion. By identifying channel 1 as the locus of global illumination encoding, LGTM realizes substantial gains in adherence to user-specified lighting cues compared to baseline methods, maintains visual/textual quality, and is composable with structural control frameworks. The methodology is notable for requiring neither retraining nor model modification, thus offering significant practical utility and opening new avenues for interpretable, user-driven generative modeling.