PractiLight: Diffusion-Based Relighting

• The paper introduces PractiLight, a relighting framework that uses a small LoRA adapter on self-attention layers of pretrained diffusion models to achieve state-of-the-art control with minimal data.
• It employs a LoRA regressor trained on synthetic data to predict a direct-irradiance map, guiding the diffusion process for coherent lighting adjustments while preserving scene identity.
• Results demonstrate robust identity and style preservation across diverse scenes with significantly reduced training parameters, data requirements, and GPU time compared to traditional methods.

Searching arXiv for the specified paper and closely related relighting work to ground the article in current research. PractiLight is a relighting framework for diffusion-generated images that treats lighting control as an operation on the internal representations of a pretrained generative model rather than as a task requiring wholesale retraining on large relighting corpora. It is described as a lightweight, data-efficient method for full-scene relighting that “taps into” implicit light-transport knowledge already present in large diffusion backbones, trains a small LoRA regressor to predict a direct-irradiance map from an input image, and then uses that regressor as a guidance signal during diffusion sampling to impose new illumination while preserving scene identity and style. The paper reports state-of-the-art performance in quality and control, together with strong parameter and data efficiency, across a wide variety of scene types (Erel et al., 1 Sep 2025).

1. Problem setting and motivating premise

Relighting an arbitrary image means changing its illumination—shadows, highlights, color of light sources, and interreflections—without disturbing scene geometry, materials, or overall style. In the formulation adopted by PractiLight, this is not reducible to local editing or color grading, because light affects the full image and the full frequency spectrum simultaneously: global low-frequency shading, mid-frequency soft shadows, and high-frequency specular glints must all change coherently.

The paper positions this as a weakness of many existing relighting systems. Traditional approaches are described as addressing one domain at a time, such as portraits, indoor rooms, or 3D objects, and as typically requiring tens or hundreds of thousands of in-domain examples. Even methods that finetune large diffusion backbones on relighting data are described as requiring millions of images and hundreds of GPU hours, with the additional risk of distorting the model’s underlying generative manifold and reducing cross-domain generalization.

PractiLight is built on the opposite premise: large text-to-image diffusion models such as Stable Diffusion and DiT are trained on billions of uncurated web images and therefore must already encode substantial information about lighting cues in order to render plausible scenes. The method does not attempt to relearn light transport from scratch. Instead, it assumes that controllable relighting can be obtained by introducing a small, targeted adapter into the pretrained backbone and by reading out the model’s existing lighting representations in a way that is useful at inference time. This directly challenges the common assumption that practical relighting necessarily requires massive, domain-specific supervision.

2. Lighting as a self-attention phenomenon

The central representational claim of PractiLight is that lighting relationships in an image are similar in nature to token interaction in self-attention layers. In the paper’s formulation, the relevant question is essentially “which pixels illuminate which other pixels?”, and this many-to-many dependency structure is argued to align better with self-attention than with cross-attention or convolution.

Two analyses are used to support this claim (Erel et al., 1 Sep 2025). The first is a layer-type analysis. Two images of the same scene under different lighting are inverted via DDIM inversion, and individual layer activations from the “target lighting” branch are swapped into the “original lighting” branch. Injecting decoder self-attention activations yields results that best match the target illumination, as measured by CLIP, DINO, and DreamSim. Cross-attention and convolutional layers perform worse.

The second is an attention-versus-timestep analysis. In early diffusion timesteps, exemplified by $t=900$ out of 1000, self-attention maps of pixels lying in specular or diffuse reflection zones strongly attend the light-source region. At later stages, exemplified by $t=100$, the attention shifts toward semantically similar areas such as texture and edges. This is interpreted to mean that low-frequency lighting cues, including shading and shadows, are resolved early, while higher-frequency shading details are refined later. On that basis, PractiLight concentrates relighting intervention in the early half of the diffusion trajectory.

This analysis has two methodological consequences. First, self-attention becomes the preferred insertion point for the relighting adapter. Second, guidance is scheduled only over a restricted timestep interval rather than uniformly across the denoising process. The design therefore treats relighting not as a generic conditioning signal, but as a temporally localized manipulation of attention-mediated latent interactions.

3. LoRA regressor and direct-irradiance supervision

PractiLight avoids full-model finetuning by inserting a Low-Rank Adapter into each self-attention block of the diffusion U-Net. The adapter is trained to regress a direct-irradiance map, defined in the paper as the per-pixel sum of first-bounce diffuse plus specular illumination from a single input image.

The training data are synthetic and intentionally simple. Geometry consists of Blender scenes with one or two primitives—sphere, cube, or cuboid—placed in a box “room.” Materials are modeled with a GGX BRDF and randomized albedo and surface roughness. Lighting is provided by one or more point sources with randomized positions and intensities. The target $I_g$ is Blender’s direct-diffuse plus direct-specular render pass, with no interreflection. The input $I_s$ is the full rendered image under the same lights. The dataset contains 4,000 image pairs at $512\times512$ resolution.

The LoRA parameterization is written as

$\Delta W = B\,A,$

where $B\in\mathbb R^{m\times r}$, $A\in\mathbb R^{r\times n}$, and $r\ll\min(m,n)$, with $r=8$. Only the LoRA matrices $t=100$0 are trained; the original U-Net weights remain frozen. During a forward pass on an image $t=100$1 at diffusion timestep $t=100$2, the adapter branches from the backbone and produces a latent prediction $t=100$3 of the direct-irradiance map.

Training is performed with an $t=100$4 latent reconstruction objective,

$t=100$5

where $t=100$6 denotes the encoder into diffusion latents. The use of latent-space supervision is significant because it keeps the regressor aligned with the representation actually used by the generative backbone. A plausible implication is that the adapter is being trained less as a standalone relighting model than as a latent readout head specialized for lighting structure.

4. Guidance-based relighting during diffusion sampling

At inference time, PractiLight turns the irradiance regressor into what the paper describes as a “soft classifier.” Rather than predicting a final relit image directly, the model uses the regressor’s output to define an energy that encourages the generated image’s lighting to match a desired control map $t=100$7.

The control map $t=100$8 is produced by depth-lifting the source image into a 3D proxy with an off-the-shelf monocular depth estimator, placing one or more point lights in that proxy scene, rendering a direct-irradiance pass, and using the result as a grayscale control image. This creates a target in irradiance space rather than in RGB image space.

Given a pretrained diffusion model with score network $t=100$9, PractiLight defines

$I_g$0

and modifies the denoiser according to

$I_g$1

Here $I_g$2 is the classifier-free guidance scale for the text condition, $I_g$3 is a timestep-dependent light guidance scale, and $I_g$4 is the unconditioned branch. The paper uses $I_g$5 for $I_g$6 and zero elsewhere, which is consistent with the earlier claim that lighting structure is most editable in the early half of the diffusion trajectory.

The inference workflow begins from a source latent $I_g$7 DDIM-invert$I_g$8. At each timestep, the model predicts conditioned and unconditioned noise, obtains the irradiance estimate $I_g$9 from the adapter, computes the gradient of the irradiance mismatch, and performs a scheduler step with the modified denoiser. Identity preservation is reinforced in two additional ways: self-attention queries are injected from the inversion trajectory, and an edge-conditioned ControlNet is applied on a composite edge map. The decoded final image is post-processed by matching mean and standard deviation to $I_s$0.

This pipeline is notable for separating lighting control from image synthesis in a specific way. The text condition controls scene content and style through standard classifier-free guidance, while the irradiance regressor contributes an auxiliary lighting-specific energy. The method therefore treats relighting as a constrained perturbation of the denoising path rather than as a separate image-to-image network.

5. Evaluation protocol, reported performance, and efficiency

The evaluation set contains 180 high-aesthetic $I_s$1 generations spanning nine domains: portraits, CG portraits, anime, animals, fantasy landscapes, indoors, outdoors, paintings, and cartoons. The images are collected from DiffusionDB. For each image, the text prompt, seed, CFG scale, and diffusion steps are fixed so that all methods are compared under identical generative conditions. A random point-light control $I_s$2 is automatically produced for each test image through the depth-lifted 3D proxy procedure (Erel et al., 1 Sep 2025).

The quantitative evaluation uses four groups of metrics. Aesthetics are measured by HPSv2, where higher is better. Control adherence is measured by $I_s$3 between the relit image and $I_s$4, LPIPS, CLIP similarity, and DINO similarity, with lower values interpreted as better mismatch. Identity preservation is measured by the same four metrics between the relit result and the original $I_s$5. Efficiency is assessed by number of trained parameters, number of data points, and GPU hours.

The reported baselines are RGBx, described as intrinsic decomposition plus recomposition; IC-Light, described as finetuned SD1.5 on 10 M relighting images; Readout Guidance, described as an MLP regressor on all features; and DiLightNet, described as object-centric diffusion relighting. Against these methods, PractiLight is reported to outperform or tie in control adherence and to significantly exceed identity preservation and aesthetics.

The efficiency claims are explicit. PractiLight uses $I_s$6 trained parameters, compared with $I_s$7 for IC-Light and $I_s$8 for RGBx. It uses 4,000 training images, compared with $I_s$9 to $512\times512$0 in other methods, and approximately 1 GPU-hour rather than hundreds or thousands of GPU hours. The LoRA regressor itself contains 784,000 parameters and is trained in approximately 1 hour on a single Quadro RTX 8000 GPU. A data-scaling ablation further reports that useful control signals emerge with only a few hundred examples and that performance saturates by 2,000 examples.

The human study, with $512\times512$1, reports that subjects choose PractiLight relit images 70–80% of the time for identity and style preservation as well as overall lighting quality. Qualitatively, the paper emphasizes generalization well beyond the simple Blender training scenes: the model edits hard shadows in medieval sketches, adds specular highlights on ceramics, repositions light on fantasy landscapes, and relights anime and painted scenes. These examples are presented as evidence that the model is not merely memorizing synthetic primitives, but is leveraging lighting structure already embedded in the foundational backbone.

6. Relation to other relighting paradigms, limitations, and open directions

PractiLight belongs to a broader relighting literature, but it occupies a distinct design point. “A Light Stage on Every Desk” formulates face relighting as subject-specific light transport acquisition under monitor illumination. It assumes Lambertian facial reflectance in its continuous model, represents monitor-to-image transfer through a light-transport matrix in its discretized form, and trains a U-Net with light encoders and AdaIN-style modulation on paired captures of a given person watching video on a monitor (Sengupta et al., 2021). By contrast, PractiLight does not rely on person-specific capture, explicit monitor patterns, or a transport matrix estimated from real observations. This suggests a shift from explicit acquisition of light transport to extraction of implicit light-transport structure from a pretrained generative model.

“Hybrelighter” addresses a different regime still: mixed-reality scene relighting on edge devices. Its architecture combines on-device depth or mesh reconstruction, a mesh-aware relight filter, a lightweight U-Net with a MobileNetV4 backbone for guidance-map extraction, guided anisotropic diffusion for image-space lighting propagation, and a separate shadow pass. It reports 10–15 ms end-to-end latency per $512\times512$2 frame, corresponding to 65–100 fps on a modern smartphone or AR tablet, and 50 fps in an iPhone 16 Pro Unity demo (Zhao et al., 19 Aug 2025). PractiLight does not target real-time mixed-reality filtering or correction of mesh artifacts. A plausible implication is that it should be understood primarily as a diffusion-native control mechanism for image generation, not as a scene-reconstruction or post-compositing method.

The paper also states several limitations of the current system. Colored lighting is not yet supported, because distinguishing material albedo from light color proved difficult with limited data and induces color shifts in the output. Higher-order global illumination, including interreflections and caustics beyond first bounce, would require richer labels or multi-bounce regressors. Removing existing hard shadows remains imperfect. Preliminary per-head analysis suggests that only certain attention heads carry strong lighting cues, raising the possibility of more selective head-level manipulation in future work. Finally, although the reported implementation uses a U-Net backbone, specifically Stable Diffusion 1.5, the paper argues that the attention-based insight extends naturally to DiT-based models such as Flux and DiT, since multi-bounce “attention chains” resemble light transport.

Taken together, these points define PractiLight as a method that reframes relighting as a problem of locating, reading, and gently steering lighting structure already present in large diffusion models. Its novelty lies less in introducing a new large-scale relighting model than in identifying a compact intervention—self-attention LoRA plus irradiance-based guidance—that appears sufficient to produce practical, cross-domain relighting from a small synthetic training set.