SpotLight: Diffusion-Based Object Relighting

• SpotLight is a diffusion-based method for controllable object relighting that leverages a coarse shadow mask as the primary illumination control to harmonize objects with their backgrounds.
• It injects shadow cues into a pre-trained latent diffusion renderer using latent shadow blending and dual-branch classifier-free guidance, ensuring precise shading control.
• The approach outperforms baselines in metrics such as PSNR, SSIM, RMSE, MAE, and LPIPS, demonstrating its effectiveness in generating visually coherent composites.

Searching arXiv for the target paper and closely related relighting/compositing work. SpotLight is a diffusion-based method for controllable object relighting that uses a coarse shadow map as the primary lighting control signal. Given a background image $I_{\mathrm{bg}}$, a virtual object specified by albedo $A_{\mathrm{obj}}$ and mask $m_{\mathrm{obj}}$, and a user-provided shadow $m_{\mathrm{shw}}$, it synthesizes a composite $I_{\mathrm{out}}$ in which the object is relit according to the intended light position while the object, shadow, and background are jointly harmonized. Its central premise is that injecting only the desired shadow into a pre-trained diffusion-based neural renderer is sufficient to induce accurate shading and plausible compositing, without additional training (Fortier-Chouinard et al., 2024).

1. Problem formulation and intended scope

SpotLight formulates object relighting as a conditional image synthesis problem. The target is not merely to paste an object into a scene, but to produce a composite in which the object’s shading is consistent with a user-specified illumination configuration and its cast shadow blends plausibly into the target background. In the reported formulation, the user specifies lighting indirectly through the object’s desired shadow, either via shadow mapping or freehand scribbles. This replaces explicit light-parameter control with a spatial cue that is easy to author and directly tied to scene geometry (Fortier-Chouinard et al., 2024).

Three properties define the method. First, it provides precise, user-controlled lighting through shadow specification alone. Second, it is training-free: it builds on off-the-shelf diffusion-based neural renderers rather than introducing fine-tuning or auxiliary optimization. Third, it targets harmonization rather than isolated relighting, jointly refining the object appearance and the shadow so that both are visually compatible with $I_{\mathrm{bg}}$.

This design addresses a specific limitation of diffusion-based neural renderers for virtual object insertion. Whereas physics-based renderers admit direct manual lighting control, diffusion-based renderers typically do not. SpotLight’s contribution is to recover such control through shadow guidance while retaining the generative flexibility of the pre-trained renderer.

2. Diffusion renderer and intrinsic conditioning

The method assumes a pre-trained latent diffusion renderer $R$. Its per-pixel conditioning signal is an intrinsic stack $i=\{A,N,D,S_{\mathrm{bg}}\}$, where $A$ denotes albedo, $N$ normals, $A_{\mathrm{obj}}$0 depth, and $A_{\mathrm{obj}}$1 background shading. An RGB image $A_{\mathrm{obj}}$2 is mapped by a VAE encoder $A_{\mathrm{obj}}$3 to a latent $A_{\mathrm{obj}}$4, typically at a lower spatial resolution such as $A_{\mathrm{obj}}$5. A U-Net denoiser $A_{\mathrm{obj}}$6 predicts $A_{\mathrm{obj}}$7-predictions at timestep $A_{\mathrm{obj}}$8 under DDIM sampling, and a VAE decoder reconstructs $A_{\mathrm{obj}}$9 from the final latent (Fortier-Chouinard et al., 2024).

Two exemplar backbones are described. ZeroComp is conditioned on intrinsic albedo, normals, depth, and masked shading via ControlNet. RGB<>X is conditioned on albedo, normals, metallic, roughness, and a masked image. SpotLight does not redefine these rendering systems; it augments them with a shadow-conditioning mechanism that perturbs the denoising trajectory in a lighting-aware manner.

This architectural choice is significant because it localizes the method’s novelty to guidance at inference time. A plausible implication is that the approach is deliberately modular: any compatible diffusion-based renderer with suitable intrinsic conditioning can serve as the substrate, provided it exposes the latent denoising process.

3. Shadow-guided conditioning mechanism

SpotLight injects the coarse shadow in two stages during denoising. The first is latent shadow blending. Let

$m_{\mathrm{obj}}$0

be a rough composite, shadowed by $m_{\mathrm{obj}}$1. At each diffusion step $m_{\mathrm{obj}}$2, the noisy latent is updated as

$m_{\mathrm{obj}}$3

where $m_{\mathrm{obj}}$4, $m_{\mathrm{obj}}$5 is the shadow mask dilated by $m_{\mathrm{obj}}$6 and then downsampled to latent size, and $m_{\mathrm{obj}}$7 applies the DDIM noise schedule. This operation anchors the latent trajectory to the desired shadow region while leaving the remainder of the latent largely unchanged (Fortier-Chouinard et al., 2024).

The second stage is dual-branch classifier-free guidance. SpotLight runs two denoising branches in parallel: a positive branch aligned with the desired shadow direction, yielding $m_{\mathrm{obj}}$8, and a negative branch aligned with the opposite direction or with no shadow, yielding $m_{\mathrm{obj}}$9. Their combination on the object region is

$m_{\mathrm{shw}}$0

with $m_{\mathrm{shw}}$1, $m_{\mathrm{shw}}$2, and $m_{\mathrm{shw}}$3 the bilinearly downsampled object mask. This guidance focuses relighting pressure on the object rather than the full scene, and the report states that high $m_{\mathrm{shw}}$4 is critical for perceptible relighting (Fortier-Chouinard et al., 2024).

A third component preserves the background. Following the underlying renderer, a shadow matte $m_{\mathrm{shw}}$5 is computed as the per-pixel ratio between renders with and without shadow guidance. During decoding, $m_{\mathrm{shw}}$6 outside $m_{\mathrm{shw}}$7 is preserved by compositing with this matte. The stated effect is artifact suppression beyond the object region.

4. Inference-only synthesis and emergence of harmonization

SpotLight introduces no fine-tuning and no additional losses. Its full procedure is inference-only. The rough composite $m_{\mathrm{shw}}$8 is first constructed; its encoded latent is noised at the highest diffusion level to initialize $m_{\mathrm{shw}}$9. Denoising then proceeds from $I_{\mathrm{out}}$0 to $I_{\mathrm{out}}$1, and each step performs latent blending, positive and negative branch evaluation, guided $I_{\mathrm{out}}$2-prediction fusion, and a standard DDIM update. The final image is obtained by decoding $I_{\mathrm{out}}$3, computing the shadow matte, and compositing the decoded result with the original background (Fortier-Chouinard et al., 2024).

The report attributes harmonization to the interaction of three mechanisms rather than to an explicit objective. Blended latent diffusion anchors the synthesis to the specified shadow region. Object-focused classifier-free guidance forces object shading to correlate with the shadow input. Matte-based background preservation prevents changes outside the intended composite area. Under this account, shadow softness and boundary consistency are emergent properties of the denoising dynamics.

This is a notable departure from optimization-heavy relighting pipelines. Instead of solving an explicit photometric inverse problem or training a dedicated relighting network, SpotLight uses the generative prior already encoded in the diffusion renderer and steers it through a low-bandwidth geometric cue. This suggests a shift from direct light-parameter estimation toward cue-based control.

5. Experimental protocol and reported performance

The quantitative evaluation uses two datasets. The “Reference-Based” set contains $I_{\mathrm{out}}$4 examples composed from ABO 3D models and Laval Indoor HDR environments, with ground-truth simulated renders. The “User-Controlled” set contains $I_{\mathrm{out}}$5 examples using the same objects under five azimuth light directions at $I_{\mathrm{out}}$6 steps and supports perceptual evaluation. Baselines are DiLightNet, IC-Light, and Neural Gaffer; each baseline is provided the ground-truth environment map and the object with shadowed background “for maximal advantage.” Evaluation metrics are PSNR, SSIM, RMSE, MAE, LPIPS, and a two-alternative forced-choice user study analyzed with Thurstone Case V $I_{\mathrm{out}}$7-scores (Fortier-Chouinard et al., 2024).

On the reference-based benchmark, the ZeroComp-backed version of SpotLight outperforms all reported baselines.

Metric	SpotLight	Next best baseline
PSNR	30.68 dB	29.64 dB
SSIM	0.973	0.965
RMSE	0.033	0.037
MAE	0.012	0.016
LPIPS	0.031	0.038

The user study uses $I_{\mathrm{out}}$8 participants with 120 pairwise comparisons each on the user-controlled set. SpotLight attains a Thurstone $I_{\mathrm{out}}$9-score of $I_{\mathrm{bg}}$0, compared with $I_{\mathrm{bg}}$1 for Neural Gaffer and $I_{\mathrm{bg}}$2 for IC-Light, with $I_{\mathrm{bg}}$3. The qualitative analysis further reports faithful highlight shifts, shadow-shape changes matching the specified light direction, and refined shadow boundaries that adapt to scene context (Fortier-Chouinard et al., 2024).

Parameter ablations indicate a trade-off structure. Higher $I_{\mathrm{bg}}$4 strengthens shading control but can reduce quantitative fidelity. The shadow blending weight $I_{\mathrm{bg}}$5 mediates shadow vividness versus subtlety, and the report recommends a small value near $I_{\mathrm{bg}}$6 to maintain softness.

6. Versatility, limitations, and projected extensions

Beyond the main object-compositing task, the method is demonstrated for hand-scribbled shadow guidance, 2D object relighting, and full-image relighting. These examples position the coarse shadow as a general-purpose control signal rather than a narrowly defined rendering primitive (Fortier-Chouinard et al., 2024).

The stated limitations are equally specific. The method relies on a plausible coarse shadow, and end-to-end free-form shadow synthesis is not addressed. The report also states that global scene relighting of the entire image is outside the current scope. Taken together with the demonstrated applications, this suggests that the current system can exploit scene-level shadow cues in selected settings but does not yet provide a general full-scene illumination model.

The future extensions proposed in the report follow naturally from those constraints. One direction is to integrate parametric light models, such as point or sphere lights, for end-to-end shadow control. Another is to extend from object-level relighting to fuller scene manipulation through multiple scribble cues or learned shadow generators. A plausible implication is that SpotLight occupies an intermediate point between fully manual compositing and explicitly physics-grounded inverse rendering: it offers controllable, training-free relighting, but its control vocabulary is presently expressed in shadows rather than in a complete light transport parameterization.