---
title: 'DreamLight: Diffusion-Based Relighting Model'
url: https://www.emergentmind.com/topics/dreamlight
type: topic
---

# DreamLight: Diffusion-Based Relighting Model

DreamLight is a diffusion-based image relighting model designed for seamless, universal, and contextually harmonious compositing of foreground subjects into new backgrounds. Its architecture supports both image-based and text-based relighting, aiming for consistent photometric realism and aesthetic unification of the foreground and background, addressing limitations in prior harmonization and relighting pipelines. DreamLight introduces the Position-Guided Light Adapter (PGLA) for explicit modeling of background light directionality and the Spectral Foreground Fixer (SFF) for post-processing adaptive frequency alignment of subject and context, and leverages semantic priors from a pretrained diffusion model to facilitate plausible results across diverse real and synthetic images [2506.14549].

## 1. Unified Problem Formulation and Input Encoding

DreamLight approaches image relighting as a universal compositing task. The inputs are:
- A foreground portrait, \( I_{\text{fg}} \in \mathbb{R}^{H\times W\times 3} \)
- Either a background image \( I_{\text{bg}} \) or a text prompt \( p \) describing the desired background/lighting (setting \( I_{\text{bg}} \) to a null/black image for text cases).

Foreground segmentation is performed via an automated model (e.g., RMBG-1.4) to obtain binary mask \( M_{\text{fg}} \); the original background is masked out, forming \( I_{\text{fg}}^m(x) = I_{\text{fg}}(x)\cdot M_{\text{fg}}(x) \).

Latent codes for \( I_{\text{fg}}^m \) and \( I_{\text{bg}} \) are produced using a VAE encoder (compatible with Stable Diffusion v1.5), yielding \( z_{\text{fg}}, z_{\text{bg}} \). These are concatenated with a random noise latent, resulting in \( Z_0 = \text{concat}(z_\text{noise},z_{\text{fg}},z_{\text{bg}}) \). Both image and text-based conditions are passed to the central diffusion U-Net, providing a unified representation for relighting control [2506.14549].

## 2. Position-Guided Light Adapter (PGLA)

PGLA is the mechanism by which DreamLight condenses spatial light information from the background and imposes it on the foreground region, allowing for directionally-aware lighting harmonization.

- **Low-frequency enhancement**: Background features are processed through a CLIP-based visual encoder, then spectrally filtered with a frequency cutoff (\( \sigma \)) to emphasize low-frequency (global illumination) components via FFT, Gaussian filtering, and IFFT with residual addition:
  \[
  f_{\text{bl}} = \text{IFFT}(\text{ReLU}(\text{Conv}(F_\text{lf}))) + f_b
  \]
  where \( f_b \) are CLIP features, and \( F_\text{lf} \) is frequency-filtered.

- **Directional light query construction**: Four query sets are defined for directions \(\{ \text{left}, \text{right}, \text{top}, \text{down}\}\), each with learnable embeddings \( f_Q^d \). Attention bias masks \( M^d \), decaying along axis-opposing directions, are used with softmax cross-attention to pool directional lighting cues:
  \[
  A^d = \text{softmax}((Q^d K_b^\top/\sqrt{C}) + M^d), \quad f_L^d = A^d V_b
  \]
  All query results are concatenated as \( f_L \).

- **Foreground injection**: In the U-Net’s bottleneck and decoder stages, standard cross-attention key/value sets are augmented by \( f_L \), and attention is restricted to foreground pixels using \( M_{\text{fg}} \) to prevent background cue leak.

This approach allows explicit, learnable modulation of the foreground’s lighting based on contextual cues from the background, critical for achieving scene-consistent relighting [2506.14549].

## 3. Spectral Foreground Fixer (SFF)

After core relighting, SFF operates as a localized, wavelet-based frequency rebalancer for the subject region:
- One-level Haar wavelet transform decomposes \( I_{\text{fg}} \) and preliminary relit output \( I_\text{out} \) into low (LQ) and high-frequency (HQ) bands.
- A modulation network \( \mathcal{M} \) takes \( HQ_{\text{in}} \) (from \( I_{\text{fg}} \)) and \( LQ_{\text{out}} \) (from \( I_{\text{out}} \)), predicting per-pixel weights \( \alpha, \beta \); the fixed high-frequency components are then
  \[
  HQ' = \alpha \odot HQ_{\text{in}} + \beta
  \]
- The final foreground is recomposed by inverse wavelet, replacing \( I'_\text{fg} \) into the foreground region of \( I_\text{out} \), yielding \( I'_\text{out} \).

SFF thus selectively preserves high-frequency facial details while harmonizing low-frequency tone with the relighted background, improving perceptual and aesthetic foreground-background unity [2506.14549].

## 4. Diffusion Model Architecture and Training

DreamLight uses a Stable Diffusion v1.5 backbone (U-Net + VAE) as the generative prior. Text prompts \( p \) are encoded via Stable Diffusion’s text encoder and, together with PGLA-derived light queries, condition the denoising U-Net at each timestep.

- Each training step optimizes:
  \[
  L_{\text{diff}} = \mathbb{E}_{t, z_0, \epsilon} [ \| \epsilon_\theta(z_t, p, I_\text{bg}, I_\text{fg}, t) - \epsilon \|_2^2 ]
  \]
  for the diffusion process, plus a multi-term spectral fixer loss:
  \[
  L_\text{sff} = \lambda_1 L_\text{mse} + \lambda_2 L_\text{perc} + \lambda_3 L_{HQ}
  \]
  where \( L_\text{mse} \) is output MSE, \( L_\text{perc} \) is a VGG-based perceptual loss, and \( L_{HQ} \) is HQ band supervision.
- Total training loss: \( L_\text{total} = L_\text{diff} + \mu\,L_\text{sff} \)
- Training data: 600k LoRA-generated pairs, 150k 3D Arnold/OLAT pairs, 300k IC-Light synthetic pairs, with batch size 512, learning rate \( 5\times10^{-5} \) [2506.14549].

## 5. Evaluation and Comparative Analysis

On a 600-pair Arnold-rendered testset, DreamLight surpasses previous harmonization and relighting benchmarks:
- PSNR: 22.15 vs. best prior 20.66
- SSIM: 0.783 vs. 0.771
- LPIPS: 0.158 vs. 0.177
- CLIP-IS: 0.908 vs. 0.896

For text-based relighting, DreamLight also delivers superior CLIP similarity (0.644), aesthetic (6.32), and ImageReward scores (3.47) on the testset. In user studies, ≥80% of participants preferred DreamLight outputs to those of IC-Light and harmonization baselines [2506.14549].

## 6. Limitations and Extensions

DreamLight assumes accurate foreground-background segmentation and typically works with 512×512 images due to model and training scale. Its explicit frequency-modeling approach is tailored to foreground-face compositing and may not trivially generalize to full-body or multi-object relighting.

The PGLA’s directional attention mechanism is currently limited to four cardinal axes. A plausible extension is to use denser or adaptive, context-sensitive directional queries. SFF currently operates only in the foreground, suggesting further extension to border/blending regions for fine-grained consistency. Incorporation of temporal consistency, learned 3D geometry priors, or object-specific adaptation remain directions for future research [2506.14549].

## 7. Usage Workflow

Operational deployment follows these steps:
1. Run segmentation to extract \( M_{\text{fg}} \) and mask \( I_{\text{fg}}^m \).
2. Encode latents: \( z_{\text{fg}} = \mathcal{E}(I_{\text{fg}}^m) \), \( z_{\text{bg}} = \mathcal{E}(I_{\text{bg}} \) or black for text).
3. Sample noise and form \( Z_0 \).
4. Forward through U-Net with PGLA: extract background light features, form directional queries, inject via masked cross-attention.
5. Denoise via diffusion steps to obtain \( I_\text{out} \).
6. Apply SFF to foreground region, recombine.
7. Composite \( I_\text{out}' \) onto the original (or new) background.

This end-to-end procedure enables text- or image-driven relighting for single portraits, supporting consistent photometric integration and controllable harmonization in diverse settings [2506.14549].

Source: https://www.emergentmind.com/topics/dreamlight