---
title: 'TokenLight: Conditional Image Relighting'
url: https://www.emergentmind.com/topics/tokenlight
type: topic
---

# TokenLight: Conditional Image Relighting

TokenLight is a relighting method that formulates image relighting as a conditional image-to-image generation problem and introduces attribute tokens to encode distinct illumination factors, including intensity, color, ambient illumination, diffuse level, and 3D light positions [2604.15310]. The model is instantiated as a latent diffusion transformer trained to approximate the conditional distribution \(p_\theta(I_r\mid I,\Delta L)\) end to end, with \(\Delta L\) parameterized in a small set of physically meaningful attributes rather than through explicit inverse rendering [2604.15310]. In the reported experiments, it is validated on synthetic and real relighting tasks, including control of in-scene lighting fixtures and editing of environment illumination using virtual light sources, and is reported to achieve state-of-the-art quantitative and qualitative performance [2604.15310].

## 1. Problem setting and formulation

TokenLight treats relighting as conditional generation. Given an input image \(I\) and a desired lighting edit \(\Delta L\), it seeks an operator \(R\) such that
\[
I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),
\]
where \(f\) denotes the physical renderer, \(g\) an unknown inverse-rendering process that extracts scene properties, \(L\) the original lighting, and \(I_r\) the relit output [2604.15310]. The central modeling choice is not to estimate geometry, BRDFs, visibility, or illumination explicitly, but instead to train a conditional generator that maps image evidence and structured lighting controls directly to a relit image.

The lighting edit \(\Delta L\) is parameterized by physically meaningful attributes: ambient scale, diffuse spread, per-light position, color, intensity, and an in-scene fixture mask [2604.15310]. These attributes are encoded as tokens and concatenated with image tokens inside the transformer. This design makes the conditioning space structured rather than textual or purely latent, and it is intended to support continuous, compositional editing of multiple illumination factors within a single model.

A common misconception would be to read the method as an inverse-rendering pipeline because the formulation invokes \(f\) and \(g\). The paper instead states that the approach bypasses explicit inverse graphics by learning the conditional distribution \(p_\theta(I_r\mid I,\Delta L)\) directly [2604.15310]. The reported behavior—respecting geometry, occlusion, and materials without explicit depth supervision—therefore reflects properties of the learned generator rather than an explicit scene decomposition.

## 2. Attribute-token representation of lighting

TokenLight defines five core lighting attributes, each encoded as one or more discrete tokens and processed jointly with the image tokens via full self-attention [2604.15310]. The tokenization scheme is the method’s principal representational contribution.

| Attribute | Meaning | Encoding |
|---|---|---|
| Ambient-scale token | Scalar \(a\in[0,1]\) scaling HDR environment map or ambient color | One Fourier-feature token |
| Global diffuse-level token | Scalar \(d_g\in[-1,1]\) interpolating between sharp and soft area-light spreads | One Fourier-feature token |
| Added-light tokens | Per-light \((\mathbf{p},\mathbf{c},\lambda,d)\) | Eight Gaussian Fourier tokens per light |
| In-scene fixture tokens | Visible fixture mask, color, intensity, on/off state | VAE mask tokens plus Fourier tokens and binary flag |
| Sequence assembly | Joint conditioning stream | Concatenated with image tokens |

The ambient-scale token uses a single scalar \(a\in[0,1]\) to uniformly scale the HDR environment map or ambient color, encoded by
\[
\phi(a) \;=\; \bigl[\sin(2\pi \mathbf{B}a),\,\cos(2\pi \mathbf{B}a)\bigr]
\quad\text{where }\mathbf{B}\sim\mathcal{N}(0,\sigma^2),
\]
with \(\sigma=5\) in practice [2604.15310]. The global diffuse-level token uses the same encoding for a scalar \(d_g\in[-1,1]\) controlling a continuum from narrow, sharp area-light spreads to wide, soft ones.

Each virtual point light is described by four parameter groups: 3D position \(\mathbf{p}=(x,y,z)\in[-1,1]^3\), chromaticity \(\mathbf{c}=(r,g,b)\in[0,1]^3\), scalar intensity \(\lambda\in[0,1]\), and per-light diffuse spread \(d\in[-1,1]\) [2604.15310]. Each scalar among \(x,y,z,r,g,b,\lambda,d\) is encoded with a separate Gaussian Fourier token; thus one added light contributes eight tokens. The model supports up to three lights by repeating this 8-token block, with inactive lights filled with \(-1\) [2604.15310].

For visible fixtures already present in the scene, TokenLight uses a binary mask \(\mathbf{m}\in\{0,1\}^{H\times W}\) to localize the fixture. The mask is encoded through the VAE encoder into latent space and flattened into image-patch tokens. Two additional Fourier tokens encode fixture color \(\mathbf{c}\) and intensity \(\lambda\), and one binary flag \(t\) indicates an on/off transition [2604.15310]. After patchifying the image \(I\) into image-latent tokens, all attribute tokens are appended into a single sequence and processed jointly [2604.15310].

The design analysis reported in the paper states that Fourier features with \(\sigma=5\), as opposed to raw scalars, were found crucial for fine position sensitivity [2604.15310]. This suggests that precise spatial relighting depends not only on supplying positional parameters, but on embedding them in a form that preserves high-frequency variation under transformer processing.

## 3. Architecture and conditioning pathway

TokenLight builds on a latent diffusion transformer of approximately 2 billion parameters, pretrained for text-to-image and video generation [2604.15310]. The image encoder is a VAE: an input image at 960 px is encoded to a \(12\times120\times120\) latent, patchified \(2\times2\) into \(48\times60\times60\) tokens, and linearly projected to 4096-dimensional vectors, yielding 3600 image tokens [2604.15310].

Lighting attribute tokens, approximately 10–30 depending on the number of lights, are concatenated to the image tokens and to the noisy target latent \(\mathbf{z}_\tau\) [2604.15310]. Positional embeddings are implemented with RoPE. The paper notes that these embeddings discriminate image-patch location but not the distinction between image and lighting tokens [2604.15310]. The transformer backbone has 64 layers with full self-attention, with feed-forward dimension and attention heads as in DiT (Peebles & Xie) [2604.15310].

Classifier-free guidance is implemented by dropping \(\Delta L\) tokens 10% of the time during training to learn an unconditional branch [2604.15310]. The decoder integrates the predicted velocity field through DDIM sampling with 50 steps to obtain an output latent, which is then decoded back to RGB [2604.15310].

Two architectural implications follow directly from these choices. First, the model couples spatial image evidence and lighting controls at the token level rather than through late fusion; this plausibly contributes to the reported sensitivity to light position, fixture localization, and diffuse spread. Second, the absence of a separate geometric module means that geometric reasoning, visibility, and material effects must be represented implicitly in the transformer’s conditional denoising dynamics.

## 4. Training data, objective, and optimization

The training setup combines a large-scale synthetic relighting corpus with a smaller real-capture fine-tuning set [2604.15310]. The synthetic corpus contains approximately 4,000 procedurally generated 3D scenes from Objaverse and procedural objects plus humans. Each scene uses 64 sampled point-light positions and an HDRI chosen from approximately 600 PolyHaven maps [2604.15310].

The rendered synthetic components include an ambient render under HDRI, on-light renders for each sampled point light and for each spread angle across 6 values, and per-fixture separation in 83 artist-authored indoor scenes, yielding approximately 100K paired renders [2604.15310]. Training pairs are synthesized on the fly according to
\[
I_r = T\bigl(a\,I_{\rm amb} + \sum_i\lambda_i\,\mathbf{c}_i\,O_i\bigr),
\]
with Reinhard tone mapping, while \(a,\lambda,\mathbf{c},d,d_g\) are randomized at load time [2604.15310].

The real-capture fine-tuning set, VisibleFixture-600, contains approximately 600 iPhone ProRAW indoor photographs under on/off toggles of visible fixtures, with manually segmented fixture masks and paired captures [2604.15310]. Its stated purpose is to bridge the domain gap and improve realism.

Training uses a conditional flow-matching objective:
\[
\mathbf{X}\sim p(\cdot\mid I,\Delta L),\quad
\boldsymbol\epsilon\sim\mathcal{N}(0,I),\quad
\mathbf{z}_\tau=(1-\tau)\,\boldsymbol\epsilon+\tau\,\mathbf{X},
\]
\[
u_\theta(\mathbf{z}_\tau,\tau,I,\Delta L)\approx\mathbf{X}-\boldsymbol\epsilon,
\]
and
\[
\mathcal{L}(\theta)
\;=\;
\mathbb{E}_{\tau,\mathbf{X},\boldsymbol\epsilon}
\Bigl\|
u_\theta(\mathbf{z}_\tau,\tau,I,\Delta L)\;-\;(\mathbf{X}-\boldsymbol\epsilon)
\Bigr\|_2^2.
\]
No explicit adversarial or perceptual losses are used; the paper attributes photorealism to the pretrained visual prior and real-data fine-tuning [2604.15310].

Optimization uses AdamW with learning rate \(1\times10^{-5}\), weight decay \(1\times10^{-2}\), and \(\beta=(0.9,0.95)\), for 15,000 steps with batch size 160, partitioned as 64 spatial, 48 fixture, and 48 diffuse [2604.15310]. Inference uses DDIM with 50 steps and guidance scale \(w=2\) [2604.15310].

## 5. Quantitative performance

The paper reports quantitative evaluation on synthetic spatial relighting, real visible-fixture editing, and human preference studies [2604.15310]. On the synthetic spatial benchmark, the test set contains 200 held-out Objaverse objects, and two target modalities are considered: PointGT, a true point-light render, and PanoGT, where a point light is approximated as an environment map [2604.15310]. On the object mask, the evaluated metrics are PSNR, SSIM, and LPIPS, and TokenLight is reported to outperform Neural Gaffer and DiffusionRenderer by a large margin, with PSNR around 21 dB versus 16–13 and LPIPS around 0.02 versus 0.04 [2604.15310].

A more specialized position-precision analysis uses 50 objects and six linear light trajectories of 32 positions each [2604.15310]. The paper constructs a confusion matrix
\[
M_{ij}=\|\hat I(\mathbf{p}_j)-I_{\rm GT}(\mathbf{p}_i)\|^2,
\]
and defines
\[
A=\tfrac1{32}\sum_i M_{ii},\quad
B=\sum_{i\neq j}|i-j|\,M_{ij}\,\Big/\sum_{i\neq j}|i-j|.
\]
Precision is measured as \(B/A\) upward, and accuracy as \(A\) downward [2604.15310]. Under this analysis, TokenLight achieves \(B/A=1.88\) versus 1.11 and \(A=0.049\) versus 0.160 for Neural Gaffer [2604.15310]. These figures support the claim that the tokenized control space preserves fine sensitivity to light placement, not merely coarse scene brightening.

On the real VisibleFixture-60 benchmark, comprising 60 on/off fixture pairs, TokenLight is compared to ScribbleLight on the masked fixture region using PSNR, SSIM, and LPIPS [2604.15310]. The reported values are PSNR 20.08, SSIM 0.848, and LPIPS 0.278 for TokenLight, versus 14.64, 0.519, and 0.611 for ScribbleLight [2604.15310].

The user study involves 18 participants, 20 in-the-wild scenes, and 5 lights, with pairwise comparison against GenLit and Careaga et al. [2604.15310]. The reported preference rates are 77.5% over GenLit and 89.2% over Careaga [2604.15310]. While user studies are inherently subjective, these results are consistent with the numerical gains reported for spatial precision and fixture editing.

## 6. Qualitative behavior, invariances, and limitations

The qualitative results emphasize several operational regimes. For spatial virtual lights, the paper shows examples of inserting a light inside a pumpkin, behind a book, behind a transparent glass object, and producing rim lighting on hair [2604.15310]. The reported interpretation is that the model respects occlusion and geometry without explicit depth. For ambient and diffuse controls, progressive ambient dimming is shown to spotlight documents and portraits, while varying the global diffuse spread \(d_g\) modulates shadow softness and sharpness [2604.15310]. For in-scene fixtures, examples include turning individual lamps on or off, adjusting per-fixture color and intensity, and suppressing a car taillight without darkening sunlight in an outdoor example [2604.15310]. The model supports up to three lights with independent position, intensity, color, and diffuse parameters, producing additively mixing colored shadows [2604.15310].

The design analysis also discusses scene-agnostic coordinates. Equations (5–8) are reported to show similarity-transform invariance, such that positions \(\mathbf{p}\), energies \(E\), and radii \(d\) scale correctly under \(\mathrm{Sim}(3)\), and the supplementary material confirms invariance under arbitrary scene transforms [2604.15310]. This suggests that the chosen coordinate parameterization is intended to remain stable under canonical rescaling and rigid similarity transformations rather than being tied to a single scene metric.

Ablations address inference-step tradeoffs and stochasticity. Supplementary results indicate that even 5–10 DDIM steps yield plausible relighting, although 50 steps are used by default [2604.15310]. Random-seed variation is reported as minor at low diffuse levels and to stabilize as shadows soften [2604.15310].

The paper also notes limited failures. Seed dependence and speed—reported as 2 s per 50 steps on A100—motivate future work in distillation [2604.15310]. More broadly, the method’s reliance on a pretrained generative prior means that physically plausible outcomes do not necessarily imply explicit physical correctness in the inverse-rendering sense. The reported success without inverse-rendering supervision demonstrates strong learned relighting behavior, but it should not be conflated with recovery of calibrated scene parameters unless such recovery is explicitly evaluated.

Source: https://www.emergentmind.com/topics/tokenlight