TokenLight: Conditional Image Relighting

• TokenLight is a relighting method that frames image editing as a conditional generation problem using physically meaningful attribute tokens for illumination factors.
• It integrates a latent diffusion transformer with attribute and image tokens, enabling continuous, compositional editing of ambient, diffuse, and fixture lighting without explicit inverse rendering.
• The approach achieves state-of-the-art quantitative and qualitative performance, demonstrating high spatial precision and effective control over in-scene lighting fixtures.

TokenLight is a relighting method that formulates image relighting as a conditional image-to-image generation problem and introduces attribute tokens to encode distinct illumination factors, including intensity, color, ambient illumination, diffuse level, and 3D light positions (Chaturvedi et al., 16 Apr 2026). The model is instantiated as a latent diffusion transformer trained to approximate the conditional distribution $p_\theta(I_r\mid I,\Delta L)$ end to end, with $\Delta L$ parameterized in a small set of physically meaningful attributes rather than through explicit inverse rendering (Chaturvedi et al., 16 Apr 2026). In the reported experiments, it is validated on synthetic and real relighting tasks, including control of in-scene lighting fixtures and editing of environment illumination using virtual light sources, and is reported to achieve state-of-the-art quantitative and qualitative performance (Chaturvedi et al., 16 Apr 2026).

1. Problem setting and formulation

TokenLight treats relighting as conditional generation. Given an input image $I$ and a desired lighting edit $\Delta L$, it seeks an operator $R$ such that

$$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$

where $f$ denotes the physical renderer, $g$ an unknown inverse-rendering process that extracts scene properties, $L$ the original lighting, and $I_r$ the relit output (Chaturvedi et al., 16 Apr 2026). The central modeling choice is not to estimate geometry, BRDFs, visibility, or illumination explicitly, but instead to train a conditional generator that maps image evidence and structured lighting controls directly to a relit image.

The lighting edit $\Delta L$0 is parameterized by physically meaningful attributes: ambient scale, diffuse spread, per-light position, color, intensity, and an in-scene fixture mask (Chaturvedi et al., 16 Apr 2026). These attributes are encoded as tokens and concatenated with image tokens inside the transformer. This design makes the conditioning space structured rather than textual or purely latent, and it is intended to support continuous, compositional editing of multiple illumination factors within a single model.

A common misconception would be to read the method as an inverse-rendering pipeline because the formulation invokes $\Delta L$1 and $\Delta L$2. The paper instead states that the approach bypasses explicit inverse graphics by learning the conditional distribution $\Delta L$3 directly (Chaturvedi et al., 16 Apr 2026). The reported behavior—respecting geometry, occlusion, and materials without explicit depth supervision—therefore reflects properties of the learned generator rather than an explicit scene decomposition.

2. Attribute-token representation of lighting

TokenLight defines five core lighting attributes, each encoded as one or more discrete tokens and processed jointly with the image tokens via full self-attention (Chaturvedi et al., 16 Apr 2026). The tokenization scheme is the method’s principal representational contribution.

Attribute	Meaning	Encoding
Ambient-scale token	Scalar $\Delta L$4 scaling HDR environment map or ambient color	One Fourier-feature token
Global diffuse-level token	Scalar $\Delta L$5 interpolating between sharp and soft area-light spreads	One Fourier-feature token
Added-light tokens	Per-light $\Delta L$6	Eight Gaussian Fourier tokens per light
In-scene fixture tokens	Visible fixture mask, color, intensity, on/off state	VAE mask tokens plus Fourier tokens and binary flag
Sequence assembly	Joint conditioning stream	Concatenated with image tokens

The ambient-scale token uses a single scalar $\Delta L$7 to uniformly scale the HDR environment map or ambient color, encoded by

$\Delta L$8

with $\Delta L$9 in practice (Chaturvedi et al., 16 Apr 2026). The global diffuse-level token uses the same encoding for a scalar $I$0 controlling a continuum from narrow, sharp area-light spreads to wide, soft ones.

Each virtual point light is described by four parameter groups: 3D position $I$1, chromaticity $I$2, scalar intensity $I$3, and per-light diffuse spread $I$4 (Chaturvedi et al., 16 Apr 2026). Each scalar among $I$5 is encoded with a separate Gaussian Fourier token; thus one added light contributes eight tokens. The model supports up to three lights by repeating this 8-token block, with inactive lights filled with $I$6 (Chaturvedi et al., 16 Apr 2026).

For visible fixtures already present in the scene, TokenLight uses a binary mask $I$7 to localize the fixture. The mask is encoded through the VAE encoder into latent space and flattened into image-patch tokens. Two additional Fourier tokens encode fixture color $I$8 and intensity $I$9, and one binary flag $\Delta L$0 indicates an on/off transition (Chaturvedi et al., 16 Apr 2026). After patchifying the image $\Delta L$1 into image-latent tokens, all attribute tokens are appended into a single sequence and processed jointly (Chaturvedi et al., 16 Apr 2026).

The design analysis reported in the paper states that Fourier features with $\Delta L$2, as opposed to raw scalars, were found crucial for fine position sensitivity (Chaturvedi et al., 16 Apr 2026). This suggests that precise spatial relighting depends not only on supplying positional parameters, but on embedding them in a form that preserves high-frequency variation under transformer processing.

3. Architecture and conditioning pathway

TokenLight builds on a latent diffusion transformer of approximately 2 billion parameters, pretrained for text-to-image and video generation (Chaturvedi et al., 16 Apr 2026). The image encoder is a VAE: an input image at 960 px is encoded to a $\Delta L$3 latent, patchified $\Delta L$4 into $\Delta L$5 tokens, and linearly projected to 4096-dimensional vectors, yielding 3600 image tokens (Chaturvedi et al., 16 Apr 2026).

Lighting attribute tokens, approximately 10–30 depending on the number of lights, are concatenated to the image tokens and to the noisy target latent $\Delta L$6 (Chaturvedi et al., 16 Apr 2026). Positional embeddings are implemented with RoPE. The paper notes that these embeddings discriminate image-patch location but not the distinction between image and lighting tokens (Chaturvedi et al., 16 Apr 2026). The transformer backbone has 64 layers with full self-attention, with feed-forward dimension and attention heads as in DiT (Peebles & Xie) (Chaturvedi et al., 16 Apr 2026).

Classifier-free guidance is implemented by dropping $\Delta L$7 tokens 10% of the time during training to learn an unconditional branch (Chaturvedi et al., 16 Apr 2026). The decoder integrates the predicted velocity field through DDIM sampling with 50 steps to obtain an output latent, which is then decoded back to RGB (Chaturvedi et al., 16 Apr 2026).

Two architectural implications follow directly from these choices. First, the model couples spatial image evidence and lighting controls at the token level rather than through late fusion; this plausibly contributes to the reported sensitivity to light position, fixture localization, and diffuse spread. Second, the absence of a separate geometric module means that geometric reasoning, visibility, and material effects must be represented implicitly in the transformer’s conditional denoising dynamics.

4. Training data, objective, and optimization

The training setup combines a large-scale synthetic relighting corpus with a smaller real-capture fine-tuning set (Chaturvedi et al., 16 Apr 2026). The synthetic corpus contains approximately 4,000 procedurally generated 3D scenes from Objaverse and procedural objects plus humans. Each scene uses 64 sampled point-light positions and an HDRI chosen from approximately 600 PolyHaven maps (Chaturvedi et al., 16 Apr 2026).

The rendered synthetic components include an ambient render under HDRI, on-light renders for each sampled point light and for each spread angle across 6 values, and per-fixture separation in 83 artist-authored indoor scenes, yielding approximately 100K paired renders (Chaturvedi et al., 16 Apr 2026). Training pairs are synthesized on the fly according to

$\Delta L$8

with Reinhard tone mapping, while $\Delta L$9 are randomized at load time (Chaturvedi et al., 16 Apr 2026).

The real-capture fine-tuning set, VisibleFixture-600, contains approximately 600 iPhone ProRAW indoor photographs under on/off toggles of visible fixtures, with manually segmented fixture masks and paired captures (Chaturvedi et al., 16 Apr 2026). Its stated purpose is to bridge the domain gap and improve realism.

Training uses a conditional flow-matching objective: $R$0

$R$1

and

$R$2

No explicit adversarial or perceptual losses are used; the paper attributes photorealism to the pretrained visual prior and real-data fine-tuning (Chaturvedi et al., 16 Apr 2026).

Optimization uses AdamW with learning rate $R$3, weight decay $R$4, and $R$5, for 15,000 steps with batch size 160, partitioned as 64 spatial, 48 fixture, and 48 diffuse (Chaturvedi et al., 16 Apr 2026). Inference uses DDIM with 50 steps and guidance scale $R$6 (Chaturvedi et al., 16 Apr 2026).

5. Quantitative performance

The paper reports quantitative evaluation on synthetic spatial relighting, real visible-fixture editing, and human preference studies (Chaturvedi et al., 16 Apr 2026). On the synthetic spatial benchmark, the test set contains 200 held-out Objaverse objects, and two target modalities are considered: PointGT, a true point-light render, and PanoGT, where a point light is approximated as an environment map (Chaturvedi et al., 16 Apr 2026). On the object mask, the evaluated metrics are PSNR, SSIM, and LPIPS, and TokenLight is reported to outperform Neural Gaffer and DiffusionRenderer by a large margin, with PSNR around 21 dB versus 16–13 and LPIPS around 0.02 versus 0.04 (Chaturvedi et al., 16 Apr 2026).

A more specialized position-precision analysis uses 50 objects and six linear light trajectories of 32 positions each (Chaturvedi et al., 16 Apr 2026). The paper constructs a confusion matrix

$R$7

and defines

$R$8

Precision is measured as $R$9 upward, and accuracy as $$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$0 downward (Chaturvedi et al., 16 Apr 2026). Under this analysis, TokenLight achieves $$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$1 versus 1.11 and $$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$2 versus 0.160 for Neural Gaffer (Chaturvedi et al., 16 Apr 2026). These figures support the claim that the tokenized control space preserves fine sensitivity to light placement, not merely coarse scene brightening.

On the real VisibleFixture-60 benchmark, comprising 60 on/off fixture pairs, TokenLight is compared to ScribbleLight on the masked fixture region using PSNR, SSIM, and LPIPS (Chaturvedi et al., 16 Apr 2026). The reported values are PSNR 20.08, SSIM 0.848, and LPIPS 0.278 for TokenLight, versus 14.64, 0.519, and 0.611 for ScribbleLight (Chaturvedi et al., 16 Apr 2026).

The user study involves 18 participants, 20 in-the-wild scenes, and 5 lights, with pairwise comparison against GenLit and Careaga et al. (Chaturvedi et al., 16 Apr 2026). The reported preference rates are 77.5% over GenLit and 89.2% over Careaga (Chaturvedi et al., 16 Apr 2026). While user studies are inherently subjective, these results are consistent with the numerical gains reported for spatial precision and fixture editing.

6. Qualitative behavior, invariances, and limitations

The qualitative results emphasize several operational regimes. For spatial virtual lights, the paper shows examples of inserting a light inside a pumpkin, behind a book, behind a transparent glass object, and producing rim lighting on hair (Chaturvedi et al., 16 Apr 2026). The reported interpretation is that the model respects occlusion and geometry without explicit depth. For ambient and diffuse controls, progressive ambient dimming is shown to spotlight documents and portraits, while varying the global diffuse spread $$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$3 modulates shadow softness and sharpness (Chaturvedi et al., 16 Apr 2026). For in-scene fixtures, examples include turning individual lamps on or off, adjusting per-fixture color and intensity, and suppressing a car taillight without darkening sunlight in an outdoor example (Chaturvedi et al., 16 Apr 2026). The model supports up to three lights with independent position, intensity, color, and diffuse parameters, producing additively mixing colored shadows (Chaturvedi et al., 16 Apr 2026).

The design analysis also discusses scene-agnostic coordinates. Equations (5–8) are reported to show similarity-transform invariance, such that positions $$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$4, energies $$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$5, and radii $$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$6 scale correctly under $$I_r \;=\; R(I,\;\Delta L)\;\approx\;f\bigl(g(I),\,L+\Delta L\bigr),$$7, and the supplementary material confirms invariance under arbitrary scene transforms (Chaturvedi et al., 16 Apr 2026). This suggests that the chosen coordinate parameterization is intended to remain stable under canonical rescaling and rigid similarity transformations rather than being tied to a single scene metric.

Ablations address inference-step tradeoffs and stochasticity. Supplementary results indicate that even 5–10 DDIM steps yield plausible relighting, although 50 steps are used by default (Chaturvedi et al., 16 Apr 2026). Random-seed variation is reported as minor at low diffuse levels and to stabilize as shadows soften (Chaturvedi et al., 16 Apr 2026).

The paper also notes limited failures. Seed dependence and speed—reported as 2 s per 50 steps on A100—motivate future work in distillation (Chaturvedi et al., 16 Apr 2026). More broadly, the method’s reliance on a pretrained generative prior means that physically plausible outcomes do not necessarily imply explicit physical correctness in the inverse-rendering sense. The reported success without inverse-rendering supervision demonstrates strong learned relighting behavior, but it should not be conflated with recovery of calibrated scene parameters unless such recovery is explicitly evaluated.