LuxDiT: Diffusion-Based HDR Lighting

• LuxDiT is a conditional generative framework that leverages a video diffusion transformer to synthesize complete 360° HDR environment maps from images or videos.
• The approach integrates latent-space denoising with a parameter-efficient LoRA finetuning stage, bridging the gap between synthetic and real HDR data.
• LuxDiT achieves state-of-the-art performance, reducing peak angular error by nearly 50% and ensuring temporally stable, high-frequency lighting inference.

LuxDiT is a conditional generative framework targeting high-dynamic-range (HDR) lighting estimation from single images or video input. The model leverages a video diffusion transformer architecture, combining latent-space denoising with global visual conditioning to synthesize 360° environment maps. LuxDiT is trained primarily on synthetic datasets rich in diverse lighting conditions, followed by a parameter-efficient LoRA finetuning stage on real HDR panoramas. The approach achieves accurate, high-frequency lighting inference, setting new benchmarks across multiple indoor and outdoor scene datasets and enabling robust applications in graphics, rendering, and augmented reality.

1. Formulation and Architectural Principles

LuxDiT conceptualizes lighting estimation as a conditional generative modeling problem, where, given an RGB image or video clip, the output is a full HDR environment map encoding the spatially varying illumination of the entire scene. The method uses a video diffusion transformer (DiT) operating in latent space, allowing for both local and global aggregation of visual cues—crucial for inferring indirect lighting attributes not directly captured by high-frequency pixel regions. The generative transformer is adapted to attend over both temporal (video-sequence) and spatial domains, supporting temporally consistent estimation in dynamic scenes.

The model architecture incorporates a dual tone-mapping representation of HDR lighting. Specifically, each environment map $E$ is encoded as both a Reinhard-like tone-mapped signal and a logarithmic intensity map:

• Reinhard mapping: $$E_{\text{Reinhard}} = (E / (1 + E)) \cdot (1 + E / M_{\text{LDR}}^2)$$
• Log-mapping: $$E_{\text{log}} = \log(1 + E) / \log(1 + M_{\text{log}})$$

where $M_{\text{LDR}} = 16$ and $M_{\text{log}} = 10000$ are fixed dynamic-range factors. These two encoded representations are passed through a variational autoencoder (VAE) and, after diffusion denoising, fused through a lightweight multilayer perceptron (MLP) to yield the reconstructed HDR output.

Conditioning is performed using learned visual tokens extracted from the input image(s) or frame(s). Angular consistency in the predicted panoramas is enforced by injecting explicit directional embeddings and leveraging the transformer’s self-attention mechanisms.

2. Training Data and Synthetic-to-Real Generalization

The scarcity of ground-truth paired HDR scene-lighting maps motivates LuxDiT’s two-stage training protocol. Initially, a large-scale synthetic dataset is constructed by physically-based rendering (PBR) of randomized 3D scenes leveraging repositories such as Objaverse. These scenes are illuminated with sampled HDR environment maps to produce input-output pairs capturing complex relationships between indirect visual effects (shadows, specularities, inter-reflections) and underlying illumination.

Additionally, training incorporates perspective crops from real HDR panoramas and low-dynamic-range panoramic video datasets (e.g., WEB360), ensuring the model learns mappings generalizable across synthetic and natural domains. This approach enables the network to capture domain-invariant priors, enhancing prediction fidelity on real-world footage.

A plausible implication is LuxDiT’s synthetic data pipeline can be extended to reflect sensor- or device-specific characteristics by appropriately parameterizing the render pipeline, thereby facilitating transfer to domain-specific use cases in graphics or mixed reality.

3. Low-Rank Adaptation (LoRA) Finetuning

To further bridge the gap between synthetic training and real-world semantics, LuxDiT employs a low-rank adaptation (LoRA) strategy. LoRA finetuning injects trainable low-rank matrices into select transformer layers, with the majority of base weights fixed, yielding efficient adaptation without catastrophic forgetting. Finetuning is supervised with a curated set of real HDR panoramas, optimizing the model to semantically match scene content—preventing mismatches where, for example, realistic lighting is necessary for urban or architectural scenes.

During the LoRA stage, only features directly involved in real-image tone mapping are used in the objective, while generalization and coverage are preserved through the fixed base architecture. This editorial configuration ensures semantic alignment and prevents overfitting to sparse HDR supervision.

4. Evaluation Metrics and Comparative Performance

LuxDiT is evaluated on standardized benchmarks ranging from indoor (Laval Indoor HDR) and outdoor (Laval Outdoor Sun) settings to panoramic datasets such as Poly Haven. Metrics include:

• Scale-invariant root mean squared error (RMSE)
• Angular error of peak luminance direction
• Normalized RMSE

The system demonstrates superior accuracy relative to contemporaneous methods, such as DiffusionLight and StyleLight. For example, on scenes with concentrated sunlight, LuxDiT reduces peak angular error by nearly 50%. Temporal consistency is maintained for video input, yielding predictions free from flickering, a frequent shortcoming in prior work.

Method	Dataset	Peak Angular Error	Scale-invariant RMSE
DiffusionLight	Laval Outdoor	Higher	Higher
StyleLight	Laval Indoor	Higher	Higher
LuxDiT	Both	Lower (down 50%)	Lower

This suggests that diffusion transformer-based methods, particularly with dual-representation tone mapping and global-context aggregation, define a new state-of-the-art for empirical lighting estimation.

5. Applications and Technical Implications

LuxDiT’s output—spatially resolved, temporally stable HDR lighting—enables robust applications across graphics, augmented reality (AR), virtual object insertion, and inverse rendering pipelines. Accurate environment maps allow for physically-plausible shadowing, reflection matching, and photometric consistency in downstream tasks. In AR, precise lighting supports seamless blending of virtual objects with live video, critically dependent on correct angular and intensity distributions.

A plausible implication is the methodology’s extensibility to synthetic data generation for robotics, enhancing perception tasks where realistic simulation of environmental illumination is required.

Beyond visual effects, LuxDiT advances unified approaches for inverse rendering, suggesting future directions where lighting, geometry, and material properties are inferred jointly using deep generative models.

6. Limitations and Prospective Directions

Noted limitations include the computational cost of diffusion-based denoising, which precludes real-time deployment. The model’s output resolution is also constricted by the spatial granularity of latent representations and available HDR ground-truth. Future improvements may focus on model distillation, architectural optimizations for acceleration, or augmenting dataset diversity for higher-fidelity outputs.

Further integration of geometry and material estimation with lighting inference may enhance reconstruction completeness, unifying inverse and forward rendering paradigms.

7. Summary and Contextual Significance

LuxDiT embodies a novel intersection of generative diffusion modeling, transformer-based global context aggregation, and efficient adaptation mechanisms for lighting estimation. Through synthetic pretraining and LoRA-empowered real-data finetuning, it synthesizes HDR environment maps characterized by high-frequency angular detail and semantic alignment with input images or video. LuxDiT outperforms contemporaneous approaches, providing a foundational technology for next-generation graphics and perception applications, and advancing the state-of-the-art in scene illumination inference from visual data (Liang et al., 3 Sep 2025).