Inferring HDR environment lighting from casually captured images or video

Develop reliable algorithms to infer high-dynamic-range (HDR) 360-degree environment illumination maps from single casually captured images or videos, achieving accurate recovery of light direction and intensity from indirect visual cues without relying on specialized capture setups.

Background

Illumination estimation is central to physically-based rendering and applications such as virtual object insertion, augmented reality, and scene relighting. A common representation is the HDR environment map, which encodes incident light from all directions but is expensive to capture and scarce in real-world datasets.

Prior supervised methods depend on paired LDR inputs and HDR ground truth, limiting generalization. Generative approaches (e.g., diffusion-based inpainting of chrome balls) reveal implicit illumination priors but are often insufficiently robust, lack direct HDR output, and require costly test-time procedures. The authors highlight that, despite progress, robustly inferring scene lighting from casual imagery remains unresolved.

References

Yet inferring lighting from casually captured images or video remains an open challenge.

— LuxDiT: Lighting Estimation with Video Diffusion Transformer  (2509.03680 - Liang et al., 3 Sep 2025) in Section 1 (Introduction)

Assuming a probe is correctly generated and segmented, quantifying the exact measurement error of our pipeline is difficult since we do not know the material properties of the generated probes.

— Shedding Light: A Benchmark for Evaluating Lighting Understanding in Generative Image Models  (2609.10787 - Giroux et al., 9 Sep 2026) in Discussion, paragraph “Approximating the measurement error”