Relit-LiVE: Relight Video by Jointly Learning Environment Video

Abstract: Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric-illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE.

• The paper presents a novel framework that fuses RGB cues with intrinsic decomposition to correct artifacts and capture complex global illumination.
• It jointly generates relit videos and per-frame environment maps without requiring explicit camera pose estimation, ensuring coherent illumination.
• Empirical results show significant improvements in PSNR, SSIM, and LPIPS on benchmarks, with practical applications in scene editing and neural rendering.

Relit-LiVE: Video Relighting by Joint Learning of Environment Video

Introduction and Motivation

Video relighting—the task of altering a video's illumination while preserving its intrinsic scene properties—remains a significant challenge, particularly with respect to achieving physically consistent, temporally stable, and material-faithful results. Prior approaches primarily fall into two paradigms: (1) direct, end-to-end relighting via large video diffusion models conditioned on text or images, which provides flexibility at the cost of lighting control and physical plausibility; and (2) two-stage pipelines that separate intrinsic decomposition and subsequent conditioned rendering, offering finer control but limited by the instability and domain gap of intrinsic estimation, especially in uncontrolled, real-world scenes. Furthermore, most methods necessitate accurate, per-frame camera pose, which is often unavailable or impractical in natural video.

Relit-LiVE addresses these challenges via two core innovations. First, it leverages a novel RGB-intrinsic fusion renderer, incorporating raw input frames as reference images to guide physically based rendering, thus effectively recapturing global illumination effects lost in intrinsic decomposition. Second, it introduces joint prediction of environment video—per-frame, viewpoint-aligned environment maps—enabling physically and geometrically consistent relighting without a requirement for prior camera pose estimation.

Figure 1: Relit-LiVE framework overview: given an input video and the environment map of the initial viewpoint, the system predicts both relit video and per-frame environment maps via intrinsic decomposition, latent mapping, partial fusion, and a DiT-based video denoising model.

Methodology

RGB-Intrinsic Fusion Renderer

Relit-LiVE begins by decomposing the input video into a suite of intrinsic properties using a pretrained inverse renderer: base color, surface normal, depth, roughness, and metallicness. These G-buffers are encoded into a structured latent space. A raw RGB reference frame is also randomly sampled from the sequence and encoded. The system performs groupwise latent fusion, particularly separating properties with shared semantics, and then concatenates the reference image latent along the frame dimension. This design enables the renderer to utilize direct visual (RGB) cues in conjunction with intrinsic physical constraints, correcting artifacts from imperfect decomposition and preserving complex real-world lighting effects.

Figure 2: Intrinsic perception enhancement: synthesizing multi-illumination data to be used as diverse reference images for training.

Joint Generation of Relit and Environment Video

Unlike prior work that assumes known camera poses and provides per-frame environment maps, Relit-LiVE reformulates relighting as a joint generation task: synthesizing the relit video and the corresponding environment video (i.e., a warped environment map for each frame) in a single forward pass of a DiT-based video diffusion network. This obviates the need for explicit camera parameter estimation, with the network implicitly learning to align geometric and illumination cues spatio-temporally via training supervision.

Lighting is encoded using three representations—Reinhard-tone-mapped LDR images, normalized log-intensity HDR maps, and directional encoding maps—processed with VAE encoders and fed both as direct features and as inputs to the network's cross-attention mechanism for dual-path lighting control.

Training Strategies

To mitigate the domain gap and data scarcity in real multi-illumination video, the authors introduce two data-centric training strategies:

1. Intrinsic Perception Enhancement (IPE): Latent-space interpolation between the outputs of the relighting (conditioned on input RGB) and rendering (unconditioned) pathways to synthesize a continuum of pseudo-realistic, multi-illumination reference data for training augmentation.
2. Self-supervised Illumination Consistency (SIC): Cycle-consistent supervision is imposed by relighting a video forward and in reverse order under controlled lighting, enforcing temporal coherence and improving generalization without additional ground-truth annotations.

   Figure 3: Self-supervised illumination consistency: forward and reverse relighting operations enforce temporal and lighting cycle-consistency.

Experimental Results

Quantitative and Qualitative Performance

Relit-LiVE achieves consistent improvements over prior methods—such as UniRelight, Diffusion Renderer, and Light-A-Video—across multiple metrics and datasets. On synthetic datasets and the MIT multi-illumination benchmark, Relit-LiVE demonstrates substantially higher PSNR, SSIM, and lower LPIPS, reflecting superior fidelity and perceptual quality. On in-the-wild videos, it outperforms both environment map-conditioned and text-prompted SOTA by large margins in terms of material consistency and lighting alignment, as further validated by user studies.

Figure 4: Qualitative comparison on MIT multi-illumination: Relit-LiVE handles complex materials and global effects, surpassing baselines in reflection and transmission quality.

Figure 5: In-the-wild video relighting: Relit-LiVE consistently surpasses both environment map and prompt-conditioned baselines in quality and physical consistency.

Figure 6: Temporal and photorealistic consistency in video relighting, outperforming alternatives.

Relit-LiVE also excels under dynamic lighting and camera motion. Its environment video predictions maintain geometric-illumination alignment across frames, enabling accurate, viewpoint-aware relighting for long, streaming sequences, and demonstrating strong generalizability even without per-frame pose annotations.

Figure 7: Robust performance under simultaneous changes in scene content and dynamic illumination.

Figure 8: Long-video segmentation and environment video tracking: Relit-LiVE maintains lighting consistency and accurate environment map propagation for sequential video chunks.

Downstream Applications

The framework naturally enables advanced scene editing (object insertion, material property manipulation), video delighting (specular highlight removal), and forward neural rendering without explicit relighting. Ablation experiments confirm that both the reference RGB signal and environment video joint modeling are critical for optimal results.

Figure 9: Scene editing: realistic lighting synthesis when inserting new objects and editing material attributes.

Figure 10: Video delighting: natural removal of scene illumination by resynthesizing under specific environmental maps.

Analysis and Ablation

Ablations show that removing either the raw reference image or the environment video estimation branch produces significant degradation—especially in physically plausible rendering and material consistency. Additionally, the dual-path lighting control (feature fusion + cross-attention) demonstrates the best trade-off between reflecting lighting effects and color temperature accuracy.

Figure 11: The raw reference image enhances handling of complex materials in relighting tasks.

Figure 12: Training strategy ablation: IPE and SIC each add substantial quantitative and qualitative improvements over the base model.

Limitations and Future Work

While merging different intrinsic latents lowers computational costs relative to frame-wise concatenation, the temporal control mechanism still leads to moderate training and memory overhead, with Relit-LiVE supporting up to 57 frames at $832 \times 480$ resolution and requiring approximately 10 minutes per video on an A800 GPU. Failure cases primarily relate to imperfect pseudo-labels in intrinsic properties or environment maps in the auto-labeled real-world training set, resulting in rare color or illumination artifacts.

Figure 13: Failure cases: color shifts and abnormal illumination due to pseudo-label errors in real-world data.

Opportunities for future work include further improving real-world pseudo-label quality, extending to higher resolutions and longer sequences, more efficient architecture for latency-critical deployment, and integrating additional scene-level priors or temporal cues. The demonstrated capacity for joint relighting and environment video estimation suggests strong potential not only in generative video editing but also inverse problem tasks such as lighting estimation and semantic scene understanding.

Conclusion

Relit-LiVE presents a cohesive and scalable framework for video relighting in unconstrained settings, characterized by physically consistent, temporally stable, and geometrically precise results without the need for camera pose priors. The fusion of reference image guidance and joint relighting-environment video generation enables robust handling of complex scenarios with diverse lighting, motion, and material properties. The system's versatility is substantiated by leading benchmark performance, extensive ablations, and applicability to a range of downstream video manipulation tasks, laying the groundwork for further advances in neural rendering, video editing, and scene-level generative modeling (2605.06658).