Relit-LiVE: Camera-Pose-Free Video Relighting

• The paper introduces Relit-LiVE, a diffusion-based video relighting framework that fuses raw RGB cues with inverse-rendered intrinsics to preserve content and achieve physically consistent illumination.
• It leverages joint RGB-intrinsic fusion and environment-video prediction to overcome the limitations of brittle intrinsic decomposition and pose dependency in complex lighting scenarios.
• Experimental evaluations demonstrate significant improvements in PSNR, SSIM, and LPIPS metrics across synthetic and real-world videos, validating its superior performance in preserving material details and temporal coherence.

Searching arXiv for the main paper and closely related context. Relit-LiVE is a video relighting framework for synthesizing a target-lit video from a source video while preserving scene content, material appearance, shadows, reflections, and temporal coherence under viewpoint changes. It is formulated as a camera-pose-free, diffusion-based system that departs from intrinsic-only relighting pipelines by explicitly injecting raw reference RGB frames into the rendering process and by jointly predicting both the relit video and a per-frame camera-aligned “environment video” in a single diffusion process. The method is presented as a response to the failure modes of inverse-rendering-based relighting on real videos, where inaccurate intrinsic decomposition can yield distorted appearances, broken materials, and temporal artifacts (Xiao et al., 7 May 2026).

1. Problem setting and conceptual position

Video relighting seeks to modify illumination while preserving scene identity and physically plausible light transport. In Relit-LiVE, the input is a source video $V^s = \{ I^s_i \}_{i=1}^n$ and a target lighting sequence $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$, and the objective is to synthesize a relit video $V^t = \{ I^t_i \}_{i=1}^n$ via

$V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$

The framework is motivated by a specific criticism of recent neural-rendering-style relighting systems: they commonly depend on intrinsic decomposition into albedo, normals, depth, roughness, metallicity, and lighting, but this decomposition is described as highly unreliable for real-world videos, especially in the presence of transparent objects, subsurface scattering, difficult global illumination, specular highlights, and shadow boundaries (Xiao et al., 7 May 2026).

Within that context, Relit-LiVE is positioned against two broad families of prior control mechanisms. One family uses brittle intrinsic decomposition followed by forward rendering; the other uses weaker lighting controls such as text prompts or rough image cues. The paper’s central claim is that the original RGB video contains scene cues that are lost or corrupted in G-buffer-style representations, so raw image evidence should remain in the rendering loop. A second claim is that known per-frame camera pose should not be a prerequisite for high-quality video relighting. This leads to the method’s two defining ideas: RGB-intrinsic fusion and joint environment-video prediction.

A common misconception is to treat relighting as a photometric adjustment problem. Relit-LiVE is explicitly framed otherwise: the target is not merely brighter or darker footage, but physically consistent changes to illumination that preserve materials, reflections, refractions, and temporal stability across frames. This emphasis places it closer to scene-level rendering than to conventional appearance transfer.

2. Scene representation and RGB-intrinsic fusion

The pipeline begins by passing the source video through a pre-trained inverse renderer from Diffusion Renderer to estimate G-buffer intrinsics: base color $V^\mathbf{a}$, normal $V^\mathbf{n}$, relative depth $V^\mathbf{d}$, roughness $V^\mathbf{r}$, and metallicity $V^\mathbf{m}$. Each intrinsic map is encoded by a pretrained VAE encoder $\mathcal{E}$ into latent tensors $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$0, with $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$1 (Xiao et al., 7 May 2026).

A notable architectural choice is the latent fusion scheme. Rather than concatenating all intrinsic latents along the frame dimension, which the paper describes as expensive, or along the channel dimension, which it says slows convergence, Relit-LiVE uses partial group-wise addition: $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$2

$\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$3

The first group combines base color, depth, and metallicity; the second combines normal and roughness. The stated rationale is that this preserves control while reducing memory and computation.

The distinctive addition is a raw reference RGB frame $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$4, randomly sampled from the source video and encoded to $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$5. This latent is concatenated along the temporal dimension with the grouped intrinsic latents. The random sampling strategy is used to reduce overfitting to source lighting: different denoising steps may sample different source frames, with the aim of retaining detail without propagating the original illumination at the pixel level. The framework therefore does not reject intrinsic representations; instead, it augments them with direct appearance evidence from the video itself.

This design suggests a broader methodological point. Relit-LiVE treats inverse-rendered intrinsics as an informative but lossy abstraction, and uses raw RGB as a corrective signal rather than as the sole conditioning stream. That interpretation is consistent with the paper’s empirical emphasis on improved handling of complex materials and transmission effects.

3. Joint prediction of relit video and environment video

The second major component is the “environment video” formulation. Standard relighting pipelines often require camera pose to warp an environment map into the proper view for each frame. Relit-LiVE instead predicts a per-frame warped environment map sequence jointly with the relit frames, where the environment maps are represented as $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$6 for frame-wise viewpoints $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$7. The combined formulation is

$\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$8

The model receives environment maps under the initial viewpoint $\mathbf{E}^t = \{ \mathbf{E}_i \}_{i=1}^n$9 and learns to infer aligned lighting for later frames, thereby removing the need for explicit pose estimation at inference time (Xiao et al., 7 May 2026).

Environment maps are encoded in three complementary forms. From HDR maps $V^t = \{ I^t_i \}_{i=1}^n$0, the method constructs LDR images $V^t = \{ I^t_i \}_{i=1}^n$1 using Reinhard tonemapping, normalized log-intensity maps

$V^t = \{ I^t_i \}_{i=1}^n$2

with $V^t = \{ I^t_i \}_{i=1}^n$3, and directional encoding images $V^t = \{ I^t_i \}_{i=1}^n$4, in which each pixel stores ray direction in camera coordinates. These are encoded with the same VAE and concatenated channel-wise to form

$V^t = \{ I^t_i \}_{i=1}^n$5

followed by a stride-1 convolution to obtain $V^t = \{ I^t_i \}_{i=1}^n$6. A second copy at $V^t = \{ I^t_i \}_{i=1}^n$7 resolution is injected through cross-attention as $V^t = \{ I^t_i \}_{i=1}^n$8.

At the core of the generator is a DiT-style video diffusion transformer. During training, both the target relit video and the target environment-video representation are encoded into latent space, noise is added independently, and the denoiser jointly predicts the relit-video latent and the environment-video latent. The stated effect is to enforce geometric-illumination alignment while supporting dynamic lighting and camera motion in a single generative process.

4. Optimization strategy and self-supervision

Relit-LiVE is trained in three stages on top of Wan2.1-T2V-1.3B. The first stage trains on synthetic data for 10,000 iterations and then on the full dataset for another 20,000 iterations to obtain the initial model. The second stage generates 8 pseudo-realistic images per real-world scene using the initial model and trains 5,000 iterations on them. The third stage applies Self-supervised Illumination Consistency with probability $V^t = \{ I^t_i \}_{i=1}^n$9 for a final 5,000 iterations. Training uses 8 A800 GPUs, batch size 16, resolution $V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$0, AdamW, learning rate $V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$1, and about 7 days of training. The synthetic phase uses fixed 17-frame clips, while later training gradually increases clip length from 1 to 57 following the $V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$2 pattern. During real-world training, $V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$3 is set to zero with probability 0.3 to simulate pure rendering (Xiao et al., 7 May 2026).

Two named strategies are used to improve generalization. The first, Intrinsic Perception Enhancement (IPE), runs the model in two modes on the same scene under random environment maps: one with the raw-reference latent $V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$4, and one with $V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$5. The paper states that the former yields realism but may leak source lighting, while the latter avoids lighting leakage but suffers from inverse-rendering errors. To construct better training data, the two outputs are interpolated in latent space with interpolation weight $V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$6, decoded by the VAE decoder $V^t = \mathcal{F_\theta}(V^s, \mathbf{E}^t).$7, and reused as new raw reference images.

The second strategy, Self-supervised Illumination Consistency (SIC), relights a video under a random environment map, reverses the frame order, and relights again using the final frame’s predicted environment map as the conditioning source. The two outputs form a cycle-consistent self-supervised pair. The intended effect is stronger temporal lighting coherence and improved alignment between light variation and viewpoint variation without extra annotations.

This training recipe reveals an important design assumption: the model is expected not only to imitate paired supervision, but also to bootstrap more realistic lighting configurations for real-world scenes and to regularize itself through temporal reversal. A plausible implication is that the framework treats lighting consistency as a sequence-level property rather than only a frame-level appearance constraint.

5. Empirical performance and ablation evidence

The evaluation spans a synthetic test set of 1,000 videos with 120 frames each, the MIT multi-illumination benchmark, and in-the-wild collections from Pexels and Sora, together with PandaSet and Bridgev2. In total, the paper states that the evaluation covers over 1,400 dynamic videos across synthetic, human, embodied, and driving domains. Baselines include UniRelight, Diffusion Renderer, Light-A-Video, TC-Light, and NeuralGaffer for relighting; RGBX and Diffusion Renderer for rendering; and StyleLight and DiffusionLight for light estimation (Xiao et al., 7 May 2026).

On relighting benchmarks, the reported quantitative gains are substantial. On synthetic images, Relit-LiVE achieves 24.85 PSNR, 0.792 SSIM, and 0.175 LPIPS, compared with Diffusion Renderer’s 17.09, 0.679, and 0.264. On synthetic video, it reports 25.39 PSNR, 0.807 SSIM, and 0.205 LPIPS, compared with 16.45, 0.665, and 0.278 for Diffusion Renderer. On MIT multi-illumination, it reports 21.86 PSNR, 0.849 SSIM, and 0.132 LPIPS, while UniRelight is reported at 20.76, 0.749, and 0.251 and Diffusion Renderer at 17.29, 0.622, and 0.355.

For in-the-wild video relighting, the paper uses Motion Preservation, CLIP-MC, and DINO-MC, together with user-study scores for Visual Realism, Physical Consistency, and Lighting Alignment. Relit-LiVE reports Motion Preservation 0.1692, CLIP-MC 0.9246, and DINO-MC 0.9091, compared with Light-A-Video’s 0.4557, 0.9150, and 0.8919, TC-Light’s 0.2405, 0.8977, and 0.8825, and Diffusion Renderer’s 0.3094, 0.9105, and 0.8754. The user study involves 37 participants and 10 comparison sets each, and the paper states that participants preferred Relit-LiVE in realism, physical consistency, and lighting alignment.

The environment-video branch is separately evaluated as a lighting-estimation component on sunlit scenes using angular error for top-5 and top-3 illumination directions. Relit-LiVE reports 20.35 mean, 7.96 median, and 14.14 std for top-5, and 20.69 mean, 7.76 median, and 14.31 std for top-3. StyleLight is reported at 66.12, 62.62, and 24.11 for top-5 and 69.27, 63.81, and 26.23 for top-3, while DiffusionLight is reported at 54.88, 51.58, and 18.83 for top-5 and 55.18, 51.82, and 18.82 for top-3.

The ablations support the paper’s central design choices. Removing the environment-video branch substantially hurts performance; removing the raw reference image also degrades results, especially on complex materials and physical transmission effects such as glass and plastic. Group-wise G-buffer addition is reported to achieve similar quality to frame concatenation while using about 25% less GPU memory. Another ablation compares lighting injection pathways and finds that using both latent fusion and cross-attention yields the best balance: fusion alone can copy content too aggressively from the raw reference, while cross-attention alone tends to miss fine reflection details.

6. Applications, distinctions, and broader significance

Relit-LiVE is presented not as a narrow relighting-only system, but as a scene decomposition and rendering framework that supports scene-level rendering, material editing, object insertion, delighting, and streaming video relighting (Xiao et al., 7 May 2026). For scene editing, the paper uses intermediate intrinsics to modify object materials, insert new objects, and preserve realistic shadows and reflections. Grounded-SAM is used to obtain masks for targeted material edits, and new objects are inserted directly into the reference image before latent interpolation is used to harmonize the edited result. For delighting, the model can remove specular highlights and reconstruct underlying material appearance by resynthesizing the video under a chosen environmental map.

Its streaming formulation is especially notable. Long videos are split into clips and relit sequentially; the environment video predicted for one clip provides the lighting condition for the next clip’s first frame. This indicates that the environment-video branch is not merely an auxiliary supervision target, but an operational representation for cross-clip illumination continuity.

The method also clarifies several distinctions within the relighting literature. It is not an intrinsic-free method, since it still depends on inverse-rendered G-buffers. It is not a pose-driven renderer, because explicit camera pose is not required at inference time. It is also not solely a diffusion prior over output frames, since its conditioning structure includes grouped intrinsics, raw RGB reference evidence, and multi-form environment-map encodings. The paper’s qualitative comparisons with NeuralGaffer, Diffusion Renderer, Light-A-Video, and TC-Light are used to argue that these differences matter most under hard conditions such as transparent materials, unusual lighting such as neon illumination, and scenes in which source shadows and highlights must be suppressed without compromising identity.

Taken together, these features define Relit-LiVE as a camera-pose-free relighting system built around a joint scene-and-light prediction principle. The broader significance is that illumination is modeled as a temporally evolving, viewpoint-aligned sequence rather than as a static control signal. This suggests a more general view of video relighting: the target of generation is not only the relit appearance, but also the latent evolution of the lighting field that makes that appearance physically and temporally coherent.