TC-Light: Coherent Video Rendering

• TC-Light is a generative video rendering framework that produces realistic video relighting and texture refinement for dynamic scenes.
• It employs a two-stage post-optimization pipeline, first aligning global exposure and then refining details via a Unique Video Tensor for temporal consistency.
• Its efficient design minimizes artifacts while supporting sim2real adaptation, AR/VR content creation, and robust performance in embodied AI applications.

TC-Light designates a temporally coherent generative video rendering framework for realistic and physically plausible world transfer, targeting applications in sim2real/real2real domain adaptation, visual content creation, and embodied AI. The model is distinguished by its robust handling of complex video dynamics and long temporal ranges, surpassing prior approaches limited to specific domains or suffering from temporal inconsistencies and high computational demands. At its core, TC-Light employs a post-optimization pipeline that first aligns global illumination and then refines fine-grained texture and lighting through a canonical video representation, enabling efficient, artifact-minimizing, and scalable video relighting (2506.18904).

1. Problem Setting and Motivation

TC-Light addresses the challenge of re-rendering videos under new lighting and texture conditions—a task central to visual data augmentation for sim2real transfer in robotics and embodied AI, film post-production, and AR/VR scenarios. Previous work on video relighting or conditioned video generation has been constrained either by domain specificity (e.g., portrait-only systems) or by the inability to maintain temporal consistency and computational efficiency in long, highly dynamic video sequences. Naively applying frame-by-frame relighting introduces temporal artifacts, such as flicker, degrading both the photometric and spatial-temporal coherence essential for effective downstream AI or human perception.

2. Model Architecture and Methodology

TC-Light implements a two-stage optimization pipeline over video data that has undergone preliminary relighting with an inflated version of the IC-Light model:

Stage I: Exposure Alignment

• Each frame receives a per-frame appearance embedding, expressed as a $3 \times 4$ affine transform matrix $E_t$, initialized to the identity. This matrix adjusts global exposure to align appearance statistics across adjacent frames.
• A joint loss supervises this optimization: a weighted combination of photometric error $L_\text{photo}$ and an $L_1$ flow-warped alignment loss penalizing misalignments between consecutive frames, masked by a flow-derived soft mask $M_t$ to avoid penalizing moving foregrounds or occlusions.
• The exposure alignment objective is:

$$L_\text{exposure} = (1 - \lambda_e)\, L_\text{photo}(\widetilde{I}_t, I_t) + \lambda_e\,L_1\big(\widetilde{I}_t \odot M_t, \text{Warp}_{t+1 \rightarrow t}(\widetilde{I}_{t+1}) \odot M_t\big)$$

where $\widetilde{I}_t(x, y) = E_t [I_t(x, y) \mid 1]^T$.

Stage II: Unique Video Tensor (UVT) Optimization

• The video is canonically compressed into the Unique Video Tensor (UVT), which serves as an indexable, low-rank representation encompassing spatiotemporal and content cues.
• Each pixel $(x, y, t)$ is mapped via an index function $\kappa(x, y, t)$—potentially incorporating flow identity, quantized color (RGB), and optionally depth or voxel indices.
• The UVT is built by averaging over all pixels sharing the same index:

$$U(\kappa_n) = \operatorname{Avg}\left\{I^{\text{in}}_t(x, y) \mid \kappa(x, y, t) = \kappa_n\right\}$$

and original frames are recovered via $I^{\text{in}}_t(x, y) = U(\kappa(x, y, t))$.

• The optimization of the UVT is supervised by a combination of total variation loss $L_\text{tv}$ (regularizing sharpness), SSIM loss $L_\text{SSIM}$ (promoting perceptual similarity), and a temporally aligned $L_1$ loss:

$$L_\text{unique} = \lambda_\text{tv} L_\text{tv}(\widetilde{I}_t) + (1 - \lambda_u) L_\text{SSIM}(\widetilde{I}_t, I_t) + \lambda_u L_1\left(\widetilde{I}_t \odot M_t, \text{Warp}_{t+1 \rightarrow t}(\widetilde{I}_{t+1}) \odot M_t\right)$$

• The model employs a multi-axis denoising scheme—balancing outputs from denoisers acting along $xy$ (spatiotemporal) and $yt$ (texture) planes—using a step-dependent coefficient $\gamma_t$ for blending, as well as adaptive instance normalization (AIN) for improved transferability.

3. Benchmark Construction and Evaluation Metrics

To evaluate temporal coherence, TC-Light introduces a dedicated long and highly dynamic video relighting benchmark, comprising 58 videos of average length 256 frames across a diverse set of illumination, scene content, and motion conditions (indoor/outdoor, synthetic/real, variable weather and lighting). This benchmark is designed to stress test temporal stability and physical plausibility in scenarios beyond the scope of existing datasets.

Evaluation metrics include:

• Motion Smoothness (Motion-S): Quantifies temporal stability across frames.
• Structural Warping SSIM (Warp-SSIM): Measures spatial structure fidelity after warping predicted frames.
• Textural Fidelity (CLIP-T): CLIP-based metric assessing the relevance of rendered textures to prompts or descriptions.
• Computation metrics: frames-per-second (FPS), runtime, and GPU VRAM usage.

Empirical results indicate state-of-the-art performance for TC-Light in both temporal and visual quality, for both relighting accuracy and computational efficiency.

4. Applications and Potential Uses

TC-Light's capacity for consistent, artifact-free relighting of long, highly dynamic video sequences has several key application areas:

• Embodied AI and Data Scaling: By generating diverse, physically plausible renderings of scenes under various lighting/textural conditions, TC-Light supports robust policy learning and domain transfer for robotics and embodied agents in both simulation-to-real and real-to-real scenarios.
• Content Creation (Film/AR/VR): Enables the adjustment of illumination and texture without introducing flicker or inconsistency, which is crucial for visual effects, post-production, and interactive experiences.
• General-Purpose Video Editing: UVT and temporally-aware optimization could be adapted for other forms of video transformation or conditional generation where spatial-temporal coherence is essential.

A plausible implication is that introducing the canonical UVT representation as a basis for video-level editing may inform future pipelines for video synthesis or editing that require high-fidelity spatial-temporal consistency.

5. Technical Innovations

Several methodological novelties distinguish TC-Light:

• Two-Stage Post-Optimization: Sequential correction of global exposure followed by fine detail alignment ensures both global and local photometric consistency across frames.
• Unique Video Tensor (UVT): Canonical, indexable structure that enables spatial-temporal compression and flexible, efficient optimization across variable-length dynamic videos.
• Decayed Multi-Axis Denoising: Separate streams for temporal smoothing and texture preservation, adaptively combined, mitigate the prompt-consistency vs. temporal-smoothness trade-off.
• Masked Temporal Consistency: Employs motion-based soft masks to apply spatiotemporal objectives only where meaningful, avoiding penalization of occluded or moving regions.

6. Limitations and Future Directions

While TC-Light demonstrates clear advantages, identified challenges and future research directions include:

• Optical Flow Dependency: The UVT and temporal alignment objectives depend on accurate flow estimation; errors can propagate into artifacts or misalignments.
• Hard Scene Cases: Difficult re-illumination scenarios such as pronounced hard shadows or low-light situations remain challenging for existing generative models.
• Balance of Prompt vs. Consistency: Further improvement of multi-axis denoising strategies could help reconcile prompt adherence with long-term temporal smoothness.

An anticipated extension is further reducing reliance on explicit flow—or integrating more flexible representation learning for temporal correspondence—and enhancing robustness to failure cases in complex real-world environments.

7. Broader Significance

TC-Light's successful integration of temporally coherent generative rendering and scalable optimization provides a generalizable approach for realistic world transfer in dynamic visual data. Its design principles—combining staged global-local alignment, canonical video-level representation, and advanced denoising—offer a foundation for further advances in video-based generative modeling and domain adaptation across scientific and creative domains.