RelightMaster: Unified Differentiable Relighting

• RelightMaster is a unified system for image, video, and 3D scene relighting that combines explicit inverse rendering with neural network refinement.
• It employs differentiable physics-based rendering and self-supervised optimization to process monocular, in-the-wild photographs with editable HDR and point light setups.
• The system bridges traditional 3D graphics precision and neural flexibility, achieving high-fidelity, controllable relighting through a hybrid rendering and refinement pipeline.

RelightMaster denotes a set of physically-motivated and generative systems for image, video, and 3D scene relighting, unified by the goal of providing explicit, physically-based, and editable lighting control in visual computing. By integrating explicit inverse rendering, differentiable physics-based rendering (PBR), and neural refinement, RelightMaster achieves end-to-end, self-supervised, and high-fidelity relighting across in-the-wild photographs, scenes, and environments. The paradigm bridges the accuracy and controllability of conventional rendering software with the flexibility and realism of neural networks, enabling workflows previously confined to 3D graphics platforms for direct use on monocular, uncalibrated, or real-world imagery (Careaga et al., 7 Aug 2025).

1. Image Formation and Scene Representation

RelightMaster fundamentally models scene appearance using the steady-state rendering equation:

$$L_o(x, \omega) = L_e(x, \omega) + \int_\Omega f_r(x, \omega' \rightarrow \omega) L_i(x, \omega') (\omega' \cdot n) d\omega'$$

This is realized with Monte Carlo path tracing (bounce depth=3 in practical settings), leveraging modern differentiable renderers. Geometry is estimated monocularly using methods such as MoGe (depth estimation, point cloud unprojection, and 2.5D triangulation), producing a mesh $M$ with vertex-wise normals. Reflectance decomposition utilizes intrinsic image networks (e.g., CID), yielding a factorization $I(x) = A(x) \cdot S(x) + R(x)$ into albedo $A$, shading $S$, and residuals $R$. The mesh is textured with projected $A$ to form an illumination-invariant colored mesh.

Lighting is parameterized as high-dynamic-range (HDR) environment maps $E$ (typically $128 \times 256$) and a finite set of point lights $\{p_i\}_{i=1}^K$, each characterized by position and intensity. The combined lighting configuration $\Psi = \{E, \{p_i\}\}$ supports both global and local edits, merging the flexibility of 3D scene authoring tools with neural inference capabilities (Careaga et al., 7 Aug 2025).

2. Differentiable Rendering and Self-Supervised Training

RelightMaster employs a differentiable PBR backend. For a given mesh $M$ and lighting $\Psi$, a forward pass synthesizes a diffuse rendering $\tilde D = \mathrm{pbr}(M, \Psi)$ using path tracing. Output channels include $\tilde D$, albedo $A$, and an invalid-pixel mask $V^c$. At training time, the pipeline solves for the unknown lighting $\Psi^*$ that minimizes $\| D - \mathrm{pbr}(M, \Psi) \|_2^2$ (where the target $D = A \cdot S$), using automatic differentiation with respect to both the HDR environment map and point light parameters via Mitsuba 3. Adam optimizer (lr=$10^{-2}$) is used, initializing $E$ gray and distributing point lights in a grid.

This differentiable rendering enables self-supervised learning: every real photograph furnishes its own supervisory signal, after decomposition and rendering, thus obviating the need for ground-truth lighting captures (Careaga et al., 7 Aug 2025).

3. Neural Refinement Network for Photorealism

The photorealistic relighting result is produced by a feed-forward encoder–decoder network, modeled after Midas and using a shift-invariant ResNet backbone and a U-Net–like decoder with skip connections. The network ingests input crops of $[\tilde D; A; V^c]$ at $384 \times 384$ resolution in linear RGB space and outputs the final relit image $\tilde I$, refined via ReLU. The self-supervised loss amalgamates pixel-level MSE and multi-scale image gradient discrepancies:

$$L = \| I - \tilde I \|_2^2 + \sum_m \| \nabla I^m - \nabla \tilde I^m \|_2^2$$

This neural module is responsible for restoring high-frequency visual attributes missed by the approximate PBR, such as soft shadows, intricate specularities, and complex materials, ensuring output fidelity and realism (Careaga et al., 7 Aug 2025).

4. Data, Training Protocol, and Evaluation

Training leverages in-the-wild raw photo corpora (RAISE, MIT-5k, PPR10k, LSMI). For each image, precomputation yields mesh, lighting parameters, and PBR output; about 15% of images with poor PBR fits (measured by the training objective) are filtered to preserve consistency. Data augmentation strategies include random cropping and flipping. The pipeline can maintain $\sim0.7$s inference time for $512 \times 512$ images using 16 spp path tracing and a single neural pass.

Evaluation comprises both quantitative (user studies, SSIM/LPIPS improvements over OutCast, RGB⇄X, ScribbleLight) and qualitative metrics, demonstrating controllability over diverse scenarios (e.g., headlights, day–night transitions) with physically plausible outcomes. Noted limitations are inability to model occluded geometry beyond the visible mesh (light leaks, shadow errors), and constraints imposed by the diffuse-only PBR which necessitate neural hallucination for specular and subsurface materials (Careaga et al., 7 Aug 2025).

5. Design Principles, Practical Guidelines, and Extensions

RelightMaster’s core design is a hybrid: it allocates explicit 3D controllability and lighting transport to a fast PBR front end and reserves all high-frequency, view-dependent, or out-of-model phenomena for the neural refinement network. Differentiable rendering not only allows learning from unannotated real photos but also stabilizes lighting optimization by constraining the parameter dimensionality (environment map + $\leq$ 16 point lights). Accurate scene reconstruction is ensured by employing current state-of-the-art monocular depth and intrinsic decomposition approaches.

Practical considerations include pre-filtering for PBR fit error, simulating only first-bounce indirect light jointly with explicit cast shadows, and targeting efficiency suitable for interactive use. To further reduce outlier influence and enhance generalization, outlier-heavy cases (e.g., deep occluders, out-of-mesh objects) are excluded from training (Careaga et al., 7 Aug 2025).

6. Impact, Limitations, and Research Directions

RelightMaster delivers explicit 3D light source control traditionally limited to parametric 3D engines, enabling such manipulations in single-view, in-the-wild photographs. This advances the state-of-the-art in relighting, allowing physically grounded editing on unconstrained inputs. However, current limitations stem from the unimodal 2.5D geometry representation, lack of explicit handling for specularities/subsurface scattering in PBR, and the need for the neural renderer to hallucinate absent details.

Future research directions include closing the gap in occlusion reasoning by incorporating multi-view geometry estimation, extending the differentiable rendering component to include specular/anisotropic BRDFs, and improving the realism of neural refinement for challenging appearance categories (notably faces or intricate reflective materials). A plausible implication is that further synergistic integration between explicit physics and learned appearance modeling will continue to push controllability, realism, and generalizability for relighting in both consumer and professional imaging workflows (Careaga et al., 7 Aug 2025).