BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering

Abstract: Inverse rendering of urban scenes from captured videos enables numerous applications, including content creation and autonomous driving simulation. Physically-based rendering methods follow and control lighting physics, but suffer from reconstruction and rendering artifacts. While generative models produce realistic videos, they offer limited consistency and controllability. We present BRDFusion, a unified framework that combines two complementary models for inverse and forward rendering. Specifically, BRDFusion recovers explicit, consistent scene properties with physical modeling and alleviates optimization ambiguity with generative priors. During forward rendering, the physical model provides controllable rendering from the scene configuration, and the generative model denoises and fixes artifacts. Therefore, our method produces high-quality videos while allowing precise control, outperforming baselines in real and synthetic scenes. Moreover, BRDFusion supports novel-view relighting, night simulation, and dynamic object insertion/editing. Project page: https://shigon255.github.io/brdfusion-page/

• The paper introduces BRDFusion, a hybrid framework that integrates physical rendering with generative diffusion to recover explicit scene geometry, material properties, and HDR lighting.
• The paper employs a multi-stage optimization pipeline combining volume rendering, physically-based rendering, and generative refinement to mitigate artifacts and noise.
• The paper demonstrates state-of-the-art performance in urban scene inverse rendering, improving intrinsic decomposition, novel view synthesis, and relighting on real and synthetic datasets.

BRDFusion: Unified Physical and Generative Inverse Rendering in Urban Scenes

Motivation and Context

Inverse rendering in dynamic, large-scale urban environments demands explicit decomposition of scene geometry, material properties, and illumination from sparse, casually captured video sequences. Conventional physically-based approaches yield physically accurate results but are susceptible to optimization ambiguities, leading to geometrical and rendering artifacts; their robustness degrades under sparse views typical in urban settings. Generative diffusion models, trained on large-scale datasets, produce photorealistic outputs, but lack explicit 3D structure and precise lighting control, hence struggle with consistent relighting and local illumination. Addressing this, BRDFusion introduces a hybrid framework, effectively integrating physical and generative models to recover explicit, consistent scene properties and enable controllable, high-quality urban scene manipulation.

Figure 1: High-fidelity inverse rendering for outdoor urban scenes; multi-view input video is decomposed into geometry, material, and HDR environment lighting for downstream tasks including relighting and object insertion.

Methodological Framework

BRDFusion models urban scenes using a scene graph of 3D Gaussians encoding geometry, albedo, roughness, metallic, and surface normals, coupled with an HDR environment map for lighting. Movable scene entities (vehicles, pedestrians) are managed via local coordinate transformations. The hybrid forward rendering process consists of:

1. Volume Rendering: Efficient rasterization of per-pixel material/geometry attributes via depth-ordered Gaussian splatting and compositing; establishes alpha, depth, normals, albedo, roughness, and metallic maps but omits global light transport.
2. Physically-Based Rendering (PBR): Simulates light transport by evaluating the rendering equation with Monte Carlo integration and importance sampling. Surface interaction is modeled using a Cook-Torrance microfacet BRDF, enabling accurate computation of emission, shading, visibility, and reflection; tone-mapping converts HDR radiance to LDR for downstream uses.
3. Generative Diffusion Denoising: Residual artifact and Monte Carlo noise from PBR are suppressed using a pretrained video diffusion model (DiffusionRenderer) via SDEdit-style partially-noised trajectory starting from PBR outputs. Conditioning on G-buffers and target environment maps preserves structural and illumination fidelity, producing temporally consistent, high-fidelity RGB renderings.

   Figure 2: Hybrid forward rendering: 3D Gaussian scene and HDR lighting rendered to G-buffers, PBR for shading, generative denoiser for final output.

Multi-Stage Optimization for Inverse Rendering

Scene reconstruction is tackled in staged optimization:

• Volume Rendering Stage: Optimization of Gaussian attributes against photometric loss and opacity supervision (sky mask), regularized by generative G-buffer priors to mitigate ambiguity. Intrinsic maps—normal, depth, albedo, roughness, metallic—are regularized via sliding-window averaged diffusion priors, compensating for short sequence length and maintaining temporal consistency.
• Generative Refinement Stage: SDEdit operates on reconstructed intrinsic maps at training time. Structure anchors from volume renderings are denoised to sharpen and harmonize the priors, ensuring consistency across views and time, before supervision is updated in successive volume rendering passes.
• Physically-Based Lighting Estimation: With geometry/material fixed, PBR is used to optimize HDR environment lighting by matching rendered shading to input frames. Generative light priors from DiffusionLight guide optimization; log-space loss stabilizes the high dynamic range fitting.
• Joint Refinement: All parameters including geometry, material, and lighting are jointly optimized, balancing explicit physical regularization and learned generative priors.

  Figure 3: Multi-stage inverse rendering with alternation between volume render/priors, generative refinement, PBR lighting optimization, and joint finetuning.

Empirical Evaluation

Extensive evaluation on real (Waymo Open Dataset) and synthetic urban datasets demonstrates the superiority of BRDFusion over physically-based (UrbanIR, InvRGB+L) and generative baselines (Gen3C+DR). Quantitative metrics include si-PSNR for albedo, RMSE for roughness/metallic, MAE for normals, and perceptual metrics for view synthesis/relighting (LPIPS, PSNR, SSIM). Strongest performances are reported for roughness, metallic, and normal estimation, and novel view relighting (PSNR: 18.33, SSIM: 0.604, LPIPS: 0.448), establishing enhanced relighting fidelity and intrinsic decomposition.

Figure 4: Qualitative comparison of NVS, inverse rendering, and relighting on synthetic data; ground truth provides reference, highlighting improvements in material and lighting decomposition.

Ablation studies substantiate the necessity of each pipeline component. Bypassing generative priors leads to catastrophic decomposition failures and shadow baking; omitting PBR optimizations results in inconsistency and relighting artifacts; skipping generative rendering degrades final output quality. The full pipeline achieves both decomposition accuracy and rendering realism.

Figure 5: Ablation: Omission of physical optimization, generative prior, or generative rendering each induces characteristic artifacts or misestimation.

Applications and Failure Modes

BRDFusion supports downstream urban scene applications: consistent relighting under diverse conditions (sunset, night, localized headlamps/streetlights), virtual object insertion with physically plausible secondary effects, and simulation for autonomous driving. Its explicit decomposition enables physically consistent scene editing and synthetic data generation.

Figure 6: BRDFusion enables controllable relighting and object insertion, simulating urban driving scenarios across various lighting conditions.

Despite generative refinement reducing temporal inconsistencies, the pipeline remains constrained in modeling explicit emissive materials (active light sources in night driving), and floaters can cast erroneous shadows in unobserved regions—limitations inherited from sparse-view scene reconstruction.

Figure 7: Failure—floaters in unobserved regions produce unexpected shadows during relighting.

Technical Contribution and Implications

BRDFusion advances urban inverse rendering by coupling explicit, physically-based modeling (geometry, material, lighting) and robust generative diffusion priors, applied symmetrically in both forward and inverse rendering. This hybrid framework resolves optimization ambiguities without sacrificing controllability, allowing for high-quality photorealism and precise scene manipulation in dynamic, real-world environments.

Practically, BRDFusion enables credible simulation frameworks for autonomous driving and urban AR/VR, supporting synthetic data creation for vision tasks requiring accurate, editable environmental context. Theoretically, this demonstrates that hybrid strategies leveraging diffusion priors as denoisers and regularizers are critical for resolving underdetermined inverse problems in high-dimensional, real-world domains, provided explicit geometry and lighting parameterizations are maintained.

Conclusion

BRDFusion achieves state-of-the-art performance in controllable inverse rendering for dynamic urban scenes by balancing the strengths of physically-based and generative models. Its design—layered optimization with generative refinement and explicit 3D representation—enables robust scene decomposition and relighting, supporting practical simulation and content creation applications in urban vision. Remaining challenges lie in unobservable region ambiguity and explicit local illumination modeling. The framework sets the precedent for future scalable, physically-accurate, generative rendering platforms for physical AI and urban simulation tasks.

(2606.17049)