BRDFusion: Unified Inverse Rendering

• BRDFusion is a framework that fuses explicit physical modeling with learned generative priors to recover geometry, material parameters, and HDR lighting.
• It employs a multi-stage pipeline combining 3D Gaussian splatting, volumetric rendering, and diffusion-based refinement to enhance rendering accuracy.
• The approach mitigates typical artifacts of pure physical or generative methods by integrating feedback loops and optimization for improved consistency and controllability.

Searching arXiv for BRDFusion and closely related BRDF inverse rendering work to ground the article in current papers. BRDFusion denotes a family of inverse-rendering formulations in which BRDF-related recovery is posed as the fusion of partial observations, geometric structure, physical image formation, and learned priors. In the most specific current usage, it refers to a unified framework for urban-scene inverse and forward rendering that combines explicit physically based scene decomposition with video diffusion, recovering geometry, material parameters, and HDR lighting from posed driving video while preserving controllability at render time (Liu et al., 15 Jun 2026). In a broader methodological sense, the term also captures earlier patterns in which observed radiance, canonical parameterizations, and generative or statistical BRDF priors are jointly constrained to reconstruct relightable appearance fields, as in single-image facial reflectance recovery in UV space (Papantoniou et al., 2023).

1. Definition and conceptual scope

BRDFusion is fundamentally a response to a recurrent split in appearance modeling. Physically based inverse-rendering methods follow and control lighting physics, but suffer from reconstruction and rendering artifacts; generative models produce realistic videos, but offer limited consistency and controllability (Liu et al., 15 Jun 2026). BRDFusion addresses that split by coupling an explicit scene representation, a physically grounded forward model, and a learned prior or denoiser.

In the urban-scene framework bearing the name, the explicit component is a relightable scene described by 3D Gaussians, Cook–Torrance BRDF parameters, and an HDR environment map; the generative component is a latent video diffusion model used both as a prior during inverse rendering and as a refiner during forward rendering (Liu et al., 15 Jun 2026). The same conceptual pattern appears in earlier work where BRDF estimation is treated as constrained generation rather than direct regression. Relightify, for example, formulates single-image facial BRDF recovery as UV-space inpainting under a diffusion prior learned over joint texture and reflectance channels, effectively fusing visible texture, geometry from a 3D morphable model, and learned appearance statistics (Papantoniou et al., 2023).

A common misconception is that BRDFusion simply means estimating a BRDF map. In the literature, it instead denotes a structured reconciliation of multiple sources of constraint: observed image evidence, canonical surface parameterization, visibility, lighting assumptions, and learned priors. Another misconception is that the generative component replaces physical modeling; in the named urban framework it does not. The physical model supplies controllable rendering from scene configuration, and the generative model denoises and fixes artifacts (Liu et al., 15 Jun 2026).

2. Physical scene model in urban BRDFusion

The urban BRDFusion framework takes as input a sequence of posed monocular video frames, optionally with sparse LiDAR, and seeks a relightable 3D scene supporting novel-view synthesis, global relighting, localized lighting, and dynamic object insertion or editing (Liu et al., 15 Jun 2026). The unknown scene variables are a set of 3D Gaussians and a global HDR environment map.

Each Gaussian stores opacity, 3D mean position, rotation quaternion, scale, surface normal, view-dependent color, albedo, roughness, and metallic. Dynamic content is organized through a scene graph with a static background node and dynamic-object nodes, each with Gaussians in local coordinates and time-dependent rigid transforms into world space (Liu et al., 15 Jun 2026). This is not merely a geometry proxy: it is the carrier for material attributes required by physically based shading.

The first rendering stage is volume rendering by 3D Gaussian splatting. Gaussians are transformed, projected, depth-sorted, and composited front-to-back to produce not only RGB but also G-buffers for opacity, depth, normals, albedo, roughness, and metallic. These buffers are then consumed by a physically based rendering stage that approximates the rendering equation,

$$L_o(x,\omega_o)=L_e(x,\omega_o)+\int_{\Omega}L_i(x,\omega_i)f(x,\omega_i,\omega_o)(\omega_i\cdot N)\,d\omega_i,$$

with incident illumination coming from the HDR environment map modulated by visibility computed through 3D Gaussian ray tracing (Liu et al., 15 Jun 2026).

The BRDF is Cook–Torrance with GGX normal distribution, Smith geometry using Schlick’s approximation, and Schlick Fresnel. In the notation of the framework,

$$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$

where $A$, $R$, $M$, and $N$ are per-pixel albedo, roughness, metallic, and normal from the G-buffers, and $F_0=(1-M)\,0.04+M\,A$ (Liu et al., 15 Jun 2026). Physically based relighting is therefore explicit rather than implicit: new environment maps and local lights are propagated through a scene representation whose geometry and materials remain individually addressable.

3. Generative priors and bidirectional coupling

The generative component is DiffusionRenderer, a latent video diffusion model with VAE encoder and decoder plus a latent denoiser. It is conditioned on sequences of normals, depth, albedo, roughness, metallic, and the target environment map (Liu et al., 15 Jun 2026). Crucially, BRDFusion does not generate from pure noise. It uses an SDEdit-style procedure that starts from a noised latent encoding of a physically rendered video, preserving physical anchoring while permitting learned correction of noise and artifacts.

During forward rendering, the procedure is straightforward in principle. The physical pipeline renders a controllable but imperfect video under the desired view and lighting. That rendering is encoded, perturbed at diffusion time $t$, and denoised under conditioning from the G-buffers and environment map. The decoded result is a temporally coherent video that remains tied to the rendered scene configuration (Liu et al., 15 Jun 2026). In this role the diffusion model is a denoiser and artifact corrector, not a replacement renderer.

During inverse rendering, the same model plays two different roles. First, its inverse side predicts normal, depth, albedo, roughness, and metallic priors from RGB clips. Because driving sequences exceed the diffusion model’s temporal window, the framework processes overlapping clips and applies sliding-window averaging to reduce temporal flicker (Liu et al., 15 Jun 2026). Second, those priors are themselves refined using a generative step anchored by current 3DGS renderings of the intrinsic maps. The result is a feedback loop in which generative priors regularize the physical optimization, while the emerging 3D reconstruction stabilizes the generative estimates.

This bidirectional coupling is the distinctive technical meaning of fusion in the 2026 framework. The physical model alleviates the consistency and controllability deficiencies of generative video synthesis; the diffusion prior alleviates the ambiguity and artifact sensitivity of large-scale outdoor inverse rendering (Liu et al., 15 Jun 2026).

4. Multi-stage optimization and objective design

BRDFusion is organized as a staged inference pipeline rather than a single monolithic optimization. After initialization from sparse LiDAR and scene-graph construction, the first major stage optimizes geometry and material under volume-rendering losses and generative priors. The loss is

$$\mathcal{L}_{\text{vol}}= \mathcal{L}_{\text{rgb}}+ \lambda_O\mathcal{L}_O+ \lambda_D\mathcal{L}_D+ \lambda_N\mathcal{L}_N+ \lambda_A\mathcal{L}_A+ \lambda_R\mathcal{L}_R+ \lambda_M\mathcal{L}_M,$$

where $\mathcal{L}_{\text{rgb}}$ is an $$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$0 photometric term, $$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$1 is a binary cross-entropy opacity loss against a non-sky mask from SegFormer, $$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$2 matches depth to LiDAR and a scale-shift-aligned generative depth prior, $$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$3 combines $$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$4 and cosine similarity for normals, and $$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$5, $$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$6, and $$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$7 impose albedo, roughness, and metallic priors (Liu et al., 15 Jun 2026).

After a first round of geometry-material optimization, the framework refines the generative intrinsic priors via the SDEdit-style intrinsic-map pass described above, then repeats the volume-rendering optimization with the improved priors. Lighting is then solved in a separate physically based stage by freezing geometry and material and optimizing the HDR environment map under

$$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$8

with

$$f(x,\omega_i,\omega_o)=(1-M)\frac{A}{\pi}+\frac{D(H)\,G(\omega_i,\omega_o)\,F(\omega_o,H)}{4(N\cdot\omega_i)(N\cdot\omega_o)},$$9

where $A$0 is an HDR lighting prior from DiffusionLight or DiffusionLight-Turbo (Liu et al., 15 Jun 2026). The log-domain term stabilizes optimization for HDR illumination.

A final joint refinement stage optimizes all parameters with

$A$1

This decomposition of the problem is central. It prevents the lighting, material, and geometry ambiguities that arise when all terms are optimized from weak initialization under a single rendering loss. The ablations make this role explicit: removing PBR optimization weakens relighting, removing generative optimization causes severe entanglement of geometry, material, and lighting, and removing generative rendering leaves Monte Carlo noise and reconstruction artifacts visible in the final outputs (Liu et al., 15 Jun 2026).

5. Capabilities, evaluation, and limitations

The explicit scene decomposition gives BRDFusion a broad application range. It supports novel-view synthesis from arbitrary camera trajectories; global relighting through replacement of the HDR environment map; localized lighting such as car headlights and streetlights through added point or spot light terms; and dynamic object insertion or editing through the 3D Gaussian scene graph (Liu et al., 15 Jun 2026). Because these operations are evaluated through Cook–Torrance shading and visibility, cast shadows and specular responses remain tied to scene geometry.

Evaluation is reported on Waymo Open Dataset front-camera sequences and on six synthetic urban scenes rendered in Blender Cycles with ground-truth PBR material maps and multiple environment maps (Liu et al., 15 Jun 2026). Baselines include UrbanIR, InvRGB+L, and a constructed generative baseline combining Gen3C with DiffusionRenderer. Metrics include PSNR, SSIM, and LPIPS for view synthesis and relighting, plus scale-invariant PSNR for albedo, RMSE for roughness and metallic, and mean angular error for normals (Liu et al., 15 Jun 2026).

The reported pattern is specific. On synthetic data, BRDFusion attains the best roughness RMSE, metallic RMSE, and normal MAE; its albedo scale-invariant PSNR is slightly below the best generative baseline but above UrbanIR and close to that baseline; its novel-view synthesis is comparable to the strongest generative method; and its novel-view relighting is best across the board in PSNR and SSIM, with LPIPS close to or better than baselines (Liu et al., 15 Jun 2026). The interpretation given by the framework is that generative methods may overfit appearance under the training illumination, but explicit light transport is required to generalize to new lighting.

The framework also has explicit limitations. It does not explicitly model emissive materials, making true nighttime inverse rendering difficult; it inherits the floater problem common to 3DGS- and NeRF-like methods in unobserved regions, which can cause unrealistic shadows under new lighting; and it remains reliant on generative priors, so catastrophic prior failures can corrupt decomposition (Liu et al., 15 Jun 2026). These limitations are structurally important: they show that BRDFusion is not a complete replacement for stronger physical observation or fully emissive scene modeling, but a hybrid regime whose success depends on maintaining tension between explicit control and learned regularization.

6. Broader BRDFFusion pattern in related literature

Outside the specific 2026 urban framework, BRDFFusion describes a broader research pattern in which BRDF estimation is solved by combining canonical parameterization, physical rendering assumptions, and learned priors. Relightify is an early example in this sense: it uses a 3D morphable model and UV-space inpainting under a latent diffusion prior over a 10-channel tensor $A$2 to recover diffuse albedo, specular albedo, and normals from a single face image, outperforming AvatarMe++ on diffuse and specular albedo PSNR while preserving identity by directly copying visible texture from the input (Papantoniou et al., 2023). The fusion there is among visible image evidence, UV geometry, and a learned joint distribution $A$3.

Other works instantiate the same pattern with different parameterizations and modalities. “A Lightweight Approach for On-the-Fly Reflectance Estimation” introduces HemiCNN and Grouplet for real-time reflectance estimation from 8-bit RGBD sequences under unknown illumination, with less than 90 ms per scene, a model size of less than 340K bytes, and the SynBRDF dataset of 500K RGBD images over 5000 materials and 5000 shapes (Kim et al., 2017). “Two-shot Spatially-varying BRDF and Shape Estimation” uses stage-wise estimation of shape and SVBRDF from unaligned flash and no-flash mobile images, explicitly fusing shape, illumination, and Cook–Torrance SVBRDF under a practical two-shot capture model (Boss et al., 2020). “One Ring to Rule Them All” jointly estimates shape and spatially varying BRDF from multi-view images with unknown generic materials using a recurrent ResNet Shape-Net and a BRDFNet, each with about 1,000 neurons, tied together by differentiable rendering and a diffeomorphic spherical parameterization (Cheng et al., 2021).

A separate branch concerns compact neural BRDF representations rather than scene-scale inverse rendering. “Neural BRDF Representation and Importance Sampling” encodes measured BRDFs as small MLPs of about 675 parameters, uses adaptive angular sampling for specular highlights, and maps a 32-D latent embedding to analytic sampling parameters for practical rendering (Sztrajman et al., 2021). “Real-Time Neural BRDF with Spherically Distributed Primitives” factorizes BRDFs into two hemisphere feature-grids, stores learnable reflectance primitives in a shared codebook, and reports strong measured BRDF compression, Monte Carlo BRDF acceleration, and extension to spatially varying effects (Dou et al., 2023). “Hypernetworks for Generalizable BRDF Representation” introduces estimation of measured BRDFs from sparse samples using a hypernetwork and set encoder, evaluated on the MERL dataset of 100 isotropic materials and compressing BRDFs into very small embeddings such as 7D (Gokbudak et al., 2023).

The term also intersects with foundational questions about what should count as the BRDF being fused. “Fresnel Microfacet BRDF” argues that the customary linear combination of Lambertian diffuse and microfacet specular reflection is physically incompatible, and instead derives a unified model in which body and surface reflection, as well as radiometric and polarimetric behavior, arise from the same Fresnel-microfacet mechanism (Ichikawa et al., 2022). By contrast, “Image-based remapping of spatially-varying material appearance” addresses fusion across renderer-specific BRDF parameter spaces, using image-based optimization against black-box renderers and a parametric regression model so that remapping can be applied per texel in SVBRDF assets (Sztrajman et al., 2018).

Taken together, these works show that BRDFFusion is not a single architecture but a technical stance. It treats reflectance recovery as the joint estimation, transfer, or compression of appearance under constraints drawn from geometry, image formation, and learned priors. The 2026 urban framework is the most explicit realization of that stance as a named system, but the underlying idea spans facial relighting, mobile SVBRDF capture, real-time RGBD reflectance estimation, neural BRDF compression, sparse-sample generalization, and even physically unified polari-radiometric reflectance modeling (Liu et al., 15 Jun 2026).