Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

Abstract: Reconstructing physics-based 3D assets -- geometry, materials, and illumination -- from multi-view images is a core problem in computer graphics and vision, and a prerequisite for realistic relighting and editing. Physics-based inverse rendering offers an accurate image-formation model, but is severely underconstrained: without strong priors, illumination is baked into materials, and reconstructions generalize poorly to novel views and lighting. Data-driven diffusion models, in contrast, predict visually plausible materials, yet their predictions rarely satisfy the rendering equation and are not directly usable for physics-based rendering. We bridge these two paradigms rather than replacing either. Our key idea is to treat the predictions of a state-of-the-art diffusion model not as target material values but as a similarity kernel for optimization: we introduce a regularization loss that penalizes deviations in the optimized material over surface regions where the diffusion predictions are near-constant, while leaving the optimization free to match the input images. Built on this regularizer, our end-to-end pipeline jointly reconstructs geometry, materials, and illumination, yielding high-quality assets that drop into standard rendering pipelines and relight faithfully. On the Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets, our method significantly outperforms state-of-the-art baselines in both reconstruction accuracy and relighting quality.

• The paper introduces a novel implicit clustering regularizer that leverages diffusion-based predictions as soft priors for physics-based inverse rendering.
• It combines neural SDF reconstruction, joint bilateral filtering, and differentiable rendering to optimize geometry and spatially-varying material parameters.
• Experimental results demonstrate state-of-the-art relighting accuracy and robust material recovery across multiple benchmark datasets.

Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

Problem Statement and Motivation

Reconstructing physics-consistent 3D assets—spanning geometry, spatially-varying materials, and illumination—from multi-view images remains a critical and unsolved challenge in computer vision and graphics. Physics-based inverse rendering, which optimizes parameterized scene representations to match observed images under physically-correct light transport, is fundamentally ill-posed. Material–lighting ambiguities, sparse observations, and the lack of strong priors drive the optimization into degenerate local minima that absorb image phenomena like cast shadows into material parameters, yielding poor generalization and non-relightable assets (“baking artifacts”).

Conversely, diffusion models and other large-scale data-driven predictors generate perceptually plausible intrinsic material maps from images, but their outputs are not physically meaningful—these maps rarely satisfy the rendering equation, and thus, relighting or integrating them into analysis-by-synthesis pipelines results in artifacts. This paper proposes a hybrid approach, using diffusion model predictions not as direct supervision, but as a structural prior: leveraging the spatial consistency of such predictions as an implicit similarity kernel in the inverse rendering optimization.

Methodology

The method operates in three stages:

1. G-buffer Prediction: For each calibrated view, diffusion-based models such as DiffusionRenderer or RGB<>X predict per-view intrinsic G-buffers (albedo, roughness, metallicity, normal).
2. Neural Shape Reconstruction: A neural SDF volume (supervised with normal prediction losses) is reconstructed using the G-buffers.
3. Physics-Based Inverse Rendering (PBIR): A differentiable renderer jointly optimizes shape, spatially-varying materials parametrized with the Disney BRDF, and an environment lighting map, minimizing a photometric loss and a novel diffusion-based material regularization loss.

Implicit Material Clustering Regularizer

Rather than enforcing rendered materials to directly match diffusion-based predictions (which leads to suboptimal consistency with physical rendering), the key contribution is an implicit clustering regularizer:

• The G-buffer predictions define a soft material similarity kernel over surface points, $k_{p,q} = \exp(-\|g_p - g_q\|^2/2\sigma^2)$, where $g_p$ concatenates [albedo, roughness, metallicity] at pixel $p$.
• For each pixel, a differentiable joint bilateral filter (JBF) aggregates rendered material parameters using this kernel, producing a per-pixel regularized map.
• The material loss penalizes the $L_1$ distance between original and kernel-filtered rendered G-buffers: $L_\mathrm{mat} = \|g - \mathrm{JBF}(g; g)\|_1$.

Albedo regularization employs a scale-agnostic log transformation to prevent ill-posed compensations between reflectance and lighting, addressing the classic inverse problem scale ambiguity.

Experimental Results

Experiments are conducted on Stanford-ORB (real-world), Synthetic4Relight, and DTC-Synthetic (both synthetic) datasets. The following baselines are compared:

• Neural-PBIR [39]: pure analysis-by-synthesis PBIR with differentiable rendering and no data-driven priors.
• MaterialFusion [27]: diffusion-guided inverse rendering with score distillation priors.

Quantitative metrics (PSNR, SSIM, LPIPS) on novel-view and novel-light relighting, aligned albedo, and roughness estimation systematically demonstrate the superiority of the proposed method:

Dataset	PSNR (Relighting)	SSIM	LPIPS ↓	Albedo PSNR	Roughness PSNR
Stanford-ORB (Ours)	27.22–34.98	0.981	0.021	—	—
Synthetic4Relight (Ours)	32.02	0.975	0.049	27.04	27.15
DTC-Synthetic (Ours)	43.21	0.9961	0.0063	—	—

Qualitative assessments show robust suppression of baked-in shadows, improved metallic and roughness recovery, and more accurate relighting, especially under strong directional lighting and for glossy/metallic surfaces. Comparative examples highlight the failure of vanilla optimization (Neural-PBIR) to eliminate baked effects and the tendency of prior-guided methods (MaterialFusion) to over-smooth materials and miss local details.

Ablation studies validate the necessity of the implicit clustering regularizer, the scale-agnostic albedo transform, and the inclusion of normal supervision for geometry recovery. Alternative regularization strategies—including direct back-projection of diffusion predictions, global scale-invariant loss, and non-data-driven diffuse-specular correlation—underperform both quantitatively and qualitatively.

Critically, the implicit clustering regularizer is model-agnostic: comparable gains are achieved when upstream diffusion models are changed (e.g., RGB<>X vs. DiffusionRenderer).

Implications and Theoretical Significance

This work addresses a crucial gap in inverse rendering: existing data-driven and physics-based approaches exhibit orthogonal failure modes, and naïve attempts at merging them (direct supervision, per-pixel losses) misalign with the goal of producing assets that are both perceptually plausible and physically valid under novel conditions. By leveraging the local constancy of diffusion-predicted intrinsic buffers as a soft constraint, the proposed implicit clustering regularizer constrains solution space without directly enforcing incorrect or view-inconsistent targets, effectively reducing the ill-posedness of the inverse problem.

Empirically, the method establishes a new state-of-the-art on Stanford-ORB, a challenging real-world relighting benchmark. The proposed regularization is robust to prediction inconsistencies, insensitive to a specific diffusion model architecture, and can be readily adopted in other mesh- or point-based PBIR pipelines (e.g., IRGS, Gaussian Splatting methods), suggesting it is generally applicable across architectures. The formalization of kernel-based regularization guided by learned priors is likely to influence future work on leveraging foundation models in photometric and geometric inverse problems.

Future Directions

Potential research avenues include:

• Incorporation of higher-resolution or multi-scale diffusion models to better constrain high-frequency material details.
• Refinement of similarity-kernel construction, potentially using model-predicted uncertainties or attention mechanisms for adaptive regularization.
• Extension to temporally consistent video relighting and dynamic scene reconstruction.
• Integration with foundation models to derive priors for more complex material/illumination decompositions, including spatially-varying index of refraction and sub-surface scattering.

Conclusion

By employing diffusion-based intrinsic predictions as a soft, kernel-driven prior, this work bridges analysis-by-synthesis and data-driven paradigms for inverse material and illumination acquisition. The implicit material clustering regularizer penalizes intra-region material deviations while enabling photo-consistency-driven corrections, resulting in physically plausible, exportable, and relightable 3D assets. The approach is flexible, improves relighting fidelity across multiple benchmarks, and generalizes across both model architectures and data domains, offering a robust paradigm for future physics-based scene reconstruction pipelines.

Reference: "Diffusion-Based Material Regularization for Physics-Based Inverse Rendering" (2606.31065)