Extracting Neural Materials from Multi-view Images

Abstract: Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initialbase color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.

• The paper introduces NeuMatEx, a novel pipeline that extracts spatially-varying neural materials using a pre-trained LMRM and uncertainty-guided optimization.
• It employs a two-stage approach by combining feed-forward initialization with differentiable Monte Carlo path tracing to accurately model complex reflectance phenomena.
• NeuMatEx achieves superior material decomposition quality and real-time rendering capabilities, outperforming traditional physically-based rendering methods.

Extracting Neural Materials from Multi-view Images: An Expert Analysis

Introduction and Motivation

The paper "Extracting Neural Materials from Multi-view Images" (2606.26715) addresses the challenge of acquiring physically-plausible, expressive material representations from multi-view images of 3D objects. Contemporary real-time interactive applications predominantly utilize physically-based rendering (PBR) material models, which decompose appearance into spatially-varying color, roughness, and metallicity. However, these models are fundamentally limited in expressiveness, often failing to accurately capture complex real-world phenomena such as clearcoat, dust, haze, fuzz, and intricate multi-lobe specular effects. Neural material representations, which parameterize reflectance using neural networks and compact latent spaces, promise richer appearance modeling but are difficult to extract from image data due to the ill-posed and highly nonlinear nature of the inverse problem.

NeuMatEx Pipeline Overview

The proposed NeuMatEx framework introduces the first full pipeline for extracting spatially-varying neural materials from multi-view images, combining strong data-driven priors and test-time inverse rendering optimization. The central problem addressed is the fractured, highly nonlinear structure of neural material latent spaces, which impedes direct optimization via conventional inverse rendering. NeuMatEx overcomes this challenge through a staged approach:

1. Feed-forward Initialization via LMRM: A large, pre-trained Large Material Reconstruction Model (LMRM) predicts initial neural material parameters—including base color, neural material latents, and per-point uncertainty—directly from a set of 17+6 view images and geometry.
2. Uncertainty-aware Test-time Optimization: The predictions are subsequently refined with differentiable Monte Carlo path tracing, using the LMRM uncertainties to anchor confident regions and regularize ambiguous ones. This prevents the optimization from baking lighting and complex effects into material parameters, and addresses local minima in the highly expressive latent space.

The neural material representation employed is based on [58], combining a Lambertian diffuse lobe and a neural network-driven specular lobe with a rich 6D latent code, enabling representation of diverse complex reflectance behaviors.

Model Architecture and Training

The LMRM leverages a diffusion transformer architecture repurposed from text-to-video generation, comprising a VAE encoder-decoder and a single-step transformer denoiser. It processes a collection of strategically chosen multi-view images to output a triplane latent feature tensor for the geometry, which is decoded via lightweight MLPs into neural material parameters and uncertainty fields. The training follows a two-stage curriculum:

• PBR Pre-training: The LMRM is first trained to predict standard PBR materials from large-scale 3D datasets, learning basic image-to-material mappings.
• Neural Material Fine-tuning: The model is then fine-tuned on assets with procedurally-generated neural materials, exposing it to highly ambiguous, multi-lobe reflectance.

Loss functions include material regression and a heteroscedastic uncertainty term, following a 3-NLL formulation to calibrate per-pixel variance estimates.

Differentiable Inverse Rendering and Uncertainty-guided Optimization

During test-time, NeuMatEx performs differentiable path tracing using a Monte Carlo estimator for the rendering equation. The only parameters optimized are the triplane latents, with all neural material decoders frozen. The key innovation is an uncertainty-guided regularization term: for confident regions (low uncertainty), optimization is tightly anchored to the LMRM’s initialization, while ambiguous regions are allowed to adapt more freely to minimize photometric error. This approach effectively prevents the typical drift and overfitting that arises when optimizing in a high-dimensional neural latent space, decoupling lighting from material and preserving plausible decompositions.

Numerical and Qualitative Results

Experimental results demonstrate a substantial improvement in material decomposition quality and rendered appearance over both feed-forward and optimization-based PBR methods. Key quantitative outcomes include:

Method	PathTrace PSNR (↑)	BaseColor PSNR (↑)
Hunyuan3D-2.1 (PBR, single)	24.42 ± 3.81	23.01 ± 4.62
TRELLIS.2 (PBR, single)	23.55 ± 2.90	23.95 ± 4.08
NVDiffRecMC++ (PBR, multi)	26.25 ± 2.42	24.89 ± 5.71
NeuMatEx (neural, multi)	34.78 ± 2.22	25.30 ± 3.93

NeuMatEx consistently produces higher PSNRs, especially in path-traced renderings, reflecting its ability to model multi-lobe specular phenomena accurately. Visual comparisons reveal that PBR-based methods frequently "bake" specular features into the base color, while NeuMatEx separately and cleanly decomposes diffuse and complex specular components.

Significant ablation studies show that both strong initialization via LMRM and uncertainty-guided regularization are critical for neural material extraction. Removing either degrades decomposition quality and introduces noticeable artifacts or local minima during optimization.

Performance measurements confirm that the neural materials produced by NeuMatEx are suitable for real-time path tracing (∼4 ms per frame at 1080p, RTX 5090, 10 bounces).

Application to Real-world Data and Generalization

NeuMatEx generalizes to high-quality multi-view photographs of real objects (Digital Twin Catalog [9]), extracting materials that preserve effects like clearcoat and layered specularity under varying illumination. The resulting neural materials are relightable, enabling rendering under novel environment maps without baked-in lighting artifacts. Some residual artifacts can appear for real-world materials exceeding the expressiveness of the neural basis, but test-time optimization typically mitigates them.

Theoretical and Practical Implications

The primary theoretical contribution is a demonstration that ill-posed inverse problems in neural material spaces can be robustly addressed by fusing strong data-driven priors and uncertainty-aware optimization. This establishes a new paradigm for reflectance decomposition beyond the PBR regime, highlighting the importance of uncertainty quantification in high-dimensional inverse rendering.

Practically, NeuMatEx enables direct acquisition of complex, real-time-renderable neural materials from standard multi-view imagery, lowering barriers for content creation in VFX, games, and AR. The method’s two-stage architecture is extensible: improvements in neural material bases or feed-forward reconstruction models will directly propagate to extraction quality.

Limitations and Future Directions

The approach inherits limitations from the underlying neural material basis [58]. Failures can arise when real-world observations fall outside this learned manifold, manifesting as color shifts or specular artifacts. While the uncertainty-guided optimization reduces these problems, complete mitigation requires further advances in neural material modeling and dataset diversity. Future work could address:

• Higher-resolution neural material triplane encoding for finer detail.
• End-to-end learning from real-world capture datasets for better domain coverage.
• Integrating more expressive or physically-motivated neural material bases.

Conclusion

NeuMatEx sets a new standard for extracting rich, spatially-varying neural materials from multi-view images. By uniting a large, pre-trained reconstruction model with uncertainty-regulated path-tracing optimization, it achieves far greater expressiveness and accuracy than traditional PBR extraction pipelines, with robust real-time rendering capabilities. The framework paves the way for scalable, photorealistic acquisition of complex material behavior, providing both a valuable asset for practical graphics pipelines and a foundational methodology for further research in neural inverse rendering.