Feed-Forward Multi-View Inverse Rendering

• Feed-forward multi-view inverse rendering is a neural approach that infers scene geometry, material, and lighting in a single forward pass from diverse viewpoints.
• It leverages advanced attention mechanisms and transformer-based modules to fuse multi-view features, enabling real-time, artifact-free editing and relighting.
• The paradigm supports practical applications like photorealistic object insertion and dynamic scene reconstruction, demonstrating improved quantitative metrics over traditional methods.

Feed-forward multi-view inverse rendering refers to the class of algorithms that, given a set of images of a scene captured from different viewpoints (multi-view), directly infer scene properties such as geometry, spatially-varying BRDFs, and illumination fields in a single forward pass through a neural network—without per-scene optimization or iterative procedures. This approach departs from traditional analysis-by-synthesis and iterative optimization methods by leveraging end-to-end trainable architectures, often using attention-based, convolutional, or transformer-based modules to fuse cross-view information. The paradigm has rapidly advanced to support challenging scene-level decompositions, volumetric and dynamic content, and photorealistic editing or insertion tasks, enabled by innovations in feature fusions, lighting parameterizations, and procedural dataset construction.

1. Rendering Equation and Scene Decomposition

Core to modern feed-forward multi-view inverse rendering systems is the explicit modeling of the image formation process via the rendering equation. The most comprehensive approaches (e.g., MAIR++ (Choi et al., 2024)) predict, for each pixel $x$ with world-space normal $n(x)$ and view direction $v(x)$, the sum of a diffuse term $I_d$ and a specular term $I_s$:

$I(x) = I_d(x) + I_s(x)$

$$I_d(x) = \frac{A(x)}{\pi} \int_{\Omega} E(x,\ell) \max\{0, n(x) \cdot \ell\} d\omega_\ell$$

$$I_s(x) = \int_{\Omega} E(x,\ell) f_s(\ell, v(x), n(x), R(x)) \max\{0, n(x) \cdot \ell\} d\omega_\ell$$

where $A(x)$ is the per-pixel diffuse albedo, $R(x)$ the roughness, $n(x)$0 is the estimated incident radiance (implicit lighting), and $n(x)$1 is the Cook–Torrance microfacet BRDF incorporating normal-distribution, Fresnel, and geometric terms.

Feed-forward systems factorize observable radiance into physically motivated, spatially-varying quantities:

• Geometry (depth, normals)
• SVBRDF maps (albedo, roughness, metallicity)
• View-consistent or per-pixel illumination fields

This complete decomposition is vital for applications such as relighting, material editing, and photorealistic object insertion.

2. Parametric Representations and Feature Aggregation

Feed-forward multi-view inverse rendering frameworks adopt richly structured parametric representations for geometry, material, and lighting:

• Geometry: Multi-view stereo (MVS) depth maps with pixel-wise confidence, refined to single-view depth and normals via tailored CNNs or U-Nets (Choi et al., 2024).
• Materials: SVBRDF parameter maps (diffuse albedo $n(x)$2, roughness $n(x)$3, sometimes metallicity or specular), regressed at full image resolution.
• Lighting: Either explicit, low-dimensional representations (e.g., spherical Gaussians, spherical harmonics), direct per-pixel or volumetric (3D) light fields (Choi et al., 2023), or, for leading methods, implicit latent codes decoded locally by MLPs (Choi et al., 2024).

The multi-view fusion problem—combining features extracted from each input image while preserving spatial alignment and occlusion reasoning—is addressed via specialized aggregation modules. Notable designs include:

• Mean-Variance/Directional Attention: Aggregates weighted per-view features, with weights modulated by depth reprojection errors and attention masking (Choi et al., 2024).
• Alternating Self-Attention Transformers: Alternates frame-wise intra-view attention with global cross-view attention to joint model appearance and consistency, as in MVInverse (Wu et al., 24 Dec 2025).
• Bidirectional Motion Encoders: For dynamic scene reconstruction, integrates context across view and time dimensions, supporting temporal fusion (Balice et al., 21 May 2026).

These schemes enable the network to resolve cross-view correspondences, suppress inconsistent features, and robustly learn from weakly supervised or synthetic data.

3. Implicit and Volumetric Lighting Representations

Realistic inverse rendering of complex scenes demands lighting models that capture non-local illumination, interreflections, and spatial variation. Early feed-forward systems relied on explicit parametric models such as spherical Gaussians (Choi et al., 2023). Recent advances include:

• Implicit Lighting Representations (ILR): Each pixel (or volume element) is assigned a high-dimensional latent vector, decoded on-the-fly via MLPs (with Fourier positional encoding) into dense environment maps or shading scalars; this supports accurate, view-dependent, and easily editable lighting and has proven crucial for artifact-free material editing and object insertion (Choi et al., 2024).
• 3D Spatially-Varying Lighting Volumes: Lighting is encoded as multi-channel volumes (with parameters for opacity, lobe axis, sharpness, intensity), composited along viewing rays via spherical Gaussian interpolation. This enables per-pixel HDR environment map synthesis and direct support for view-dependent effects (Choi et al., 2023, Choi et al., 2024).
• Low-Rank Tensorial Volumes: For highly scattering and heterogeneous media, e.g., in TensoIS (Tiwari et al., 4 Sep 2025), scattering parameters ($n(x)$4, $n(x)$5) are represented using low-rank tensor decompositions and predicted from as few as six multi-view images.

This evolution from parametric to implicit/volumetric lighting—facilitated by self-supervised learning on synthetic datasets—marks a key step in achieving practical, high-fidelity inverse rendering.

4. Feed-Forward Architectures and Training Paradigms

Feed-forward multi-view inverse rendering architectures typically comprise several coordinated modules:

• Per-view Backbones: DenseNet, ResNeXt, or ViT encoders process individual images, optionally sharing weights (Choi et al., 2024, Wu et al., 24 Dec 2025).
• Multi-view Aggregation: Attention blocks, mean-variance modules, or bidirectional transformers integrate per-view features (Choi et al., 2024, Wu et al., 24 Dec 2025, Balice et al., 21 May 2026).
• Decoders: U-Net variants and specialized MLP heads yield full-resolution SVBRDF and lighting maps.
• Additional Modules: Lighting encoders/decoders (for ILR), 3D volumetric lighting networks using 3D U-Nets, or tensor decomposition heads (for scattering) (Tiwari et al., 4 Sep 2025).

Training is supervised on large-scale synthetic datasets with ground-truth geometry, BRDF, and sometimes per-pixel HDR lighting (e.g., OpenRooms FF (Choi et al., 2024, Choi et al., 2023), HeteroSynth (Tiwari et al., 4 Sep 2025)). Targeted losses include:

• Pixel-wise MSE and scale-invariant (log) MSE on outputs
• Cosine or angular error for normals
• Regularization terms (entropy, smoothness, KL)
• Temporal/consistency losses for real video adaptation (by warping predictions and enforcing invariance) (Wu et al., 24 Dec 2025)

No adversarial or perceptual losses are needed. Multi-stage training is sometimes employed: synthetic pretraining for accurate physical decomposition, followed by self-supervised temporal consistency finetuning on unlabeled video (Wu et al., 24 Dec 2025).

5. Quantitative Results and Qualitative Performance

State-of-the-art feed-forward multi-view inverse rendering systems have established strong baselines across several metrics and tasks:

Method	Normal MSE (×10⁻²)	Albedo MSE (×10⁻²)	Rough. MSE (×10⁻²)	HDR Re-render (MSE)	Real Re-render (MSE)
CIS 2020 (SV)	3.78	0.87	6.50	0.49	≈1.15
Zhu et al. 22	1.74	0.53	4.21	3.57(*)	≈3.6(*)
MAIR	1.01	0.64	3.46	0.78	-
MAIR++	1.01	0.48	2.50	0.058	0.32

(*) Monte Carlo renderer; real-data error ≈3.6 (Choi et al., 2024).

Significance and qualitative effects:

• MAIR++ and similar models yield artifact-free, temporally stable material edits, with robust generalization to real scenes—attributable to their use of ILR, 3D lighting, and attention-based feature aggregation.
• Material and lighting consistency across views is substantially better than in single-view models, enabling reliable object insertion, relighting, and editing.
• Feed-forward inference is orders of magnitude faster than per-scene optimization (under 100 ms/view for MAIR++).
• For scattering and volumetric effects, TensoIS achieves $n(x)$6 and MSE $n(x)$7 on unseen shapes and Perlin fields, showing strong generalization with minimal multi-view data (Tiwari et al., 4 Sep 2025).

6. Extensions: Scattering, Dynamics, and Unposed Setups

Recent expansion of the paradigm includes:

• Tensorial Inverse Scattering: TensoIS (Tiwari et al., 4 Sep 2025) addresses heterogeneous volumetric media, learning to reconstruct spatially multiscale scattering coefficients in 3D from minimal multi-view observations. This leverages fractal Perlin noise models and low-rank tensor decompositions for representation and enables fast inference for subsurface phenomena.
• Dynamic Scene Modeling: NoPo4D (Balice et al., 21 May 2026) extends feed-forward multi-view reconstruction to the 4D spatiotemporal domain, integrating motion estimation (via image-plane and depth shifts), bidirectional temporal aggregation, and view-dependent opacity. This supports real-time modeling of dynamic content without pose estimation, outperforming both feed-forward and per-scene optimization methods on dynamic benchmarks, with PSNR up to 29.15 dB.
• Physics-based Indirect Illumination: Approaches such as (Deng et al., 2022) model indirect lighting via neural expansions of the rendering equation, including explicit handling of boundary illuminants and application of the Leibniz rule for differentiability.

These advances broaden the applicability of feed-forward multi-view inverse rendering to cover material heterogeneity, dynamics, general pose, and intricate indirect effects.

7. Challenges, Limitations, and Future Directions

Despite significant milestones, notable challenges remain:

• Domain shift and generalization: Synthetic-to-real transfer is limited by dataset realism. Consistency-based finetuning on real video improves robustness but does not fully close the gap (Wu et al., 24 Dec 2025).
• Extreme occlusions and coverage: Sparse view coverage or dynamic occlusions may degrade feature fusion and volumetric modeling, leading to artifacts.
• Representation accuracy: Linear velocity models (NoPo4D) cannot capture strong accelerations or non-linear dynamics; tensor decompositions may miss fine-scale heterogeneity (Balice et al., 21 May 2026, Tiwari et al., 4 Sep 2025).
• Surface/boundary effects: Systems lacking explicit BSSRDF for surfaces (TensoIS) may have limited fidelity at boundaries or in mixed surface/subsurface scenarios (Tiwari et al., 4 Sep 2025).
• Lighting parameterizations: Existing ILRs and volumetric lighting fields are highly expressive but may be hard to supervise for indirect multiple-bounce effects without exhaustive datasets.

A plausible implication is that continued advances will depend on (a) better bridging of the real-to-synthetic gap, (b) richer neural representations for challenging phenomena (e.g., wavelength-dependent scattering, non-linear motion), and (c) integration of geometry, material, and lighting estimation in unified, scalable frameworks. The feed-forward paradigm continues to support real-time, scene-level relighting and editing in vision, graphics, and AR workflows.