- The paper introduces a feed-forward inverse rendering approach that predicts a structured 3D Gaussian representation with intrinsic PBR material parameters for physically-based scene reconstruction.
- It employs a dual-branch network architecture, combining a ResNet-transformer pipeline for geometry with a DINOv2-based intrinsic branch, ensuring robust multi-view consistency.
- Experimental results show high fidelity in novel view synthesis, real-time relighting, and material editing across both synthetic and real-world datasets.
InvSplat: A Feed-Forward Architecture for Fast Physically-Based Inverse Scene Reconstruction
Introduction
Inverse rendering—recovering 3D geometry and physically meaningful material properties from images—is crucial for applications requiring scene relighting, novel view synthesis, and physically grounded editing. Traditional solutions demand either expensive per-scene optimization or operate in 2D image space, sacrificing multi-view consistency and explicit 3D scene structure. "InvSplat: Inverse Feed-Forward Scene Splatting" (2607.02301) proposes the first feed-forward inverse rendering architecture that predicts a structured 3D Gaussian representation with intrinsic material properties directly from multi-view posed images, achieving real-time, multi-view consistent scene decomposition.
Methodology
3D Gaussian Representation with PBR Intrinsics
InvSplat builds upon 3D Gaussian Splatting (3DGS), encoding each primitive as a Gaussian parameterized by position, orientation, scale, and opacity, and critically extends the representation to include physically-based rendering (PBR) material parameters: albedo, metallicity, and roughness, as well as explicit surface normals. This disentangled, physically grounded representation allows for stable rendering from novel viewpoints as well as physics-based relighting and material editing in downstream applications.
Feed-Forward Dual-Branch Architecture
The primary architectural innovation is a dual-branch network, consisting of:
- Geometry Branch: A stereo-inspired, multi-view pipeline incorporating a ResNet backbone and a transformer-based encoder. Features from multiple views are fused through cross-view self-attention and a cost volume mechanism to resolve depth, geometric context, and to construct explicit 3D primitives.
- Intrinsic Branch: Adopts a DINOv2 ViT-L/14-based encoder, followed by a deep transformer "translator" that fuses intrinsic cues (for albedo, metallic, roughness) across views.
- Decoding and Fusion: Cross-branch features are decoded by multiple heads to predict depth, normals, and material parameters per Gaussian primitive. The predicted primitives are then rasterized using a differentiable Gaussian renderer, producing all required property maps in a single pass.
Training Protocol
Supervision is enforced via pixel-wise and perceptual (LPIPS) losses on material properties, affine-invariant depth loss for geometry, and cosine similarity for surface normals. The design leverages both synthetic (Interior Verse, Structured3D, Infinigen) and real-world datasets (RealEstate10K, DL3DV), and uses strong pretrained backbones for initialization, with end-to-end fine-tuning improving the multi-task performance.
Experimental Results
Material and Geometry Prediction
InvSplat achieves strong quantitative results for per-view albedo, metallicity, roughness, and normal estimation. On Interior Verse, it yields comparable PSNR and SSIM values for albedo (PSNR ≈ 22.2, SSIM ≈ 0.87) and closely matches the fine-tuned 2D multi-view baseline (MVInverse), while benefiting from explicit 3D scene structure and rapid inference.
Multi-View Consistency
A key characteristic is markedly improved multi-view consistency across reflectant and specular material regions, as measured by reprojection RMSE across views in Structured3D. InvSplat matches or outperforms MVInverse on consistency metrics for albedo, metallic, and roughness maps, providing substantial robustness in applications demanding seamless viewpoint synthesis.
Generalizability & Downstream Applications
InvSplat generalizes to both synthetic and real-world scenes, performing well even with sparse multi-view input, and enabling physically-plausible relighting, robust material editing, and stable novel view synthesis (demonstrated on RealEstate10K and DL3DV). The method supports real-time relighting under novel illuminants via PBR-compliant rendering, significantly augmenting the expressivity and utility over RGB-centric feed-forward 3DGS approaches.
Ablative Analysis
Ablations confirm that explicit normal prediction heads, joint geometry-material fusion, and unified fine-tuning outperform naïve combinations of previous per-view (MVInverse) and geometric (ReSplat) baselines. End-to-end learning ensures that each Gaussian primitive's intrinsic property estimates are view-consistent and geometrically anchored.
Discussion and Implications
Theoretical Implications
InvSplat bridges a fundamental gap: efficient, 3D-consistent, physically interpretable inverse rendering in a single forward pass. By reconstructing disentangled geometry and material factors, InvSplat enables physically-correct rendering and editing, fostering further separation of illumination, material, and geometry in neural scene representations. This shift from direct RGB modeling to intrinsic decomposition naturally extends to applications in material-aware scene understanding, photorealistic content generation, and AR/VR asset creation.
Practical Implications
The capability to recover editable, physically-grounded 3D scenes from few images, without per-scene optimization, has direct impact for scalable content creation pipelines in entertainment, industrial design, and robotics. Fast relighting and material editing—demonstrated via BRDF-based relighting—are crucial for interactive 3D authoring tools. However, InvSplat also inherits limitations from feed-forward pipelines: sensitivity to camera pose quality, resolution constraints, and limited coverage for occluded or unobserved scene regions.
Speculation on Future Developments
Advances may follow in several directions:
- Learning Lighting Priors: Integration of explicit illumination estimation would further complete the inverse rendering pipeline.
- Generative Expansion: Hybridization with generative models could allow hallucination of unobserved geometry and texture.
- Higher-Order BRDFs and Dynamic Scenes: Extension to richer material models and dynamic scene content.
- Self-supervised or Active-View Training: Exploiting weak or self-supervision to further alleviate the need for specialized synthetic datasets.
Conclusion
InvSplat presents a practical, feed-forward inverse rendering method that reconstructs geometry and intrinsic material properties into a differentiable 3D Gaussian representation from multi-view images. By enabling physically-based relighting, material editing, and robust novel view synthesis with strong cross-view consistency, it sets a new standard for efficiency and fidelity in neural 3D scene reconstruction. This work substantiates the potential of explicit intrinsic-material-based representations, laying the groundwork for future developments in physically-based, real-time scene understanding and synthesis (2607.02301).