Mesh-Based Inverse Rendering

• Mesh-based inverse rendering frameworks are computational techniques that recover 3D geometry, materials, and lighting from images using explicit surface meshes and differentiable optimization.
• They integrate differentiable rasterization, neural inverse modeling, and physics-based rendering to optimize scene parameters with edge-aware and topology-informed loss functions.
• These methods yield high-fidelity, topology-consistent models compatible with modern graphics pipelines, enhancing applications in AR, CAD, and photorealistic rendering.

Mesh-based inverse rendering frameworks represent a major category of computational techniques aimed at recovering 3D geometry, materials, and illumination from images, leveraging explicit surface meshes as the medium for optimization and differentiable rendering. By combining advances in differentiable rasterization, mesh-based physics-based rendering, neural inverse modeling, and topological priors, these frameworks offer practical routes to reconstructing high-fidelity, topology-consistent, physically parameterized models, directly compatible with modern rendering and graphics pipelines.

1. Mathematical Foundations of Mesh-Based Inverse Rendering

Mesh-based inverse rendering recasts the image formation process as a differentiable mapping from scene parameters (vertex positions, faces, per-surface materials, and lighting) to observed intensities. The key mathematical formulation is the rendering equation, which, for surface mesh $\mathcal{M}$, outgoing direction %%%%1%%%%, and input illumination $L_i$, is written for each point $\mathbf{x}$ as

$$L_o(\mathbf{x}, \omega_o) = L_e(\mathbf{x}, \omega_o) + \int_H f_r(\mathbf{x}, \omega_o, \omega_i) L_i(\mathbf{x}, \omega_i) (\mathbf{n} \cdot \omega_i) \, d\omega_i$$

where $f_r$ is the bidirectional reflectance distribution function (typically microfacet, GGX, or neural-parametric), $L_e$ is emitted radiance, and $(\mathbf{n}\cdot\omega_i)$ accounts for surface orientation.

The mesh $\mathcal{M} = (V, F)$ is characterized by vertices $V$ and faces $F$. Per-vertex or per-face attributes—their positions, normals, albedo, roughness, and optionally specular parameters—are optimized so that a differentiable rendering layer produces images matching the observed multi-view input. The optimization objective is generally

$$\min_{\mathcal{M}, \theta} \sum_{v \in \text{views}} \mathcal{L}_\text{render}(R(\mathcal{M}, \theta; v), I_v) + \mathcal{L}_\text{reg}(\mathcal{M}, \theta)$$

where $\theta$ encodes material and lighting model parameters, $R(\cdot)$ is the differentiable renderer, $\mathcal{L}_\text{render}$ computes photometric error, and regularization $\mathcal{L}_\text{reg}$ imposes smoothness, flattening, and topological or geometric priors.

Notable frameworks such as Soft Rasterizer (Liu et al., 2019) use a non-parametric differentiable rasterization: $S^i = 1 - \prod_{j=1}^N (1 - D_j^i)$ where $D_j^i$ is the per-triangle probability that pixel $i$ falls inside face $j$, governed by a soft logistic function of pixel-edge distance, supporting smooth backpropagation to mesh vertices.

2. Differentiable Rasterization and Physics-Based Rendering

Traditional mesh renderers are not differentiable due to the discrete point-in-triangle test and z-buffering. The Soft Rasterizer (Liu et al., 2019) converts these discrete steps to continuous, probabilistic, differentiable functions by replacing the binary silhouette with a “soft” aggregate over all triangles. This enables gradient-based learning of mesh geometry from image-level losses—including silhouette (IoU), Laplacian (vertex smoothness), and flattening (coplanarity of faces).

Advanced frameworks extend mesh differentiability to full PBR, leveraging explicit gradients through rasterization (nvdiffrast), multi-bounce Monte Carlo integration, and automatic differentiation of BRDFs. For example, Triplet (Yang, 2024) replaces standard winner-takes-all depth compositing with alpha-blended energy accumulation across triangles, enabling gradients to flow through occlusions and surface overlap.

Complex scene lighting and materials (e.g., Disney-GGX, neural BRDFs) are incorporated with mesh-parameterized textures, per-vertex SH coefficients, and direct–indirect illumination models, allowing optimization of physically plausible reflectance and global illumination (Sun et al., 2023, Li et al., 2022).

3. Mesh Generation, Extraction, and Refinement

Initial meshes are generated either via template primitives (sphere, genus-$g$ model), marching cubes from neural SDFs, or mesh extraction from radiance fields or Gaussian splatting (Wang et al., 2023, Choi et al., 2024). Subsequent refinement employs explicit mesh optimization—vertices and materials are updated via differentiable rendering losses.

Adaptive mesh refinement schemes (edge splits/collapses/flips, curvature-guided subdivision) are crucial for handling fine geometry and complex topology. Topology-aware frameworks incorporate explicit genus enforcement and persistent homology priors (Gao et al., 24 Nov 2025, Gao et al., 17 Jan 2026). Remeshing operations are performed without changing the Euler characteristic, preserving genus by local operations informed by curvature and connectivity statistics. For high-genus surfaces, cameras are adaptively placed around critical loops to expose and stabilize photometric gradients during optimization (Gao et al., 17 Jan 2026).

Mesh-based neural rendering modules introduce per-vertex learned features and MLP-based deferred shading to capture photorealistic details and intricate indirect lighting (Ming et al., 2024).

4. Material, Lighting Model Disentanglement and Losses

Material estimation is achieved via SVBRDF parameterization over mesh vertices/UVs (base color, roughness, metallic, specular), often with MLP regressors and strong priors/regularization (Bayesian, Laplace) to prevent collapse. Lighting decomposition includes HDR environment maps (mesh-probe or MLP), spatially-varying SH lobes, and indirect global illumination caches (Sun et al., 2023, Li et al., 2022).

To prevent “baked” interreflections and shadowing errors, frameworks such as JOC (Sun et al., 2023) and MIRReS (Dai et al., 2024) employ neural radiance caches and multi-bounce path tracing. Edge-aware rendering and silhouette-based losses are essential for sharp contour recovery (Zhang et al., 2022). Regularization is further enhanced by bilateral solvers over mesh UVs—smoothing textures while preserving essential details.

Material–lighting disentanglement relies on careful design of loss terms:

• Photometric losses (e.g., $L_1$, SSIM, LPIPS),
• Laplacian/flattening for vertex smoothness,
• Geometry inversion penalty (face Jacobian determinant),
• Regularization on SH coefficients and indirect illumination cache,
• Topological consistency via Wasserstein distance on persistent diagrams (Gao et al., 17 Jan 2026).

5. Handling Topology, High-Genus Surfaces, and Persistent Homology Priors

A key challenge in mesh-based inverse rendering is the preservation and accurate reconstruction of surface topology, especially for high-genus shapes (with tunnels and handles). Standard frameworks suffer from vanishing gradients near occluded regions or self-intersecting structures, often collapsing tunnels or oversmoothing (Gao et al., 24 Nov 2025). Persistent homology integration offers a principled mechanism to regularize optimization:

• PH computes topological features via simplicial filtrations and persistence diagrams;
• Losses penalize deviation of birth/death times for critical cycles (handles/tunnels) from expected values;
• Camera placement adapts to view topology-critical regions for robust photometric supervision (Gao et al., 17 Jan 2026);
• Remeshing schemes preserve genus by topology-preserving edge operations, enforced via discrete Gauss–Bonnet constraints.

Such topology-informed loss integration significantly reduces geometric errors (Chamfer Distance) and improves Volume IoU over baseline approaches, especially on complex shapes.

6. Engineering and Applications

Mesh-based frameworks are computationally efficient and scalable. Pseudo-code examples in the literature detail GPU-based rasterization, differentiable path tracing, fused volume and mesh optimization, and locality/sparsity-aware neural modules for practical runtime (<1–5 min per scene on RTX-class GPUs) (Yang, 2024, Dai et al., 2024).

Output meshes with baked SVBRDFs and HDR illumination can be exported in standard formats (glTF, FBX, USD) for use in game engines, CAD, animation, and AR/VR pipelines. Applications span photorealistic object insertion, relighting, material editing, blendshape deformation recovery, urban/indoor scene reconstruction, and high-genus artifact preservation.

Empirical results consistently show improvements in image synthesis quality (PSNR, SSIM), geometry accuracy (sub-millimeter Chamfer), and material estimation fidelity, with topological robustness unique to mesh frameworks. Ablation analyses validate the necessity of edge-aware and topology-informed terms.

7. Ongoing Developments and Limitations

Recent research continues to extend mesh-based inverse rendering to multi-bounce illumination, volumetric scene components, adaptive mesh refinement, and dynamic topology adjustment (Dai et al., 2024, Gao et al., 17 Jan 2026). Limitations include gradient instability for malformed meshes, restricted support for volumetric media (scattering, transparency), and dependence on initial mesh fidelity.

Future work targets coupling dynamic remeshing/backbone changes with full adjoint differentiation, volumetric radiance field integration, neural radiance caches for global illumination, and broader support for scene types (transparent, participating media). Robustness to limited views and reduced image datasets also remains an area of research.

---

In summary, mesh-based inverse rendering frameworks fuse explicit mesh parameterization, differentiable rendering, sophisticated material and lighting disentanglement, topology-aware priors, and advanced optimization strategies. The corpus of arXiv research demonstrates that these methods yield robust, accurate, and physically meaningful scene reconstructions, with broad utility in graphics, vision, and AR applications (Liu et al., 2019, Gao et al., 24 Nov 2025, Gao et al., 17 Jan 2026, Sun et al., 2023, Yang, 2024, Li et al., 2022, Dai et al., 2024, Wang et al., 2023, Zhang et al., 2022, Ming et al., 2024).