MeshSplatting: Differentiable Rendering with Opaque Meshes (2512.06818v1)

Abstract: Primitive-based splatting methods like 3D Gaussian Splatting have revolutionized novel view synthesis with real-time rendering. However, their point-based representations remain incompatible with mesh-based pipelines that power AR/VR and game engines. We present MeshSplatting, a mesh-based reconstruction approach that jointly optimizes geometry and appearance through differentiable rendering. By enforcing connectivity via restricted Delaunay triangulation and refining surface consistency, MeshSplatting creates end-to-end smooth, visually high-quality meshes that render efficiently in real-time 3D engines. On Mip-NeRF360, it boosts PSNR by +0.69 dB over the current state-of-the-art MiLo for mesh-based novel view synthesis, while training 2x faster and using 2x less memory, bridging neural rendering and interactive 3D graphics for seamless real-time scene interaction. The project page is available at https://meshsplatting.github.io/.

• The paper introduces a fully differentiable pipeline for mesh-based rendering that directly optimizes connected opaque triangle meshes using restricted Delaunay triangulation.
• It achieves state-of-the-art results with significant improvements in PSNR, memory usage, and training time compared to previous mesh-based view synthesis methods.
• The method integrates seamlessly with game engines, enabling real-time novel view synthesis, efficient object segmentation, and immediate deployment in interactive applications.

Authoritative Essay on "MeshSplatting: Differentiable Rendering with Opaque Meshes" (2512.06818)

Introduction

The "MeshSplatting: Differentiable Rendering with Opaque Meshes" paper introduces a fully differentiable framework for end-to-end optimization of mesh-based scene representations tailored for mesh-centric pipelines in computer graphics, AR/VR, and real-time applications. This method, called MeshSplatting, advances the state-of-the-art by obviating the need for a posteriori conversion of 3D Gaussian splats or triangle soups to classical meshes, instead directly optimizing connected, opaque, colored triangle meshes robustly compatible with standard game engines.

Background and Motivation

Primitive-based approaches, specifically 3D Gaussian Splatting, have enabled high-fidelity, real-time novel view synthesis. However, such representations are point-based and inherently incompatible with mesh-dominant workflows. Prior attempts at extracting meshes from Gaussian scene representations introduce considerable complexity, increase training overhead, and often fail to maintain high visual fidelity due to non-differentiable post-processing. Early differentiable mesh rasterizers suffered from poor scalability, non-convex optimization, and limited applicability to complex, real-world scenes.

MeshSplatting addresses core limitations by introducing:

• Connected, opaque mesh construction via restricted Delaunay triangulation directly during optimization.
• Joint geometry and appearance refinement compatible with differentiable rendering.
• An optimization curriculum that transitions from triangle soups to fully connected opaque meshes, supporting immediate deployment in physics-based simulations, interactive environments, and scene editing.

Methodology

MeshSplatting's architecture centers on a two-stage pipeline: an initial triangle soup optimization followed by mesh connectivity enforcement and refinement.

Vertex-Sharing Mesh Parameterization:

Unlike unstructured soups, each triangle references a shared vertex set, with each vertex $\mathbf{v}_i$ encoding position, color (via spherical harmonics), and opacity. During backward propagation, gradients from all incident faces update each shared vertex, promoting smooth transitions and geometric consistency. The transition from triangle soup to mesh is realized via restricted Delaunay triangulation, which preserves surface geometry while imposing connectivity.

Figure 1: Mesh parametrization transitions from independent triangle soups (left) to vertices shared among multiple triangles (right), enabling gradient accumulation and smooth global geometry.

Optimization Schedule:

Opacity and triangle window parameters ($\sigma$) are dynamically scheduled. Initially, triangles are semi-transparent with smooth window functions to ensure stable gradient flow, then progressively annealed to yield sharp, fully opaque triangles.

Figure 2: Window parameter scheduling illustrates the transition from smooth to sharp triangles by annealing $\sigma$ during the optimization pipeline.

Losses and Regularization:

A composite loss over photometric metrics, depth alignment, opacity, and normal consistency is applied. Notably, the depth-alignment term robustly penalizes discrepancies between predicted and reference depth, acting on individual vertices rather than local mesh topology—a property critical for scalable end-to-end mesh learning.

Mesh Densification and Pruning:

Adopting strategies similar to 3DGS-MCMC, MeshSplatting adaptively densifies regions insufficiently covered by the mesh and prunes irrelevant or occluded primitives based on opacity and blending weights, ensuring both mesh compactness and visual fidelity.

Empirical Results

View Synthesis and Mesh Quality

MeshSplatting is benchmarked on Mip-NeRF360 and Tanks and Temples datasets, outperforming previous mesh-based view synthesis approaches on PSNR, LPIPS, SSIM, and mesh compactness. On Mip-NeRF360, it achieves a +0.69dB PSNR boost over MiLo and significant reductions in memory usage and training time. Compared to all mesh-discrete methods, MeshSplatting yields higher visual quality with fewer vertices and direct compatibility with game engines.

Application to Object Segmentation

MeshSplatting's opaque mesh structure enables trivial mapping between 2D masks and mesh triangles, making object extraction and scene editing straightforward and robust, unlike Gaussian-based methods where pixel contributions are highly entangled.

Figure 3: Demonstration of segmentation in the Garden scene, with direct extraction of objects and corresponding mesh representations.

Figure 4: Further qualitative examples of object extraction capabilities across diverse scenes.

Regularization and Mesh Quality Ablation

Careful balance of normal and depth regularization terms is crucial:

Figure 5: Regularization ablation showing the trade-off between visual fidelity and geometric accuracy. Under-regularization yields non-physical geometry, while over-regularization smooths surfaces but can attenuate high-frequency appearance details.

Efficiency and Practicality

MeshSplatting demonstrates compelling computational advantages, with over 2x reduction in both memory footprint and training time vis-à-vis MiLo, and real-time rendering performance on commodity hardware. The output meshes are directly usable for downstream tasks including physical simulation, interactive walkthroughs, and hardware-accelerated ray tracing, with no retraining or manual conversion.

Implications and Future Perspectives

MeshSplatting fundamentally aligns the high-fidelity neural scene representations of modern view synthesis with the practical requirements of interactive graphics systems. By outputting natively connected, colored, and opaque triangle meshes, it closes the compatibility gap between neural rendering and established graphics engines.

On the theoretical side, integrating restricted Delaunay triangulation within the optimization loop presents a pathway for learning manifold mesh topology in complex scenes, with clear implications for further exploration in differentiable geometry processing and joint optimization of appearance and shape.

For future work, the authors note limitations in the handling of highly transparent surfaces or extremely sparse backgrounds, where representation fidelity may still lag behind volumetric approaches. Future research may integrate learned texturing, sky domes, or explicit watertightness constraints, potentially enhancing both the visual and geometric robustness of neural mesh reconstructions.

Conclusion

MeshSplatting provides a practical, efficient, and fully differentiable pipeline for generating high-fidelity, connected, and directly usable mesh representations of complex real-world scenes. This work not only achieves state-of-the-art performance in mesh-based novel view synthesis but also establishes a new standard for interoperability between neural scene representations and real-time interactive applications. MeshSplatting is poised to be foundational for the next generation of AR/VR, simulation, and 3D content creation, where seamless mesh integration and visual realism are paramount.