Differentiable Splatting in Neural Rendering

• Differentiable splatting is a rendering framework that projects geometric primitives onto lower-dimensional grids using continuous, differentiable kernels to enable end-to-end optimization.
• It leverages methods like anisotropic Gaussian, triangle, and tetrahedron splatting to facilitate high-fidelity neural scene capture, mesh recovery, and vector graphics synthesis.
• Recent advancements introduce learnable kernel forms such as GEF and adaptive blending schemes, enhancing efficiency, detail preservation, and rendering fidelity.

Differentiable splatting is a class of rendering and transformation operations in which continuous or discrete geometric primitives (e.g., points, curves, Gaussians, triangles, tetrahedra) are projected or "splatted" onto a lower-dimensional domain (typically an image or raster grid), with all stages of the process supporting analytic or autodiff-based backpropagation. This property enables direct optimization of geometry, appearance, and topology for tasks such as neural rendering, inverse graphics, vector graphics synthesis, and multi-view reconstruction. Modern differentiable splatting techniques are central to a diverse range of state-of-the-art methods, including Gaussian Splatting for 3D scene capture, softmax splatting for video interpolation, triangle and tetrahedron splatting for differentiable mesh optimization, and differentiable curve splatting for vector graphics.

1. Theoretical Foundations of Differentiable Splatting

At its core, differentiable splatting involves mapping high-dimensional primitives to a lower-dimensional raster (image grid, point cloud, or range-view) using a continuous, locally-supporting kernel. The most canonical example is anisotropic Gaussian splatting, with each primitive described by a set of parameters $(\mu, \Sigma, o, c)$ for mean, covariance, opacity, and color. The projected image-space density at location $p$ for splat $i$ is

$$G_i(p) = o_i \exp\left(-\frac{1}{2}(p - \mu_i)^\top \Sigma_i^{-1} (p - \mu_i)\right).$$

Color and opacity contributions from all primitives are composed using front-to-back alpha blending: $$C(p) = \sum_{i=1}^{N} T_{i-1}(p) \, G_i(p) \, c_i, \quad T_{i}(p) = \prod_{j < i} (1 - G_j(p)).$$ All parts of this mapping, including projection, kernel evaluation, and front-to-back compositing, are differentiable with respect to the parameters of each splat (Held et al., 25 May 2025, Huang et al., 26 Mar 2024, Djeacoumar et al., 24 Feb 2025, Sheng et al., 23 Jun 2025, Xu et al., 17 Oct 2024).

Recent work advocates replacing the Gaussian kernel with more flexible functional forms. Generalized Exponential Splatting (GES) introduces learnable "shape" exponents: $$\text{GEF}(x; \mu, \alpha, \beta, A) = A \exp\left( -\left| \frac{x - \mu}{\alpha} \right|^{\beta} \right),$$ enabling sharper local support and memory-efficient modeling of high-frequency details (Hamdi et al., 15 Feb 2024).

Not all splatting kernels are Gaussian; triangle and tetrahedron splatting use SDF-based window functions or NeuS-style sigmoidal thickness as the blending kernel, maintaining analytic forms for gradients and silhouette sharpness (Held et al., 29 Sep 2025, Held et al., 25 May 2025, Gu et al., 3 Jun 2024).

2. Methodologies: Primitives, Kernels, and Composition

The table below summarizes representative primitive types and their splatting kernels.

Method/Domain	Primitive Type	Kernel / Composition
3D Gaussian Splatting (Held et al., 25 May 2025, Huang et al., 26 Mar 2024)	Anisotropic Gaussians	$\exp(-½ (x-\mu)^\top\Sigma^{-1}(x-\mu))$, alpha blending
GES (Hamdi et al., 15 Feb 2024)	Gen. exp. blobs	$\exp(-|x-\mu|^\beta)$, effective Gaussian scaling
Triangle Splatting (Held et al., 25 May 2025, Held et al., 29 Sep 2025)	2D/3D Triangles	SDF-based window: $[\text{ReLU}(\phi/\phi(s))]^\sigma$
Tetrahedron Splatting (Gu et al., 3 Jun 2024)	3D Tetrahedra	NeuS-style SDF-to-opacity slab, barycentric blending
Bézier Splatting (Liu et al., 20 Mar 2025)	2D Bézier Curves	Sampled Gaussians along curve, alpha blend
SketchSplat (Ying et al., 18 Mar 2025)	3D lines/Béziers	Sampled Gaussians on 3D curves, compositing via transparency
Softmax Splatting (Niklaus et al., 2020, Wang et al., 2023)	Flowed pixels/features	Softmax-importance over splatting kernel, feature sum

The differentiability in these pipelines arises from kernel properties (analytic derivatives with respect to center, orientation, and scale) and the use of soft, continuous blending that avoids hard occlusion or discontinuous rasterization (Held et al., 25 May 2025, Held et al., 29 Sep 2025, Xu et al., 17 Oct 2024).

3. Differentiable Splatting in 3D Neural Rendering and Reconstruction

Differentiable splatting underpins cutting-edge 3D scene capture, neural rendering, and physical simulation:

3D Gaussian Splatting achieves fast, high-fidelity multi-view reconstruction by representing scenes as millions of anisotropic Gaussians, compositing via analytic projection and depth-ordered blending (Huang et al., 26 Mar 2024, Xu et al., 17 Oct 2024, Xie et al., 14 Oct 2025). The forward rendering is fully differentiable, enabling optimization against photometric, depth, normal, or semantic losses. Recent advances incorporate analytic ray-ellipsoid intersections for differentiable depth and normal computation (Xie et al., 14 Oct 2025), and introduce learnable pruning attributes for sparsity and efficiency.

Triangle and Tetrahedron Splatting enable direct mesh-reconstruction pipelines. In Triangle Splatting, both vertex positions and per-triangle opacity/smoothness are learned via gradients from the differentiable renderer, resulting in crisp, watertight surfaces compatible with classical mesh engines (Held et al., 29 Sep 2025, Held et al., 25 May 2025, Sheng et al., 23 Jun 2025). Tetrahedron Splatting couples a deformable, SDF-predicting tetrahedral grid with NeuS-type volume rendering and differentiable alpha-blending for topology-robust mesh extraction (Gu et al., 3 Jun 2024).

Integration with Signed Distance Fields is achieved by pulling the parameters of explicit primitives (e.g., Gaussians) to the SDF zero-level set, jointly optimizing the explicit representation and the implicit field via multi-view consistency and geometry constraints (Zhang et al., 18 Oct 2024).

4. Computer Vision and Graphics Applications

Differentiable splatting is foundational in:

• Radiance field learning: Real-time rendering for neural scene representation, with or without explicit mesh or surfel extraction (Huang et al., 26 Mar 2024, Held et al., 25 May 2025, Gu et al., 3 Jun 2024).
• Simulated sensor modeling: LiDAR-GS applies Gaussian splatting for simulating LiDAR returns, including intensity and ray-drop channels, via differentiable, micro-cross-section splatting per laser beam and learns complex incidence- or environment-dependent properties (Chen et al., 7 Oct 2024).
• Point cloud rendering: Neural network-predicted splats enable low-latency, high-fidelity rendering of dynamic point clouds, robust to sparsity and compression artifacts (Hu et al., 24 Sep 2024).
• Vector graphics and curve rendering: Bézier Splatting and SketchSplat represent SVG-style curves or 3D CAD edges as sampled splats, achieving orders-of-magnitude speedup over optimization-via-differentiable rasterization (e.g., DiffVG), with closed-form control point gradients and adaptive disparity-aware resource allocation (Liu et al., 20 Mar 2025, Ying et al., 18 Mar 2025).
• Video frame interpolation and optical flow: Softmax splatting seamlessly blends features at occlusion or multi-mapping sites in forward warps and delivers best-in-class flow and interpolation performance (Niklaus et al., 2020, Wang et al., 2023).

5. Differentiability, Gradients, and Optimization

Because every function involved in splatting—from kernel evaluation to compositing—is built from differentiable primitives (exp, matmul, normalization, product/sum recurrences), autodiff frameworks (PyTorch, LibTorch, custom CUDA) provide analytic gradients with respect to all primitive parameters. This enables:

• End-to-end learning: Geometry, appearance, opacity, and auxiliary attributes (normals, semantic logits) are optimized directly by minimizing multi-modal losses (photometric, SSIM, normal consistency, semantic cross-entropy, etc.) (Xie et al., 14 Oct 2025).
• Gradient flow through geometric and appearance parameters: Gradients reach vertex positions, SDF values, rotation, scale, and color coefficients, yielding joint optimization for both geometry and appearance (Held et al., 25 May 2025, Huang et al., 26 Mar 2024, Xu et al., 17 Oct 2024, Sheng et al., 23 Jun 2025).
• Effective handling at occlusion boundaries: Multi-layer or depth-aware splatting schemes avoid zero gradients at visibility discontinuities, yielding unbiased and stable updates near silhouettes and depth steps (Cole et al., 2021, Held et al., 25 May 2025).
• Efficient backward computation: Hardware-accelerated pipelines using programmable blending, quad- and subgroup-level atomic reduction, and mixed-precision render targets deliver 10x backward speedups over naive atomic designs and >3x end-to-end acceleration for training and inference tasks on commodity GPUs (Yuan et al., 24 May 2025).

6. Regularization, Topology Adaptation, and Mesh/Multi-Modal Outputs

Differentiable splatting pipelines often incorporate regularization and dynamic adaptation:

• Geometry regularizers: Depth-distortion, normal-consistency, eikonal (unit-gradient), tangent alignment, and opacity sparsity encourage more accurate, robust, and interpretable reconstructions (Huang et al., 26 Mar 2024, Gu et al., 3 Jun 2024, Zhang et al., 18 Oct 2024).
• Adaptive pruning and densification: Pruning removes primitives with little contribution, while densification—e.g., error-driven subdivision or midpoint insertion—allocates more capacity where needed, producing sparser, higher-quality representations (Liu et al., 20 Mar 2025, Held et al., 25 May 2025, Xie et al., 14 Oct 2025).
• Topology adaptation: SketchSplat and Bézier Splatting use adaptive merging (endpoints, overlap, colinearity), visibility filtering, and error-based curve addition to yield compact, topologically meaningful vector/edge primitives with high coverage (Ying et al., 18 Mar 2025, Liu et al., 20 Mar 2025).
• Mesh Extraction: Meshes are extracted from tetrahedral/SDF representations via Marching Tetrahedra, or triangle splat sets are exported directly as standard mesh formats after differentiable training (Gu et al., 3 Jun 2024, Held et al., 25 May 2025, Sheng et al., 23 Jun 2025).
• Multi-modal rendering: Recent methods render and optimize for multiple outputs jointly, including color, depth, surface normal, and semantic class logits (Xie et al., 14 Oct 2025).

7. Limitations, Performance, and Future Directions

Key limitations and directions in differentiable splatting research include:

• Support and pruning: Gaussians, even with adaptive splitting or GEF kernels, may diffuse at edges or require memory-intensive overcomplete sets for complex geometry. Triangles and tetrahedra achieve sharper boundaries but can leave unconnected floaters or need topology fusion (Sheng et al., 23 Jun 2025, Gu et al., 3 Jun 2024).
• Mesh connectivity: Triangle and tetrahedron splatting optimizes disconnected "soups"; global mesh manifoldification is an open challenge (Held et al., 29 Sep 2025, Held et al., 25 May 2025).
• Advanced physical rendering: Effects such as shadows and indirect lighting require additional volumetric or hybrid pipelines for high-fidelity rendering beyond pure splatting (Sheng et al., 23 Jun 2025).
• Hardware/efficiency: Hardware-friendly designs (direct GPU rasterization, mixed-precision, tile-based acceleration) drive real-time inference, but portability to mobile platforms depends on future GPU software support (Yuan et al., 24 May 2025).
• Generalization and multimodal tasks: Unified multimodal splatting (color, depth, normals, semantics), and integration into 3D generation, AR/VR, and point cloud streaming remain active research directions (Xie et al., 14 Oct 2025, Chen et al., 7 Oct 2024, Hu et al., 24 Sep 2024).

In summary, differentiable splatting unifies analytic rendering, rasterization, and compositing for a broad range of geometric primitives, supporting direct, gradient-based optimization of both geometry and appearance. This technique underpins key advances across neural rendering, mesh recovery, vector graphics, and sensor simulation, with accelerating innovation in kernel design, differentiable pipelines, and scalable GPU infrastructure (Huang et al., 26 Mar 2024, Held et al., 25 May 2025, Held et al., 29 Sep 2025, Gu et al., 3 Jun 2024, Ying et al., 18 Mar 2025, Xie et al., 14 Oct 2025).