Differentiable Splat Rendering

• Differentiable splat rendering is a technique that replaces traditional non-differentiable rasterization with continuous, gradient-friendly kernels to optimize geometric and appearance parameters.
• It employs various primitives—such as 3D Gaussians, triangles, and Bézier curves—projected as density functions with differentiable compositing rules that enhance occlusion and boundary handling.
• The approach underpins high-fidelity view synthesis, neural rendering, vector graphics, and mesh extraction, achieving notable efficiency improvements through hardware acceleration and optimized gradient propagation.

Differentiable splat rendering comprises a family of rendering techniques that express rasterization and compositing as fully differentiable operators, allowing direct optimization of geometric and appearance parameters via gradient-based learning. Central to this paradigm is "splatting," in which geometric primitives—such as points, 3D Gaussians, triangles, or parametric curves—are projected onto the image plane as density functions with finite or infinite support. The differentiable construction of the splat operator, its compositing rule, and its treatment of occlusion and boundaries have enabled breakthroughs in 3D scene reconstruction, neural rendering, vector graphics, real-time rendering, and mesh extraction.

1. Mathematical Formulations and Splat Kernels

Differentiable splat rendering generalizes the classical graphics pipeline by replacing hard, non-differentiable rasterization and z-buffering with continuous, differentiable kernels and accumulation rules.

Splat Representations

• 3D Gaussian Splatting: Each primitive is a 3D anisotropic Gaussian, parameterized by mean $\mu \in \mathbb{R}^3$, positive-definite covariance %%%%1%%%%, and per-splat appearance/opacity parameters (Hanson et al., 2024, Feng et al., 2024). The projection to 2D for pixel $p$ yields:

$$g_i(p) = \exp\left( -\frac{1}{2} (p - \mu'_i)^{\top} (\Sigma'_i)^{-1} (p - \mu'_i) \right)$$

Composited with standard front-to-back $\alpha$-blending:

$$C(p) = \sum_{i=1}^N c_i \alpha_i(p) \prod_{j=1}^{i-1} [1 - \alpha_j(p)]$$

where $\alpha_i(p)$ is the projected per-pixel opacity.

• Triangle Splatting: Each primitive is a triangle with projected vertices $q_1, q_2, q_3$. Pixel weights are determined by a window function $I(p; \{q_i\}, \sigma)$ based on the triangle SDF (Held et al., 25 May 2025, Held et al., 29 Sep 2025, Held et al., 7 Dec 2025):

$$I(p) = \left[ \max_{i=1,2,3} (n_i \cdot p + d_i)/\phi(s) \right]_+^{\sigma}$$

This kernel is strictly zero outside the triangle and smoothly varies inside; $\sigma$ controls sharpness.

• Bézier Splatting: Parametric curves $B(t)$ are sampled at $K$ points, with each sample rendered as an anisotropic 2D Gaussian with data-driven covariance (Liu et al., 20 Mar 2025). Differentiability is maintained through the entire rasterization and compositing chain.
• One-pixel Splatting (ADOP): Extreme case in which each point is projected and contributes to exactly one pixel. The rounding operation is non-differentiable, so finite-difference approximations of pixel-wise changes are used for the backward pass (Rückert et al., 2021).

Compositing and Accumulation

The common theme is differentiable accumulation—splat contributions are weighted and composited per pixel through differentiable "over" operators (alpha blending), differentiable occlusion models, or commutative weighted summation (for efficiency) (Hou et al., 2024). Sorting by depth preserves correct transmittance; sort-free approximations improve performance at the expense of physically correct occlusion.

Gradient Computation

All splatting frameworks derive per-pixel derivatives with respect to primitive parameters—mean, covariance, color, opacity, triangle vertices, or curve control points—using chain rule and (for most kernels) standard autodiff. Exact expressions are available for all core parameters, including challenging cases like control-point gradients for discontinuity-aware boundaries or region-of-influence for triangle edges (Held et al., 29 Sep 2025, Qu et al., 2024, Liu et al., 20 Mar 2025).

2. Algorithmic Pipelines and Performance Optimizations

Splat renderers are implemented either as tile-based CUDA/OpenCL pipelines or on the graphics hardware via programmable fragment shaders with custom blending and gradient backpropagation.

Core Steps

• Preprocessing: Project all splats (Gaussians, triangles, points, curve samples) to image space and identify affected camera tiles or pixels. Efficient culling based on per-splat $\alpha$ threshold or tight bounding ellipses/boxes drastically reduces redundant computation (Hanson et al., 2024, Feng et al., 2024).
• Compositing: For each tile or pixel, accumulate sorted (or sort-free) per-splat contributions via hardware or software alpha blending.
• Gradient Accumulation: Backpropagate loss through accumulation—via programmable blending (hardware fragment interlocks) for 3DGS (Yuan et al., 24 May 2025), or explicit CUDA kernels, or autodiff on the full compositing formula.
• Pruning and Densification: Iterative training-time pruning removes redundant primitives, reducing model size and compute (e.g., per-splat sensitivity scoring, hardness annealing for triangles, error-driven densification for curves) (Hanson et al., 2024, Held et al., 29 Sep 2025, Liu et al., 20 Mar 2025).

Advances in Efficiency

• Hardware Rasterization and Programmable Blending: Exploiting fixed-function blending and per-pixel synchronized buffers (via Vulkan/DirectX fragment interlocks) enables efficient, memory-scalable differentiable rasterization on GPUs (Yuan et al., 24 May 2025).
• Hybrid Reductions: Local (quad/subgroup) aggregation of fragment gradients minimizes atomic write pressure and accelerates the backward pass (Yuan et al., 24 May 2025).
• Sparse Primitive Scheduling: Tile-based and AccuTile methods avoid unnecessary workload—precision ellipse-tile intersection removes >90% of redundant splat-tile pairs (Hanson et al., 2024, Feng et al., 2024).
• Sort-free Weighted-Sum Rendering: Eliminating per-pixel depth sorting and non-commutative alpha-compositing, at the price of approximate occlusion (Hou et al., 2024).

Benchmark Results

Renderer/System	Device	Forward+Backward Time	Speedup vs. Baseline	Memory Reduction
FlashGS (3DGS, CUDA)	RTX 3090	7.4x faster	7.4x	49% less
Speedy-Splat (3DGS, CUDA)	RTX 4090	6.71x faster	6.71x	10.6x model reduction
Hardware-diff Splatting (VK)	RTX 4080	3.07x faster	3.07x (float16)	4–5x
Bézier Splatting (VG, CUDA/Py)	—	20x (fwd), 150x (bwd)	20–150x	2–3x (vs. LIVE)

3. Occlusion, Boundaries, and Discontinuity Treatment

Handling occlusion boundaries, silhouette discontinuities, and hard edges is critical for high-fidelity view synthesis and mesh extraction.

• Multi-Layer Splatting: Multi-layer depth peeling combined with per-layer splatting allows gradients to be properly defined at occlusion/disocclusion boundaries—key for mesh, parametric, and surface rendering (Cole et al., 2021).
• Discontinuity-aware Splatting: DisC-GS augments each Gaussian with learnable Bézier-curve boundary masks; the indicator function gates splat influence to only one side of an explicit discontinuity. A custom pseudo-gradient operator provides meaningful gradient flow through the piecewise-constant mask (Qu et al., 2024).
• Sharp Boundaries with Triangles: Triangle and Mesh Splatting rely on SDF-based window functions with learnable sharpness. Annealing $σ\to 0$ during optimization yields hard-edged, opaque geometry suitable for export to standard pipelines (Held et al., 29 Sep 2025, Held et al., 7 Dec 2025).

Approach	Boundary Model	Discontinuity Derivative
Standard 3DGS	Gaussian tails (blurred)	Smooth, no exact edges
DisC-GS	Bézier curve masking	Gradient by parametric search
Triangle/ Mesh Splatting	SDF-based, sharp-edged kernel	Chain rule through window SDF
Bézier Splatting (VG)	Control-point diff on curves	Full chain rule via Bernstein

4. Primitive Types and Mesh Extraction

Recent innovations have focused on bridging the gap between neural splat representations and mesh-based pipelines required in AR/VR and CG.

• Opaque Triangle Splatting: Opacity and sharpness are annealed to enforce hard, mesh-compatible geometry, enabling direct rasterizer import and physics/interaction support (Held et al., 29 Sep 2025, Held et al., 7 Dec 2025).
• Mesh Splatting and Restricted Delaunay: Joint geometry-color optimization, connectivity-enforcing mesh extraction via restricted Delaunay triangulation, and SDF-based regularization yield visually and topologically consistent output meshes—improving efficiency and mesh quality over conventional meshification of Gaussians (Held et al., 7 Dec 2025).
• Comparison to Gaussians: Standard 3DGS offers volumetric coverage and smooth view extrapolation but leads to blurry edges and complex mesh extraction; triangle/mesh splatting achieves sharper edges, mesh compatibility, and real-time rasterization at the cost of lower expressivity for transparency and background holes (Held et al., 29 Sep 2025, Held et al., 25 May 2025).

5. Practical Applications and Experimental Results

Differentiable splat rendering has become foundational in:

• Novel View Synthesis: 3DGS, triangle splatting, and mesh splatting support fast, high-fidelity view synthesis, with PSNR/SSIM/LPIPS gains over prior approaches in both unbounded/outdoor and bounded/indoor datasets (Hanson et al., 2024, Held et al., 29 Sep 2025, Held et al., 7 Dec 2025, Held et al., 25 May 2025).
• Vector Graphics: Bézier Splatting achieves large speedups and higher fidelity in differentiable vectorization and SVG export (Liu et al., 20 Mar 2025).
• Real-Time Rendering: Hardware-differentiable pipelines enable low-latency rendering and gradient computation on edge and mobile hardware (Yuan et al., 24 May 2025, Hou et al., 2024, Feng et al., 2024).
• Mesh Extraction/AR/VR Integration: End-to-end mesh pipelines (MeshSplatting, Triangle Splatting+) bridge neural rendering and interactive graphics, producing compact, rendered meshes with real-time 3D engine compatibility (Held et al., 29 Sep 2025, Held et al., 7 Dec 2025).

Method	PSNR↑	LPIPS↓	Vertices (M)	Train Time	FPS (HD)
3DGS (no mesh)	27.21	0.214	—	39 min	134
RaDe-GS (mesh)	23.56	0.361	31	—	OOM
MiLo (mesh)	24.09	0.323	7	106 min	170
MeshSplatting (mesh)	24.78	0.310	3	48 min	220
Triangle Splatting+ (mesh)	25.21	0.294	2	39 min	—

6. Extensions and Limitations

Research Directions

• Occlusion Handling: Multi-layer splatting and hybrid depth compositing techniques continue to evolve for more accurate and efficient occlusion boundary differentiation (Cole et al., 2021, Qu et al., 2024).
• Primitive Versatility: Work is ongoing to support complex materials (transparency), adaptive fill-rules (vector graphics), and hierarchical/multi-resolution primitives (Liu et al., 20 Mar 2025).
• Hardware Integration and Mobile Efficiency: Mixed-precision, fixed-capacity memory models, and programmable blending are actively explored for better deployment on resource-constrained platforms (Yuan et al., 24 May 2025, Feng et al., 2024).
• Hybrid Representations: Experiments with discontinuity-aware and curve/surface hybridization point toward future frameworks marrying explicit geometry and neural radiance field techniques (Qu et al., 2024).

Limitations

• Blurring and Transparency: Gaussian splatting blurs sharp features and is challenged on hard discontinuities; triangles struggle to represent translucency and background fill.
• View Extrapolation: Opaque triangles exhibit strong artifacts when rendering far outside the training trajectory, as opposed to the graceful falloff of volumetric Gaussians.
• Scalability: While one-pixel splatting (ADOP) is extremely fast, it lacks the expressivity of higher-order parametric kernels, and is less effective for richly textured or smoothly varying surfaces (Rückert et al., 2021).

---

Differentiable splat rendering has become an essential tool for modern 3D vision, neural rendering, and graphics, enabling high-fidelity, real-time, and mesh-compatible pipelines by unifying efficient, gradient-friendly primitive rasterization with application-driven compositional objectives. Recent advances deliver strong performance on conventional and resource-constrained hardware, robust mesh extraction, and sharper, more controllable representations suitable for both photorealistic and stylized output. Continued work focuses on hybrid primitives, occlusion-robust differentiation, and system integration spanning rendering, editing, and scene understanding.