Splattable Neural Primitives

• Splattable neural primitives are parameterized, differentiable spatial functions with a finite receptive domain that efficiently encode radiance, deformation, and semantic fields.
• They leverage localized mixtures of bump functions—such as anisotropic Gaussians, ellipsoids, triangles, or billboards—and shallow MLPs for direct analytical splatting and compositing.
• Their design supports end-to-end differentiable optimization, rapid GPU implementations, and applications in real-time view synthesis, 3D reconstruction, and medical image registration.

Splattable neural primitives are parameterized, differentiable spatial functions or geometric objects, typically associated with a finite receptive domain, designed for explicit and efficient encoding of radiance fields, deformation fields, or structured geometric and semantic information. Unlike traditional neural volumetric representations that rely on global, shared implicit neural networks, splattable primitives localize computation through mixtures of bump functions—commonly anisotropic Gaussians, ellipsoids, triangles, or planar billboards—and may further parameterize internal density or appearance fields using shallow multilayer perceptrons (MLPs). This paradigm enables efficient compositing via analytical splatting instead of ray marching, allows differential optimization of both position and internal structure, and provides a direct bridge between classical computer graphics and modern neural field methods. Splattable neural primitives are the foundational abstraction for current state-of-the-art real-time view synthesis, semantic 3D reconstruction, medical image registration, manipulation, and articulated object modeling.

1. Mathematical and Algorithmic Foundations

Let $P$ denote a splattable neural primitive with support in $\mathbb{R}^3$, equipped with learnable geometric parameters (e.g., center, scale, rotation) and, optionally, an internal function (typically a shallow neural network or analytic profile):

• Gaussian primitives: $P$ is a 3D ellipsoid parameterized by center $\mu \in \mathbb{R}^3$, covariance $\Sigma \in \mathbb{R}^{3 \times 3}$, opacity $\alpha \in \mathbb{R}$, and color or SH coefficients $c \in \mathbb{R}^d$ (Lewis et al., 13 Jun 2025, Ji et al., 3 Sep 2024, Svitov et al., 13 Nov 2024, Sheng et al., 29 May 2025).
• Neural ellipsoids: Each $P$ defines a bounded ellipsoid, with density $\sigma(x) = f_\sigma(u(x))$ where $u$ is a normalized coordinate inside the ellipsoid and $f_\sigma$ is a compact MLP with periodic activations (Zhou et al., 9 Oct 2025).
• Planar/textured billboards: Each $P$ is a 2D local patch in 3D, parameterized by center, orientation, scale, and learnable $S_T \times S_T$ textures for color and alpha (Svitov et al., 13 Nov 2024).
• Triangles: Each $P$ is a triangle with three vertices in $\mathbb{R}^3$, sharpness, opacity, and SH color coefficients (Held et al., 25 May 2025).
• Mixtures: For generality, functions $f:\mathbb{R}^d \rightarrow \mathbb{R}^p$ can be represented as finite sums $f(x)=\sum_{i=1}^N w_i\phi(x;p_i,A_i)$, where $\phi$ is a smooth, localized bump function (Gaussian or otherwise), and the $N$ primitives are adapted by gradient descent in both geometry and parameters (Daniels et al., 18 Nov 2025).

Splatting refers to projecting each $P$ to the screen, computing its contribution analytically in image (or world) space, and compositing these via alpha blending or transmittance accumulations. Optimization is end-to-end differentiable with respect to all primitive parameters.

2. Design, Parameterization, and Internal Structure

Splattable neural primitives generalize analytic splats by associating internal fields—neural or semantic—with each primitive. Exemplary design choices include:

• Ellipsoid primitives with neural density fields: Each $P$ is a bounded ellipsoid with a density field learned via a shallow MLP. The analytic solution for the line integral through $P$ eliminates ray marching, and the MLP is designed for closed-form integration (e.g., cosine activation as in SIREN) (Zhou et al., 9 Oct 2025).
• Hashed-grid features with trilinear splatting: For dense deformation fields (e.g., medical image registration), multi-resolution hashed grids and trilinear splatting synthesize feature vectors for any query point. All grids contribute via splatting; features are then interpolated by a shallow MLP for the final output (e.g., displacement vector) (Li et al., 8 Feb 2024).
• Textured planar and triangle splats: Splatting is not restricted to ellipsoids; 2D billboards with learnable textures and sharp, arbitrarily-shaped alpha maps and triangles with learned coverage, opacity, and compact support have been deployed for enhanced surface detail and efficient compression (Svitov et al., 13 Nov 2024, Held et al., 25 May 2025).
• Generalized splats and mixtures: Within Splat Regression Models, any $C^\infty$ bump function may be used as a "mother splat," and affine transformations provide spatial localization and directional adaptivity. Finite mixtures approximate arbitrary functions, with parameters updated by Wasserstein–Fisher–Rao gradient flow (Daniels et al., 18 Nov 2025).

The choice of primitive and internal parameterization is dictated by the trade-off between local flexibility, computational efficiency, and the demands of the target task.

3. Rendering and Compositing Techniques

Rendering with splattable neural primitives relies on direct analytical compositing, no explicit voxelization or point rasterization is required.

• Alpha compositing: For a pixel $p$, overlapping primitives $\{P_i\}$ are composited front-to-back:

$C(p) = \sum_i c_i K_i \prod_{j < i} (1 - K_j)$

where $K_i$ is the analytic opacity kernel for $P_i$ at $p$ (Zhou et al., 9 Oct 2025Svitov et al., 13 Nov 2024Lewis et al., 13 Jun 2025).

• Closed-form kernels: For ellipsoidal neural primitives, the line integral of the neural density field is analytic. For planar/textured splats, the intersection yields a (u,v) coordinate for bilinear sampling of texture and alpha (Zhou et al., 9 Oct 2025, Svitov et al., 13 Nov 2024).
• Triangle splats: A window function $I_T(p)$ derived from a signed distance field gives each triangle its screen-space coverage; sharpness and opacity control blending (Held et al., 25 May 2025).
• Neural feature compositing: In part segmentation, grasp synthesis, or semantic reconstruction, the feature output of a primitive (typically post-MLP) is composited similarly to color, supporting direct semantic and instance-aware rendering (Ji et al., 3 Sep 2024, Sheng et al., 29 May 2025).

Efficient GPU implementations employ bounding-box culling, tile-based rasterization, and dynamic pruning/densification.

4. Training and Optimization Protocols

Both analytical and neural splattable primitives are amenable to end-to-end differentiable optimization.

• Supervision signals: Supervision may include photometric losses ($\ell_1$, SSIM, LPIPS), depth (geometry) losses, CNCC or similar image-matching terms, semantic consistency, instance segmentation, and cross-entropy over softmaxed assignments (Li et al., 8 Feb 2024, Lewis et al., 13 Jun 2025, Svitov et al., 13 Nov 2024, Sheng et al., 29 May 2025).
• Splat regression and gradient flows: Optimization in parameter space (center, affine/shape, mixing weights) is formalized as a Wasserstein–Fisher–Rao gradient flow in mixture space; this unifies direct composition-based fitting and modern SGD/Adam-based backpropagation (Daniels et al., 18 Nov 2025).
• Population control: Learnable importance scores or variance-based splitting/pruning determine the set of active primitives, enabling compact representations and rapid adaptation (Zhou et al., 9 Oct 2025, Sheng et al., 29 May 2025).
• Multi-field and hierarchical features: Dual fields with coarse (semantic) and fine (instance/appearance) features, as in SpatialSplat, are jointly trained to minimize redundancy and maximize expressivity (Sheng et al., 29 May 2025).
• Fast per-case adaptation: Several works show that splattable primitive systems can optimize scene representations in seconds or minutes, notably outperforming NeRFs or field-based models both in speed and final accuracy (Li et al., 8 Feb 2024, Ji et al., 3 Sep 2024).

5. Applications Across Domains

Splattable neural primitives serve as the core computational abstraction in a range of domains:

• Novel view synthesis: Real-time radiance field rendering with Gaussian, neural, triangle, or planar splats achieves state-of-the-art perceptual scores (LPIPS, PSNR) at 100–1000+ FPS, often with $10\times$ fewer primitives than prior analytic methods (Zhou et al., 9 Oct 2025, Svitov et al., 13 Nov 2024, Held et al., 25 May 2025).
• Semantic 3D reconstruction: Instance-aware semantic fields (dual field: coarse and fine) and selective Gaussian pruning enable efficient feed-forward 3D reconstruction and segmentation from sparse, unposed images (Sheng et al., 29 May 2025).
• Medical image registration: Multi-level splatting with hashed features and shallow interpolation networks realizes sub-millimeter accuracy for deformable registration at sub-2s runtime, with improved adaptation at sliding boundaries (Li et al., 8 Feb 2024).
• Articulated object modeling: Gaussian splat representations with part-aware semantic weights and kinematic tree parameterization allow learning, segmentation, and manipulation of highly articulated objects with deep kinematic chains (Lewis et al., 13 Jun 2025).
• Robotics and grasping: Gaussians with attached latent/semantic features enable open-vocabulary promptable segmentation and grasp planning, supporting real-time manipulation and dynamic tracking (Ji et al., 3 Sep 2024).
• Function regression and modeling: Splat regression models generalize mixture basis methods, providing fast, interpretable solutions to regression, inverse problems, and physics-informed PDE modeling (Daniels et al., 18 Nov 2025).

6. Advantages, Limitations, and Outlook

Splattable neural primitives provide several unique advantages over both global neural field models and classical mesh/voxel approaches:

• Efficiency and scalability: Analytical splatting kernels enable orders-of-magnitude speedup compared to ray marching. Systematic pruning and densification yield compact, storage-efficient models (Zhou et al., 9 Oct 2025, Svitov et al., 13 Nov 2024, Sheng et al., 29 May 2025).
• Locality, adaptivity, and interpretability: Each primitive is spatially localized, with learnable anisotropy and internal fields, facilitating fine adaptation to complex surfaces, multiscale features, and semantic regions (Daniels et al., 18 Nov 2025, Zhou et al., 9 Oct 2025).
• Directly editable and transformable: Rigid and non-rigid transforms are applied directly to primitive parameters, streamlining editing, animation, and dynamic manipulation (Lewis et al., 13 Jun 2025, Ji et al., 3 Sep 2024).
• Limitations: Optimization may exhibit local minima or slow convergence for highly under-constrained neural mixtures; calibration of regularization and splitting thresholds can be sensitive; existing methods may require scene-specific tuning for best performance or specialized feature networks for high-frequency detail (Zhou et al., 9 Oct 2025, Lewis et al., 13 Jun 2025).
• Extensions: Promising directions include learned controllers for population management, multi-band MLPs for within-primitive expressivity, hybrid architectures mixing grid and primitive features, incorporation of explicit boundary-aware kernels, and generalization to temporally dynamic and relightable representations (Zhou et al., 9 Oct 2025, Li et al., 8 Feb 2024).

Splattable neural primitives thus comprise a unifying abstraction at the intersection of neural field modeling and explicit differentiable graphics, with growing theoretical and empirical justification across high-impact domains.