Neural Gaussian Splatting

• Neural Gaussian Splatting is a technique that integrates neural networks with differentiable 3D Gaussian representations to overcome limitations in scene radiance and geometry parameterization.
• It replaces traditional parameterizations with injective submanifold field embeddings, ensuring a well-behaved latent space for stable neural learning and accurate scene reconstruction.
• The method leverages a Variational Auto-Encoder with point-cloud encoding to deliver dramatic improvements in PSNR, LPIPS, and cross-domain generalization across 3D applications.

Neural Gaussian Splatting (NGS) refers to the integration of learning-based methods, especially neural network models, with the explicit, differentiable 3D Gaussian Splatting framework that underpins state-of-the-art view synthesis and 3D scene representation pipelines. This approach addresses both representational and practical limitations of parameterizing scene radiance and geometry purely by the native parameters of 3D Gaussian Splatting (3DGS), enabling stable, robust, and high-fidelity learning and manipulation of complex 3D environments. The union of neural methods and Gaussian splats facilitates injective, homogeneous feature representations, improved generalization, and latent structure, overcoming the significant issues of non-uniqueness and parameter heterogeneity inherent in the standard parameterization.

1. Limitations of Parameter-Based Representations in 3D Gaussian Splatting

The canonical parameterization of 3DGS encodes each Gaussian splat as a tuple θ = (μ, q, s, c, o) consisting of position μ ∈ ℝ³, rotation q ∈ SO(3), anisotropic scale s ∈ (ℝ⁺)³, spherical-harmonic (SH) color coefficients c ∈ ℝ³×K, and opacity o ∈ ℝ. This native form, while compact and efficient for explicit modeling and fast differentiable rasterization, is problematic for learning systems and neural representations:

• Non-uniqueness ("embedding collisions"): Multiple distinct θ can generate the same rendered radiance field φG(x, d). Equivalences arise from symmetries in q (e.g., q ↦ –q), ellipsoidal symmetries, and couplings amongst SH coefficients and spatial parameters. Formally, ∃ θ₁ ≠ θ₂ such that φG(θ₁) = φG(θ₂).
• Numerical heterogeneity: The components of θ live on distinct manifolds (Euclidean, rotation group, spheres) and have radically different scaling sensitivities. For example, small perturbations in q can drastically alter the object orientation, while similar-magnitude shifts in higher-order SH have little visible effect.
• Learning instabilities: Standard MLP encoders and reconstruction losses such as ‖θ_pred – θ_targ‖ₚ treat these coordinates uniformly, which results in training instability, poor convergence, sensitivity to domain shift, and non-interpretable latent embeddings.

These fundamental mismatches motivate the search for alternative representations more amenable to neural processing.

2. Submanifold Field Embedding: Injective, Homogeneous Encodings

To resolve the above representational failures, the "Learning Unified Representation of 3D Gaussian Splatting" framework (Xin et al., 26 Sep 2025) introduces a submanifold field ("SF") encoding for 3D Gaussian primitives. This approach re-encodes each Gaussian, not by its raw analytic parameters, but as an object-surface field plus its color embedding:

• Geometric field: The iso-probability ellipsoid

$$\mathcal{M} = \{ x \in \mathbb{R}^3 \mid (x - \mu)^\top \Sigma^{-1} (x - \mu) = r^2 \}$$

where $Σ = R(q)\, \mathrm{diag}(s)^2\,R(q)^\top$.

• Color field: A view-dependent field on $\mathcal{M}$,

$$F(x) = \sigma(o)\cdot \mathrm{Color}(d_x),\quad d_x = \frac{x-\mu}{\|x-\mu\|}$$

with Color(·) synthesized from the SH expansion evaluated at the viewing direction.

• Injectivity: The mapping θ ↦ $\mathcal{E} = (\mathcal{M}, F) \in \mathcal{E}$ is injective: no two distinct θ corresponding to distinct radiance fields have the same surface+color field. Thus, $\mathcal{E}$ forms a structure-aware, unique, and Euclidean-compliant embedding.

This submanifold field representation yields a well-behaved input space for neural learning: the encoded manifold is invariant to parameter redundancies and undergirds stable neural approximation via established point-cloud network architectures.

3. Variational Auto-Encoder on Submanifold Fields

The practical instantiation of neural learning over submanifold fields employs a Variational Auto-Encoder (VAE):

• Discretization: Uniformly sample P points $\{x_m\}$ on $\mathcal{M}$, recording their corresponding colors $F(x_m)$ to form a colored point cloud $P = \{ (x_m, c_m) \}$.
• PointNet Encoder: $f(P) \to (\mu^z, \sigma^z)$, enabling latent reparameterization $z \in \mathbb{R}^D$ (typically D = 32).
• Decoder: Two implicit decoders, $$g_c: \mathbb{R}^3 \times \mathbb{R}^D \to \mathbb{R}^3$$ (geometry) and $$g_f: \mathbb{R}^3 \times \mathbb{R}^D \to \mathbb{R}^3$$ (color), deform a unit-sphere template to reconstruct both surface points and their colors.
• Reconstruction Loss: Wasserstein-2 distance between the source and reconstructed colored point clouds:

$$W_2^2(P, \hat{P}) = \min_{\Gamma} \sum_{i, j} \Gamma_{ij}[ \|x_i - y_j\|^2 + \lambda \|c_i - d_j\|^2 ]$$

• Full VAE Loss:

$$\mathcal{L} = \mathbb{E}_{z \sim q(z|P)} [ W_2^2(P, \hat{P}) ] + \beta \cdot \mathrm{KL}(q(z|P)\|\mathcal{N}(0, I))$$

• Invertibility: One can recover θ from a decoded manifold via PCA (for position and axes) and least squares (for SH), ensuring that the representation preserves the original rendering semantics.

4. Empirical Evaluation and Benchmarking

Integrating submanifold field embeddings into neural learning yields dramatic improvements across several axes of evaluation:

Metric/Experiment	Parametric baseline	SF-VAE (Field-based)
PSNR ShapeSplat (object-level)	≈37.5	≈63.4
LPIPS ShapeSplat	≈0.15	≈0.01
PSNR Mip-NeRF360 (scene-level)	≈18.8	≈29.8
LPIPS Mip-NeRF360	≈0.45	≈0.08
Cross-domain generalization (PSNR)	≈9.8	≈19.2

Further findings:

• Latent robustness: Reconstructions degrade gracefully under latent noise injection; parametric methods become unstable.
• Latent space semantics: Clustering SF-VAE embeddings cleanly separates foreground and background components; direct θ or parametric VAE embeddings mix these.
• Interpolation: Linear interpolation in SF-VAE latent space morphs one Gaussian to another smoothly. Direct θ-space interpolation causes discontinuous, jittery geometric transitions.

These results are obtained for SF-VAE with a latent dimension D=32 and P=144 sampled points, across random Gaussians and zero-shot tests on object and scene-level datasets.

5. Implications for Learning Systems and Downstream Applications

The unified neural embedding of Gaussian Splatting primitives introduced by submanifold field representations unlocks several benefits for learning systems:

• Model stability: Elimination of parameter ambiguity and numerical heterogeneity yields stable convergence and improved resistance to domain shift.
• Semantic structure: The latent space naturally encodes meaningful scene categories and supports semantic interpolation between geometric/appearance archetypes.
• Generalization and robustness: Superior cross-domain generalization is demonstrated, with markedly better PSNR and structural metrics in zero-shot transfer settings.
• Editing and synthesis: Since embeddings live in well-behaved Euclidean spaces, traditional learning, clustering, and generative modeling tools are directly applicable.
• Recoverability: The ability to invert from the SF-VAE latent back to the full analytic parameters θ means existing 3DGS renderers are strictly compatible with this new regime.

6. Connections with Related Work and Extensions

While this submanifold field approach is tailored for 3D Gaussian Splatting, the core principles are extensible to any explicit primitive-based 3D representation exhibiting parameter redundancy and numeric heterogeneity. The general recipe—injective surface field encoding, neural point-cloud aggregation, and metric learning via optimal transport—can inform hybrid explicit/implicit methods and latent-geometry architectures. Future directions plausibly include conditional generative modeling, semantic manipulation, and multi-modal learning scenarios, benefiting from the unique and homogeneous nature of the learned latent space.