Anisotropic 3D Gaussians

• Anisotropic 3D Gaussians are parametric models that encode directional variances using covariance matrices to capture ellipsoidal shapes and spatial correlations.
• They employ eigen-decomposition and neural network-based parameterization to accurately define scale and orientation for applications in graphics and statistics.
• Their practical applications include real-time neural rendering, non-stationary Gaussian random fields, and enhanced modeling of specular effects in diverse domains.

Anisotropic 3D Gaussians are a class of parametric models in which the uncertainty, density, or appearance at a point in three-dimensional space is described by a Gaussian distribution whose spatial properties vary directionally. Unlike isotropic Gaussians, which exhibit equal variance in all spatial directions, anisotropic 3D Gaussians encode their orientation and scale via a symmetric positive-definite covariance matrix or, in some applications, via directional lobes. These formulations enable the modeling of ellipsoidal shapes and directionally dependent correlations, intensities, or appearance phenomena, with applications in probabilistic graphics, spatial statistics, and machine learning.

1. Mathematical Formulations of Anisotropic 3D Gaussians

The core representation of an anisotropic 3D Gaussian is as a density function:

$$G(x) = \exp\left(- (x-\mu)^\top \Sigma^{-1} (x-\mu) \right)$$

where $\mu \in \mathbb{R}^3$ is the mean and $\Sigma \in \mathbb{R}^{3\times3}$ is a symmetric, positive-definite covariance matrix. The eigenvalues of $\Sigma$ determine the squared radii along principal axes, while the eigenvectors determine orientation. Eigen-decomposition $\Sigma = U\Lambda U^\top$ (with $U \in SO(3)$ and diagonal $\Lambda$) provides a parameterization amenable to learning and interpretation. This framework is used to represent 3D parts in object-centric methods such as GaussiGAN (Mejjati et al., 2021), where per-part covariance controls the scale and rotational alignment of semantic object components.

A distinct extension is the anisotropic spherical Gaussian (ASG) kernel for appearance modeling:

$$\mathrm{ASG}(\nu \mid [x, y, z], [\lambda, \mu], \xi) = \xi \cdot S(\nu; z) \cdot \exp\big(-\lambda (\nu \cdot x)^2 - \mu (\nu \cdot y)^2\big)$$

with $\nu \in \mathbb{R}^3$ a unit query direction (often the reflection direction), $[x, y, z]$ an orthonormal frame, $\lambda$, $\mu$ positive concentration parameters, and $\xi$ an amplitude (Yang et al., 24 Feb 2024). When $\lambda \ne \mu$, anisotropy arises—concentration and sharpness differ along the $x$ ("tangent") and $y$ ("bitangent") axes orthogonal to the lobe peak $z$.

2. Parameterization and Learning Mechanisms

In the context of geometry-aware learning or view-dependent graphics, parameterizing anisotropy efficiently is crucial. For geometric proxy models, each anisotropic Gaussian is defined by predicting the eigenvalues (for scale) and orthonormal eigenvectors (for orientation) for each object part using neural networks, ensuring positive definiteness through bounded activations and orthonormalization (Mejjati et al., 2021).

For appearance modeling, as in Spec-Gaussian, each 3D Gaussian is assigned a local feature vector $f \in \mathbb{R}^{24}$, from which multiple ASG lobe parameters $\{\lambda_i, \mu_i, \xi_i\}_{i=1..N}$ are generated via a multilayer perceptron (MLP) conditioned on the view direction. This produces a flexible, high-frequency, and view-dependent response, essential for accurate specular and anisotropic effects (Yang et al., 24 Feb 2024).

Large-scale statistical models of spatial phenomena, such as non-stationary Gaussian random fields (GRFs), use spatially varying anisotropy encoded in a matrix field $H(x)$, parameterized as

$$H(x) = \gamma(x) I_3 + v(x) v(x)^\top + \omega(x) \omega(x)^\top$$

with $\gamma(x)>0$, $v(x)$, and $\omega(x)$ as smooth scalar and vector fields, allowing locally ellipsoidal correlation (Berild et al., 2023).

3. Applications Across Disciplines

Computer Graphics and Neural Rendering:

Anisotropic 3D Gaussians are foundational in 3D Gaussian Splatting (3D-GS), which uses millions of Gaussians for real-time photo-realistic rendering through GPU rasterization. Spec-Gaussian replaces the standard low-frequency SH-based appearance model with per-Gaussian ASG fields, achieving superior modeling of specular and directional phenomena without increasing the Gaussian count. The color at each pixel is composed as $c = c_d + c_s$, with $c_s$ predicted by ASG-based MLPs (Yang et al., 24 Feb 2024). GaussiGAN leverages anisotropic 3D Gaussians as a part-based, interpretable geometric proxy for controllable, multi-view consistent object synthesis from 2D silhouettes (Mejjati et al., 2021).

Spatial Statistics and Environmental Modeling:

Anisotropic 3D Gaussians form the core building block of Gaussian random fields (GRFs) used for modeling spatial processes in environmental and geophysical domains. Via SPDEs, spatially varying anisotropy is incorporated for realistic covariances, with the matrix $H(x)$ controlling local directionality of correlation and range parameter $\kappa(x)$ controlling local length-scale (Berild et al., 2023). Such parametric GRFs outperform stationary models in real-world tasks like ocean salinity prediction.

4. Computational Schemes and Optimization Strategies

Differentiable Rendering and Projection:

In geometry-centric deep learning pipelines, 3D anisotropic Gaussians are analytically projected via the camera model to 2D, with closed-form expressions for projected mean and 2D covariance, yielding soft 2D mask or image components preserving differentiability for learning (Mejjati et al., 2021).

Coarse-to-Fine Training and Anchor-Based Growth:

To efficiently scale high-capacity 3D-GS models, an anchor-based algorithm clusters Gaussians into voxels, spawning neural Gaussians adaptively based on the local data and using regularization to avoid excessive overlap. Coarse-to-fine rendering resolution schedules are critical to avoid “floater” artifacts and overfitting, with empirical ablations demonstrating a $>90\%$ reduction in floaters and significant storage savings versus per-Gaussian parameterization (Yang et al., 24 Feb 2024).

SPDE-Based GMRF Discretization:

In spatial statistics, the SPDE $$(\kappa(x)^2 - \nabla \cdot [H(x)\nabla])^{\alpha/2} u(x) = W(x)$$ enables construction of GRFs. Discretization on a regular grid yields a Gaussian Markov random field (GMRF) with sparse precision matrix, scalable to large data volumes (Berild et al., 2023). Parameters are efficiently estimated by marginal likelihood maximization leveraging analytic gradients and sparse linear algebra.

5. Empirical Properties and Comparative Results

Spec-Gaussian demonstrates increased PSNR (by $\sim$0.8 dB on NeRF synthetic, 1.3 dB on NSVF synthetic with high specularity) and lower LPIPS and SSIM error compared to traditional SH appearance models, all while maintaining real-time inference speeds (70–130 FPS on an NVIDIA 3090 GPU) (Yang et al., 24 Feb 2024). Ablations indicate that direct use of ASG, without MLP feature decoupling, produces visually implausible specular highlights, while the anchor+coarse-to-fine regime is essential for artifact suppression.

In non-stationary GRF modeling for 3D phenomena such as ocean physics, spatially varying anisotropy leads to lower RMSE and CRPS for prediction of unobserved locations, outperforming stationary priors, particularly with limited observational coverage (Berild et al., 2023).

In self-supervised 3D structure learning, GaussiGAN’s retention of an explicit, interpretable scale+rotation representation for each object part facilitates disentanglement of shape, pose, and camera, as demonstrated in multi-view synthesis and mask generation tasks (Mejjati et al., 2021).

6. Limitations and Potential Extensions

The computational overhead for anisotropic 3D Gaussian models is higher than their isotropic or SH-based counterparts due to per-Gaussian MLPs and ASG lobe evaluations (∼2–4× for Spec-Gaussian), managed via parameter sharing and early culling (Yang et al., 24 Feb 2024). Current approaches cannot capture true mirror-like global reflections, due to lack of explicit mesh or environment models. In statistical modeling, fitting non-stationary, high-parameter anisotropic SPDEs requires extensive data coverage to avoid overfitting (Berild et al., 2023).

Future research directions include integration of per-Gaussian environment probes for dynamic or relightable appearance, Bernstein-basis compression to reduce memory usage, and incorporation of ground-truth surface cues to separate specular from reflective phenomena (Yang et al., 24 Feb 2024). In spatial statistics, further regularization and exploitation of domain-specific structure may enable reliable high-resolution fitting in data-scarce regimes (Berild et al., 2023).