View-Dependent Gaussian Splats

• View-dependent Gaussian splats are a 3D scene representation that uses angularly parameterized Gaussians to accurately model effects like specular highlights and anisotropic surfaces.
• Techniques extend classical Gaussian splatting by incorporating spherical and anisotropic Gaussians, view-dependent opacity matrices, and neural descriptors for efficient high-fidelity rendering.
• Empirical evaluations demonstrate improved rendering speeds and visual fidelity across benchmarks, making these methods effective for novel view synthesis and realistic scene reconstruction.

View-dependent Gaussian splats are a class of scene representations and rendering methodologies that enable accurate, efficient synthesis of novel views in 3D graphics and vision. These approaches extend classical 3D Gaussian splatting by parameterizing the appearance and opacity of each Gaussian as explicit functions of viewer direction, thereby allowing high-fidelity modeling of effects such as specular highlights, reflections, anisotropic surfaces, directional opacity, and uncertainty. The following sections present an organized survey of the principal methods, mathematical foundations, and empirical results in view-dependent Gaussian splatting.

1. View-dependent Color Modeling: Spherical Gaussians and Generalizations

A fundamental challenge in 3D Gaussian splatting (3DGS) is expressing per-Gaussian color as a function of viewpoint direction to correctly render appearance phenomena such as specular highlights. Traditional 3DGS encodes color using low-order spherical harmonics (SH), which are effective for diffuse or low-frequency variation but exhibit a poor tradeoff between parameter count and expressive power for complex, high-frequency effects.

SG-Splatting replaces SH color functions with one or more spherical Gaussians (SGs), which provide compact, high-resolution angular modeling (Wang et al., 31 Dec 2024). A single-lobe SG has the form

$$C(d; D, \alpha, \lambda, \mu) = D + \alpha \exp[\lambda(d \cdot \mu - 1)]$$

where $d$ is the unit sight direction, $D$ the diffuse offset, $\alpha$ the color amplitude, $\mu$ the SG axis, and $\lambda$ the concentration. For purely view-dependent color, $D$ may be omitted.

Advanced approaches introduce multiple SG lobes, with mean directions $\mu_i$ constrained to be mutually orthogonal, efficiently covering multi-directional specular structure. Organizing three SGs, each with learned amplitude and concentration, yields a parameter budget ($10$–$15$) drastically reduced relative to degree-$3$ SH ($48$ parameters). Orthogonality among $\mu_i$ axes is maintained via Gram–Schmidt in training.

Recent extensions employ mixed models combining SG and low-degree SH. For each Gaussian, color is given by

$$C(d) = SH_L(d) + \sum_{i=1}^{M} \alpha_i \exp[\lambda_i(d \cdot \mu_i - 1)]$$

where $L$ is adaptively chosen by splat size, and $M$ is typically $3$. This split captures low-frequency smoothness and high-frequency highlights with minimal redundancy (Wang et al., 31 Dec 2024).

Spec-Gaussian further generalizes the SG representation using anisotropic spherical Gaussians (ASGs), parameterized by three orthonormal axes and two sharpness (concentration) values, naturally encoding anisotropy in view-dependent reflectance (Yang et al., 24 Feb 2024). Small neural networks map per-Gaussian features and viewpoint to ASG parameters, enabling sharp and anisotropic highlight modeling without increasing the splat count.

2. View-dependent Opacity and Uncertainty

Beyond color, opacity and uncertainty are increasingly modeled as explicit view-dependent functions on each splat.

VoD-3DGS introduces a per-Gaussian, view-dependent opacity function,

$$\hat{\alpha}_i(\omega) = \sigma(\gamma_i + \omega^T \hat{S}_i \omega)$$

where $\omega$ is the viewing direction, $\gamma_i$ the density bias, and $\hat{S}_i$ a learned symmetric matrix (Nowak et al., 29 Jan 2025). This formulation enables directional suppression of Gaussians, critical for handling specular highlights and reflections. $\hat{S}_i$ is learned jointly with all other scene parameters, and regularization ensures view-consistency.

Uncertainty estimation as a view-dependent scalar feature is modeled in 3DGS by appending an additional low-order SH channel per Gaussian, encoding the confidence of appearance from a given direction (Han et al., 10 Apr 2025). The uncertainty function is

$$\sigma_i(\theta, \phi) = \sum_{l=0}^{L} \sum_{m=-l}^{l} c_{i,lm}^\sigma Y_{l,m}(\theta,\phi)$$

where $Y_{l,m}$ are SH basis functions. This channel is optimized via a loss that encourages low uncertainty for well-observed views and high uncertainty elsewhere. Integration incurs no runtime overhead beyond storage for SH coefficients.

In some hybrid models, an MLP is used to directly modulate both color and opacity as a function of viewing direction and geometric context, inheriting expressive power from neural radiance field (NeRF) paradigms (Malarz et al., 2023).

3. Feature-based and Image-based Splatting Architectures

Beyond analytical angular models, neural-feature-based and image-based architectures have been proposed to further ameliorate limitations in classical Gaussian splatting.

Latent-SpecGS replaces SH coefficients with compact neural descriptors per Gaussian ($16$-dimensional) (Wang et al., 23 Aug 2024). These descriptors are split into diffuse and specular codes, which are splatted as feature maps. Parallel CNN decoders reconstruct diffuse and specular colors from feature maps and viewpoint embeddings. A learned mask (from a small MLP, using SH-embedded view and normal direction) blends the two outputs into the final rendered image, yielding sharp highlights and correct anisotropy.

Image-Based Gaussian Splatting (IBGS) leverages high-resolution source images for residual learning (Nguyen et al., 18 Nov 2025). Base color is given by standard SH rasterization, and a CNN infers per-pixel color residuals from warped source images aligned via splat normals and depth consistency tests. This image-based residual enables high-frequency detail and complex view-dependence without excessive per-Gaussian parameterization or storage. IBGS employs multi-view pooling and deep convolutional decoding to produce spatially and angularly faithful renders.

4. Quantitative and Empirical Evaluation

Benchmarking across standard datasets (Mip-NeRF360, Tanks & Temples, Deep Blending, NeRF Synthetic, Shiny/Specular scenes) reveals tradeoffs in speed, memory, and visual fidelity.

SG-Splatting leads to $1.4$–$1.5\times$ speed-up over baseline 3DGS (e.g. $334$ FPS vs. $247$ FPS on Mip-NeRF360; model size $416$ MB vs. $781$ MB) with negligible PSNR/SSIM impact (Wang et al., 31 Dec 2024). VoD-3DGS exhibits minor slows ($\sim$20–50 FPS drop) and $100$–$200$ MB memory increases relative to 3DGS but achieves superior reflection and highlight quality (Nowak et al., 29 Jan 2025). On the Shiny dataset, Latent-SpecGS achieves maximum fidelity: PSNR up to $27.23$, SSIM $0.884$, and LPIPS $0.109$, outperforming Spec-Gaussian and baseline 3D-GS in challenging specular scenes (Wang et al., 23 Aug 2024). IBGS achieves state-of-the-art PSNR/SSIM/LPIPS across all novel-view synthesis benchmarks without storage bloat, but incurs a $\sim$2–3$\times$ compute slowdown due to deep residual inference (Nguyen et al., 18 Nov 2025).

Uncertainty modeling produces interpretable uncertainty maps with minimal cost (AUSE as low as $0.41$ in single-GPU/$20$ min training, vs. $0.11$ for ensembles/$169$ min), indicating scalable pose/novelty estimation (Han et al., 10 Apr 2025).

5. Integration, Practical Considerations, and Limitations

The majority of view-dependent splatting methodologies are designed as “plug-and-play” upgrades to core 3DGS pipelines. SG-Splatting, VoD-3DGS, and Latent-SpecGS require only minor per-Gaussian data-structure extensions (parameter slots for SG/ASG lobes, opacity matrices, latent codes) and simple subroutine replacements in render loops. Spec-Gaussian and Latent-SpecGS decoders add moderate GPU overhead; image-based splatting architectures incur higher latency and memory due to per-frame CNN inference.

Observed limitations include modest fidelity drops for SG-only models in highly glossy scenes, risk of overfitting (opacity matrices) in purely diffuse contexts, and extra runtime/memory overhead for deep feature/image-based decoders. Tuning of lobe count, SH-degree, regularization, and mask learning requires empirical verification. Support for dynamic or unbounded scenes is under active investigation (Wang et al., 23 Aug 2024, Nguyen et al., 18 Nov 2025).

6. Significance and Future Directions

View-dependent Gaussian splatting has substantially advanced the capability of scene representations for photorealistic rendering, novel view synthesis, asset extraction, and scene completion. High-resolution angular modeling via SG/ASG, directional opacity via matrix parameterization, and neural/image-based residual inference jointly yield representations that outperform classical methods in both qualitative and quantitative terms.

Future work aims to further compress per-splat parameterization (e.g. latent code sharing, neural distillation of image-based features), better disentangle lighting/BRDF parameters, accommodate dynamism and deformation, and integrate uncertainty fields for active view planning. Incorporation of physically based parameterizations (e.g. microfacet models) and next-level hierarchical/parallel computation architectures are considered promising for further gains in scalability and realism.

Model/Method	Parameterization	Core Innovation
SG-Splatting	Spherical Gaussians	Compact, fast view-dependent color
VoD-3DGS	Opacity Matrix ($\hat S$)	View-dependent suppression of splats
Spec-Gaussian	ASG/MLP	High-frequency, anisotropic appearance
Latent-SpecGS	Neural descriptor + CNN	Two-stage decoding, learned masking
IBGS	SH + Image-based CNN	Residual learning from source images
VDGS (NeRF-GS Hybrid)	MLP	Joint learning of color/opacity residuals