Learning View-Dependent Splatting Kernels

Abstract: We present a differentiable framework to automatically learn view-dependent 2D kernels in a splatting-based pipeline to improve reconstruction quality and representation efficiency for novel 3D view synthesis. Our volumetric primitive is defined as a bounding ellipsoid and a 3D-kernel latent vector. We first learn a projection network to output a 2D-kernel latent, taking the attributes of the ellipsoid and the 3D-kernel latent as input. Next, the result is sent to a decoder to produce a radially symmetric 2D kernel in terms of Mahalanobis distance, bounded by the projected ellipsoid. The neural networks along with per-primitive attributes are jointly optimized. The effectiveness of our approach is demonstrated on standard benchmarks, comparing favorably against state-of-the-art techniques on both analytical and learned kernels. Finally, we extend the idea to learn general 2D kernels for 2D splatting as well as image representation.

• The paper proposes a novel differentiable framework for automatically discovering view-dependent splatting kernels that overcome limitations of fixed analytical methods.
• The methodology employs dual MLPs to project and decode kernel latent codes, creating adaptive kernel shapes tailored to specific view conditions.
• Empirical evaluations show that the approach delivers improved reconstruction quality and efficient representation, achieving superior PSNR, SSIM, and LPIPS metrics across benchmarks.

Learning View-Dependent Splatting Kernels: An Authoritative Summary

Problem Formulation and Motivation

The paper "Learning View-Dependent Splatting Kernels" (2605.25426) introduces a novel differentiable framework for discovering view-dependent splatting kernels, aiming to enhance both reconstruction quality and representation efficiency in neural radiance field rendering pipelines. The underlying motivation is to surpass the expressiveness limitations imposed by analytical kernels—such as Gaussians, Student's t, Beta, and B-splines—in 3D or 2D splatting approaches. Prior state-of-the-art methods define volumetric primitives with fixed, view-independent kernel shapes, restricting their adaptability to scene geometry and view conditions. This work directly confronts two open questions: the search for optimal splatting kernels and the development of a mechanism for their automatic discovery, addressing both with a unified, data-driven strategy.

Methodology: Unified Differentiable Framework

Volumetric Primitive Parameterization

Each learned primitive consists of:

• A 3D bounding ellipsoid, parameterized by position, scale, and rotation,
• A kernel latent code ($\mathbf{z}_{3D}$) serving as a compact, learnable representation for the primitive's density profile.

Neural Projection

The pipeline first projects the ellipsoid onto the image plane, yielding a 2D elliptical footprint via affine transforms. Crucially, it deploys a global, lightweight MLP ($\Phi_{\operatorname{proj}}$) that consumes the kernel latent code, spatial, scale, and orientation attributes (all transformed into camera space), outputting a 2D kernel latent ($\mathbf{z}_{2D}$) customized for each view.

Figure 1: The pipeline for splatting volumetric primitives: attributes are projected, and a learned MLP transforms kernel latents for view-aware kernel decoding.

Kernel Decoder and Splatting

A second MLP ($\Phi_{\operatorname{dec}}$) uses the 2D kernel latent and the Mahalanobis distance in elliptical screen space to synthesize a radially symmetric opacity profile for splatting. To accelerate rendering, the kernel is pre-sampled across $k$ points, with opacity values interpolated during rasterization. This design decouples per-pixel calls from expensive neural evaluations, allowing scalability with primitive count and memory footprint.

Extension to Planar Primitives

The framework generalizes to 2D splatting. Planar primitives are defined by a bounding ellipse and a higher-dimensional kernel latent, projected with an analogous MLP, and decoded (with increased complexity) to produce general, potentially asymmetric kernels, permitting both 2D image and surface representation.

Empirical Evaluation and Results

Reconstruction Quality

The method is evaluated on four benchmarks (Mip-NeRF360, Tanks & Temples, DeepBlending, NeRF Synthetic), comparing both volumetric and planar variants against leading analytical and learned primitives. Across PSNR, SSIM, and LPIPS, the approach consistently ranks best or second-best for all metrics, with strong quantitative gains attributable to its adaptive kernel learning mechanism.

Figure 2: Qualitative comparisons showing output views from multiple state-of-the-art approaches versus the learned kernels, including error metrics.

The view-dependency of kernel shapes yields robust improvements, especially in scenes characterized by sharp transitions, high-frequency textures, or challenging occlusions—factors that degrade performance for fixed-kernel methods.

Representation Efficiency and Scalability

Matching memory consumption across competing pipelines, the learned primitives achieve higher PSNR with fewer primitives, demonstrating superior geometric expressiveness and efficiency.

Figure 3: Comparative renderings and error statistics under varying memory footprints and kernel sample counts, highlighting scalability and quality.

Increasing kernel sample count ($k$) enhances expressiveness up to a threshold, beyond which diminishing returns are noted; this is attributed to shrinking screen-space primitive sizes with higher quantities, simplifying kernel profiles.

Kernel Profile Analysis

The paper provides analyses and visualizations illustrating how kernel profiles adapt across views and scenes, distinctly diverging from the static profiles of Gaussians, Beta, or Student's t kernels.

Figure 4: Visualization of kernel profiles and corresponding views, demonstrating learned view-dependency and adaptability.

Adaptation is especially pronounced in specialized scene classes (e.g., hair or vegetation), where similar distributions are learned, suggesting the approach's suitability for compressing or tailoring representations to domain-specific tasks.

Extension to 2D Image Representation

The method also outperforms analytical Gabor Splatting in high-fidelity image representation benchmarks, delivering lower reconstruction error with similar memory footprint, facilitated by its general (non-radial) kernel learning.

Ablation Studies

Systematic ablations on kernel latent dimension, decoder/projection MLP width, pre-training shape (cosine vs. Gaussian vs. polynomial), and input configuration deliver insights into balancing quality and footprint. The combination of spatial, scale, and orientation-aware input for the projection MLP consistently improves output metrics. Pre-training on cosine shapes yields optimal performance.

Limitations and Future Directions

The main practical limitation is slower rendering speed, due primarily to neural decoding overhead. The current pipeline can be accelerated via distillation, weight pruning, or analytical approximation, as well as improved anti-aliasing. Extensions to positional/frequency encoding and pipeline auto-design (e.g., RenderFormer) may further enhance expressiveness and generalization. The method's ability to automatically learn domain-specialized kernel distributions opens avenues for scene-specific compression and generative modeling in graphics.

Conclusion

This paper delivers a unified, differentiable framework for learning view-dependent splatting kernels, providing both theoretical and empirical advancement in image-based rendering. By leveraging learnable kernel latents and neural projection/decoding, the approach achieves improved reconstruction quality, representation efficiency, and scalability across scene and view domains. The implications for neural graphics research are multifold: automatic discovery of optimal primitives, adaptive specialization for scene classes, and potential integration with broader rendering and relighting tasks.