RaySplats: Ray Tracing Gaussian Splatting

• The paper introduces a novel integration of ray tracing with 3D Gaussian Splatting, enabling physically based rendering with accurate shadows, reflections, and refractions.
• It employs tailored acceleration structures and ray–Gaussian intersection algorithms to scale efficiently on GPU hardware, managing millions of primitives.
• The framework supports differentiable optimization for inverse rendering and material inference, driving advances in real-time novel-view synthesis and editing.

RaySplats is a class of rendering frameworks that incorporates ray tracing directly into the 3D Gaussian Splatting (3DGS) paradigm, enabling physically-based effects such as shadows, reflections, refractions, and complex indirect illumination in particle‐based or point cloud scene representations. This integration leverages acceleration structures and specialized ray-Gaussian intersection algorithms, extending Gaussian Splatting from efficient rasterization to physically plausible image synthesis, volumetric effects, and advanced editing scenarios. RaySplats frameworks support differentiable optimization pipelines and are compatible with existing GPU ray tracing hardware.

1. Gaussian Splatting: Data Representation and Motivation

RaySplats builds upon the 3DGS representation, where a scene is modeled as a collection of anisotropic 3D Gaussian primitives, each defined by a center $\mu \in \mathbb{R}^3$, a covariance $\Sigma \in \mathbb{R}^{3\times 3}$ (parameterized via rotation $R\in SO(3)$ and diagonal scaling $S=\operatorname{diag}(s_{x},s_{y},s_{z})$), per-primitive opacity $\alpha$, and view-dependent color, typically as spherical harmonics coefficients (Moenne-Loccoz et al., 9 Jul 2024, Byrski et al., 31 Jan 2025). The core density function for a Gaussian at point $x$ is:

$\phi(x) = \exp[-(x-\mu)^\top \Sigma^{-1} (x-\mu)]$

RaySplats replaces classical screen-space rasterization with direct world-space ray tracing, allowing for physically-based effects and consistent ray-level compositing. By design, the representation is amenable to both rasterization and ray-based volume rendering, and can encode material properties such as albedo and roughness for PBR applications (Gu et al., 20 Dec 2024, Zhang et al., 13 Oct 2025).

2. Acceleration Structures and Ray–Primitive Intersection

To efficiently handle large numbers of semi-transparent Gaussian particles, RaySplats frameworks construct bounding volume hierarchies (BVH) over either triangle meshes that tightly approximate each Gaussian’s extent or direct ellipsoidal AABBs (Moenne-Loccoz et al., 9 Jul 2024, Byrski et al., 15 Mar 2025, Sun et al., 9 Apr 2025). Common practice involves:

• Encapsulating each Gaussian with a tight proxy mesh (e.g., stretched icosahedron or octagon), sized to enclose the support where $\phi(x) \geq \phi_{\min}$.
• Building a SAH-guided, top-down BVH over all proxy meshes or triangles, facilitating high-performance GPU ray tracing.
• In frameworks such as REdiSplats, flat (2D) Gaussians are attached to mesh vertices and represented as polygons in their local tangent plane for efficient intersection (Byrski et al., 15 Mar 2025).

Ray–Gaussian intersection is realized through either analytical ray–ellipsoid intersection (solving a quadratic for $t$) or ray–triangle intersection over proxy meshes. This approach leverages native hardware support for ray–triangle intersections and yields scalability to millions of primitives with efficient traversal and hit recording.

3. Rendering Algorithm and Volume Compositing

RaySplats employs a slab-based or k-buffered volume rendering pipeline:

• For each ray, intersections with up to $k$ proxy meshes are collected and sorted in depth order.
• At each intersection, the algorithm computes the sample position maximizing the Gaussian kernel along the ray, evaluates local density, computes particle opacity, and accumulates view-dependent color via alpha compositing:

$$L = \sum_{i=1}^N T_{i-1} \cdot \alpha_i \cdot c_i, \quad T_i = \prod_{j=1}^i (1-\alpha_j)$$

• The pipeline can be implemented with “any-hit” and “closest-hit” OptiX programs, utilizing slab-wise traversal to avoid full per-sample re-traversal (Moenne-Loccoz et al., 9 Jul 2024, Byrski et al., 31 Jan 2025).
• Stochastic RaySplats (Sun et al., 9 Apr 2025) advances this by sampling only a single intersection per ray through Russian roulette, yielding unbiased Monte Carlo estimators and drastically reducing memory and sorting requirements.

Extensions include generalized kernel choices (e.g., $n$-order or cosine-modulated Gaussians), surface kernels, and differentiable backpropagation by replaying the same ray traversal in the backward pass (Moenne-Loccoz et al., 9 Jul 2024).

4. Lighting, Shadows, Reflections, and Indirect Effects

RaySplats renderers natively support:

• Secondary rays (e.g., for hard/soft shadows, reflection, refraction) via recursive BVH traversal and secondary ray spawning.
• Ray-geometry unification, allowing mixed mesh and particle scenes within a single BVH infrastructure (Byrski et al., 31 Jan 2025, Moenne-Loccoz et al., 9 Jul 2024).
• Physically-based shading models, including support for the Disney BRDF and integration with environment maps for both direct and indirect illumination (Gu et al., 20 Dec 2024, Zhang et al., 13 Oct 2025).
• Indirect lighting and visibility are handled by tracing rays from shading points using the same Gaussian-dense BVH, aggregating both occlusion and secondary radiance contributions.

In MaterialRefGS (Zhang et al., 13 Oct 2025), this paradigm is extended to deferred PBR rendering with multi-view consistent material inference, estimating per-splat metallicity and roughness, enforcing photometric and material consistency across views, and integrating both direct and indirect radiance via ray tracing through the 2D Gaussian splats.

5. Differentiable Optimization and Inverse Rendering

RaySplats pipelines are universally designed for end-to-end differentiability. Forward passes (ray intersection, hit sorting, compositional color accumulation) are replayable in the backward pass, enabling efficient gradient accrual with atomic operations. This underpins their use in inverse rendering and scene relighting:

• IRGS (Gu et al., 20 Dec 2024) demonstrates differentiable 2D Gaussian ray tracing for accurate indirect radiance modeling, showing improved convergence, photometric reconstruction (~35.5 dB PSNR at $N_r=256$ samples), and improved albedo accuracy over rasterization-based methods.
• Training proceeds via large-batch SGD with random mini-batch rays, loss terms on photometry, normal consistency, occupancy, and material smoothness.
• Efficient BVH rebuilds and analytic gradient formulas ensure scalability to rapid scene optimization and dynamic content (Moenne-Loccoz et al., 9 Jul 2024).

6. Applications, Extensions, and Performance

RaySplats unlocks new capabilities for Gaussian Splatting-based representations:

• Real-time novel-view synthesis with physically plausible lighting, even at HD to 4K resolutions ($20$–$78$ FPS on RTX hardware; memory use $\sim$50%–100% that of 3DGS) (Moenne-Loccoz et al., 9 Jul 2024, Byrski et al., 31 Jan 2025).
• Inverse rendering for photorealistic relighting, multi-view consistent material property estimation, and environment-aware effects.
• Hybrid editing and simulation: mesh-compatible representations enable manual control (vertex adjustment), export to Blender/Nvdiffrast, and integration with physics engines for dynamic or physically-driven scenes (Byrski et al., 15 Mar 2025).
• Ray matrixes seamless composition with mesh-based objects and allows instancing for efficient scene duplication.
• Stochastic strategies favor high parallelism and reduced memory—supporting both high-end and low-end hardware (Sun et al., 9 Apr 2025).

Limitations include increased complexity relative to rasterization, higher per-frame computation times (particularly for multi-bounce illumination or dense volumetric effects), and challenges in modeling thin or highly anisotropic Gaussians without shape artifacts (Byrski et al., 15 Mar 2025). Emerging research addresses hybrid representations, adaptive polygon granularity, and hardware quantization.

7. State of the Art and Future Research Directions

The RaySplats paradigm has yielded state-of-the-art results for both view synthesis and photometric/physical realism:

Study	Dataset	SSIM	PSNR (dB)	LPIPS	FPS	Features
RaySplats (Byrski et al., 31 Jan 2025)	MipNeRF360, T&T, DeepBlending	0.846	27.3	0.237	20–40	Shadows, reflections, mesh integration
RaySplats (Moenne-Loccoz et al., 9 Jul 2024)	MipNeRF360 (“HD”)	≈0.85	~27	–	55–78	Secondary rays, dof, stochastic sampling
REdiSplats (Byrski et al., 15 Mar 2025)	T&T, MipNeRF360, DeepBlending	0.82–0.91	22–30	0.21–0.32	20–33	Real-time editing, PBR, physics/BVH
IRGS (Gu et al., 20 Dec 2024)	Synthetic4Relight	–	35.5	–	~1/syn	Accurate indirect, differentiable GRT
Stochastic RaySplats (Sun et al., 9 Apr 2025)	3DGS assets	–	–	–	75–110	MC sampling, low register, path tracer

Continued research focuses on scalable ray traversal for larger ($>1$M-particle) scenes, dynamic and non-rigid content, multi-level LOD, better modeling of volumetric scattering, importance-sampled MC integration, adaptive mesh proxies, and efficient global optimization in differentiable frameworks. RaySplats positions the 3DGS family as a robust, extensible foundation for physically-driven neural rendering and hybrid 3D scene technologies.

---

References:

(Moenne-Loccoz et al., 9 Jul 2024, Byrski et al., 31 Jan 2025, Gu et al., 20 Dec 2024, Byrski et al., 15 Mar 2025, Sun et al., 9 Apr 2025, Zhang et al., 13 Oct 2025)