RaySplats: Gaussian Ray-Traced Rendering

• RaySplats is a rendering approach that represents 3D scenes using anisotropic Gaussian primitives to accurately simulate lighting phenomena such as shadows, reflections, and refractions.
• It employs an advanced ray-tracing pipeline to compute exact ray-ellipsoid and ray-mesh intersections, enabling precise physical light transport and realistic shading effects.
• The method facilitates editable, differentiable scene manipulation and extends its applications beyond vision, including radiomap extrapolation with state-of-the-art performance.

RaySplats are a family of rendering and modeling methods that extend the 3D Gaussian Splatting (3DGS) paradigm, replacing the conventional rasterization pipeline with a ray-tracing-based approach acting directly on volumetric Gaussian primitives. This unlocks accurate modeling of lighting phenomena—such as shadows, reflections, and refractions—previously unattainable in standard 3DGS workflows, and enables seamless integration of Gaussian-based volumes with analytic mesh geometry. In RaySplats, each scene primitive is an anisotropic Gaussian (possibly "flattened" into disk-like ellipsoids) defined by a center, covariance, learned opacity, and color, with rendering achieved by computing exact ray-ellipsoid or ray-mesh intersections. RaySplats have facilitated advances in neural scene representation, editable volumetric tools, and extended physical simulation, with active research applying these techniques beyond vision, including to radio-frequency signal extrapolation.

1. Underlying Representation: Gaussian Primitives

RaySplats represents 3D scenes using a set of weighted, anisotropic Gaussians:

$$G = \{ (N(m_i, \Sigma_i), \hat{\alpha}_i, c_i) \}_{i=1}^N$$

where:

• $m_i \in \mathbb{R}^3$: Gaussian center,
• $\Sigma_i \in \mathbb{R}^{3 \times 3}$: positive-definite covariance, factorized as $\Sigma_i = R_i S_i S_i R_i^T$ with $R_i \in SO(3)$ and $S_i = \text{diag}(s_{i,1}, s_{i,2}, s_{i,3})$,
• $\hat{\alpha}_i \in [0,1]$: learned peak opacity,
• $c_i \in \mathbb{R}^3$: RGB color (as diffuse albedo, sometimes parameterized via spherical harmonics in extended settings).

A key distinction from rasterized 3DGS is that RaySplats treats each Gaussian as a true 3D ellipsoid or "confidence ellipsoid" corresponding to a quantile $Q$ of the multivariate Gaussian, only considering those points $x$ satisfying $(x - m_i)^T \Sigma_i^{-1} (x - m_i) \le Q$ as within the splat's effective spatial support (Byrski et al., 31 Jan 2025).

REdiSplats (HyConEx) introduces "flat" Gaussians by collapsing one principal axis to a very small value ($s_{i,1} = \varepsilon$), yielding mesh-approximated ellipses parameterized by polygons in local frames for exact ray-tracing intersection (Byrski et al., 15 Mar 2025).

2. Ray-Tracing-Based Rendering Core

RaySplats replaces the rasterization-based rendering loop of 3DGS with a full ray tracing pipeline. Primary rays generated from the camera are traced through a bounding-volume hierarchy (BVH) containing both mesh and Gaussian primitives (as ellipsoids or polygonal meshes). For each ray:

• Ray-ellipsoid intersection is solved analytically. Given $r(t) = o + t d$, the intersection with a Gaussian ellipsoid is the smallest $t \ge 0$ satisfying $(r(t) - m)^T \Sigma^{-1} (r(t) - m) = Q$. This is transformed into a unit-sphere quadratic for $t$ and solved robustly (Byrski et al., 31 Jan 2025).
• For flat (polygonal) Gaussians, intersection is performed via classical ray-triangle methods (e.g., Möller–Trumbore), traversing each mesh Gaussian as its triangulated disk (Byrski et al., 15 Mar 2025).
• At each Gaussian hit, the opacity is evaluated as:

$$\alpha_i \approx \hat{\alpha}_i \exp\left(-\frac{1}{2}\| \Sigma_i^{-1/2}(p - m_i) \|^2\right)$$

where $p$ is the intersection point (Byrski et al., 31 Jan 2025, Byrski et al., 15 Mar 2025).

• Color compositing follows front-to-back alpha blending: $C \leftarrow C + T \alpha_i c_i$, $T \leftarrow T (1 - \alpha_i)$, terminating when $T$ falls below a small threshold (Byrski et al., 31 Jan 2025, Byrski et al., 15 Mar 2025).

Because both meshes and Gaussians are traversed in the same BVH, all classical ray tracing phenomena—shadows (via shadow rays), reflections and refractions (via recursive rays), analytic material BRDFs—are supported without modification (Byrski et al., 31 Jan 2025).

3. Lighting, Shading, and Physical Effects

RaySplats allows physically correct lighting computation at any intersection, leveraging standard BRDFs (e.g., Lambertian, Phong, GGX). Normals at Gaussian intersections can be estimated from the covariance structure ($\Sigma_i$) or, for mesh-approximated flats, taken from mesh triangles. The full rendering equation is sampled by launching secondary rays, enabling per-pixel effects such as shadows, inter-reflections, and environment illumination (Byrski et al., 31 Jan 2025, Byrski et al., 15 Mar 2025).

Because all primitives are explicit geometry (meshes or ellipsoids), RaySplats supports exporting to standard graphics tools (OBJ, PLY, GLTF), with each Gaussian mapped to a mesh patch carrying per-vertex color and opacity. Integration with physics engines is direct, as mesh vertices may be manipulated and physical properties assigned (Byrski et al., 15 Mar 2025).

4. Editable and Differentiable Scene Manipulation

REdiSplats extends RaySplats with a mesh-based editing pipeline, enabling direct, local adjustment of Gaussian parameters by moving mesh vertices in 3D modeling tools. Modifying a flat Gaussian's ellipse is accomplished by tracking which control points (vertices) are moved and reconstructing the corresponding rotation and scale matrices, maintaining geometric and statistical consistency (Byrski et al., 15 Mar 2025).

End-to-end differentiability is preserved: by rasterizing mesh-based Gaussians using tools such as Nvdiffrast, gradients can be routed from the output image back to underlying Gaussian parameters, enabling both optimization from images and real-time interactive editing (Byrski et al., 15 Mar 2025).

5. Applications Beyond Vision: Radiomap Extrapolation

RaySplats-style splatting underpins RadSplatter, extending the method to electromagnetic radiomap extrapolation from sparse, noisy measurements (Wang et al., 18 Feb 2025). Here, each scatterer is represented as an anisotropic 3D Gaussian with complex attenuation and beam-directional gain, parameterized by spherical harmonic coefficients. RadSplatter introduces a relaxed-mean (RM) reparameterization to robustly extract Gaussian centers from point clouds, alongside a learned MLP-based 3D-to-2D projection that directly maps Gaussians to radio beam grids (no camera matrices).

Electromagnetic splatting accumulates complex multipath contributions for each grid cell, and the loss function incorporates both data fidelity and structural regularization. Experiments on synthetic and real datasets show RadSplatter achieving state-of-the-art mean absolute error (e.g., 7.56 dB on synthetic Shanghai data, 7.04 dB on real Hangzhou), along with real-time inference speed, outpacing reference methods by up to 30× (Wang et al., 18 Feb 2025).

6. Quantitative Evaluation and Limitations

RaySplats demonstrates strong competitive performance on standard 3D scene benchmarks, matching or surpassing prior 3DGS variants:

Benchmark	3DGS-30K	RaySplats	Best result
Mip-NeRF360	27.21 dB	27.31 dB	RaySplats
Tanks & Temples	23.14 dB	22.20 dB	3DGS-30K
Deep Blending	29.41 dB	29.57 dB	RaySplats

RaySplats achieves interactive rendering rates on typical GPUs (OptiX 8.0), with overheads within a factor of 2 relative to rasterization. Limitations include the requirement for ray-tracing hardware and slightly increased per-pixel costs, mitigated through early-exit heuristics. Shading on Gaussians is basic; future work may explore learned view-dependent emission and richer BRDFs (Byrski et al., 31 Jan 2025).

7. Impact and Future Research

RaySplats and its variants (including REdiSplats/HyConEx and RadSplatter) have advanced the intersection of neural rendering, analytic geometry, and physics-based simulation. By bridging mesh representations with volumetric splatting in a unified ray-tracing core, they enable both photorealistic rendering and cross-modal applications such as electromagnetic field estimation. Ongoing research directions include advanced light transport (participating media, full path tracing atop Gaussian primitives), extension to new sensing domains, and more expressive latent representations within each primitive (Byrski et al., 31 Jan 2025, Byrski et al., 15 Mar 2025, Wang et al., 18 Feb 2025).