RayGaussX: Accelerating Gaussian-Based Ray Marching for Real-Time and High-Quality Novel View Synthesis (2509.07782v1)

Abstract: RayGauss has achieved state-of-the-art rendering quality for novel-view synthesis on synthetic and indoor scenes by representing radiance and density fields with irregularly distributed elliptical basis functions, rendered via volume ray casting using a Bounding Volume Hierarchy (BVH). However, its computational cost prevents real-time rendering on real-world scenes. Our approach, RayGaussX, builds on RayGauss by introducing key contributions that accelerate both training and inference. Specifically, we incorporate volumetric rendering acceleration strategies such as empty-space skipping and adaptive sampling, enhance ray coherence, and introduce scale regularization to reduce false-positive intersections. Additionally, we propose a new densification criterion that improves density distribution in distant regions, leading to enhanced graphical quality on larger scenes. As a result, RayGaussX achieves 5x to 12x faster training and 50x to 80x higher rendering speeds (FPS) on real-world datasets while improving visual quality by up to +0.56 dB in PSNR. Project page with videos and code: https://raygaussx.github.io/.

• The paper introduces novel algorithmic and system-level optimizations—such as empty-space skipping, adaptive sampling, and ray reordering—to accelerate both training and rendering.
• The paper demonstrates substantial empirical gains, achieving 5× to 12× faster training and 50× to 80× higher rendering FPS with up to +0.56 dB PSNR improvement.
• The paper provides a scalable framework that supports real-time, high-quality rendering in complex, large-scale scenes while maintaining efficient GPU utilization.

Accelerating Gaussian-Based Ray Marching for Real-Time and High-Quality Novel View Synthesis: The RayGaussX Approach

Introduction and Motivation

RayGaussX addresses the computational bottlenecks of volumetric Gaussian-based ray marching for novel view synthesis, building upon the RayGauss framework. While RayGauss achieves high-fidelity rendering by representing radiance and density fields with irregularly distributed elliptical Gaussians and leveraging a BVH for efficient intersection, its computational cost precludes real-time performance, especially on large-scale or outdoor scenes. RayGaussX introduces a suite of algorithmic and systems-level optimizations to enable real-time, high-quality rendering, with a focus on both training and inference acceleration, while also improving visual quality in challenging scenarios.

Core Contributions

RayGaussX introduces several key innovations:

• Empty-space skipping and adaptive sampling to minimize unnecessary ray samples in transparent regions and dynamically allocate computational effort where it most impacts image quality.
• Ray and primitive reordering to enhance memory access patterns and ray coherence, maximizing GPU parallelism and reducing warp divergence.
• Scale regularization to penalize highly anisotropic Gaussians, reducing false-positive BVH intersections and improving traversal efficiency.
• A novel densification criterion that corrects for distance bias, ensuring even density distribution of Gaussians across the scene, particularly benefiting large-scale and outdoor environments.

These contributions are summarized visually in the following figure.

Figure 1: Main contributions of RayGaussX approach to improve training and rendering speed.

Efficient Ray Sampling: Empty-Space Skipping and Adaptive Sampling

RayGaussX leverages the explicit spatial support of Gaussian primitives and the BVH structure to implement efficient empty-space skipping. By alternating between closest-hit and any-hit BVH queries, the algorithm rapidly advances rays through empty regions, only sampling where nonzero density is present. This is particularly effective for scenes with large transparent volumes, as it avoids unnecessary computation without impacting rendering quality.

Adaptive sampling is implemented at the segment level, with the step size $\Delta S_i$ for the $i$-th segment determined by both the transmittance $T_i$ and the distance to the camera $d_i$. The adaptive criterion

$$\Delta S_i = N_S \cdot \min\left(\max\left(\frac{d_i}{\beta}, \Delta t_{\min}\right) \cdot T_i^{-1/3}, \Delta t_{\max}\right)$$

allocates more samples to regions with high transmittance or close to the camera, and fewer samples to distant or highly occluded regions. This approach is computationally lightweight and compatible with the dynamic nature of scene optimization during training.

Ray Coherence and Memory Access Optimization

RayGaussX improves GPU efficiency by spatially reordering Gaussian primitives in memory using Z-order (Morton) curves, ensuring that spatially adjacent Gaussians are stored contiguously. This enhances memory coalescing during ray marching, as rays from the same image are likely to intersect nearby Gaussians.

Ray coherence is further improved by organizing rays in tiles rather than scanlines, reducing warp divergence in CUDA execution. The OptiX API's 2D grid launch is leveraged for initial grouping, and the combination of ray and primitive reordering yields significant throughput gains.

Scale Regularization for Anisotropic Gaussians

Highly anisotropic Gaussians result in large AABBs relative to their actual volume, increasing the number of false-positive BVH intersections and degrading performance. RayGaussX introduces an isotropic loss term that penalizes Gaussians with high AABB-to-ellipsoid volume ratios:

$$L_s = \frac{1}{N} \sum_{l}(\max(r_{\mathrm{max}, l}, r_0) - r_0)$$

where $r_{\mathrm{max}, l}$ is a rotation-invariant upper bound on the volume ratio, and $r_0$ is a threshold. This regularization is critical for maintaining traversal efficiency, especially in real-world scenes with complex geometry.

Distance-Aware Densification Criterion

The original RayGauss densification criterion is biased against distant Gaussians due to the smaller image-space gradient induced by world-space perturbations at large distances. RayGaussX introduces a corrective factor $\alpha_i = \|\mu_l - o_i\|/f$ to the gradient norm, ensuring that densification is not suppressed for distant primitives:

$$\frac{1}{I_l} \sum_{i=1}^{I_l} \alpha_i \|\nabla_{\mu_l} L_i\| > \tau$$

This adjustment leads to more uniform Gaussian coverage across the scene, directly improving rendering quality in outdoor and large-scale environments.

Empirical Results and Ablation

RayGaussX demonstrates substantial improvements over both RayGauss and state-of-the-art NVS baselines. On real-world datasets, RayGaussX achieves 5× to 12× faster training and 50× to 80× higher rendering FPS compared to RayGauss, with up to +0.56 dB PSNR improvement. Notably, RayGaussX approaches the rendering speed of rasterization-based methods (e.g., Spec-Gaussian) while delivering higher visual quality and avoiding rasterization artifacts such as flickering.

Ablation studies confirm that each contribution—empty-space skipping, adaptive sampling, Z-order reordering, ray coherence, isotropic loss, and the new densification criterion—provides measurable gains in speed and/or quality. The combination of all techniques enables interactive rendering (27 FPS at 1297×840 resolution) with state-of-the-art fidelity.

Implementation Considerations

RayGaussX is implemented atop the OptiX API, exploiting hardware-accelerated BVH traversal and custom intersection shaders for ellipsoidal Gaussians. The memory layout of primitives is periodically updated to maintain spatial locality. Adaptive sampling and empty-space skipping are integrated into the ray marching loop, with minimal overhead. The isotropic loss and densification criterion are incorporated into the training objective, with hyperparameters tuned for real-world datasets.

Resource requirements are dominated by GPU memory for storing Gaussian parameters and BVH structures. The approach scales well with scene complexity due to the logarithmic BVH traversal and the explicit empty-space skipping. For deployment, RayGaussX supports both offline and interactive rendering scenarios, with real-time performance achievable on high-end consumer GPUs.

Implications and Future Directions

RayGaussX demonstrates that volumetric ray marching with explicit Gaussian primitives can achieve real-time performance and high visual quality, challenging the dominance of rasterization-based splatting in the NVS domain. The explicit handling of empty space, adaptive sampling, and memory access optimization are broadly applicable to other particle-based and volumetric rendering frameworks.

The distance-aware densification criterion addresses a key limitation in prior work, suggesting that further improvements in scene representation and sampling strategies could yield additional gains, particularly for unbounded or highly non-uniform scenes. The isotropic loss highlights the importance of geometric regularization for efficient hardware utilization.

Potential future directions include integrating learned importance sampling, extending the approach to support dynamic scenes, and exploring hybrid representations that combine the strengths of volumetric and rasterization-based methods. The explicit, physically consistent formulation of RayGaussX also opens avenues for incorporating advanced light transport phenomena, such as participating media and global illumination, within the same framework.

Conclusion

RayGaussX advances the state of the art in Gaussian-based volumetric ray marching for novel view synthesis by introducing a set of targeted algorithmic and systems-level optimizations. These enable real-time, high-fidelity rendering across a range of scene types, with strong empirical results and practical implementation strategies. The approach provides a robust foundation for further research in efficient, high-quality neural rendering and scene representation.