Radiant Foam: Real-Time Differentiable Ray Tracing (2502.01157v1)

Abstract: Research on differentiable scene representations is consistently moving towards more efficient, real-time models. Recently, this has led to the popularization of splatting methods, which eschew the traditional ray-based rendering of radiance fields in favor of rasterization. This has yielded a significant improvement in rendering speeds due to the efficiency of rasterization algorithms and hardware, but has come at a cost: the approximations that make rasterization efficient also make implementation of light transport phenomena like reflection and refraction much more difficult. We propose a novel scene representation which avoids these approximations, but keeps the efficiency and reconstruction quality of splatting by leveraging a decades-old efficient volumetric mesh ray tracing algorithm which has been largely overlooked in recent computer vision research. The resulting model, which we name Radiant Foam, achieves rendering speed and quality comparable to Gaussian Splatting, without the constraints of rasterization. Unlike ray traced Gaussian models that use hardware ray tracing acceleration, our method requires no special hardware or APIs beyond the standard features of a programmable GPU.

• The paper introduces a novel framework that leverages a differentiable Voronoi volumetric mesh to model neural radiance fields and enable gradient-based optimization.
• It reformulates volume rendering as an exact summation over piecewise constant fields, eliminating the need for complex importance sampling.
• The work adapts an efficient ray traversal algorithm for real-time performance without specialized hardware while dynamically optimizing the mesh.

The paper presents a novel framework for real-time differentiable ray tracing that exploits a volumetric mesh representation, parameterized as a 3D Voronoi diagram, to model neural radiance fields. The approach is formulated to combine the efficiency of rasterization-based methods (such as 3D Gaussian Splatting) with the flexibility inherent in ray tracing for accurate simulation of light transport phenomena, including reflections, refractions, and complex camera models.

The core contributions and technical details of the work are as follows:

• Volumetric Mesh Representation and Continuity The method departs from the traditional use of discrete representations by leveraging a differentiable formulation of a volumetric mesh. Instead of directly optimizing a Delaunay triangulation—which suffers from discontinuities due to discrete edge flips—the paper employs the dual Voronoi diagram. This representation partitions 3D space into convex polyhedral cells, each corresponding to a unique Voronoi site. Importantly, while the underlying Delaunay connectivity changes discretely, these changes only occur when cell faces collapse to zero surface area, ensuring that the overall geometry (and subsequently the radiance field) varies continuously with the positions of the sites. This property is instrumental in facilitating stable gradient-based optimization.
• Exact Volume Rendering with Piecewise Constant Fields The authors adopt a volume rendering formulation where the density ($\sigma$) and radiance ($c$) are assumed to be constant within each cell. This assumption permits a reformulation of the continuous volume rendering integral into an exact summation over segments:

$$c_r = \sum_{n=1}^{N} T_n \bigl(1 - \exp(-\sigma_n \delta_n)\bigr) c_n,$$

where: - $c_r$ is the observed color along a ray, - $\delta_n$ represents the travel distance in cell $n$, - $T_n$ is the accumulated transmittance up to the $n$-th cell. The exact equivalence between this formulation and the continuous integral removes the need for complex importance sampling schemes, directly leading to computational efficiency.

• Efficient Ray Tracing without Dedicated Hardware The paper revisits a ray traversal algorithm originally proposed for tetrahedral meshes and adapts it for use with the convex polyhedral cells of the Voronoi diagram. This adaptation permits robust and efficient ray tracing without relying on specialized hardware such as NVIDIA RTX cores or APIs like OptiX. By prefetching neighboring vertices and iteratively testing intersections with cell faces, the algorithm efficiently computes the exit point along a ray with an $O(1)$ query cost per cell.
• Differentiable Mesh Optimization via Densification and Pruning
  • Densification: Voronoi sites are initially seeded (often using a Structure-from-Motion (SfM) point cloud for better initialization) and are then densified in regions where the gradient of the reconstruction loss is high. This gradient norm, weighted by the cell radius, identifies underfit regions that require higher resolution.
  • Pruning: Cells with persistently low density, which do not contribute significantly to the rendered output or define object boundaries, are pruned. This ensures that the model maintains a parsimonious representation without sacrificing fidelity at object boundaries.
• Training Pipeline and Loss Formulation The training leverages standard gradient-based optimization using the Adam optimizer within the PyTorch framework, along with custom CUDA kernels for operations such as incremental Delaunay updates and ray tracing over the Voronoi cells. The model optimizes per-site parameters, including position, density, and view-dependent color (expressed via spherical harmonics up to degree three). To encourage surfaces and suppress artifacts such as floaters, the method employs an additional regularization loss analogous to the distortion loss found in Mip-NeRF 360. This loss is computed based on the quantile function of the volume rendering weight distribution:

$$L_{\text{quantile}} = \mathbb{E}_{t_1,\, t_2 \sim \mathcal{U}[0,1]} \Bigl[\bigl|W^{-1}(t_1)-W^{-1}(t_2)\bigr|\Bigr],$$

where: - $W^{-1}(\cdot)$ denotes the inverse cumulative distribution function (quantile function) of the weight distribution along a ray, effectively concentrating density at surfaces.

• Performance and Empirical Validation Quantitative evaluations on benchmarks such as Mip-NeRF 360 and Deep Blending indicate that the proposed method achieves reconstruction quality (PSNR, SSIM, LPIPS) comparable to state-of-the-art approaches, with PSNR values in the high 20s to low 30s and SSIM scores around 0.9 in several cases. Notably, the proposed method demonstrates significant speed improvements, achieving frame rates in excess of 300 FPS on consumer-grade GPUs. Ablation studies further confirm the critical role of components such as SfM initialization, adaptive densification, and the quantile-based regularization in obtaining accurate and artifact-free reconstructions.

In summary, the paper introduces a principled method for real-time differentiable ray tracing that combines the continuous optimization advantages of Voronoi-based volumetric meshes with an efficient ray tracing algorithm. The approach maintains high fidelity in radiance field reconstruction and enables integration of advanced light transport effects, all while operating without reliance on dedicated ray tracing hardware. This work therefore provides a compelling alternative to rasterization-based methods by offering both enhanced physical accuracy and real-time rendering performance.