Don't Splat your Gaussians: Volumetric Ray-Traced Primitives for Modeling and Rendering Scattering and Emissive Media (2405.15425v2)

Abstract: Efficient scene representations are essential for many computer graphics applications. A general unified representation that can handle both surfaces and volumes simultaneously, remains a research challenge. Inspired by recent methods for scene reconstruction that leverage mixtures of 3D Gaussians to model radiance fields, we formalize and generalize the modeling of scattering and emissive media using mixtures of simple kernel-based volumetric primitives. We introduce closed-form solutions for transmittance and free-flight distance sampling for different kernels, and propose several optimizations to use our method efficiently within any off-the-shelf volumetric path tracer. We demonstrate our method as a compact and efficient alternative to other forms of volume modeling for forward and inverse rendering of scattering media. Furthermore, we adapt and showcase our method in radiance field optimization and rendering, providing additional flexibility compared to current state of the art given its ray-tracing formulation. We also introduce the Epanechnikov kernel and demonstrate its potential as an efficient alternative to the traditionally-used Gaussian kernel in scene reconstruction tasks. The versatility and physically-based nature of our approach allows us to go beyond radiance fields and bring to kernel-based modeling and rendering any path-tracing enabled functionality such as scattering, relighting and complex camera models.

• The paper presents a novel method replacing grid-based representations with compact 3D Gaussian primitives.
• It achieves efficient closed-form transmittance calculations, reducing memory usage and accelerating rendering speeds.
• The approach supports both forward and inverse rendering, enabling real-time applications and scalable production pipelines.

Volumetric Primitives: Efficient Rendering and Modeling of Media

Background and Motivation

The representation of complex volumetric scenes—think clouds, smoke, or detailed datasets from Disney Animation—has long been done using grid-based methods like voxel grids. These methods store data in 3D grids, which can be computationally expensive and memory-intensive. Each time we double the resolution, we're increasing the amount of data by eight times.

This paper proposes an alternative approach via volumetric primitives—specifically using 3D Gaussians. Instead of representing a scene using a massive grid of voxels, the scene is represented by a smaller, more efficient set of volumetric primitives.

How It Works: A High-Level Overview

The core idea here is to use a mix of 3D Gaussian primitives to model volumetric media. Each Gaussian is a small packet that efficiently captures details of the scene. This representation has remarkable benefits:

• Compactness: It uses less memory than traditional grid-based methods.
• Speed: Evaluations and renderings are faster due to closed-form solutions for essential computations like transmittance and free-flight distance sampling.

Key Contributions

Let's break down the most significant contributions from this research:

1. Novel Volumetric Representation:
   • Instead of grid cells, the scene is broken down into 3D Gaussian primitives.
   • This approach fits well into the radiative transfer framework, making it suitable for physically accurate rendering.
2. Efficient Transmittance Calculation:
   • The researchers developed closed-form solutions for calculating how light transmits through the media.
   • This bypasses the need for computationally expensive tracking techniques that are standard with voxel grids.
3. Volume Rendering:
   • The method provides a way to handle forward rendering (for generating images from a model) and inverse rendering (for deducing the model from images).
4. Improved Performance and Memory Efficiency:
   • The representation is far less memory-intensive. For instance, the cloud dataset example achieves significant compression from 550.4MBs (voxel grid) to just 215KBs (Gaussian primitives).

Strong Numerical Results

One of the paper's standout results is the performance improvement:

• Memory Compression: The 3D Gaussian approach can represent complex scenes using significantly less memory.
• Speed: Achieving average framerates of 400-500 frames per second (FPS) with minimal loss in quality (just 2-3 dB PSNR in synthetic datasets).

Practical and Theoretical Implications

Practical Implications:

• Real-Time Applications: With the drastic reduction in memory and increased rendering speed, this method is particularly compelling for real-time applications, such as virtual reality and interactive graphics.
• Production Pipelines: The approach simplifies the data handling in production environments, reducing storage and computational costs.

Theoretical Implications:

• Extension to Other Kernels: Although this paper focuses on Gaussian kernels, the authors hint at the potential to use other kernel types, which may further generalize and optimize performance.
• Future Algorithms: This opens up new avenues for both the forward and inverse rendering algorithms to better handle complex lighting scenarios.

Future Directions

1. Exploring Other Kernels: There's potential for other types of kernels that could provide even better performance or compactness.
2. Handling Anisotropic and Correlated Media: Generalizing the current framework to more complex media types.
3. Level-of-Detail Representations: A promising application for the film and gaming industry, where different levels of detail are often needed.

Conclusion

This paper provides a novel method for modeling and rendering volumetric media, showing significant improvements over traditional grid-based approaches. Through volumetric primitives, particularly 3D Gaussians, the research demonstrates not only a memory-efficient representation but also faster rendering, making it highly suitable for real-time applications and scalable production environments. Future work on this framework could further extend its capabilities and efficiency.