Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis (2401.02436v2)

Abstract: Recently, high-fidelity scene reconstruction with an optimized 3D Gaussian splat representation has been introduced for novel view synthesis from sparse image sets. Making such representations suitable for applications like network streaming and rendering on low-power devices requires significantly reduced memory consumption as well as improved rendering efficiency. We propose a compressed 3D Gaussian splat representation that utilizes sensitivity-aware vector clustering with quantization-aware training to compress directional colors and Gaussian parameters. The learned codebooks have low bitrates and achieve a compression rate of up to $31\times$ on real-world scenes with only minimal degradation of visual quality. We demonstrate that the compressed splat representation can be efficiently rendered with hardware rasterization on lightweight GPUs at up to $4\times$ higher framerates than reported via an optimized GPU compute pipeline. Extensive experiments across multiple datasets demonstrate the robustness and rendering speed of the proposed approach.

• The paper introduces a compressed 3D Gaussian splatting method that reduces data size by up to 31x while maintaining visual quality.
• It employs sensitivity-aware clustering, quantization-aware fine-tuning, and entropy encoding to optimize memory and training efficiency.
• Improved GPU rasterization boosts rendering speed by up to 4x, making the method ideal for real-time novel view synthesis.

Introduction

A novel approach to 3D scene reconstruction is presented in this paper, where optimized 3D Gaussian splatting plays a vital role. The main focus is on enhancing the practicality of view synthesis from sparse image data sets for various applications. This is achieved through a new compressed 3D Gaussian splat representation that promises significant reductions in memory requirements and substantial improvements in rendering performance, even on less powerful GPUs.

Scene Compression Methodology

The compression strategy is thorough, incorporating several innovative steps:

1. Sensitivity-aware clustering: A pivotal first step where the color information and Gaussian parameters are inspected for their influence on training images. This allows for a precise and tailored compression of these elements into compact codebooks.
2. Quantization-aware fine-tuning: Following the compression, the system undergoes fine-tuning. This stage recovers details that were lost during clustering, with an emphasis on maintaining a low bit-rate for the training phase.
3. Entropy encoding: This final compression step leverages the spatial coherence of 3D Gaussian parameters. Algorithms such as entropy and run-length encoding are utilized to efficiently compact the data.

These steps collectively result in up to 31 times reduction of data size with nominal impact on visual quality.

Rendering Performance

Rendering is a critical aspect of novel view synthesis, particularly for real-time applications. The proposed method showcases impressive rendering capabilities. By employing GPU rasterization, rendering is accomplished with up to four times higher frame rates compared to previous techniques, without compromising the visual fidelity of the synthesized views.

Empirical Validation

Extensive testing confirms the robustness and speed of the proposed method. Benchmarks across multiple data sets, including various real-world scenes, demonstrate that the compression retains high image quality with dramatic reductions in required memory. A set of ablation studies further corroborates the contribution of each individual stage within the compression pipeline.

Conclusion and Future Work

The paper concludes by highlighting the substantial compression rates and rendering speed improvements made possible by the new method. The resulting solution is well-suited for applications that involve network streaming or operate on devices with limited processing capabilities. Looking ahead, the authors intend to explore additional compression techniques, especially for spatial positioning of Gaussians, to broaden the scope of applications for their 3D Gaussian splatting based reconstruction and rendering methods.