R$^2$-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction

Abstract: 3D Gaussian splatting (3DGS) has shown promising results in image rendering and surface reconstruction. However, its potential in volumetric reconstruction tasks, such as X-ray computed tomography, remains under-explored. This paper introduces R$^2$-Gaussian, the first 3DGS-based framework for sparse-view tomographic reconstruction. By carefully deriving X-ray rasterization functions, we discover a previously unknown integration bias in the standard 3DGS formulation, which hampers accurate volume retrieval. To address this issue, we propose a novel rectification technique via refactoring the projection from 3D to 2D Gaussians. Our new method presents three key innovations: (1) introducing tailored Gaussian kernels, (2) extending rasterization to X-ray imaging, and (3) developing a CUDA-based differentiable voxelizer. Experiments on synthetic and real-world datasets demonstrate that our method outperforms state-of-the-art approaches in accuracy and efficiency. Crucially, it delivers high-quality results in 4 minutes, which is 12$\times$ faster than NeRF-based methods and on par with traditional algorithms. Code and models are available on the project page https://github.com/Ruyi-Zha/r2_gaussian.

• The paper introduces a rectification technique that mitigates integration bias in 3D Gaussian splatting for CT reconstruction.
• It leverages tailored Gaussian kernels, expanded X-ray rasterization, and a CUDA-based differentiable voxelizer to enhance accuracy and speed.
• Experimental results show improvements of 0.93 dB in PSNR and 0.014 in SSIM while reducing training time from over 30 to about 10 minutes.

R$^2$-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction

Abstract

The paper presents a novel framework known as R$^2$-Gaussian, which leverages 3D Gaussian Splatting (3DGS) for sparse-view tomographic reconstruction, specifically addressing the limitations of 3DGS in volumetric reconstruction tasks such as X-ray computed tomography (CT). R$^2$-Gaussian introduces a critical rectification technique to mitigate integration bias in the standard 3DGS formulation and proposes three innovations: tailored Gaussian kernels, expanded rasterization for X-ray imagery, and a CUDA-based differentiable voxelizer. Experimental results demonstrate that the approach outperforms existing state-of-the-art (SOTA) methods, offering a notable improvement in PSNR and SSIM metrics while significantly reducing computation time.

Introduction

Tomographic reconstruction with 3DGS confronts unique challenges due to the intrinsic limitations of sparse-view CT and the necessity for high computational efficiency. Traditional algorithms suffer from notable artifacts or slow performance, while neural radiance field (NeRF) based methods, although precise, are prohibitively time-consuming.

This study emphasizes that existing 3DGS techniques, while suitable for image rendering, are inadequate for volumetric reconstruction due to an inherent integration bias. By identifying and rectifying this bias, the paper establishes a robust framework for direct CT reconstruction using 3DGS.

Methodology

Radiative Gaussian Representation

R$2$-Gaussian employs radiative Gaussians to represent the density field of scanned objects. Each Gaussian is parameterized by position, covariance, and central density, allowing for precise and flexible modeling of complex structures. The Gaussian parameters are initialized using the FDK algorithm, providing a low-quality but useful starting point for further optimization.

X-ray Rasterization

The paper introduces a rectification technique to address the integration bias found in standard 3DGS. This bias impedes volumetric reconstruction due to the misrepresentation of 3D densities when projected onto 2D planes. By deriving new X-ray rasterization functions and applying them within the ray space, R$^2$-Gaussian fundamentally resolves this issue. The new methodology ensures a bias-free representation, thereby enabling consistent volumetric properties across different views.

Density Voxelization

To further enhance volumetric reconstruction, R$2$-Gaussian incorporates a CUDA-based differentiable voxelizer, which extracts density volumes from the radiative Gaussians. This voxelizer allows for efficient querying of density values and enables regularization with 3D priors during training.

Results and Evaluation

The experimental results overwhelmingly support the R$2$-Gaussian framework's efficacy. Across various modalities and sparsity scenarios, R$^2$-Gaussian consistently outperforms baseline and SOTA methods. Key results include:

• An average improvement of 0.93 dB in PSNR and 0.014 in SSIM compared to the best-performing NeRF-based methods.
• A remarkable reduction in training time, achieving optimal results within approximately 10 minutes, significantly faster than the >30-minute requirement of NeRF-based methods.

Detailed evaluations on diverse datasets, encompassing human organs, animals, plants, and artificial objects, underscore the method’s robustness and practical value in real-world applications such as medical diagnostics and industrial inspections.

Discussion and Implications

The findings indicate that the rectification of integration bias is crucial not only for improving volumetric reconstruction but also for enhancing the consistency and accuracy of 3DGS-based representations. By ensuring that the density values remain unbiased and view-independent, the proposed method facilitates more reliable and precise reconstructions.

The implications of this research extend beyond CT reconstructions. The rectification technique and innovations introduced herein could serve to improve any 3DGS-related tasks that require precise volumetric analyses, such as MRI reconstruction and complex surface modeling.

Conclusion

The R$2$-Gaussian framework represents a significant advancement in applying 3D Gaussian splatting to volumetric reconstruction tasks. By introducing critical rectifications and methodological innovations, it overcomes the limitations of standard 3DGS, achieving superior performance and efficiency. This study not only sets a new benchmark for sparse-view tomographic reconstruction but also opens avenues for further research into 3DGS applications, highlighting the potential for broad-scale improvements across various domains of volumetric analysis.