Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Rendering (2412.04459v3)

Abstract: We propose an efficient radiance field rendering algorithm that incorporates a rasterization process on adaptive sparse voxels without neural networks or 3D Gaussians. There are two key contributions coupled with the proposed system. The first is to adaptively and explicitly allocate sparse voxels to different levels of detail within scenes, faithfully reproducing scene details with $65536^3$ grid resolution while achieving high rendering frame rates. Second, we customize a rasterizer for efficient adaptive sparse voxels rendering. We render voxels in the correct depth order by using ray direction-dependent Morton ordering, which avoids the well-known popping artifact found in Gaussian splatting. Our method improves the previous neural-free voxel model by over 4db PSNR and more than 10x FPS speedup, achieving state-of-the-art comparable novel-view synthesis results. Additionally, our voxel representation is seamlessly compatible with grid-based 3D processing techniques such as Volume Fusion, Voxel Pooling, and Marching Cubes, enabling a wide range of future extensions and applications.

• The paper introduces a dynamic Morton ordering technique that minimizes rendering artifacts by accurately sorting sparse voxels along pixel rays.
• It achieves adaptive voxel fidelity with over 4 dB PSNR improvement and more than a 10x speedup in rendering compared to previous methods.
• The approach integrates seamlessly with classical algorithms like TSDF-Fusion and Marching Cubes, enhancing multi-view synthesis and mesh reconstruction.

Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Rendering

This paper introduces SVRaster, a framework designed to enhance multi-view reconstruction and novel view synthesis through an innovative approach leveraging sparse voxels. This method eschews neural networks and 3D Gaussian splats in favor of a rasterization algorithm applied to sparse voxels, resulting in improvements in rendering speed and fidelity.

Key Contributions

1. Dynamic Morton Ordering: The paper outlines a method to ensure correct depth ordering of sparse voxels for rendering. This is achieved by dynamically sorting voxel positions along pixel rays, effectively mitigating artifacts such as the popping artifacts prevalent in configurations that use Gaussian splatting based on centroid sorting.
2. Adaptive Sparse Voxel Fidelity: The sparse voxels are fitted adaptively to various levels of scene detail. This allows the method to reproduce scene intricacies reliably while maintaining high frame rates, thereby enhancing previous neural-free voxel grid representations by over 4 dB in PSNR and offering more than a 10x speedup in rendering frames per second.
3. Compatibility with Classical Algorithms: The sparse voxel representation integrates naturally with established grid-based 3D processing algorithms like TSDF-Fusion and Marching Cubes. This compatibility extends the usability of the method across applications requiring robust mesh reconstruction.

Methodological Insights

The ingenious combination of rasterization efficiency with volumetric voxel representation stands at the core of SVRaster. Traditional grid-based techniques solve the depth ordering and voxel density issues inherently but suffer from inferior rendering speeds compared to Gaussian splatting. SVRaster achieves a harmonious blend by revisiting voxel representations, refining them into a sparser format compatible with fast rasterization methods derived from graphics processing. Importantly, the paper demonstrates the adaptability of the voxel size to cater to varying detail levels, enhancing both rendering performance and visual accuracy.

Experimental Results

In terms of novel-view synthesis, SVRaster achieves performance metrics comparable to state-of-the-art Gaussian splatting techniques, with notable efficiency in rendering speed. Performance benchmarking across the MipNeRF360 and other datasets indicates superior LPIPS scores and competitive PSNR and SSIM values against de facto standards like 3DGS, although with nuanced disparities attributable to initialization strategies and progressive optimization differences.

In mesh reconstruction tasks, the application of sparse-voxel TSDF-Fusion and Marching Cubes demonstrates promising accuracy, showing competitiveness with NeRF and SDF-based frameworks, which traditionally possess advantages due to embedded surface geometry modeling.

Implications and Prospective Directions

Practically, SVRaster's improvements in real-time rendering and scene detail fidelity make it highly applicable in domains such as virtual reality and real-time scene synthesis for gaming engines or simulation environments. The theoretical integration with classical algorithms highlights its versatility and potential to bridge newer radiance field models with existing 3D processing pipelines.

Future work could explore optimizing anti-aliasing techniques within the rasterization process and investigate more parameter-efficient models of view-dependent appearance. There's also room to directly model signed distance fields within SVRaster to leverage the density field's full capability, potentially aligning the approach more closely with emerging methodologies in neural implicit surfaces.

Overall, this paper presents a compelling advancement in the field, marrying the efficiency of rasterization with the robustness of sparse volumetric representations, opening new avenues for high-fidelity, real-time 3D rendering solutions.