Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB (2409.15689v1)

Abstract: The goal of this paper is to encode a 3D scene into an extremely compact representation from 2D images and to enable its transmittance, decoding and rendering in real-time across various platforms. Despite the progress in NeRFs and Gaussian Splats, their large model size and specialized renderers make it challenging to distribute free-viewpoint 3D content as easily as images. To address this, we have designed a novel 3D representation that encodes the plenoptic function into sinusoidal function indexed dense volumes. This approach facilitates feature sharing across different locations, improving compactness over traditional spatial voxels. The memory footprint of the dense 3D feature grid can be further reduced using spatial decomposition techniques. This design combines the strengths of spatial hashing functions and voxel decomposition, resulting in a model size as small as 150 KB for each 3D scene. Moreover, PPNG features a lightweight rendering pipeline with only 300 lines of code that decodes its representation into standard GL textures and fragment shaders. This enables real-time rendering using the traditional GL pipeline, ensuring universal compatibility and efficiency across various platforms without additional dependencies.

• The paper introduces a novel spectral encoding method that compresses 3D scenes into a 150 KB neural representation.
• It utilizes spatial decomposition techniques like tensor-rank and tri-plane decomposition to achieve efficient, real-time WebGL rendering.
• Benchmark results demonstrate up to a 100x reduction in model size while maintaining competitive visual quality on standard datasets.

Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB

The paper "Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB" delineates a novel method for encoding 3D scenes into compact representations, facilitating real-time rendering across multiple platforms. Traditional approaches such as Neural Radiance Fields (NeRF) and Gaussian Splatting exhibit limitations in terms of model size and rendering dependencies, often necessitating specialized hardware and software for optimal performance. This research addresses these challenges through the development of Plenoptic PNG (PPNG), a portable neural graphics framework that achieves a significant reduction in model size while maintaining real-time rendering capabilities.

Overview

The key innovation of PPNG lies in its encoding of the plenoptic function into a dense volumetric grid indexed by sinusoidal functions. This spectral domain representation allows for substantial feature sharing across different spatial locations, enhancing compactness relative to traditional voxel-based methods. The authors implement spatial decomposition techniques, combining the strengths of spatial hashing and tensor-rank decomposition, to further minimize the memory footprint. The resultant model achieves a size as small as 150 KB per 3D scene.

Methodology

PPNG employs the follow steps for 3D scene representation:

1. Spectral Domain Encoding: The input Euclidean coordinates of a 3D point are mapped into a multi-scale, multi-dimensional Fourier embedding. This continuous, periodic representation enables efficient feature sharing.
2. Feature Volume and Decomposition: The dense 3D feature grid, composed of voxel volumes indexed in the Fourier domain, is reduced through CP-decomposition or tri-plane decomposition. This maintains the expressive power while significantly reducing the parameter count.
3. Real-time Rendering: PPNG representations can be decoded into GL textures and shaders for efficient rendering on WebGL-compatible platforms. The fast decoding and lightweight implementation ensure real-time performance on a variety of devices, including mobile phones and laptops.

Performance and Benchmarks

The proposed methods, PPNG-1, PPNG-2, and PPNG-3, showcase varying degrees of factorization and compactness:

• PPNG-1: Utilizes tensor-rank decomposition, resulting in the smallest model size (~151 KB) with slight trade-offs in rendering quality.
• PPNG-2: Employs tri-plane decomposition, balancing model size (~2.49 MB) and rendering quality.
• PPNG-3: Maintains the densest grid representation (~32.8 MB), providing the highest rendering quality.

The extensive experimentation on datasets such as Synthetic NeRF, Blended MVS, and Tanks and Temples demonstrates the effectiveness of PPNG in achieving a favorable balance between model size, rendering quality, and training time. PPNG-3, for instance, achieves real-time WebGL rendering while maintaining visual fidelity comparable to state-of-the-art methods, but with a significantly reduced memory requirement.

Compared to existing methods:

• PPNG-1 achieves a 100x reduction in model size with competitive visual quality.
• PPNG-3 offers the best trade-offs with a 5x smaller size and rapid training times.

Implications and Future Directions

The implications of Plenoptic PNG are twofold. Practically, it paves the way for widespread distribution and rendering of photorealistic 3D content on platforms with limited computational resources. Theoretically, it introduces an efficient framework for neural field compression and real-time rendering, highlighting the potential of spectral domain representations in compacting high-dimensional data.

Future research could explore:

• Contracted Space Modeling: Extending PPNG to handle unbounded scenes efficiently by incorporating models that adaptively contract space.
• Integration with Advanced Hardware: Optimizations to leverage the evolving capabilities of mobile GPUs and specialized processors.

The advancements demonstrated by Plenoptic PNG are poised to facilitate new applications in augmented reality (AR), virtual reality (VR), and real-time graphics, democratizing access to immersive 3D experiences across diverse platforms.