NeX: Real-time View Synthesis with Neural Basis Expansion

Abstract: We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects -- in real time. Unlike traditional MPI that uses a set of simple RGB$\alpha$ planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forward-facing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000$\times$ faster rendering time than the state of the art. For real-time demos, visit https://nex-mpi.github.io/

• The paper introduces neural basis expansion to enhance MPI for real-time view synthesis with complex view-dependent effects.
• It employs a hybrid implicit-explicit modeling strategy that optimizes memory use while capturing fine details for photorealism.
• Experimental results demonstrate over 1000x faster rendering and robust performance on benchmark and Shiny datasets.

NeX: Real-time View Synthesis with Neural Basis Expansion

The paper "NeX: Real-time View Synthesis with Neural Basis Expansion" presents a novel approach to view synthesis focusing on real-time rendering capabilities while addressing limitations of previous techniques such as traditional Multiplane Image (MPI) representation. This work is significant for those in the field of computer vision and graphics, especially those addressing the challenges of dynamic view-dependent effects in novel view synthesis.

The primary innovation involves enriching the traditional MPI format by integrating neural basis expansions instead of relying solely on RGB$\alpha$ planes. This expansion enables the accurate modeling of complex view-dependent effects, such as reflections, without sacrificing computational efficiency. Notably, each pixel is parameterized through a linear combination of basis functions derived from a neural network. This parameterization not only enhances real-time capabilities but also supports photorealism in view synthesis.

The authors introduce a hybrid implicit-explicit modeling strategy, wherein certain model parameters are stored explicitly, such as base color coefficients, while others are learned through implicit representation, refining fine detail and achieving high accuracy. This blend optimizes memory use and accelerates rendering speed, resulting in over 1000 times faster rendering than current state-of-the-art methods like NeRF.

Experimental results demonstrate superior performance across various datasets, including the benchmark forward-facing and the newly introduced Shiny dataset, specifically designed to test intricate view-dependent effects like rainbow reflections. Consistently, NeX achieves the highest scores on major qualitative and quantitative metrics, indicating its effectiveness in capturing scene intricacies others fail to manage.

Significantly, the results show the enhanced capability of NeX in rendering realistic images with complex optical phenomena in real-time, supported by an average rendering speed exceeding 200 frames per second on standard graphics hardware. The use of learned neural basis functions over fixed functions shows substantive improvement, demonstrating a critical advantage in balancing complexity and computational cost.

In the broader context, this approach represents a shift towards integrating deep learning techniques to optimize traditional rendering pipelines, suggesting potential pathways for future research. The adaptability of the neural basis expansion framework hints at applications extending beyond MPI, potentially influencing the development of efficient light field representations and augmenting neural rendering techniques.

This research exemplifies a practical step toward real-time, photorealistic graphics rendering and sets a foundation for exploring neural approaches in complex scene representations and visual effects modeling. Future work might explore augmenting this framework with additional neural structures or leveraging more diverse datasets to generalize capabilities further.