MIMO-NeRF: Fast Neural Rendering with Multi-input Multi-output Neural Radiance Fields (2310.01821v1)
Abstract: Neural radiance fields (NeRFs) have shown impressive results for novel view synthesis. However, they depend on the repetitive use of a single-input single-output multilayer perceptron (SISO MLP) that maps 3D coordinates and view direction to the color and volume density in a sample-wise manner, which slows the rendering. We propose a multi-input multi-output NeRF (MIMO-NeRF) that reduces the number of MLPs running by replacing the SISO MLP with a MIMO MLP and conducting mappings in a group-wise manner. One notable challenge with this approach is that the color and volume density of each point can differ according to a choice of input coordinates in a group, which can lead to some notable ambiguity. We also propose a self-supervised learning method that regularizes the MIMO MLP with multiple fast reformulated MLPs to alleviate this ambiguity without using pretrained models. The results of a comprehensive experimental evaluation including comparative and ablation studies are presented to show that MIMO-NeRF obtains a good trade-off between speed and quality with a reasonable training time. We then demonstrate that MIMO-NeRF is compatible with and complementary to previous advancements in NeRFs by applying it to two representative fast NeRFs, i.e., a NeRF with sample reduction (DONeRF) and a NeRF with alternative representations (TensoRF).
- FLIP: A difference evaluator for alternating images. Proc. ACM Comput. Graph. Interact. Tech., 3(2), 2020.
- Learning neural light fields with ray-space embedding networks. In CVPR, 2022.
- SAL: Sign agnostic learning of shapes from raw data. In CVPR, 2020.
- Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
- Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
- Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In ECCV, 2020.
- Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022.
- pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In CVPR, 2021.
- TensoRF: Tensorial radiance fields. In ECCV, 2022.
- Hallucinated neural radiance fields in the wild. In CVPR, 2022.
- Learning implicit fields for generative shape modeling. In CVPR, 2019.
- Neural unsigned distance fields for implicit function learning. In NeurIPS, 2020.
- GRAM: Generative radiance manifolds for 3D-aware image generation. In CVPR, 2022.
- NeuSample: Neural sample field for efficient view synthesis. arXiv preprint arXiv:2111.15552, 2021.
- Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
- Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. In CVPR, 2021.
- FastNeRF: High-fidelity neural rendering at 200FPS. In ICCV, 2021.
- Learning shape templates with structured implicit functions. In ICCV, 2019.
- Generative adversarial nets. In NIPS, 2014.
- Implicit geometric regularization for learning shapes. In ICML, 2020.
- StyleNeRF: A style-based 3D-aware generator for high-resolution image synthesis. In ICLR, 2022.
- Baking neural radiance fields for real-time view synthesis. In ICCV, 2021.
- Denoising diffusion probabilistic models. In NeurIPS, 2020.
- EfficientNeRF: Efficient neural radiance fields. In CVPR, 2022.
- Local implicit grid representations for 3D scenes. In CVPR, 2020.
- Takuhiro Kaneko. AR-NeRF: Unsupervised learning of depth and defocus effects from natural images with aperture rendering neural radiance fields. In CVPR, 2022.
- Adam: A method for stochastic optimization. In ICLR, 2015.
- ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
- AdaNeRF: Adaptive sampling for real-time rendering of neural radiance fields. In ECCV, 2022.
- SDF-SRN: Learning signed distance 3D object reconstruction from static images. In NeurIPS, 2020.
- AutoInt: Automatic integration for fast neural volume rendering. In CVPR, 2021.
- Neural sparse voxel fields. In NeurIPS, 2020.
- Learning to infer implicit surfaces without 3D supervision. In NeurIPS, 2019.
- DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In CVPR, 2020.
- NeRF in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
- Nelson Max. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph., 1(2), 1995.
- Occupancy networks: Learning 3D reconstruction in function space. In CVPR, 2019.
- Implicit surface representations as layers in neural networks. In ICCV, 2019.
- NeRF in the dark: High dynamic range view synthesis from noisy raw images. In CVPR, 2022.
- Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph., 38(4), 2019.
- NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
- Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4), 2022.
- Rectified linear units improve restricted Boltzmann machines. In ICML, 2010.
- DONeRF: Towards real-time rendering of compact neural radiance fields using depth oracle networks. Comput. Graph. Forum, 40(4), 2021.
- CAMPARI: Camera-aware decomposed generative neural radiance fields. In 3DV, 2021.
- GIRAFFE: Representing scenes as compositional generative neural feature fields. In CVPR, 2021.
- Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In CVPR, 2020.
- DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
- Convolutional occupancy networks. In ECCV, 2020.
- TermiNeRF: Ray termination prediction for efficient neural rendering. In 3DV, 2021.
- DreamFusion: Text-to-3D using 2D diffusion. In ICLR, 2023.
- D-NeRF: Neural radiance fields for dynamic scenes. In CVPR, 2021.
- DeRF: Decomposed radiance fields. In CVPR, 2021.
- KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In ICCV, 2021.
- PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019.
- GRAF: Generative radiance fields for 3D-aware image synthesis. In NeurIPS, 2020.
- Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
- Implicit neural representations with periodic activation functions. In NeurIPS, 2020.
- Light field networks: Neural scene representations with single-evaluation rendering. In NeurIPS, 2021.
- Scene representation networks: Continuous 3D-structure-aware neural scene representations. In NeurIPS, 2019.
- EpiGRAF: Rethinking training of 3D GANs. In NeurIPS, 2022.
- Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
- Light field neural rendering. In CVPR, 2022.
- Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
- Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020.
- Ref-NeRF: Structured view-dependent appearance for neural radiance fields. In CVPR, 2022.
- R2L: Distilling neural radiance field to neural light field for efficient novel view synthesis. In ECCV, 2022.
- Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process., 13(4), 2004.
- NeX: Real-time view synthesis with neural basis expansion. In CVPR, 2021.
- DIVeR: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In CVPR, 2022.
- Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021.
- GIRAFFE HD: A high-resolution 3D-aware generative model. In CVPR, 2022.
- Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurIPS, 2020.
- PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
- PixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021.
- NeRF++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
- The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.