Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MIMO-NeRF: Fast Neural Rendering with Multi-input Multi-output Neural Radiance Fields (2310.01821v1)

Published 3 Oct 2023 in cs.CV, cs.AI, cs.GR, cs.LG, and eess.IV

Abstract: Neural radiance fields (NeRFs) have shown impressive results for novel view synthesis. However, they depend on the repetitive use of a single-input single-output multilayer perceptron (SISO MLP) that maps 3D coordinates and view direction to the color and volume density in a sample-wise manner, which slows the rendering. We propose a multi-input multi-output NeRF (MIMO-NeRF) that reduces the number of MLPs running by replacing the SISO MLP with a MIMO MLP and conducting mappings in a group-wise manner. One notable challenge with this approach is that the color and volume density of each point can differ according to a choice of input coordinates in a group, which can lead to some notable ambiguity. We also propose a self-supervised learning method that regularizes the MIMO MLP with multiple fast reformulated MLPs to alleviate this ambiguity without using pretrained models. The results of a comprehensive experimental evaluation including comparative and ablation studies are presented to show that MIMO-NeRF obtains a good trade-off between speed and quality with a reasonable training time. We then demonstrate that MIMO-NeRF is compatible with and complementary to previous advancements in NeRFs by applying it to two representative fast NeRFs, i.e., a NeRF with sample reduction (DONeRF) and a NeRF with alternative representations (TensoRF).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. FLIP: A difference evaluator for alternating images. Proc. ACM Comput. Graph. Interact. Tech., 3(2), 2020.
  2. Learning neural light fields with ray-space embedding networks. In CVPR, 2022.
  3. SAL: Sign agnostic learning of shapes from raw data. In CVPR, 2020.
  4. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
  5. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
  6. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In ECCV, 2020.
  7. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022.
  8. pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In CVPR, 2021.
  9. TensoRF: Tensorial radiance fields. In ECCV, 2022.
  10. Hallucinated neural radiance fields in the wild. In CVPR, 2022.
  11. Learning implicit fields for generative shape modeling. In CVPR, 2019.
  12. Neural unsigned distance fields for implicit function learning. In NeurIPS, 2020.
  13. GRAM: Generative radiance manifolds for 3D-aware image generation. In CVPR, 2022.
  14. NeuSample: Neural sample field for efficient view synthesis. arXiv preprint arXiv:2111.15552, 2021.
  15. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
  16. Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. In CVPR, 2021.
  17. FastNeRF: High-fidelity neural rendering at 200FPS. In ICCV, 2021.
  18. Learning shape templates with structured implicit functions. In ICCV, 2019.
  19. Generative adversarial nets. In NIPS, 2014.
  20. Implicit geometric regularization for learning shapes. In ICML, 2020.
  21. StyleNeRF: A style-based 3D-aware generator for high-resolution image synthesis. In ICLR, 2022.
  22. Baking neural radiance fields for real-time view synthesis. In ICCV, 2021.
  23. Denoising diffusion probabilistic models. In NeurIPS, 2020.
  24. EfficientNeRF: Efficient neural radiance fields. In CVPR, 2022.
  25. Local implicit grid representations for 3D scenes. In CVPR, 2020.
  26. Takuhiro Kaneko. AR-NeRF: Unsupervised learning of depth and defocus effects from natural images with aperture rendering neural radiance fields. In CVPR, 2022.
  27. Adam: A method for stochastic optimization. In ICLR, 2015.
  28. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
  29. AdaNeRF: Adaptive sampling for real-time rendering of neural radiance fields. In ECCV, 2022.
  30. SDF-SRN: Learning signed distance 3D object reconstruction from static images. In NeurIPS, 2020.
  31. AutoInt: Automatic integration for fast neural volume rendering. In CVPR, 2021.
  32. Neural sparse voxel fields. In NeurIPS, 2020.
  33. Learning to infer implicit surfaces without 3D supervision. In NeurIPS, 2019.
  34. DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In CVPR, 2020.
  35. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
  36. Nelson Max. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph., 1(2), 1995.
  37. Occupancy networks: Learning 3D reconstruction in function space. In CVPR, 2019.
  38. Implicit surface representations as layers in neural networks. In ICCV, 2019.
  39. NeRF in the dark: High dynamic range view synthesis from noisy raw images. In CVPR, 2022.
  40. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph., 38(4), 2019.
  41. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  42. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4), 2022.
  43. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010.
  44. DONeRF: Towards real-time rendering of compact neural radiance fields using depth oracle networks. Comput. Graph. Forum, 40(4), 2021.
  45. CAMPARI: Camera-aware decomposed generative neural radiance fields. In 3DV, 2021.
  46. GIRAFFE: Representing scenes as compositional generative neural feature fields. In CVPR, 2021.
  47. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In CVPR, 2020.
  48. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
  49. Convolutional occupancy networks. In ECCV, 2020.
  50. TermiNeRF: Ray termination prediction for efficient neural rendering. In 3DV, 2021.
  51. DreamFusion: Text-to-3D using 2D diffusion. In ICLR, 2023.
  52. D-NeRF: Neural radiance fields for dynamic scenes. In CVPR, 2021.
  53. DeRF: Decomposed radiance fields. In CVPR, 2021.
  54. KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In ICCV, 2021.
  55. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019.
  56. GRAF: Generative radiance fields for 3D-aware image synthesis. In NeurIPS, 2020.
  57. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  58. Implicit neural representations with periodic activation functions. In NeurIPS, 2020.
  59. Light field networks: Neural scene representations with single-evaluation rendering. In NeurIPS, 2021.
  60. Scene representation networks: Continuous 3D-structure-aware neural scene representations. In NeurIPS, 2019.
  61. EpiGRAF: Rethinking training of 3D GANs. In NeurIPS, 2022.
  62. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
  63. Light field neural rendering. In CVPR, 2022.
  64. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
  65. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020.
  66. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. In CVPR, 2022.
  67. R2L: Distilling neural radiance field to neural light field for efficient novel view synthesis. In ECCV, 2022.
  68. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process., 13(4), 2004.
  69. NeX: Real-time view synthesis with neural basis expansion. In CVPR, 2021.
  70. DIVeR: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In CVPR, 2022.
  71. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021.
  72. GIRAFFE HD: A high-resolution 3D-aware generative model. In CVPR, 2022.
  73. Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurIPS, 2020.
  74. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
  75. PixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021.
  76. NeRF++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
  77. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Citations (6)

Summary

We haven't generated a summary for this paper yet.