Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering (2310.01881v1)
Abstract: Recent advances in Neural Radiance Fields (NeRF) have demonstrated significant potential for representing 3D scene appearances as implicit neural networks, enabling the synthesis of high-fidelity novel views. However, the lengthy training and rendering process hinders the widespread adoption of this promising technique for real-time rendering applications. To address this issue, we present an effective adaptive multi-NeRF method designed to accelerate the neural rendering process for large scenes with unbalanced workloads due to varying scene complexities. Our method adaptively subdivides scenes into axis-aligned bounding boxes using a tree hierarchy approach, assigning smaller NeRFs to different-sized subspaces based on the complexity of each scene portion. This ensures the underlying neural representation is specific to a particular part of the scene. We optimize scene subdivision by employing a guidance density grid, which balances representation capability for each Multilayer Perceptron (MLP). Consequently, samples generated by each ray can be sorted and collected for parallel inference, achieving a balanced workload suitable for small MLPs with consistent dimensions for regular and GPU-friendly computations. We aosl demonstrated an efficient NeRF sampling strategy that intrinsically adapts to increase parallelism, utilization, and reduce kernel calls, thereby achieving much higher GPU utilization and accelerating the rendering process.
- [n. d.]. Mixture of volumetric primitives for efficient neural rendering | ACM Transactions on Graphics. https://dl.acm.org/doi/abs/10.1145/3450626.3459863
- Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5855–5864.
- NeRD: Neural Reflectance Decomposition from Image Collections. https://doi.org/10.48550/arXiv.2012.03918 arXiv:2012.03918 [cs].
- pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis. https://doi.org/10.48550/arXiv.2012.00926 arXiv:2012.00926 [cs].
- Depth-supervised NeRF: Fewer Views and Faster Training for Free. https://doi.org/10.48550/arXiv.2107.02791 arXiv:2107.02791 [cs].
- Adaptive Voronoi NeRFs. arXiv preprint arXiv:2303.16001 (2023).
- FastNeRF: High-Fidelity Neural Rendering at 200FPS. 14346–14355. https://openaccess.thecvf.com/content/ICCV2021/html/Garbin_FastNeRF_High-Fidelity_Neural_Rendering_at_200FPS_ICCV_2021_paper.html
- Baking Neural Radiance Fields for Real-Time View Synthesis. 5875–5884. https://openaccess.thecvf.com/content/ICCV2021/html/Hedman_Baking_Neural_Radiance_Fields_for_Real-Time_View_Synthesis_ICCV_2021_paper.html
- AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields. https://doi.org/10.48550/ARXIV.2207.10312
- AutoInt: Automatic Integration for Fast Neural Volume Rendering. 14556–14565. https://openaccess.thecvf.com/content/CVPR2021/html/Lindell_AutoInt_Automatic_Integration_for_Fast_Neural_Volume_Rendering_CVPR_2021_paper.html
- Neural sparse voxel fields. Advances in Neural Information Processing Systems 33 (2020), 15651–15663.
- Deep Appearance Models for Face Rendering. ACM Transactions on Graphics 37, 4 (Aug. 2018), 1–13. https://doi.org/10.1145/3197517.3201401 arXiv:1808.00362 [cs].
- Mixture of Volumetric Primitives for Efficient Neural Rendering. https://doi.org/10.48550/arXiv.2103.01954 arXiv:2103.01954 [cs].
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. CoRR abs/2003.08934 (2020). arXiv:2003.08934 https://arxiv.org/abs/2003.08934
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph. 41, 4, Article 102 (jul 2022), 15 pages. https://doi.org/10.1145/3528223.3530127
- Real-time Neural Radiance Caching for Path Tracing. CoRR abs/2106.12372 (2021). arXiv:2106.12372 https://arxiv.org/abs/2106.12372
- DONeRF: Towards Real-Time Rendering of Neural Radiance Fields using Depth Oracle Networks. CoRR abs/2103.03231 (2021). arXiv:2103.03231 https://arxiv.org/abs/2103.03231
- DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Computer Graphics Forum 40, 4 (2021), 45–59. https://doi.org/10.1111/cgf.14340 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14340.
- Nicholas Pilkington. 2022. DroneDeploy NeRF Dataset. https://github.com/nickponline/dd-nerf-dataset.
- KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. CoRR abs/2103.13744 (2021). arXiv:2103.13744 https://arxiv.org/abs/2103.13744
- Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations. CoRR abs/1906.01618 (2019). arXiv:1906.01618 http://arxiv.org/abs/1906.01618
- Direct Voxel Grid Optimization: Super-Fast Convergence for Radiance Fields Reconstruction. 5459–5469. https://openaccess.thecvf.com/content/CVPR2022/html/Sun_Direct_Voxel_Grid_Optimization_Super-Fast_Convergence_for_Radiance_Fields_Reconstruction_CVPR_2022_paper.html
- Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8248–8258.
- Advances in Neural Rendering. https://doi.org/10.48550/arXiv.2111.05849 arXiv:2111.05849 [cs].
- Alex Trevithick and Bo Yang. 2021. GRF: Learning a General Radiance Field for 3D Representation and Rendering. https://doi.org/10.48550/arXiv.2010.04595 arXiv:2010.04595 [cs].
- Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII. Springer, 106–122.
- PlenOctrees for Real-Time Rendering of Neural Radiance Fields. 5752–5761. https://openaccess.thecvf.com/content/ICCV2021/html/Yu_PlenOctrees_for_Real-Time_Rendering_of_Neural_Radiance_Fields_ICCV_2021_paper.html
- NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination. https://doi.org/10.1145/3478513.3480496 arXiv:2106.01970 [cs].