Papers
Topics
Authors
Recent
Search
2000 character limit reached

IL-NeRF: Incremental Learning for Neural Radiance Fields with Camera Pose Alignment

Published 10 Dec 2023 in cs.CV and cs.AI | (2312.05748v1)

Abstract: Neural radiance fields (NeRF) is a promising approach for generating photorealistic images and representing complex scenes. However, when processing data sequentially, it can suffer from catastrophic forgetting, where previous data is easily forgotten after training with new data. Existing incremental learning methods using knowledge distillation assume that continuous data chunks contain both 2D images and corresponding camera pose parameters, pre-estimated from the complete dataset. This poses a paradox as the necessary camera pose must be estimated from the entire dataset, even though the data arrives sequentially and future chunks are inaccessible. In contrast, we focus on a practical scenario where camera poses are unknown. We propose IL-NeRF, a novel framework for incremental NeRF training, to address this challenge. IL-NeRF's key idea lies in selecting a set of past camera poses as references to initialize and align the camera poses of incoming image data. This is followed by a joint optimization of camera poses and replay-based NeRF distillation. Our experiments on real-world indoor and outdoor scenes show that IL-NeRF handles incremental NeRF training and outperforms the baselines by up to $54.04\%$ in rendering quality.

Authors (4)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3366–3375, 2017.
  2. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139–154, 2018.
  3. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  4. Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
  5. Clnerf: Continual learning meets nerf. arXiv preprint arXiv:2308.14816, 2023.
  6. Tensorf: Tensorial radiance fields. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 333–350. Springer, 2022.
  7. Meil-nerf: Memory-efficient incremental learning of neural radiance fields. arXiv preprint arXiv:2212.08328, 2022.
  8. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882–12891, 2022.
  9. Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999.
  10. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501–5510, 2022.
  11. The orienteering problem. Naval Research Logistics (NRL), 34(3):307–318, 1987.
  12. Incremental learning for neural radiance field with uncertainty-filtered knowledge distillation. arXiv preprint arXiv:2212.10950, 2022.
  13. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 32, 2019.
  14. The categorized orienteering problem with count-dependent profits. Applied Soft Computing, 113:107962, 2021.
  15. Self-calibrating neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5846–5854, 2021.
  16. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  17. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  18. kwea123. ngp-pl: a pytorch implementation of instant-ngp, 2022.
  19. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5741–5751, 2021.
  20. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
  21. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  22. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pages 67–82, 2018.
  23. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210–7219, 2021.
  24. Electric vehicle routing, arc routing, and team orienteering problems in sustainable transportation. Energies, 14(16):5131, 2021.
  25. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.
  26. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  27. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16190–16199, 2022.
  28. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  29. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics, 33(5):1255–1262, 2017.
  30. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  31. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589–5599, 2021.
  32. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  33. Dubins orienteering problem. IEEE Robotics and Automation Letters, 2(2):1210–1217, 2017.
  34. Instant continual learning of neural radiance fields. arXiv preprint arXiv:2309.01811, 2023.
  35. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  36. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14153–14161, 2021.
  37. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12892–12901, 2022.
  38. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
  39. Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv:2210.13641, 2022.
  40. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016.
  41. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
  42. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558–16569, 2021.
  43. Orienteering problems. EURO Advanced Tutorials on Operational Research, 2019.
  44. The orienteering problem: A survey. European Journal of Operational Research, 209(1):1–10, 2011.
  45. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021a.
  46. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  47. Nerf–: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021b.
  48. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5610–5619, 2021.
  49. Neutex: Neural texture mapping for volumetric neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7119–7128, 2021.
  50. Reinforced continual learning. Advances in Neural Information Processing Systems, 31, 2018.
  51. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805–4815, 2021.
  52. inerf: Inverting neural radiance fields for pose estimation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1323–1330. IEEE, 2021.
  53. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3713–3722, 2019.
  54. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
Citations (2)
View on Semantic Scholar

Summary

  • The paper introduces IL-NeRF, a novel framework that incrementally aligns camera poses without relying on pre-estimated values.
  • It employs replay-based distillation to preserve learned information and effectively mitigate catastrophic forgetting.
  • Experimental results demonstrate a rendering quality improvement of up to 54% over traditional models, validated by metrics like PSNR, SSIM, and LPIPS.

Introduction

Neural Radiance Fields (NeRF) have gained popularity for their ability to construct three-dimensional scenes from two-dimensional images using neural networks. However, a significant challenge faced by NeRF is what is known as catastrophic forgetting, which occurs when the model forgets previously learned information upon exposure to new data. Moreover, traditional NeRF models necessitate pre-estimated camera poses for their training, which is not always practical since camera poses may not be known in advance.

Addressing Incremental Learning Challenges

A proposed solution to these problems is the novel framework named IL-NeRF. It distinguishes itself by addressing incremental learning challenges directly. This framework focuses on a practical scenario, where camera poses arrive sequentially and are not pre-estimated. To achieve this, it introduces an incremental camera pose alignment module that selects and initialises a set of reference camera poses from the past learning. It allows for newly arriving data to have the camera poses aligned within an established coordinate system. Additionally, a method of replay-based NeRF distillation is utilized, which preserves knowledge through the learning process of sequential data sets.

Technical Approach

In terms of technical approach, IL-NeRF implements a graph-based reward-collection optimization to determine the optimal selection of camera poses, and a greedy algorithm is used for practicality. For aligning camera poses, transfer matrices are calculated based on the rotation and translation of selected camera poses, ensuring continuity within the same coordinate system. Joint optimization of camera poses and the NeRF model is performed to refine the accuracy and mitigate potential errors in the alignments.

Experimental Findings

Experimental results across various datasets demonstrate IL-NeRF's proficiency in minimizing catastrophic forgetting and maintaining camera pose alignment. The framework outperforms baseline methodologies, including traditional NeRF, EWC, and NeRF-SLAM in terms of rendering quality. It has been shown to improve key metrics such as PSNR, SSIM, and LPIPS remarkably. For example, it outperformed the original NeRF by up to 54.04% in rendering quality.

Significance and Applications

IL-NeRF's ability to learn incrementally is highly significant for real-world applications in fields like automotive and remote sensing. By handling both the challenges of catastrophic forgetting and alignment of camera poses without requiring the entire dataset, IL-NeRF broadens the potential for on-the-go 3D scene reconstruction using NeRF methodologies. This innovation meets practical needs where data might be scarce or sequential and opens avenues for more robust continual learning models in artificial intelligence.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free