Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (2405.18715v2)

Published 29 May 2024 in cs.CV

Abstract: Neural Radiance Fields (NeRFs) have shown remarkable success in synthesizing photorealistic views from multi-view images of static scenes, but face challenges in dynamic, real-world environments with distractors like moving objects, shadows, and lighting changes. Existing methods manage controlled environments and low occlusion ratios but fall short in render quality, especially under high occlusion scenarios. In this paper, we introduce NeRF On-the-go, a simple yet effective approach that enables the robust synthesis of novel views in complex, in-the-wild scenes from only casually captured image sequences. Delving into uncertainty, our method not only efficiently eliminates distractors, even when they are predominant in captures, but also achieves a notably faster convergence speed. Through comprehensive experiments on various scenes, our method demonstrates a significant improvement over state-of-the-art techniques. This advancement opens new avenues for NeRF in diverse and dynamic real-world applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
  2. Robust dense mapping for large-scale dynamic environments. In ICRA, 2018.
  3. Dynaslam: Tracking, mapping, and inpainting in dynamic scenes. RA-L, 2018.
  4. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
  5. Hallucinated neural radiance fields in the wild. In CVPR, 2022.
  6. Uncertainty estimation for data-driven visual odometry. IEEE Transactions on Robotics, 2020.
  7. Neural radiance flow for 4d view synthesis and video processing. In ICCV, 2021.
  8. Direct sparse odometry. PAMI, 2017.
  9. The stdyn-slam: a stereo vision and semantic segmentation approach for vslam in dynamic outdoor environments. IEEE Access, 2022.
  10. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021.
  11. Bayes’ rays: Uncertainty quantification for neural radiance fields. In CVPR, 2024.
  12. Kubric: a scalable dataset generator. In CVPR, 2022a.
  13. Kubric: A scalable dataset generator. In CVPR, 2022b.
  14. Identity mappings in deep residual networks. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 630–645. Springer, 2016.
  15. Gradient-based uncertainty for monocular depth estimation. In ECCV, 2022.
  16. Clusterslam: A slam backend for simultaneous rigid body clustering and motion estimation. In CVPR, 2019.
  17. Efficient uncertainty estimation for semantic segmentation in videos. In ECCV, 2018.
  18. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV, 2021.
  19. Neu-nbv: Next best view planning using uncertainty estimation in image-based neural rendering. In IROS, 2023.
  20. What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS, 2017.
  21. Up-nerf: Unconstrained pose-prior-free neural radiance fields. In NeurIPS, 2023.
  22. Segment anything. In ICCV, 2023.
  23. Neural 3d video synthesis from multi-view video. In CVPR, 2022.
  24. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
  25. Neural scene chronology. In CVPR, 2023.
  26. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021a.
  27. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021b.
  28. Symmetry and uncertainty-aware object slam for 6dof object pose estimation. In CVPR, 2022.
  29. Keypointnerf: Generalizing image-based volumetric avatars using relative spatial encoding of keypoints. In ECCV, 2022.
  30. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  31. Deep deterministic uncertainty for semantic segmentation. In CVPR, 2023.
  32. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 2017.
  33. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  34. Activenerf: Learning where to see with uncertainty estimation. In ECCV, 2022.
  35. Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields. ACM TOG, 2021.
  36. On the uncertainty of self-supervised monocular depth estimation. In CVPR, 2020.
  37. Neurar: Neural uncertainty for autonomous 3d reconstruction with implicit neural representations. RA-L, 2023.
  38. Urban radiance fields. In CVPR, 2022.
  39. Robustnerf: Ignoring distractors with robust losses. In CVPR, 2023.
  40. Uncle-slam: Uncertainty learning for dense neural slam. In ICCV Workshops, 2023.
  41. Structure-from-motion revisited. In CVPR, 2016.
  42. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016.
  43. Graf: Generative radiance fields for 3d-aware image synthesis. In NeurIPS, 2020.
  44. Estimating 3d uncertainty field: Quantifying uncertainty for neural radiance fields. In ICRA, 2024.
  45. Dytanvo: Joint refinement of visual odometry and motion segmentation in dynamic environments. In ICRA, 2023.
  46. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
  47. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. In NeurIPS, 2021.
  48. NeuralDiff: Segmenting 3D objects that move in egocentric videos. In 3DV, 2021.
  49. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2105.05994, 2021.
  50. Neural point-based volumetric avatar: Surface-guided neural points for efficient and photorealistic volumetric head avatar. In SIGGRAPH Asia Conference Papers, 2023.
  51. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004.
  52. D2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. In NeurIPS, 2022.
  53. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021.
  54. Debsdf: Delving into the details and bias of neural indoor scene reconstruction. arXiv preprint arXiv:2308.15536, 2023.
  55. S3im: Stochastic structural similarity and its unreasonable effectiveness for neural fields. In ICCV, 2023.
  56. Sinnerf: Training neural radiance fields on complex scenes from a single image. In ECCV, 2022.
  57. Desrf: Deformable stylized radiance field. In CVPR, 2023.
  58. Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. In ICLR, 2024.
  59. D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry. In CVPR, 2020.
  60. Cubeslam: Monocular 3-d object slam. IEEE Transactions on Robotics, 2019.
  61. Deflowslam: Self-supervised scene motion decomposition for dynamic dense slam. arXiv preprint arXiv:2207.08794, 2022.
  62. Pvo: Panoptic visual odometry. In CVPR, 2023.
  63. Ds-slam: A semantic visual slam towards dynamic environments. In IROS, 2018.
  64. Vdo-slam: a visual dynamic object-aware slam system. arXiv preprint arXiv:2005.11052, 2020.
  65. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  66. Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild. In ECCV, 2022.
Citations (15)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now