Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization (2308.10001v2)

Published 19 Aug 2023 in cs.CV

Abstract: Neural Radiance Fields (NeRF) have shown promise in generating realistic novel views from sparse scene images. However, existing NeRF approaches often encounter challenges due to the lack of explicit 3D supervision and imprecise camera poses, resulting in suboptimal outcomes. To tackle these issues, we propose AltNeRF -- a novel framework designed to create resilient NeRF representations using self-supervised monocular depth estimation (SMDE) from monocular videos, without relying on known camera poses. SMDE in AltNeRF masterfully learns depth and pose priors to regulate NeRF training. The depth prior enriches NeRF's capacity for precise scene geometry depiction, while the pose prior provides a robust starting point for subsequent pose refinement. Moreover, we introduce an alternating algorithm that harmoniously melds NeRF outputs into SMDE through a consistence-driven mechanism, thus enhancing the integrity of depth priors. This alternation empowers AltNeRF to progressively refine NeRF representations, yielding the synthesis of realistic novel views. Extensive experiments showcase the compelling capabilities of AltNeRF in generating high-fidelity and robust novel views that closely resemble reality.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibrium. In CVPR.
  2. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 5470–5479.
  3. Fast training of neural lumigraph representations using meta learning. NIPS, 34: 172–186.
  4. Unsupervised scale-consistent depth and ego-motion learning from monocular video. NIPS, 32.
  5. NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior. In CVPR.
  6. DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields. In CVPR.
  7. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 5828–5839.
  8. Depth-supervised nerf: Fewer views and faster training for free. In CVPR, 12882–12891.
  9. Digging into self-supervised monocular depth estimation. In ICCV, 3828–3838.
  10. Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras. In ICCV.
  11. 3d packing for self-supervised monocular depth estimation. In CVPR, 2485–2494.
  12. Multi-Frame Self-Supervised Depth With Transformers. In CVPR, 160–170.
  13. Deep residual learning for image recognition. In CVPR, 770–778.
  14. RA-Depth: Resolution Adaptive Self-Supervised Monocular Depth Estimation. In ECCV, 565–581. Springer.
  15. Huber, P. J. 1964. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1): 73 – 101.
  16. Self-calibrating neural radiance fields. In ICCV, 5846–5854.
  17. Monoindoor: Towards good practice of self-supervised monocular depth estimation for indoor environments. In ICCV, 12787–12796.
  18. Ray tracing volume densities. ACM SIGGRAPH, 18(3): 165–174.
  19. Neural lumigraph rendering. In CVPR, 4287–4297.
  20. AdaNeRF: Adaptive Sampling for Real-Time Rendering of Neural Radiance Fields. In ECCV, 254–270. Springer.
  21. Read: Large-scale neural scene rendering for autonomous driving. In AAAI, 2.
  22. Barf: Bundle-adjusting neural radiance fields. In ICCV.
  23. Self-supervised monocular depth estimation for all day images using domain separation. In ICCV, 12737–12746.
  24. Robust Dynamic Radiance Fields. In CVPR.
  25. Hr-depth: High resolution self-supervised monocular depth estimation. In AAAI, 3.
  26. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 7210–7219.
  27. Gnerf: Gan-based neural radiance field without posed camera. In ICCV, 6351–6361.
  28. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In CVPR.
  29. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ToG, 38(4): 1–14.
  30. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 405–421.
  31. Instant neural graphics primitives with a multiresolution hash encoding. ToG, 41(4): 1–15.
  32. Indoor Segmentation and Support Inference from RGBD Images. In ECCV.
  33. Exploiting Pseudo Labels in a Self-Supervised Learning Framework for Improved Monocular Depth Estimation. In CVPR, 1578–1588.
  34. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 10318–10327.
  35. Vision transformers for dense prediction. In ICCV, 12179–12188.
  36. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In CVPR, 12240–12249.
  37. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. arXiv preprint arXiv:2302.12249.
  38. Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction. In ICCV.
  39. Adaptive Co-teaching for Unsupervised Monocular Depth Estimation. In ECCV, 89–105. Springer.
  40. Dense depth priors for neural radiance fields from sparse input views. In CVPR.
  41. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 234–241. Springer.
  42. Structure-from-Motion Revisited. In CVPR.
  43. Feature-metric loss for self-supervised learning of depth and egomotion. In ECCV, 572–588. Springer.
  44. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 8248–8258.
  45. Learned initializations for optimizing coordinate-based neural representations. In CVPR, 2846–2855.
  46. Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs. In CVPR, 12922–12931.
  47. Unsupervised monocular depth estimation for night-time images using adversarial domain feature adaptation. In ECCV, 443–459. Springer.
  48. Digging into Depth Priors for Outdoor Neural Radiance Fields. In ACM MM, 1221–1230.
  49. Regularizing nighttime weirdness: Efficient self-supervised monocular depth estimation in the dark. In ICCV, 16055–16064.
  50. Can Scale-Consistent Monocular Depth Be Learned in a Self-Supervised Scale-Invariant Manner? In ICCV, 12727–12736.
  51. Ibrnet: Learning multi-view image-based rendering. In CVPR, 4690–4699.
  52. Image quality assessment: from error visibility to structural similarity. TIP, 13(4): 600–612.
  53. NeRF–: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064.
  54. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV, 5610–5619.
  55. Unsupervised Depth Completion From Visual Inertial Odometry. IEEE Robotics and Automation Letters, 5(2): 1899–1906.
  56. Toward Practical Monocular Indoor Depth Estimation. In CVPR, 3814–3824.
  57. Desnet: Decomposed scale-consistent network for unsupervised depth completion. In AAAI, 3, 3109–3117.
  58. Plenoctrees for real-time rendering of neural radiance fields. In ICCV, 5752–5761.
  59. pixelnerf: Neural radiance fields from one or few images. In CVPR, 4578–4587.
  60. Anisotropic fourier features for neural image-based rendering and relighting. In AAAI, volume 36, 3152–3160.
  61. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492.
  62. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 586–595.
  63. Unsupervised learning of depth and ego-motion from video. In CVPR, 1851–1858.
  64. View synthesis by appearance flow. In ECCV.
  65. On the continuity of rotation representations in neural networks. In CVPR, 5745–5753.
  66. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In ECCV, 36–53.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Kun Wang (355 papers)
  2. Zhiqiang Yan (43 papers)
  3. Huang Tian (1 paper)
  4. Zhenyu Zhang (250 papers)
  5. Xiang Li (1003 papers)
  6. Jun Li (780 papers)
  7. Jian Yang (506 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now