Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CorresNeRF: Image Correspondence Priors for Neural Radiance Fields (2312.06642v1)

Published 11 Dec 2023 in cs.CV

Abstract: Neural Radiance Fields (NeRFs) have achieved impressive results in novel view synthesis and surface reconstruction tasks. However, their performance suffers under challenging scenarios with sparse input views. We present CorresNeRF, a novel method that leverages image correspondence priors computed by off-the-shelf methods to supervise NeRF training. We design adaptive processes for augmentation and filtering to generate dense and high-quality correspondences. The correspondences are then used to regularize NeRF training via the correspondence pixel reprojection and depth loss terms. We evaluate our methods on novel view synthesis and surface reconstruction tasks with density-based and SDF-based NeRF models on different datasets. Our method outperforms previous methods in both photometric and geometric metrics. We show that this simple yet effective technique of using correspondence priors can be applied as a plug-and-play module across different NeRF variants. The project page is at https://yxlao.github.io/corres-nerf.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Learning implicit fields for generative shape modeling. In CVPR, 2019.
  2. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
  3. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
  4. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  5. View independent generative adversarial network for novel view synthesis. In ICCV, 2019.
  6. Deep 3d portrait from a single image. In CVPR, 2020.
  7. Smartportraits: Depth powered handheld smartphone dataset of human portraits for state estimation, reconstruction and synthesis. In CVPR, 2022.
  8. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
  9. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In CVPR, 2022.
  10. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
  11. pixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021.
  12. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV, 2021.
  13. Depth-supervised nerf: Fewer views and faster training for free. In CVPR, 2022.
  14. Dense depth priors for neural radiance fields from sparse input views. In CVPR, 2022.
  15. InfoNeRF: Ray entropy minimization for few-shot neural volume rendering. In CVPR, 2022.
  16. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction. In NeurIPS, 2022.
  17. NeRF++: Analyzing and improving neural radiance fields. arXiv:2010.07492, 2020.
  18. Neuris: Neural reconstruction of indoor scenes using normal priors. In ECCV, 2022.
  19. Structure-from-motion revisited. In CVPR, 2016.
  20. UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, 2021.
  21. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021.
  22. superglue: Learning feature matching with graph neural networks. In CVPR, 2020.
  23. Loftr: Detector-free local feature matching with transformers. In CVPR, 2021.
  24. Dkm: Dense kernelized feature matching for geometry estimation. arXiv:2202.00667, 2022.
  25. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. TOG, 2019.
  26. Large scale multi-view stereopsis evaluation. In CVPR, 2014.
  27. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV, 2021.
  28. Ray priors through reprojection: Improving neural radiance fields for novel view extrapolation. In CVPR, 2022.
  29. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
  30. Neural 3d scene reconstruction with the manhattan-world assumption. In CVPR, 2022.
  31. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021.
  32. Ibrnet: Learning multi-view image-based rendering. In CVPR, 2021.
  33. Learned initializations for optimizing coordinate-based neural representations. In CVPR, 2021.
  34. ConsistentNeRF: Enhancing neural radiance fields with 3d consistency for sparse view synthesis. arXiv:2305.11031, 2023.
  35. SPARF: neural radiance fields from sparse and noisy poses. In CVPR, 2023.
  36. David G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004.
  37. Histograms of oriented gradients for human detection. In CVPR, 2005.
  38. SURF: speeded up robust features. In ECCV, 2006.
  39. LIFT: learned invariant feature transform. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, 2016.
  40. Superpoint: Self-supervised interest point detection and description. In CVPR, 2018.
  41. Neural-guided RANSAC: learning where to sample model hypotheses. In ICCV, 2019.
  42. Quadtree attention for vision transformers. In ICLR, 2022.
  43. Matchformer: Interleaving attention in transformers for feature matching. In ACCV, 2022.
  44. Dgc-net: Dense geometric correspondence network. In WACV, 2019.
  45. Glu-net: Global-local universal network for dense flow and correspondences. In CVPR, 2020.
  46. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
  47. MegaDepth: Learning single-view depth prediction from internet photos. In CVPR, 2018.
  48. Self-calibrating neural radiance fields. In ICCV, 2021.
  49. Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurIPS, 2020.
  50. Image quality assessment: from error visibility to structural similarity. TIP, 2004.
  51. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  52. Volume rendering of neural implicit surfaces. In NeurIPS, 2021.
  53. Nerf-supervision: Learning dense object descriptors from neural radiance fields. In ICRA, 2022.
  54. Template nerf: Towards modeling dense shape correspondences from category-specific object images. arXiv:2111.04237, 2021.
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. CorresNeRF: Image Correspondence Priors for Neural Radiance Fields