Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 73 tok/s
Gemini 2.5 Pro 39 tok/s Pro
GPT-5 Medium 27 tok/s Pro
GPT-5 High 19 tok/s Pro
GPT-4o 115 tok/s Pro
Kimi K2 226 tok/s Pro
GPT OSS 120B 461 tok/s Pro
Claude Sonnet 4 38 tok/s Pro
2000 character limit reached

BoostMVSNeRFs: Boosting MVS-based NeRFs to Generalizable View Synthesis in Large-scale Scenes (2407.15848v1)

Published 22 Jul 2024 in cs.CV

Abstract: While Neural Radiance Fields (NeRFs) have demonstrated exceptional quality, their protracted training duration remains a limitation. Generalizable and MVS-based NeRFs, although capable of mitigating training time, often incur tradeoffs in quality. This paper presents a novel approach called BoostMVSNeRFs to enhance the rendering quality of MVS-based NeRFs in large-scale scenes. We first identify limitations in MVS-based NeRF methods, such as restricted viewport coverage and artifacts due to limited input views. Then, we address these limitations by proposing a new method that selects and combines multiple cost volumes during volume rendering. Our method does not require training and can adapt to any MVS-based NeRF methods in a feed-forward fashion to improve rendering quality. Furthermore, our approach is also end-to-end trainable, allowing fine-tuning on specific scenes. We demonstrate the effectiveness of our method through experiments on large-scale datasets, showing significant rendering quality improvements in large-scale scenes and unbounded outdoor scenarios. We release the source code of BoostMVSNeRFs at https://su-terry.github.io/BoostMVSNeRFs/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (94)
  1. Neural point-based graphics. In ECCV.
  2. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV.
  3. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR.
  4. Zip-NeRF: Anti-aliased grid-based neural radiance fields. In ICCV.
  5. Nerd: Neural reflectance decomposition from image collections. In ICCV.
  6. Fwd: Real-time novel view synthesis with forward warping and depth. In CVPR.
  7. Depth synthesis and local warps for plausible image-based navigation. ACM TOG (2013).
  8. Tensorf: Tensorial radiance fields. In ECCV.
  9. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV.
  10. Point-based multi-view stereo network. In ICCV.
  11. Explicit Correspondence Matching for Generalizable Neural Radiance Fields. arXiv preprint arXiv:2304.12294 (2023).
  12. Improving Robustness for Joint Optimization of Camera Poses and Decomposed Low-Rank Tensorial Radiance Fields. In AAAI.
  13. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In CVPR.
  14. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR.
  15. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2.
  16. Depth-supervised nerf: Fewer views and faster training for free. In CVPR.
  17. Peeking behind objects: Layered depth prediction from a single image. Pattern Recognition Letters (2019).
  18. Deepview: View synthesis with learned gradient descent. In CVPR.
  19. Deepstereo: Learning to predict new views from the world’s imagery. In CVPR.
  20. SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes. In CVPR.
  21. Layered depth images. In SIGGRAPH.
  22. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In CVPR.
  23. Baking neural radiance fields for real-time view synthesis. In ICCV.
  24. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV.
  25. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In CVPR.
  26. Geonerf: Generalizing nerf with geometry priors. In CVPR.
  27. Learning-based view synthesis for light field cameras. ACM TOG (2016).
  28. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR.
  29. Neural 3d video synthesis from multi-view video. In CVPR.
  30. Crowdsampling the plenoptic function. In ECCV.
  31. Im4d: High-fidelity and real-time novel view synthesis for dynamic scenes. arXiv preprint arXiv:2310.08585 (2023).
  32. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia.
  33. Vision transformer for nerf-based view synthesis from a single input image. In WACV.
  34. Neural rendering and reenactment of human actor videos. ACM TOG (2019).
  35. Neural rays for occlusion-aware image-based rendering. In CVPR.
  36. Robust dynamic radiance fields. In CVPR.
  37. Neural volumes: Learning dynamic renderable volumes from images. ACM TOG (2019).
  38. Mixture of volumetric primitives for efficient neural rendering. ACM TOG (2021).
  39. Progressively optimized local radiance fields for robust view synthesis. In CVPR.
  40. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM TOG (2019).
  41. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.
  42. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG (2022).
  43. Extracting triangular 3d models, materials, and lighting from images. In CVPR.
  44. An analysis of approximations for maximizing submodular set functions—I. Mathematical programming (1978).
  45. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR.
  46. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV.
  47. Nerfies: Deformable neural radiance fields. In ICCV.
  48. Eric Penner and Li Zhang. 2017. Soft 3d reconstruction for view synthesis. ACM TOG (2017).
  49. Surfels: Surface elements as rendering primitives. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques.
  50. D-nerf: Neural radiance fields for dynamic scenes. In CVPR.
  51. Gernot Riegler and Vladlen Koltun. 2020. Free view synthesis. In ECCV.
  52. Dense depth priors for neural radiance fields from sparse input views. In CVPR.
  53. MixNeRF: Modeling a Ray with Mixture Density for Novel View Synthesis from Sparse Inputs. In CVPR.
  54. Garf: Geometry-aware generalized neural radiance field. arXiv preprint arXiv:2212.02280 (2022).
  55. 3d photography using context-aware layered depth inpainting. In CVPR.
  56. Deepvoxels: Learning persistent 3d feature embeddings. In CVPR.
  57. Nagabhushan Somraj and Rajiv Soundararajan. 2023. ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields. (2023).
  58. Pushing the boundaries of view extrapolation with multiplane images. In CVPR.
  59. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR.
  60. Block-nerf: Scalable large scene neural view synthesis. In CVPR.
  61. Deferred neural rendering: Image synthesis using neural textures. ACM TOG (2019).
  62. Alex Trevithick and Bo Yang. 2021. Grf: Learning a general radiance field for 3d representation and rendering. In ICCV.
  63. Richard Tucker and Noah Snavely. 2020. Single-view view synthesis with multiplane images. In CVPR.
  64. Layer-structured 3d scene inference via view synthesis. In ECCV.
  65. SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates. In CVPR.
  66. Let there be color! Large-scale texturing of 3D reconstructions. In ECCV.
  67. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In ICCV.
  68. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS.
  69. F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories. In CVPR.
  70. Ibrnet: Learning multi-view image-based rendering. In CVPR.
  71. Image quality assessment: from error visibility to structural similarity. IEEE TIP (2004).
  72. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV.
  73. Nex: Real-time view synthesis with neural basis expansion. In CVPR.
  74. Surface light fields for 3D photography. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2.
  75. ReconFusion: 3D Reconstruction with Diffusion Priors. arXiv preprint arXiv:2312.02981 (2023).
  76. Jamie Wynn and Daniyar Turmukhambetov. 2023. Diffusionerf: Regularizing neural radiance fields with denoising diffusion models. In CVPR.
  77. Space-time neural irradiance fields for free-viewpoint video. In CVPR.
  78. Point-nerf: Point-based neural radiance fields. In CVPR.
  79. FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization. In CVPR.
  80. Mvsnet: Depth inference for unstructured multi-view stereo. In ECCV.
  81. Recurrent mvsnet for high-resolution multi-view stereo depth inference. In CVPR.
  82. Volume rendering of neural implicit surfaces. In NeurIPS.
  83. Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurIPS.
  84. Plenoxels: Radiance fields without neural networks. In CVPR.
  85. Plenoctrees for real-time rendering of neural radiance fields. In ICCV.
  86. pixelnerf: Neural radiance fields from one or few images. In CVPR.
  87. Zehao Yu and Shenghua Gao. 2020. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. In CVPR.
  88. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR.
  89. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 (2020).
  90. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR.
  91. Nerfusion: Fusing radiance fields for large-scale scene reconstruction. In CVPR.
  92. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM TOG (2021).
  93. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH.
  94. Vdn-nerf: Resolving shape-radiance ambiguity via view-dependence normalization. In CVPR.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 2 posts and received 16 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube