Papers
Topics
Authors
Recent
Search
2000 character limit reached

Global Latent Neural Rendering

Published 13 Dec 2023 in cs.CV | (2312.08338v2)

Abstract: A recent trend among generalizable novel view synthesis methods is to learn a rendering operator acting over single camera rays. This approach is promising because it removes the need for explicit volumetric rendering, but it effectively treats target images as collections of independent pixels. Here, we propose to learn a global rendering operator acting over all camera rays jointly. We show that the right representation to enable such rendering is a 5-dimensional plane sweep volume consisting of the projection of the input images on a set of planes facing the target camera. Based on this understanding, we introduce our Convolutional Global Latent Renderer (ConvGLR), an efficient convolutional architecture that performs the rendering operation globally in a low-resolution latent space. Experiments on various datasets under sparse and generalizable setups show that our approach consistently outperforms existing methods by significant margins.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. Learning neural light fields with ray-space embedding. In CVPR, pages 19819–19829, 2022.
  2. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, pages 5855–5864, 2021.
  3. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV, 2023.
  4. Basicvsr: The search for essential components in video super-resolution and beyond. In CVPR, pages 4947–4956, 2021.
  5. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, pages 14124–14133, 2021.
  6. Tensorf: Tensorial radiance fields. In ECCV, pages 333–350. Springer, 2022.
  7. Deep stereo using adaptive thin volume representation with uncertainty awareness. In CVPR, pages 2524–2534, 2020.
  8. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In CVPR, pages 7911–7920, 2021.
  9. Robert T Collins. A space-sweep approach to true multi-image matching. In CVPR, pages 358–363, 1996.
  10. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. ACM Transactions on Graphics (TOG), pages 11–20, 1996.
  11. Depth-supervised nerf: Fewer views and faster training for free. In CVPR, pages 12882–12891, 2022.
  12. Learning to render novel views from wide-baseline stereo pairs. In CVPR, 2023.
  13. Deepstereo: Learning to predict new views from the world’s imagery. In CVPR, pages 5515–5524, 2016.
  14. Deepview: View synthesis with learned gradient descent. In CVPR, pages 2367–2376, 2019.
  15. Plenoxels: Radiance fields without neural networks. In CVPR, pages 5501–5510, 2022.
  16. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, page 43–54, 1996.
  17. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In CVPR, pages 2495–2504, 2020.
  18. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. ICCV, 2023.
  19. Single-view view synthesis in the wild with learned adaptive multiplane images. In ACM SIGGRAPH, 2022.
  20. Multiple view geometry in computer vision. Cambridge university press, 2003.
  21. Casual 3d photography. ACM Transactions on Graphics (TOG), 36(6):1–15, 2017.
  22. Worldsheet: Wrapping the world in a 3d sheet for view synthesis from a single image. In ICCV, pages 12528–12537, 2021.
  23. Dpsnet: End-to-end deep plane sweep stereo. In ICLR, 2019.
  24. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV, pages 5885–5894, 2021.
  25. To nerf or not to nerf. https://codalab.lisn.upsaclay.fr/competitions/14427, 2023a.
  26. Vschh 2023: A benchmark for the view synthesis challenge of human heads. In ICCV, pages 1121–1128, 2023b.
  27. Large scale multi-view stereopsis evaluation. In CVPR, pages 406–413, 2014.
  28. Geonerf: Generalizing nerf with geometry priors. In CVPR, pages 18365–18375, 2022.
  29. Learning-based view synthesis for light field cameras. ACM TOG, 35(6):1–10, 2016.
  30. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1–14, 2023.
  31. Stereo magnification with multi-layer images. In CVPR, pages 8687–8696, 2022.
  32. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, page 31–42, 1996.
  33. Mine: Towards continuous depth mpi with nerf for novel view synthesis. In ICCV, pages 12578–12588, 2021.
  34. Dynibar: Neural dynamic image-based rendering. In CVPR, 2023.
  35. Enhanced deep residual networks for single image super-resolution. In CVPR workshop, pages 136–144, 2017.
  36. Efficient neural radiance fields for interactive free-viewpoint video. In ACM TOG, pages 1–9, 2022.
  37. Deep multi depth panoramas for view synthesis. In ECCV, pages 328–344. Springer, 2020.
  38. An intriguing failing of convolutional neural networks and the coordconv solution. 31, 2018.
  39. Neural rays for occlusion-aware image-based rendering. In CVPR, 2022.
  40. Leonard McMillan Jr. An image-based approach to three-dimensional computer graphics. The University of North Carolina at Chapel Hill, 1997.
  41. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM TOG, 38(4):1–14, 2019.
  42. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  43. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG, 41(4):1–15, 2022.
  44. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, pages 5480–5490, 2022.
  45. Soft 3d reconstruction for view synthesis. 36(6), 2017a.
  46. Soft 3d reconstruction for view synthesis. ACM TOG, 36(6):1–11, 2017b.
  47. Diner: Depth-aware image-based neural radiance fields. In CVPR, pages 12449–12459, 2023.
  48. Learning transferable visual models from natural language supervision. pages 8748–8763, 2021.
  49. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In ICCV, pages 10901–10911, 2021.
  50. Free view synthesis. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16, pages 623–640. Springer, 2020.
  51. Stable view synthesis. In CVPR, pages 12216–12225, 2021.
  52. Ganerf: Leveraging discriminators to optimize neural radiance fields. ACM Trans. Graph., 2023.
  53. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  54. Structure-from-motion revisited. In CVPR, pages 4104–4113, 2016.
  55. Flipnerf: Flipped reflection rays for few-shot novel view synthesis. In ICCV, pages 22883–22893, 2023a.
  56. Mixnerf: Modeling a ray with mixture density for novel view synthesis from sparse inputs. In CVPR, pages 20659–20668, 2023b.
  57. Layered depth images. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pages 231–242, 1998.
  58. Review of image-based rendering techniques. In Visual Communications and Image Processing 2000, pages 2–13. SPIE, 2000.
  59. Scene representation networks: Continuous 3d-structure-aware neural scene representations. 32, 2019.
  60. Light field networks: Neural scene representations with single-evaluation rendering. 34:19313–19325, 2021.
  61. Self-improving multiplane-to-layer images for novel view synthesis. In WACV, pages 4309–4318, 2023.
  62. Pushing the boundaries of view extrapolation with multiplane images. In CVPR, pages 175–184, 2019.
  63. Generalizable patch-based neural rendering. In ECCV, 2022a.
  64. Light field neural rendering. In CVPR, pages 8269–8279, 2022b.
  65. Stereo matching with transparency and matting. In ICCV, pages 517–524, 1998.
  66. Efficient view synthesis and 3d-based multi-frame denoising with multiplane feature representations. In CVPR, 2023.
  67. Grf: Learning a general radiance field for 3d representation and rendering. In CVPR, pages 15182–15192, 2021.
  68. Single-view view synthesis with multiplane images. In CVPR, pages 551–560, 2020.
  69. Attention is all you need. 30, 2017.
  70. Ibrnet: Learning multi-view image-based rendering. In CVPR, pages 4690–4699, 2021.
  71. Nex: Real-time view synthesis with neural basis expansion. In CVPR, pages 8534–8543, 2021.
  72. Learning inverse depth regression for multi-view stereo with correlation cost volume. In AAAI, pages 12508–12515, 2020.
  73. Cost volume pyramid based depth inference for multi-view stereo. In CVPR, pages 4877–4886, 2020.
  74. Mvsnet: Depth inference for unstructured multi-view stereo. In ECCV, pages 767–783, 2018.
  75. Recurrent mvsnet for high-resolution multi-view stereo depth inference. In CVPR, pages 5525–5534, 2019.
  76. pixelnerf: Neural radiance fields from one or few images. In CVPR, pages 4578–4587, 2021.
  77. Ilsh: The imperial light-stage head dataset for human head view synthesis. In ICCV, pages 1112–1120, 2023.
  78. Stereo magnification: learning view synthesis using multiplane images. ACM TOG, 37(4):1–12, 2018.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 5 likes about this paper.

Sign Up for Free