Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
3 tokens/sec
DeepSeek R1 via Azure Pro
51 tokens/sec
2000 character limit reached

Voxel-Mesh Hybrid Representation for Real-Time View Synthesis (2403.06505v2)

Published 11 Mar 2024 in cs.CV

Abstract: The neural radiance fields (NeRF) have emerged as a prominent methodology for synthesizing realistic images of novel views. While neural radiance representations based on voxels or mesh individually offer distinct advantages, excelling in either rendering quality or speed, each has limitations in the other aspect. In response, we propose a hybrid representation named Vosh, seamlessly combining both voxel and mesh components in hybrid rendering for view synthesis. Vosh is meticulously crafted by optimizing the voxel grid based on neural rendering, strategically meshing a portion of the volumetric density field to surface. Therefore, it excels in fast rendering scenes with simple geometry and textures through its mesh component, while simultaneously enabling high-quality rendering in intricate regions by leveraging voxel component. The flexibility of Vosh is showcased through the ability to adjust hybrid ratios, providing users the ability to control the balance between rendering quality and speed based on flexible usage. Experimental results demonstrate that our method achieves commendable trade-off between rendering quality and speed, and notably has real-time performance on mobile devices. The interactive web demo and code are available at https://zyyzyy06.github.io/Vosh.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of CVPR. 5470–5479.
  2. Efficient geometry-aware 3D generative adversarial networks. In Proceedings of CVPR. 16123–16133.
  3. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proceedings of CVPR. 16569–16578.
  4. Plenoxels: Radiance fields without neural networks. In Proceedings of CVPR. 5501–5510.
  5. Nerf: Neural radiance field in 3d vision, a comprehensive review. arXiv preprint arXiv:2210.00379 (2022).
  6. VMesh: Hybrid volume-mesh representation for efficient view synthesis. In SIGGRAPH Asia 2023 Conference Papers. 1–11.
  7. Baking neural radiance fields for real-time view synthesis. In Proceedings of ICCV. 5875–5884.
  8. NeRFFaceLighting: Implicit and Disentangled Face Lighting Representation Leveraging Generative Prior in Neural Radiance Fields. ACM Transactions on Graphics 42, 3 (2023), 1–18.
  9. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics 42, 4 (2023), 1–14.
  10. Renderable Neural Radiance Map for Visual Navigation. In Proceedings of CVPR. 9099–9108.
  11. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics 39, 6 (2020), 1–14.
  12. RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering. In Proceedings of ICCAD. 1–9.
  13. EyeNeRF: a hybrid representation for photorealistic synthesis, animation and relighting of human eyes. ACM Transactions on Graphics 41, 4 (2022), 1–16.
  14. Neural sparse voxel fields. Proceedings of NeurIPS 33 (2020), 15651–15663.
  15. William E Lorensen and Harvey E Cline. 1998. Marching cubes: A high resolution 3D surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field. 347–353.
  16. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (2021), 99–106.
  17. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics 41, 4 (2022), 1–15.
  18. Dynamic mesh-aware radiance fields. In Proceedings of ICCV. 385–396.
  19. NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes. arXiv preprint arXiv:2303.09431 (2023).
  20. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of ICCV. 14335–14345.
  21. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics 42, 4 (2023), 1–12.
  22. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of CVPR. 5459–5469.
  23. Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis. ACM Transactions on Graphics (ToG) 41, 6 (2022), 1–10.
  24. Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement. In Proceedings of ICCV. 17739–17749.
  25. Advances in neural rendering. In Computer Graphics Forum, Vol. 41. 703–735.
  26. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.
  27. Adaptive Shells for Efficient Neural Radiance Field Rendering. ACM Transactions on Graphics 42, 6 (2023), 1–15.
  28. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In Proceedings of ECCV. 597–614.
  29. BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis. In ACM SIGGRAPH 2023 Conference Proceedings. 1–9.
  30. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of ICCV. 5752–5761.
  31. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of CVPR. 586–595.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now