Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF (2404.00874v1)

Published 1 Apr 2024 in cs.CV

Abstract: We present DiSR-NeRF, a diffusion-guided framework for view-consistent super-resolution (SR) NeRF. Unlike prior works, we circumvent the requirement for high-resolution (HR) reference images by leveraging existing powerful 2D super-resolution models. Nonetheless, independent SR 2D images are often inconsistent across different views. We thus propose Iterative 3D Synchronization (I3DS) to mitigate the inconsistency problem via the inherent multi-view consistency property of NeRF. Specifically, our I3DS alternates between upscaling low-resolution (LR) rendered images with diffusion models, and updating the underlying 3D representation with standard NeRF training. We further introduce Renoised Score Distillation (RSD), a novel score-distillation objective for 2D image resolution. Our RSD combines features from ancestral sampling and Score Distillation Sampling (SDS) to generate sharp images that are also LR-consistent. Qualitative and quantitative results on both synthetic and real-world datasets demonstrate that our DiSR-NeRF can achieve better results on NeRF super-resolution compared with existing works. Code and video results available at the project website.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
  2. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
  3. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. arXiv preprint arXiv:2103.15595, 2021.
  4. Tensorf: Tensorial radiance fields. arXiv preprint arXiv:2203.09517, 2022a.
  5. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  6. Local-to-global registration for bundle-adjusting neural radiance fields. arXiv preprint arXiv:2211.11505, 2022b.
  7. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501–5510, 2022.
  8. Super-nerf: View-consistent detail generation for nerf super-resolution, 2023.
  9. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  10. Denoising diffusion probabilistic models, 2020.
  11. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106.15282, 2021.
  12. Learning to stylize novel views, 2021.
  13. Refsr-nerf: Towards high fidelity and super resolution view synthesis. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8244–8253, Los Alamitos, CA, USA, 2023a. IEEE Computer Society.
  14. Dreamtime: An improved optimization strategy for text-to-3d content creation, 2023b.
  15. Self-calibrating neural radiance fields. In ICCV, 2021.
  16. Srflow-da: Super-resolution using normalizing flow with deep convolutional block. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021.
  17. Nerf-vae: A geometry aware 3d scene generative model. https://arxiv.org/abs/2104.00587, 2021.
  18. Photo-realistic single image super-resolution using a generative adversarial network, 2017.
  19. Srdiff: Single image super-resolution with diffusion probabilistic models, 2021a.
  20. Nemi: Unifying neural radiance fields with multiplane images for novel view synthesis. CoRR, abs/2103.14910, 2021b.
  21. Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling. In IEEE International Conference on Computer Vision, 2021.
  22. Barf: Bundle-adjusting neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
  23. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  24. Zero-1-to-3: Zero-shot one image to 3d object, 2023.
  25. Srflow: Learning the super-resolution space with normalizing flow, 2020.
  26. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600, 2022.
  27. Switch-nerf: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In International Conference on Learning Representations (ICLR), 2023.
  28. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
  29. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  30. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 20(3):209–212, 2013.
  31. Differentiable image parameterizations. Distill, 2018. https://distill.pub/2018/differentiable-parameterizations.
  32. Instant neural graphics primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.05989, 2022.
  33. Do deep generative models know what they don’t know?, 2019.
  34. Snerf: Stylized neural implicit representations for 3d scenes, 2022.
  35. Improved denoising diffusion probabilistic models, 2021.
  36. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
  37. High-resolution image synthesis with latent diffusion models, 2022.
  38. Image super-resolution via iterative refinement, 2021.
  39. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
  40. Denoising diffusion implicit models, 2022.
  41. Generative modeling by estimating gradients of the data distribution, 2020.
  42. Score-based generative modeling through stochastic differential equations, 2021.
  43. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. CVPR, 2022.
  44. Block-NeRF: Scalable large scene neural view synthesis. arXiv, 2022.
  45. Nerf-sr: High-quality neural radiance fields using supersampling. arXiv, 2021a.
  46. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation, 2022.
  47. Ibrnet: Learning multi-view image-based rendering. arXiv preprint arXiv:2102.13090, 2021b.
  48. Esrgan: Enhanced super-resolution generative adversarial networks, 2018.
  49. Real-esrgan: Training real-world blind super-resolution with pure synthetic data, 2021c.
  50. 4k-nerf: High fidelity neural radiance fields at ultra high resolutions, 2023a.
  51. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.16213, 2023b.
  52. Diffir: Efficient diffusion model for image restoration, 2023.
  53. Sinnerf: Training neural radiance fields on complex scenes from a single image. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII, pages 736–753. Springer, 2022a.
  54. Point-nerf: Point-based neural radiance fields. arXiv preprint arXiv:2201.08845, 2022b.
  55. Local implicit normalizing flow for arbitrary-scale image super-resolution, 2023.
  56. pixelNeRF: Neural radiance fields from one or few images. https://arxiv.org/abs/2012.02190, 2020.
  57. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  58. Hifa: High-fidelity text-to-3d with advanced diffusion guidance, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now