Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Phase Coded Image Prior (2404.03906v1)

Published 5 Apr 2024 in eess.IV and cs.CV

Abstract: Phase-coded imaging is a computational imaging method designed to tackle tasks such as passive depth estimation and extended depth of field (EDOF) using depth cues inserted during image capture. Most of the current deep learning-based methods for depth estimation or all-in-focus imaging require a training dataset with high-quality depth maps and an optimal focus point at infinity for all-in-focus images. Such datasets are difficult to create, usually synthetic, and require external graphic programs. We propose a new method named "Deep Phase Coded Image Prior" (DPCIP) for jointly recovering the depth map and all-in-focus image from a coded-phase image using solely the captured image and the optical information of the imaging system. Our approach does not depend on any specific dataset and surpasses prior supervised techniques utilizing the same imaging system. This improvement is achieved through the utilization of a problem formulation based on implicit neural representation (INR) and deep image prior (DIP). Due to our zero-shot method, we overcome the barrier of acquiring accurate ground-truth data of depth maps and all-in-focus images for each new phase-coded system introduced. This allows focusing mainly on developing the imaging system, and not on ground-truth data collection.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9446–9454.
  2. A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” ACM transactions on graphics (TOG), vol. 26, no. 3, pp. 70–es, 2007.
  3. H. Haim, S. Elmalem, R. Giryes, A. M. Bronstein, and E. Marom, “Depth estimation from a single image using deep learned phase coded mask,” IEEE Transactions on Computational Imaging, vol. 4, no. 3, pp. 298–310, 2018.
  4. Y. Gil, S. Elmalem, H. Haim, E. Marom, and R. Giryes, “Online training of stereo self-calibration using monocular depth estimation,” IEEE Transactions on Computational Imaging, vol. 7, pp. 812–823, 2021.
  5. J. Chang and G. Wetzstein, “Deep optics for monocular depth estimation and 3d object detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 10 193–10 202.
  6. Y. Wu, V. Boominathan, H. Chen, A. Sankaranarayanan, and A. Veeraraghavan, “Phasecam3d — learning phase masks for passive single view depth estimation,” in 2019 IEEE International Conference on Computational Photography (ICCP), 2019, pp. 1–12.
  7. S. Elmalem, R. Giryes, and E. Marom, “Learned phase coded aperture for the benefit of depth of field extension,” Optics express, vol. 26, no. 12, pp. 15 316–15 331, 2018.
  8. U. Akpinar, E. Sahin, M. Meem, R. Menon, and A. Gotchev, “Learning wavefront coding for extended depth of field imaging,” IEEE Transactions on Image Processing, vol. 30, pp. 3307–3320, 2021.
  9. I. Gkioulekas, J. Chen, J. T. Barron, N. Wadhwa, P. Srinivasan, R. Garg, S. Xin, and T. Xue, “Defocus map estimation and blur removal from a single dual-pixel image,” 2021.
  10. V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Heidrich, F. Heide, and G. Wetzstein, “End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1–13, 2018.
  11. E. Yosef, S. Elmalem, and R. Giryes, “Video reconstruction from a single motion blurred image using learned dynamic phase coding,” arXiv preprint arXiv:2112.14768, 2021.
  12. Y. Gandelsman, A. Shocher, and M. Irani, “” double-dip”: unsupervised image decomposition via coupled deep-image-priors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 11 026–11 035.
  13. D. Ren, K. Zhang, Q. Wang, Q. Hu, and W. Zuo, “Neural blind deconvolution using deep priors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3341–3350.
  14. T. Yokota, K. Kawai, M. Sakata, Y. Kimura, and H. Hontani, “Dynamic pet image reconstruction using nonnegative matrix factorization incorporated with deep image prior,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  15. F. Hashimoto, K. Ote, and Y. Onishi, “PET image reconstruction incorporating deep image prior and a forward projection model,” IEEE Transactions on Radiation and Plasma Medical Sciences, vol. 6, no. 8, pp. 841–846, nov 2022. [Online]. Available: https://doi.org/10.1109%2Ftrpms.2022.3161569
  16. K. Gong, C. Catana, J. Qi, and Q. Li, “Pet image reconstruction using deep image prior,” IEEE Transactions on Medical Imaging, vol. 38, no. 7, pp. 1655–1665, 2019.
  17. ——, “Direct reconstruction of linear parametric images from dynamic pet using nonlocal deep image prior,” IEEE Transactions on Medical Imaging, vol. 41, no. 3, pp. 680–689, 2022.
  18. G. Mataev, M. Elad, and P. Milanfar, “Deepred: Deep image prior powered by red,” ArXiv, vol. abs/1903.10176, 2019.
  19. R. Fermanian, M. Le Pendu, and C. Guillemot, “Regularizing the Deep Image Prior with a Learned Denoiser for Linear Inverse Problems,” in MMSP 2021 - IEEE 23rd International Workshop on Multimedia Siganl Processing.   Tampere, Finland: IEEE, Oct. 2021, pp. 1–6. [Online]. Available: https://hal.archives-ouvertes.fr/hal-03310533
  20. E. Kurniawan, Y. Park, and S. Lee, “Noise-resistant demosaicing with deep image prior network and random rgbw color filter array,” Sensors, vol. 22, no. 5, p. 1767, 2022.
  21. Y.-C. Chen, C. Gao, E. Robb, and J.-B. Huang, “Nas-dip: Learning deep image prior with neural architecture search,” in European Conference on Computer Vision.   Springer, 2020, pp. 442–459.
  22. N. Shabtay, E. Schwartz, and R. Giryes, “Pip: Positional-encoding image prior,” arXiv preprint arXiv:2211.14298, 2022.
  23. M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng, “Fourier features let networks learn high frequency functions in low dimensional domains,” Advances in Neural Information Processing Systems, vol. 33, pp. 7537–7547, 2020.
  24. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, 2020, pp. 405–421.
  25. V. Sitzmann, J. N. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” in arXiv, 2020.
  26. A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, “Plenoctrees for real-time rendering of neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 5752–5761.
  27. J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Zip-nerf: Anti-aliased grid-based neural radiance fields,” arXiv preprint arXiv:2304.06706, 2023.
  28. T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Transactions on Graphics (ToG), vol. 41, no. 4, pp. 1–15, 2022.
  29. J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 5855–5864.
  30. J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 5470–5479.
  31. M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-nerf: Scalable large scene neural view synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 8248–8258.
  32. A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields from one or few images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4578–4587.
  33. B. Mildenhall, P. Hedman, R. Martin-Brualla, P. P. Srinivasan, and J. T. Barron, “Nerf in the dark: High dynamic range view synthesis from noisy raw images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 16 190–16 199.
  34. S. Ramasinghe and S. Lucey, “Beyond periodicity: towards a unifying framework for activations in coordinate-mlps,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII.   Springer, 2022, pp. 142–158.
  35. A. Hertz, O. Perel, R. Giryes, O. Sorkine-Hornung, and D. Cohen-Or, “Sape: Spatially-adaptive progressive encoding for neural optimization,” Advances in Neural Information Processing Systems, vol. 34, pp. 8820–8832, 2021.
  36. D. B. Lindell, D. Van Veen, J. J. Park, and G. Wetzstein, “Bacon: Band-limited coordinate networks for multiscale scene representation,” in CVPR, 2022.
  37. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  38. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  39. V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” Advances in Neural Information Processing Systems, vol. 33, pp. 7462–7473, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now