Curvature regularization for Non-line-of-sight Imaging from Under-sampled Data (2301.00406v4)
Abstract: Non-line-of-sight (NLOS) imaging aims to reconstruct the three-dimensional hidden scenes from the data measured in the line-of-sight, which uses photon time-of-flight information encoded in light after multiple diffuse reflections. The under-sampled scanning data can facilitate fast imaging. However, the resulting reconstruction problem becomes a serious ill-posed inverse problem, the solution of which is highly possibility to be degraded due to noises and distortions. In this paper, we propose novel NLOS reconstruction models based on curvature regularization, i.e., the object-domain curvature regularization model and the dual (signal and object)-domain curvature regularization model. In what follows, we develop efficient optimization algorithms relying on the alternating direction method of multipliers (ADMM) with the backtracking stepsize rule, for which all solvers can be implemented on GPUs. We evaluate the proposed algorithms on both synthetic and real datasets, which achieve state-of-the-art performance, especially in the compressed sensing setting. Based on GPU computing, our algorithm is the most effective among iterative methods, balancing reconstruction quality and computational time. All our codes and data are available at https://github.com/Duanlab123/CurvNLOS.
- D. Faccio, A. Velten, and G. Wetzstein, “Non-line-of-sight imaging,” Nature Reviews Physics, vol. 2, no. 6, pp. 318–327, 2020.
- R. Geng, Y. Hu, Y. Chen et al., “Recent advances on non-line-of-sight imaging: Conventional physical models, deep learning, and new scenes,” APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1, 2021.
- N. Scheiner, F. Kraus, F. Wei, B. Phan, F. Mannan, N. Appenrodt, W. Ritter, J. Dickmann, K. Dietmayer, B. Sick et al., “Seeing around street corners: Non-line-of-sight detection and tracking in-the-wild using doppler radar,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2068–2077.
- M. Isogawa, Y. Yuan, M. O’Toole, and K. M. Kitani, “Optical non-line-of-sight physics-based 3d human pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7013–7022.
- W. Chen, F. Wei, K. N. Kutulakos, S. Rusinkiewicz, and F. Heide, “Learned feature embeddings for non-line-of-sight imaging and recognition,” ACM Transactions on Graphics (ToG), vol. 39, no. 6, pp. 1–18, 2020.
- S. Zhu, Y. M. Sua, P. Rehain, and Y.-P. Huang, “Single photon imaging and sensing of highly obscured objects around the corner,” Optics Express, vol. 29, no. 25, pp. 40 865–40 877, 2021.
- C. Wu, J. Liu, X. Huang, Z.-P. Li, C. Yu, J.-T. Ye, J. Zhang, Q. Zhang, X. Dou, V. K. Goyal et al., “Non–line-of-sight imaging over 1.43 km,” Proceedings of the National Academy of Sciences, vol. 118, no. 10, p. e2024468118, 2021.
- R. Cao, F. de Goumoens, B. Blochet, J. Xu, and C. Yang, “High-resolution non-line-of-sight imaging employing active focusing,” Nature Photonics, pp. 1–7, 2022.
- B. Wang, M.-Y. Zheng, J.-J. Han, X. Huang, X.-P. Xie, F. Xu, Q. Zhang, and J.-W. Pan, “Non-line-of-sight imaging with picosecond temporal resolution,” Physical Review Letters, vol. 127, no. 6, p. 053602, 2021.
- M. O’Toole, D. B. Lindell, and G. Wetzstein, “Confocal non-line-of-sight imaging based on the light-cone transform,” Nature, vol. 555, no. 7696, pp. 338–341, 2018.
- D. B. Lindell and G. Wetzstein, “Three-dimensional imaging through scattering media based on confocal diffuse tomography,” Nature Communications, vol. 11, no. 1, pp. 1–8, 2020.
- O. Gupta, T. Willwacher, A. Velten, A. Veeraraghavan, and R. Raskar, “Reconstruction of hidden 3d shapes using diffuse reflections,” Optics Express, vol. 20, no. 17, pp. 19 096–19 108, 2012.
- M. La Manna, F. Kine, E. Breitbach, J. Jackson, T. Sultan, and A. Velten, “Error backprojection algorithms for non-line-of-sight imaging,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 7, pp. 1615–1626, 2018.
- V. Arellano, D. Gutierrez, and A. Jarabo, “Fast back-projection for non-line of sight reconstruction,” Optics express, vol. 25, no. 10, pp. 11 574–11 583, 2017.
- X. Liu, I. Guillén, M. La Manna, J. H. Nam, S. A. Reza, T. Huu Le, A. Jarabo, D. Gutierrez, and A. Velten, “Non-line-of-sight imaging using phasor-field virtual wave optics,” Nature, vol. 572, no. 7771, pp. 620–623, 2019.
- D. B. Lindell, G. Wetzstein, and M. O’Toole, “Wave-based non-line-of-sight imaging using fast fk migration,” ACM Transactions on Graphics (ToG), vol. 38, no. 4, pp. 1–13, 2019.
- S. Xin, S. Nousias, K. N. Kutulakos, A. C. Sankaranarayanan, S. G. Narasimhan, and I. Gkioulekas, “A theory of fermat paths for non-line-of-sight shape reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 6800–6809.
- F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin, “Diffuse mirrors: 3d reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors,” in IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3222–3229.
- B. Ahn, A. Dave, A. Veeraraghavan, I. Gkioulekas, and A. C. Sankaranarayanan, “Convolutional approximations to the general non-line-of-sight imaging operator,” in IEEE/CVF International Conference on Computer Vision. IEEE, 2019, pp. 7888–7898.
- X. Liu, J. Wang, Z. Li, Z. Shi, X. Fu, and L. Qiu, “Non-line-of-sight reconstruction with signal–object collaborative regularization,” Light: Science & Applications, vol. 10, no. 1, pp. 1–20, 2021.
- J.-T. Ye, X. Huang, Z.-P. Li, and F. Xu, “Compressed sensing for active non-line-of-sight imaging,” Optics Express, vol. 29, no. 2, pp. 1749–1763, 2021.
- J. Liu, Y. Zhou, X. Huang, Z. Li, and F. Xu, “Photon-efficient non-line-of-sight imaging,” IEEE Transactions on Computational Imaging, vol. 8, pp. 639–650, 2022.
- J. G. Chopite, M. B. Hullin, M. Wand, and J. Iseringhausen, “Deep non-line-of-sight reconstruction,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 957–966.
- W. Chen, F. Wei, K. N. Kutulakos, S. Rusinkiewicz, and F. Heide, “Learned feature embeddings for non-line-of-sight imaging and recognition,” ACM Transactions on Graphics, vol. 39, no. 6, pp. 230:1–230:18, 2020.
- M. Isogawa, Y. Yuan, M. O’Toole, and K. M. Kitani, “Optical non-line-of-sight physics-based 3d human pose estimation,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020, pp. 7011–7020.
- S. Shen, Z. Wang, P. Liu, Z. Pan, R. Li, T. Gao, S. Li, and J. Yu, “Non-line-of-sight imaging via neural transient fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 7, pp. 2257–2268, 2021.
- S. Zhu, Y. M. Sua, T. Bu, and Y.-P. Huang, “Compressive non-line-of-sight imaging with deep learning,” Physical Review Applied, vol. 19, no. 3, p. 034090, 2023.
- J. H. Nam, E. Brandt, S. Bauer, X. Liu, M. Renna, A. Tosi, E. Sifakis, and A. Velten, “Low-latency time-of-flight non-line-of-sight imaging at 5 frames per second,” Nature Communications, vol. 12, no. 1, pp. 1–10, 2021.
- C. Pei, A. Zhang, Y. Deng, F. Xu, J. Wu, U. David, L. Li, H. Qiao, L. Fang, and Q. Dai, “Dynamic non-line-of-sight imaging system based on the optimization of point spread functions,” Optics Express, vol. 29, no. 20, pp. 32 349–32 364, 2021.
- M. Isogawa, D. Chan, Y. Yuan, K. Kitani, and M. O’Toole, “Efficient non-line-of-sight imaging from transient sinograms,” in European Conference on Computer Vision. Springer, 2020, pp. 193–208.
- M. Zhang, Y. Shi, W. Sheng, J. Liu, J. Li, Y. Wei, B. Wang, D. Zhang, and Y. Liu, “Non-line-of-sight imaging based on archimedean spiral scanning,” Optics Communications, vol. 537, p. 129450, 2023.
- J. Liu, R. W. Liu, J. Sun, and T. Zeng, “Rank-one prior: Real-time scene recovery,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8845 – 8860, 2023.
- J. Liu, M. Yan, and T. Zeng, “Surface-aware blind image deblurring,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 3, pp. 1041–1055, 2019.
- A. Chambolle and T. Pock, “Total roto-translational variation,” Numerische Mathematik, vol. 142, no. 3, pp. 611–666, 2019.
- B. Dong, H. Ju, Y. Lu, and Z. Shi, “CURE: curvature regularization for missing data recovery,” SIAM Journal on Imaging Sciences, vol. 13, no. 4, pp. 2169–2188, 2020.
- Q. Zhong, K. Yin, and Y. Duan, “Image reconstruction by minimizing curvatures on image surface,” Journal of Mathematical Imaging Vision, vol. 63, no. 1, pp. 30–55, 2021.
- S. Masnou and J. Morel, “Level lines based disocclusion,” in Proceedings of the 1998 IEEE International Conference on Image Processing, Chicago, Illinois, USA, October 4-7, 1998, pp. 259–263.
- Q. Zhong, Y. Li, Y. Yang, and Y. Duan, “Minimizing discrete total curvature for image processing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9474–9482.
- K. Schrader, T. Alt, J. Weickert, and M. Ertel, “Cnn-based euler’s elastica inpainting with deep energy and deep image prior,” in 10th European Workshop on Visual Information Processing. IEEE, 2022, pp. 1–6.
- D. Lefloch, M. Kluge, H. Sarbolandi, T. Weyrich, and A. Kolb, “Comprehensive use of curvature for robust and accurate online surface reconstruction,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2349–2365, 2017.
- Y. He, S. H. Kang, and H. Liu, “Curvature regularized surface reconstruction from point clouds,” SIAM Journal on Imaging Sciences, vol. 13, no. 4, pp. 1834–1859, 2020.
- Z. Zheng, Y. Hu, A. Cai, W. Zhang, J. Li, B. Yan, and G. Hu, “Few-view computed tomography image reconstruction using mean curvature model with curvature smoothing and surface fitting,” IEEE Transactions on Nuclear Science, vol. 66, no. 2, pp. 585–596, 2018.
- M. Yan and Y. Duan, “Nonlocal elastica model for sparse reconstruction,” Journal of Mathematical Imaging Vision, vol. 62, no. 4, pp. 532–548, 2020.
- W. Zhu, X. Tai, and T. F. Chan, “Image segmentation using euler’s elastica as the regularization,” Journal of Scientific Computing, vol. 57, no. 2, pp. 414–438, 2013.
- L. Liu, D. Chen, M. Shu, B. Li, H. Shu, M. Pâques, and L. D. Cohen, “Trajectory grouping with curvature regularization for tubular structure tracking,” IEEE Transactions on Image Processing, vol. 31, pp. 405–418, 2022.
- C.-Y. Tsai, K. N. Kutulakos, S. G. Narasimhan, and A. C. Sankaranarayanan, “The geometry of first-returning photons for non-line-of-sight imaging,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 7216–7224.
- C.-Y. Tsai, A. C. Sankaranarayanan, and I. Gkioulekas, “Beyond volumetric albedo-a surface optimization framework for non-line-of-sight imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1545–1555.
- F. Heide, M. O’Toole, K. Zang, D. B. Lindell, S. Diamond, and G. Wetzstein, “Non-line-of-sight imaging with partial occluders and surface normals,” ACM Transactions on Graphics (ToG), vol. 38, no. 3, pp. 1–10, 2019.
- S. I. Young, D. B. Lindell, B. Girod, D. Taubman, and G. Wetzstein, “Non-line-of-sight surface reconstruction using the directional light-cone transform,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1407–1416.
- A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM Journal on Imaging Sciences, vol. 2, no. 1, pp. 183–202, 2009.