Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Revisiting Nonlocal Self-Similarity from Continuous Representation (2401.00708v1)

Published 1 Jan 2024 in cs.CV and eess.IV

Abstract: Nonlocal self-similarity (NSS) is an important prior that has been successfully applied in multi-dimensional data processing tasks, e.g., image and video recovery. However, existing NSS-based methods are solely suitable for meshgrid data such as images and videos, but are not suitable for emerging off-meshgrid data, e.g., point cloud and climate data. In this work, we revisit the NSS from the continuous representation perspective and propose a novel Continuous Representation-based NonLocal method (termed as CRNL), which has two innovative features as compared with classical nonlocal methods. First, based on the continuous representation, our CRNL unifies the measure of self-similarity for on-meshgrid and off-meshgrid data and thus is naturally suitable for both of them. Second, the nonlocal continuous groups can be more compactly and efficiently represented by the coupled low-rank function factorization, which simultaneously exploits the similarity within each group and across different groups, while classical nonlocal methods neglect the similarity across groups. This elaborately designed coupled mechanism allows our method to enjoy favorable performance over conventional NSS methods in terms of both effectiveness and efficiency. Extensive multi-dimensional data processing experiments on-meshgrid (e.g., image inpainting and image denoising) and off-meshgrid (e.g., climate data prediction and point cloud recovery) validate the versatility, effectiveness, and efficiency of our CRNL as compared with state-of-the-art methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” IEEE Transactions on Image Processing, vol. 16, no. 8, pp. 2080–2095, 2007.
  2. W. Dong, G. Shi, X. Li, Y. Ma, and F. Huang, “Compressive sensing via nonlocal low-rank regularization,” IEEE Transactions on Image Processing, vol. 23, no. 8, pp. 3618–3632, 2014.
  3. S. Gu, Q. Xie, D. Meng, W. Zuo, X. Feng, and L. Zhang, “Weighted nuclear norm minimization and its applications to low level vision,” International Journal of Computer Vision, vol. 121, p. 183–208, 2017.
  4. Y. Chen, W. Cao, L. Pang, and X. Cao, “Hyperspectral image denoising with weighted nonlocal low-rank model and adaptive total variation regularization,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.
  5. W.-J. Zheng, X.-L. Zhao, Y.-B. Zheng, and Z.-F. Pang, “Nonlocal patch-based fully connected tensor network decomposition for multispectral image inpainting,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022.
  6. T.-X. Jiang, M. K. Ng, X.-L. Zhao, and T.-Z. Huang, “Framelet representation of tensor nuclear norm for third-order tensor completion,” IEEE Transactions on Image Processing, vol. 29, pp. 7233–7244, 2020.
  7. X. Gong, W. Chen, and J. Chen, “A low-rank tensor dictionary learning method for hyperspectral image denoising,” IEEE Transactions on Signal Processing, vol. 68, pp. 1168–1180, 2020.
  8. Y. Liu, X. Yuan, J. Suo, D. J. Brady, and Q. Dai, “Rank minimization for snapshot compressive imaging,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 12, pp. 2990–3006, 2019.
  9. W. He, H. Zhang, L. Zhang, and H. Shen, “Total-variation-regularized low-rank matrix factorization for hyperspectral image restoration,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 1, pp. 178–188, 2016.
  10. J. Peng, Q. Xie, Q. Zhao, Y. Wang, L. Yee, and D. Meng, “Enhanced 3DTV regularization and its applications on hsi denoising and compressed sensing,” IEEE Transactions on Image Processing, vol. 29, pp. 7889–7903, 2020.
  11. T. Yokota, R. Zdunek, A. Cichocki, and Y. Yamashita, “Smooth nonnegative matrix and tensor factorizations for robust multi-way data analysis,” Signal Processing, vol. 113, pp. 234–249, 2015.
  12. O. Debals, M. Van Barel, and L. De Lathauwer, “Nonnegative matrix factorization using nonnegative polynomial approximations,” IEEE Signal Processing Letters, vol. 24, no. 7, pp. 948–952, 2017.
  13. Q. Xie, Q. Zhao, Z. Xu, and D. Meng, “Color and direction-invariant nonlocal self-similarity prior and its application to color image denoising,” Science China Information Sciences, vol. 63, p. 222101, 2020.
  14. Q. Zhao, P. Tan, Q. Dai, L. Shen, E. Wu, and S. Lin, “A closed-form solution to retinex with nonlocal texture constraints,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 7, pp. 1437–1444, 2012.
  15. X. Zhang, X. Yuan, and L. Carin, “Nonlocal low-rank tensor factor analysis for image restoration,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8232–8241.
  16. J. Xue, Y. Zhao, W. Liao, and J. C.-W. Chan, “Nonlocal low-rank regularized tensor decomposition for hyperspectral image denoising,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 7, pp. 5174–5189, 2019.
  17. Y. Chang, L. Yan, and S. Zhong, “Hyper-laplacian regularized unidirectional low-rank tensor recovery for multispectral image denoising,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5901–5909.
  18. L. Feng, H. Sun, Q. Sun, and G. Xia, “Compressive sensing via nonlocal low-rank tensor regularization,” Neurocomputing, vol. 216, pp. 45–60, 2016.
  19. L. Wang, Z. Xiong, G. Shi, F. Wu, and W. Zeng, “Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 10, pp. 2104–2111, 2017.
  20. L. Zhang, L. Song, B. Du, and Y. Zhang, “Nonlocal low-rank tensor completion for visual data,” IEEE Transactions on Cybernetics, vol. 51, no. 2, pp. 673–685, 2021.
  21. Z. Zha, B. Wen, X. Yuan, J. Zhang, J. Zhou, Y. Lu, and C. Zhu, “Nonlocal structured sparsity regularization modeling for hyperspectral image denoising,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–16, 2023.
  22. W. Cao, J. Yao, J. Sun, and G. Han, “A tensor-based nonlocal total variation model for multi-channel image recovery,” Signal Processing, vol. 153, pp. 321–335, 2018.
  23. Y. Wang, T.-Z. Huang, X.-L. Zhao, and T.-X. Jiang, “Video deraining via nonlocal low-rank regularization,” Applied Mathematical Modelling, vol. 79, pp. 896–913, 2020.
  24. J. Lu, C. Xu, Z. Hu, X. Liu, Q. Jiang, D. Meng, and Z. Lin, “A new nonlocal low-rank regularization method with applications to magnetic resonance image denoising,” Inverse Problems, vol. 38, no. 6, p. 065012, 2022.
  25. W. Cui, S. Liu, F. Jiang, and D. Zhao, “Image compressed sensing using non-local neural network,” IEEE Transactions on Multimedia, vol. 25, pp. 816–830, 2023.
  26. Y. Sun, Y. Yang, Q. Liu, J. Chen, X.-T. Yuan, and G. Guo, “Learning non-locally regularized compressed sensing network with half-quadratic splitting,” IEEE Transactions on Multimedia, vol. 22, no. 12, pp. 3236–3248, 2020.
  27. L. Wang, C. Sun, M. Zhang, Y. Fu, and H. Huang, “Dnu: Deep non-local unrolling for computational spectral imaging,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1658–1668.
  28. A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, 2005, pp. 60–65 vol. 2.
  29. W. Dong, G. Shi, and X. Li, “Nonlocal image restoration with bilateral variance estimation: A low-rank approach,” IEEE Transactions on Image Processing, vol. 22, no. 2, pp. 700–711, 2013.
  30. X. Liu, X. Chen, J. Li, and Y. Chen, “Nonlocal weighted robust principal component analysis for seismic noise attenuation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 2, pp. 1745–1756, 2021.
  31. L. Zhuang, X. Fu, M. K. Ng, and J. M. Bioucas-Dias, “Hyperspectral image denoising based on global and nonlocal low-rank factorizations,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 12, pp. 10 438–10 454, 2021.
  32. H. Chen, M. Wei, Y. Sun, X. Xie, and J. Wang, “Multi-patch collaborative point cloud denoising via low-rank recovery with graph constraint,” IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 11, pp. 3255–3270, 2020.
  33. D. Zhu, H. Chen, W. Wang, H. Xie, G. Cheng, M. Wei, J. Wang, and F. L. Wang, “Nonlocal low-rank point cloud denoising for 3-D measurement surfaces,” IEEE Transactions on Instrumentation and Measurement, 2022, doi=10.1109/TIM.2021.3139686.
  34. M. Imaizumi and K. Hayashi, “Tensor decomposition with smoothness,” in International Conference on Machine Learning (ICML), vol. 70, 2017, pp. 1597–1606.
  35. N. Kargas and N. D. Sidiropoulos, “Supervised learning and canonical decomposition of multivariate functions,” IEEE Transactions on Signal Processing, vol. 69, pp. 1097–1107, 2021.
  36. I. V. Oseledets, “Constructive representation of functions in low-rank tensor formats,” Constructive Approximation, vol. 37, pp. 1–18, 2013.
  37. B. Hashemi and L. N. Trefethen, “Chebfun in three dimensions,” SIAM Journal on Scientific Computing, vol. 39, no. 5, pp. C341–C363, 2017.
  38. V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 7462–7473.
  39. J. Lee, J. Tack, N. Lee, and J. Shin, “Meta-learning sparse implicit neural representations,” in Advances in Neural Information Processing Systems (NeurIPS), M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34, 2021, pp. 11 769–11 780.
  40. C. Ma, P. Yu, J. Lu, and J. Zhou, “Recovering realistic details for magnification-arbitrary image super-resolution,” IEEE Transactions on Image Processing, vol. 31, pp. 3669–3683, 2022.
  41. J. Chibane, T. Alldieck, and G. Pons-Moll, “Implicit functions in feature space for 3D shape reconstruction and completion,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 6968–6979.
  42. W. Zhao, X. Liu, Z. Zhong, J. Jiang, W. Gao, G. Li, and X. Ji, “Self-supervised arbitrary-scale point clouds upsampling via implicit neural representation,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 1989–1997.
  43. Y. Chen, S. Liu, and X. Wang, “Learning continuous image representation with local implicit image function,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 8624–8634.
  44. J. Yang, S. Shen, H. Yue, and K. Li, “Implicit transformer network for screen content image continuous super-resolution,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 13 304–13 315.
  45. K. Zhang, D. Zhu, X. Min, and G. Zhai, “Implicit neural representation learning for hyperspectral image super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–12, 2023.
  46. C. Kim, J. Lee, and J. Shin, “Zero-shot blind image denoising via implicit neural representations,” arXiv:2204.02405, 2022.
  47. T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM Review, vol. 51, no. 3, pp. 455–500, 2009.
  48. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), 2015.
  49. Y. Luo, X. Zhao, Z. Li, M. K. Ng, and D. Meng, “Low-rank tensor function representation for multi-dimensional data recovery,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 10.1109/TPAMI.2023.3341688.
  50. Y. Chen, W. He, N. Yokoya, T.-Z. Huang, and X.-L. Zhao, “Nonlocal tensor-ring decomposition for hyperspectral image denoising,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 2, pp. 1348–1362, 2020.
  51. L. Yuan, C. Li, D. P. Mandic, J. Cao, and Q. Zhao, “Tensor ring decomposition with rank minimization on latent space: An efficient approach for tensor completion,” in AAAI Conference on Artificial Intelligence (AAAI), 2019, pp. 9151–9158.
  52. D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” International Journal of Computer Vision, vol. 128, p. 1867–1888, 2020.
  53. Y. Luo, X.-L. Zhao, D. Meng, and T.-X. Jiang, “HLRTF: Hierarchical low-rank tensor factorization for inverse problems in multi-dimensional imaging,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 19 303–19 312.
  54. M. Yang, Q. Luo, W. Li, and M. Xiao, “3-D array image data completion by tensor decomposition and nonconvex regularization approach,” IEEE Transactions on Signal Processing, vol. 70, pp. 4291–4304, 2022.
  55. H. Wang, J. Peng, W. Qin, J. Wang, and D. Meng, “Guaranteed tensor recovery fused low-rankness and smoothness,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, doi=10.1109/TPAMI.2023.3259640.
  56. F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar, “Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum,” IEEE Transactions on Image Processing, vol. 19, no. 9, pp. 2241–2253, 2010.
  57. Y. Wang, J. Peng, Q. Zhao, Y. Leung, X.-L. Zhao, and D. Meng, “Hyperspectral image restoration via total variation regularized low-rank tensor decomposition,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 11, no. 4, pp. 1227–1243, 2018.
  58. J. Peng, H. Wang, X. Cao, X. Liu, X. Rui, and D. Meng, “Fast noise removal in hyperspectral images via representative coefficient total variation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–17, 2022.
  59. Q. Yuan, Q. Zhang, J. Li, H. Shen, and L. Zhang, “Hyperspectral image denoising employing a spatial–spectral deep residual convolutional neural network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 2, pp. 1205–1218, 2019.
  60. A. Maffei, J. M. Haut, M. E. Paoletti, J. Plaza, L. Bruzzone, and A. Plaza, “A single model CNN for hyperspectral image denoising,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 4, pp. 2516–2529, 2020.
  61. S. Lee, G. Wolberg, and S. Shin, “Scattered data interpolation with multilevel b-splines,” IEEE Transactions on Visualization and Computer Graphics, vol. 3, no. 3, pp. 228–244, 1997.
  62. Y.-B. Zheng, T.-Z. Huang, X.-L. Zhao, Q. Zhao, and T.-X. Jiang, “Fully-connected tensor network decomposition and its application to higher-order tensor completion,” in AAAI Conference on Artificial Intelligence (AAAI), 2021, pp. 11 071–11 078.
  63. Q. Xie, M. Zhou, Q. Zhao, Z. Xu, and D. Meng, “MHF-Net: An interpretable deep network for multispectral and hyperspectral image fusion,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp. 1457–1473, 2022.
  64. J. Li, C. Huang, R. Chan, H. Feng, M. K. Ng, and T. Zeng, “Spherical image inpainting with frame transformation and data-driven prior deep networks,” SIAM Journal on Imaging Sciences, vol. 16, no. 3, pp. 1177–1194, 2023.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now