Papers
Topics
Authors
Recent
2000 character limit reached

Deep Diversity-Enhanced Feature Representation of Hyperspectral Images (2301.06132v3)

Published 15 Jan 2023 in cs.CV and eess.IV

Abstract: In this paper, we study the problem of efficiently and effectively embedding the high-dimensional spatio-spectral information of hyperspectral (HS) images, guided by feature diversity. Specifically, based on the theoretical formulation that feature diversity is correlated with the rank of the unfolded kernel matrix, we rectify 3D convolution by modifying its topology to enhance the rank upper-bound. This modification yields a rank-enhanced spatial-spectral symmetrical convolution set (ReS$3$-ConvSet), which not only learns diverse and powerful feature representations but also saves network parameters. Additionally, we also propose a novel diversity-aware regularization (DA-Reg) term that directly acts on the feature maps to maximize independence among elements. To demonstrate the superiority of the proposed ReS$3$-ConvSet and DA-Reg, we apply them to various HS image processing and analysis tasks, including denoising, spatial super-resolution, and classification. Extensive experiments show that the proposed approaches outperform state-of-the-art methods both quantitatively and qualitatively to a significant extent. The code is publicly available at https://github.com/jinnh/ReSSS-ConvSet.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. M. Shimoni, R. Haelterman, and C. Perneel, “Hypersectral imaging for military and security applications: Combining myriad processing and sensing techniques,” IEEE Geoscience and Remote Sensing Magazine, vol. 7, no. 2, pp. 101–117, 2019.
  2. Z. P. Lee, W. Rhea, R. Arnone, and W. Goode, “Absorption coefficients of marine waters: expanding multiband information to hyperspectral data,” IEEE Transactions on Geoscience and Remote Sensing, vol. 43, no. 1, pp. 118–124, 2005.
  3. Y. Chang, L. Yan, H. Fang, S. Zhong, and W. Liao, “Hsi-denet: Hyperspectral image restoration via convolutional neural network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 2, pp. 667–682, 2019.
  4. J. Jiang, H. Sun, X. Liu, and J. Ma, “Learning spatial-spectral prior for super-resolution of hyperspectral imagery,” IEEE Transactions on Computational Imaging, vol. 6, pp. 1082–1096, 2020.
  5. X. Cao, X. Fu, C. Xu, and D. Meng, “Deep spatial-spectral global reasoning network for hyperspectral image denoising,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1–14, 2021.
  6. R. Hang, F. Zhou, Q. Liu, and P. Ghamisi, “Classification of hyperspectral images via multitask generative adversarial networks,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 2, pp. 1424–1436, 2021.
  7. L. Mou, P. Ghamisi, and X. X. Zhu, “Deep recurrent neural networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 7, pp. 3639–3655, 2017.
  8. D. Hong, Z. Han, J. Yao, L. Gao, B. Zhang, A. Plaza, and J. Chanussot, “Spectralformer: Rethinking hyperspectral image classification with transformers,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.
  9. M. Li, Y. Fu, and Y. Zhang, “Spatial-spectral transformer for hyperspectral image denoising,” in Proc. AAAI Conference on Artificial Intelligence, vol. 37, no. 1, 2023, pp. 1368–1376.
  10. M. Zhang, C. Zhang, Q. Zhang, J. Guo, X. Gao, and J. Zhang, “Essaformer: Efficient transformer for hyperspectral image super-resolution,” in Proc. IEEE/CVF International Conference on Computer Vision, 2023, pp. 23 073–23 084.
  11. Q. Yuan, Q. Zhang, J. Li, H. Shen, and L. Zhang, “Hyperspectral image denoising employing a spatial-spectral deep residual convolutional neural network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 2, pp. 1205–1218, 2019.
  12. N. He, M. E. Paoletti, J. M. Haut, L. Fang, S. Li, A. Plaza, and J. Plaza, “Feature extraction with multiscale covariance maps for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 2, pp. 755–769, 2019.
  13. Q. Zhang, Q. Yuan, J. Li, X. Liu, H. Shen, and L. Zhang, “Hybrid noise removal in hyperspectral imagery with a spatial-spectral gradient network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 10, pp. 7317–7329, 2019.
  14. X. Wang, K. Tan, P. Du, C. Pan, and J. Ding, “A unified multiscale learning framework for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–19, 2022.
  15. M. Zhu, L. Jiao, F. Liu, S. Yang, and J. Wang, “Residual spectral?spatial attention network for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 1, pp. 449–462, 2021.
  16. M. E. Paoletti, S. Moreno-Álvarez, and J. M. Haut, “Multiple attention-guided capsule networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–20, 2022.
  17. W. Liu and J. Lee, “A 3-d atrous convolution neural network for hyperspectral image denoising,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 8, pp. 5701–5715, 2019.
  18. Q. Shi, X. Tang, T. Yang, R. Liu, and L. Zhang, “Hyperspectral image denoising using a 3-d attention denoising network,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1–16, 2021.
  19. S. Mei, X. Yuan, J. Ji, Y. Zhang, S. Wan, and Q. Du, “Hyperspectral image spatial super-resolution via 3d full convolutional neural network,” Remote Sensing, vol. 9, no. 11, p. 1139, 2017.
  20. A. B. Hamida, A. Benoit, P. Lambert, and C. B. Amar, “3-d deep learning approach for remote sensing image classification,” IEEE Transactions on geoscience and remote sensing, vol. 56, no. 8, pp. 4420–4434, 2018.
  21. Q. Wang, Q. Li, and X. Li, “Hyperspectral image superresolution using spectrum and feature context,” IEEE Transactions on Industrial Electronics, vol. 68, no. 11, pp. 11 276–11 285, 2021.
  22. Z. Qiu, T. Yao, and T. Mei, “Learning spatio-temporal representation with pseudo-3d residual networks,” in Proc. IEEE International Conference on Computer Vision, 2017, pp. 5533–5541.
  23. D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in Proc. IEEE/CVF conference on Computer Vision and Pattern Recognition, 2018, pp. 6450–6459.
  24. F. Gonda, D. Wei, T. Parag, and H. Pfister, “Parallel separable 3d convolution for video and volumetric data understanding,” arXiv preprint arXiv:1809.04096, 2018.
  25. W. Dong, H. Wang, F. Wu, G. Shi, and X. Li, “Deep spatial–spectral representation learning for hyperspectral image denoising,” IEEE Transactions on Computational Imaging, vol. 5, no. 4, pp. 635–648, 2019.
  26. M. Zhang, W. Li, and Q. Du, “Diverse region-based cnn for hyperspectral image classification,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2623–2634, 2018.
  27. F. Yang, X. Zhou, Y. Wang, A. Atawulla, and R. Bi, “Diversity features enhanced prototypical network for few-shot intent detection,” in Proc. International Joint Conference on Artificial Intelligence, 7 2022, pp. 4447–4453.
  28. P. Ghamisi, N. Yokoya, J. Li, W. Liao, S. Liu, J. Plaza, B. Rasti, and A. Plaza, “Advances in hyperspectral image and signal processing: A comprehensive overview of the state of the art,” IEEE Geoscience and Remote Sensing Magazine, vol. 5, no. 4, pp. 37–78, 2017.
  29. S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Deep learning for hyperspectral image classification: An overview,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6690–6709, 2019.
  30. J. Peng, W. Sun, H.-C. Li, W. Li, X. Meng, C. Ge, and Q. Du, “Low-rank and sparse representation for hyperspectral image processing: A review,” IEEE Geoscience and Remote Sensing Magazine, vol. 10, no. 1, pp. 10–43, 2022.
  31. K. Wei, Y. Fu, and H. Huang, “3-d quasi-recurrent neural network for hyperspectral image denoising,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, pp. 363–375, 2021.
  32. H. Zhang, H. Chen, G. Yang, and L. Zhang, “Lr-net: Low-rank spatial-spectral network for hyperspectral image denoising,” IEEE Transactions on Image Processing, vol. 30, pp. 8743–8758, 2021.
  33. X. Rui, X. Cao, Q. Xie, Z. Yue, Q. Zhao, and D. Meng, “Learning an explicit weighting scheme for adapting complex hsi noise,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2021, pp. 6739–6748.
  34. T. Bodrito, A. Zouaoui, J. Chanussot, and J. Mairal, “A trainable spectral-spatial sparse coding model for hyperspectral image restoration,” in Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  35. J. Li, R. Cui, B. Li, R. Song, Y. Li, Y. Dai, and Q. Du, “Hyperspectral image super-resolution by band attention through adversarial learning,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 6, pp. 4304–4318, 2020.
  36. J. Hu, X. Jia, Y. Li, G. He, and M. Zhao, “Hyperspectral image super-resolution via intrafusion network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 10, pp. 7459–7471, 2020.
  37. Q. Li, Q. Wang, and X. Li, “Exploring the relationship between 2d/3d convolution for hyperspectral image super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1–11, 2021.
  38. K. Li, D. Dai, and L. Van Gool, “Hyperspectral image super-resolution with rgb image super-resolution as an auxiliary task,” in Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022, pp. 4039–4048.
  39. X. Wang, J. Ma, and J. Jiang, “Hyperspectral image super-resolution via recurrent feedback embedding and spatial?spectral consistency regularization,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.
  40. J. Hou, Z. Zhu, J. Hou, H. Zeng, J. Wu, and J. Zhou, “Deep posterior distribution-based embedding for hyperspectral image super-resolution,” IEEE Transactions on Image Processing, vol. 31, pp. 5720–5732, 2022.
  41. F. Melgani and L. Bruzzone, “Classification of hyperspectral remote sensing images with support vector machines,” IEEE Transactions on geoscience and remote sensing, vol. 42, no. 8, pp. 1778–1790, 2004.
  42. J. Ham, Y. Chen, M. M. Crawford, and J. Ghosh, “Investigation of the random forest framework for classification of hyperspectral data,” IEEE Transactions on Geoscience and Remote Sensing, vol. 43, no. 3, pp. 492–501, 2005.
  43. Y. Bazi and F. Melgani, “Gaussian process approach to remote sensing image classification,” IEEE transactions on geoscience and remote sensing, vol. 48, no. 1, pp. 186–197, 2009.
  44. L. Ma, M. M. Crawford, and J. Tian, “Local manifold learning-based k𝑘kitalic_k -nearest-neighbor for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 48, no. 11, pp. 4099–4109, 2010.
  45. W. Hu, Y. Huang, L. Wei, F. Zhang, and H. Li, “Deep convolutional neural networks for hyperspectral image classification,” Journal of Sensors, vol. 2015, 2015.
  46. Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, “Deep feature extraction and classification of hyperspectral images based on convolutional neural networks,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 10, pp. 6232–6251, 2016.
  47. M. E. Paoletti, J. M. Haut, R. Fernandez-Beltran, J. Plaza, A. J. Plaza, and F. Pla, “Deep pyramidal residual networks for spectral–spatial hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 2, pp. 740–754, 2019.
  48. D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, “Graph convolutional networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 7, pp. 5966–5978, 2021.
  49. Y. Chen, X. Jin, J. Feng, and S. Yan, “Training group orthogonal neural networks with privileged information,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2017.
  50. J. Wang, Y. Chen, R. Chakraborty, and S. X. Yu, “Orthogonal convolutional neural networks,” in Proc. IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 505–11 515.
  51. F. Heide, W. Heidrich, and G. Wetzstein, “Fast and flexible convolutional sparse coding,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2015, pp. 5135–5143.
  52. K. Yanai, R. Tanno, and K. Okamoto, “Efficient mobile implementation of a cnn-based object recognition system,” in Proc. 24th ACM international conference on Multimedia, 2016, pp. 362–366.
  53. M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. De Freitas, “Predicting parameters in deep learning,” Advances in neural information processing systems, vol. 26, 2013.
  54. W. Shang, K. Sohn, D. Almeida, and H. Lee, “Understanding and improving convolutional neural networks via concatenated rectified linear units,” in Proc. international conference on machine learning.   PMLR, 2016, pp. 2217–2225.
  55. M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao, “Hrank: Filter pruning using high-rank feature map,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1526–1535.
  56. L. Zhang, Q. Zhang, B. Du, X. Huang, Y. Y. Tang, and D. Tao, “Simultaneous spectral-spatial feature selection and extraction for hyperspectral images,” IEEE Transactions on Cybernetics, vol. 48, no. 1, pp. 16–28, 2016.
  57. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention.   Springer, 2015, pp. 234–241.
  58. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in Proc. IEEE conference on computer vision and pattern recognition, 2018, pp. 2472–2481.
  59. H. Zhang, W. He, L. Zhang, H. Shen, and Q. Yuan, “Hyperspectral image restoration using low-rank matrix recovery,” IEEE Transactions on Geoscience and Remote Sensing, vol. 52, no. 8, pp. 4729–4743, 2014.
  60. Y. Chen, X. Cao, Q. Zhao, D. Meng, and Z. Xu, “Denoising hyperspectral image with non-i.i.d. noise structure,” IEEE Transactions on Cybernetics, vol. 48, no. 3, pp. 1054–1066, 2018.
  61. Y. Wang, J. Peng, Q. Zhao, Y. Leung, X.-L. Zhao, and D. Meng, “Hyperspectral image restoration via total variation regularized low-rank tensor decomposition,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 11, no. 4, pp. 1227–1243, 2018.
  62. Q. Xie, Q. Zhao, D. Meng, Z. Xu, S. Gu, W. Zuo, and L. Zhang, “Multispectral images denoising by intrinsic tensor sparsity regularization,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016, pp. 1692–1700.
  63. W. He, Q. Yao, C. Li, N. Yokoya, Q. Zhao, H. Zhang, and L. Zhang, “Non-local meets global: An iterative paradigm for hyperspectral image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 4, pp. 2089–2107, 2022.
  64. M. Li, J. Liu, Y. Fu, Y. Zhang, and D. Dou, “Spectral enhanced rectangle transformer for hyperspectral image denoising,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5805–5814.
  65. B. Arad and O. Ben-Shahar, “Sparse recovery of hyperspectral signal from natural rgb images,” in Proc. EuropeanEuropean Conference on Computer Vision.   Springer, 2016, pp. 19–34.
  66. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  67. J. Jiang, D. Liu, J. Gu, and S. Süsstrunk, “What is the space of spectral sensitivity functions for digital color cameras?” in Proc. IEEE Workshop on Applications of Computer Vision.   IEEE, 2013, pp. 168–179.
  68. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  69. R. H. Yuhas, A. F. Goetz, and J. W. Boardman, “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm,” in Proc. Summaries 3rd Annu. JPL Airborne Geosci. Workshop, vol. 1, 1992, pp. 147–149.
  70. F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar, “Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum,” IEEE Transactions on Image Processing, vol. 19, no. 9, pp. 2241–2253, 2010.
  71. S. K. Roy, S. Manna, T. Song, and L. Bruzzone, “Attention-based adaptive spectral–spatial kernel resnet for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 9, pp. 7831–7843, 2021.
  72. Z. Zheng, Y. Zhong, A. Ma, and L. Zhang, “Fpga: Fast patch-free global learning framework for fully end-to-end hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 8, pp. 5612–5626, 2020.
  73. J. Nalepa, M. Myller, and M. Kawulok, “Validating hyperspectral image segmentation,” IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 8, pp. 1264–1268, 2019.
  74. L. Zou, X. Zhu, C. Wu, Y. Liu, and L. Qu, “Spectral–spatial exploration for hyperspectral image classification via the fusion of fully convolutional networks,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, pp. 659–674, 2020.
  75. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  76. J. Li, X. Huang, and L. Tu, “Whu-ohs: A benchmark dataset for large-scale hersepctral image classification,” International Journal of Applied Earth Observation and Geoinformation, vol. 113, p. 103022, 2022.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Slide Deck Streamline Icon: https://streamlinehq.com

Whiteboard

Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - jinnh/ReSSS-ConvSet (20 stars)