Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LiDAR-Guided Cross-Attention Fusion for Hyperspectral Band Selection and Image Classification (2404.03883v2)

Published 5 Apr 2024 in eess.IV and cs.CV

Abstract: The fusion of hyperspectral and LiDAR data has been an active research topic. Existing fusion methods have ignored the high-dimensionality and redundancy challenges in hyperspectral images, despite that band selection methods have been intensively studied for hyperspectral image (HSI) processing. This paper addresses this significant gap by introducing a cross-attention mechanism from the transformer architecture for the selection of HSI bands guided by LiDAR data. LiDAR provides high-resolution vertical structural information, which can be useful in distinguishing different types of land cover that may have similar spectral signatures but different structural profiles. In our approach, the LiDAR data are used as the "query" to search and identify the "key" from the HSI to choose the most pertinent bands for LiDAR. This method ensures that the selected HSI bands drastically reduce redundancy and computational requirements while working optimally with the LiDAR data. Extensive experiments have been undertaken on three paired HSI and LiDAR data sets: Houston 2013, Trento and MUUFL. The results highlight the superiority of the cross-attention mechanism, underlining the enhanced classification accuracy of the identified HSI bands when fused with the LiDAR features. The results also show that the use of fewer bands combined with LiDAR surpasses the performance of state-of-the-art fusion models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. T. Lai, “A review on visual-SLAM: Advancements from geometric modelling to learning-based semantic scene understanding using multi-modal sensor fusion,” Sensors, vol. 22, no. 19, p. 7265, 2022.
  2. J. Gao, P. Li, Z. Chen, and J. Zhang, “A survey on deep learning for multimodal data fusion,” Neural Computation, vol. 32, no. 5, pp. 829–864, 2020.
  3. W. Yu, L. Gao, H. Huang, Y. Shen, and G. Shen, “HI2D2FNet: Hyperspectral intrinsic image decomposition guided data fusion network for hyperspectral and LiDAR classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–15, 2023.
  4. W. Sun and Q. Du, “Hyperspectral band selection: A review,” IEEE Geoscience and Remote Sensing Magazine, vol. 7, no. 2, pp. 118–139, 2019.
  5. F. Jahan, J. Zhou, M. Awrangjeb, and Y. Gao, “Fusion of hyperspectral and LiDAR data using discriminant correlation analysis for land cover classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 11, no. 10, pp. 3905–3917, 2018.
  6. S. Khan, S. Nazir, I. García-Magariño, and A. Hussain, “Deep learning-based urban big data fusion in smart cities: Towards traffic monitoring and flow-preserving fusion,” Computers & Electrical Engineering, vol. 89, p. 106906, 2021.
  7. S. Falahatnejad and A. Karami, “Deep fusion of hyperspectral and LiDAR images using attention-based CNN,” SN Computer Science, vol. 4, no. 1, p. 1, 2022.
  8. Z. Wang, D. Ziou, C. Armenakis, D. Li, and Q. Li, “A comparative analysis of image fusion methods,” IEEE Transactions on Geoscience and Remote Sensing, vol. 43, no. 6, pp. 1391–1402, 2005.
  9. H. Kaur, D. Koundal, and V. Kadyan, “Image fusion techniques: a survey,” Archives of Computational Methods in Engineering, vol. 28, pp. 4425–4447, 2021.
  10. L. Liu, C.-f. Li, Y.-m. Lei, J.-y. Yin, and J.-j. Zhao, “Feature extraction for hyperspectral remote sensing image using weighted PCA-ICA,” Arabian Journal of Geosciences, vol. 10, pp. 1–10, 2017.
  11. S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Deep learning for hyperspectral image classification: An overview,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6690–6709, 2019.
  12. D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chanussot, Q. Du, and B. Zhang, “More diverse means better: Multimodal deep learning meets remote-sensing imagery classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 5, pp. 4340–4354, 2020.
  13. Z. Xue, X. Tan, X. Yu, B. Liu, A. Yu, and P. Zhang, “Deep hierarchical vision transformer for hyperspectral and LiDAR data classification,” IEEE Transactions on Image Processing, vol. 31, pp. 3095–3110, 2022.
  14. J. Zhang, Z. Meng, F. Zhao, H. Liu, and Z. Chang, “Convolution transformer mixer for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022.
  15. C. Debes, A. Merentitis, R. Heremans, J. Hahn, N. Frangiadakis, T. van Kasteren, W. Liao, R. Bellens, A. Pižurica, S. Gautama, W. Philips, S. Prasad, Q. Du, and F. Pacifici, “Hyperspectral and LiDAR data fusion: Outcome of the 2013 GRSS data fusion contest,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 7, no. 6, pp. 2405–2418, 2014.
  16. University of Trento, “Theses of the university of trento,” [Data set], 2022, original work published 2020. [Online]. Available: http://data.europa.eu/88u/dataset/theses-of-the-university-of-trento
  17. X. Du and A. Zare, “Technical report: Scene label ground truth map for MUUFL Gulfport data set,” University of Florida, Gainesville, FL, Tech. Rep. 20170417, Apr. 2017, available: http://ufdc.ufl.edu/IR00009711/00001.
  18. A. M. Kim, F. A. Kruse, and R. C. Olsen, “Integrated analysis of light detection and ranging (LiDAR) and hyperspectral imagery (HSI) data,” in Laser Radar Technology and Applications XXI, vol. 9832.   SPIE, 2016, pp. 271–285.
  19. C. V. Poulton, M. J. Byrd, P. Russo, E. Timurdogan, M. Khandaker, D. Vermeulen, and M. R. Watts, “Long-range lidar and free-space data communication with high-performance optical phased arrays,” IEEE Journal of Selected Topics in Quantum Electronics, vol. 25, no. 5, pp. 1–8, 2019.
  20. F. Qamar and G. Dobler, “Atmospheric correction of vegetation reflectance with simulation-trained deep learning for ground-based hyperspectral remote sensing,” Plant Methods, vol. 19, no. 1, p. 74, 2023.
  21. S. S. Sawant, P. Manoharan, and A. Loganathan, “Band selection strategies for hyperspectral image classification based on machine learning and artificial intelligent techniques–survey,” Arabian Journal of Geosciences, vol. 14, pp. 1–10, 2021.
  22. J.-H. Kim, J. Kim, Y. Yang, S. Kim, and H. S. Kim, “Covariance-based band selection and its application to near-real-time hyperspectral target detection,” Optical Engineering, vol. 56, no. 5, pp. 053 101–053 101, 2017.
  23. X. Sun, H. Zhang, F. Xu, Y. Zhu, and X. Fu, “Constrained-target band selection with subspace partition for hyperspectral target detection,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 9147–9161, 2021.
  24. B. Xu, X. Li, W. Hou, Y. Wang, and Y. Wei, “A similarity-based ranking method for hyperspectral band selection,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 11, pp. 9585–9599, 2021.
  25. Z. Zhang, D. Wang, X. Sun, L. Zhuang, R. Liu, and L. Ni, “Spatial sampling and grouping information entropy strategy based on kernel fuzzy C-Means clustering method for hyperspectral band selection,” Remote Sensing, vol. 14, no. 19, p. 5058, 2022.
  26. W. Zhang, X. Li, Y. Dou, and L. Zhao, “A geometry-based band selection approach for hyperspectral image analysis,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 8, pp. 4318–4333, 2018.
  27. J. Wang, C. Tang, X. Liu, W. Zhang, W. Li, X. Zhu, L. Wang, and A. Y. Zomaya, “Region-aware hierarchical latent feature representation learning-guided clustering for hyperspectral band selection,” IEEE Transactions on Cybernetics, 2022.
  28. X. Cao, B. Wu, D. Tao, and L. Jiao, “Automatic band selection using spatial-structure information and classifier-based clustering,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 9, no. 9, pp. 4352–4360, 2016.
  29. J. Wang, C. Tang, Z. Li, X. Liu, W. Zhang, E. Zhu, and L. Wang, “Hyperspectral band selection via region-aware latent features fusion based clustering,” Information Fusion, vol. 79, pp. 162–173, 2022.
  30. A. Tahraoui, R. Kheddam, A. Bouakache, and A. Belhadj-Aissa, “Affinity propagation for unsupervised classification of remotely sensed images,” in 2017 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP).   IEEE, 2017, pp. 1–6.
  31. Y. Cai, Z. Zhang, Z. Cai, X. Liu, X. Jiang, and Q. Yan, “Graph convolutional subspace clustering: A robust subspace clustering framework for hyperspectral image,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 5, pp. 4191–4202, 2020.
  32. Q. Wang, F. Zhang, and X. Li, “Optimal clustering framework for hyperspectral band selection,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 10, pp. 5910–5922, 2018.
  33. M. Zhang, J. Ma, and M. Gong, “Unsupervised hyperspectral band selection by fuzzy clustering with particle swarm optimization,” IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 5, pp. 773–777, 2017.
  34. J. Wang, C. Tang, X. Zheng, X. Liu, W. Zhang, and E. Zhu, “Graph regularized spatial-spectral subspace clustering for hyperspectral band selection,” Neural Networks, vol. 153, pp. 292–302, 2022.
  35. Y. Cai, X. Liu, and Z. Cai, “Bs-nets: An end-to-end framework for band selection of hyperspectral image,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 3, pp. 1969–1984, 2019.
  36. L. Mou, S. Saha, Y. Hua, F. Bovolo, L. Bruzzone, and X. X. Zhu, “Deep reinforcement learning for band selection in hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–14, 2022.
  37. C. O. Ayna, R. Mdrafi, Q. Du, and A. C. Gurbuz, “Learning-based optimization of hyperspectral band selection for classification,” Remote Sensing, vol. 15, no. 18, p. 4460, 2023.
  38. C. Yu, S. Zhou, M. Song, and C.-I. Chang, “Semisupervised hyperspectral band selection based on dual-constrained low-rank representation,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2021.
  39. H. Zhao, L. Bruzzone, R. Guan, F. Zhou, and C. Yang, “Spectral-spatial genetic algorithm-based unsupervised band selection for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 11, pp. 9616–9632, 2021.
  40. Y.-L. Chang, J.-N. Liu, Y.-L. Chen, W.-Y. Chang, T.-J. Hsieh, and B. Huang, “Hyperspectral band selection based on parallel particle swarm optimization and impurity function band prioritization schemes,” Journal of Applied Remote Sensing, vol. 8, no. 1, pp. 084 798–084 798, 2014.
  41. Q. Wang, J. Lin, and Y. Yuan, “Salient band selection for hyperspectral image classification via manifold ranking,” IEEE Transactions on Neural Networks and Learning Systems, vol. 27, no. 6, pp. 1279–1289, 2016.
  42. F. Jahan, J. Zhou, M. Awrangjeb, and Y. Gao, “Inverse coefficient of variation feature and multilevel fusion technique for hyperspectral and LiDAR data classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, pp. 367–381, 2020.
  43. R. Hang, Z. Li, P. Ghamisi, D. Hong, G. Xia, and Q. Liu, “Classification of hyperspectral and LiDAR data using coupled CNNs,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 7, pp. 4939–4950, 2020.
  44. D. Hong, L. Gao, R. Hang, B. Zhang, and J. Chanussot, “Deep encoder-decoder networks for classification of hyperspectral and LiDAR data,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2020.
  45. S. Jia, X. Zhou, S. Jiang, and R. He, “Collaborative contrastive learning for hyperspectral and LiDAR classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–14, 2023.
  46. W. Liao, F. Van Coillie, L. Gao, L. Li, B. Zhang, and J. Chanussot, “Deep learning for fusion of APEX hyperspectral and full-waveform LiDAR remote sensing data for tree species mapping,” IEEE Access, vol. 6, pp. 68 716–68 729, 2018.
  47. L. Zhang, L. Zhang, and B. Du, “Deep learning for remote sensing data: A technical tutorial on the state of the art,” IEEE Geoscience and Remote Sensing Magazine, vol. 4, no. 2, pp. 22–40, 2016.
  48. S. Mohla, S. Pande, B. Banerjee, and S. Chaudhuri, “FusAtNet: Dual attention based spectrospatial multimodal fusion network for hyperspectral and LiDAR classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 92–93.
  49. L. Zhou, J. Geng, and W. Jiang, “Joint classification of hyperspectral and LiDAR data based on position-channel cooperative attention network,” Remote Sensing, vol. 14, no. 14, p. 3247, 2022.
  50. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  51. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  52. D. Hong, Z. Han, J. Yao, L. Gao, B. Zhang, A. Plaza, and J. Chanussot, “SpectralFormer: Rethinking hyperspectral image classification with transformers,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2021.
  53. D. Hong, B. Zhang, X. Li, Y. Li, C. Li, J. Yao, N. Yokoya, H. Li, X. Jia, A. Plaza et al., “Spectralgpt: Spectral foundation model,” arXiv preprint arXiv:2311.07113, 2023.
  54. X. Zhou, M. Cui, A. Singhania, and D. Fernández Díaz, Juan Carlos, “Houston 2013 data set,” National Center for Airborne Laser Mapping (NCALM), University of Houston, Jun. 2013, dataset collected over the University of Houston campus and pre-processed with the assistance of the authors. [Online]. Available: https://hyperspectral.ee.uh.edu/?page_id=459
  55. Q. Wang, Q. Li, and X. Li, “Hyperspectral band selection via adaptive subspace partition strategy,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 12, pp. 4940–4950, 2019.
  56. S. Yu, S. Jia, and C. Xu, “Convolutional neural networks for hyperspectral image classification,” Neurocomputing, vol. 219, pp. 88–98, 2017.
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now