Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Transformers Fusion across Disjoint Samples for Hyperspectral Image Classification (2405.01095v1)

Published 2 May 2024 in cs.CV and eess.IV

Abstract: 3D Swin Transformer (3D-ST) known for its hierarchical attention and window-based processing, excels in capturing intricate spatial relationships within images. Spatial-spectral Transformer (SST), meanwhile, specializes in modeling long-range dependencies through self-attention mechanisms. Therefore, this paper introduces a novel method: an attentional fusion of these two transformers to significantly enhance the classification performance of Hyperspectral Images (HSIs). What sets this approach apart is its emphasis on the integration of attentional mechanisms from both architectures. This integration not only refines the modeling of spatial and spectral information but also contributes to achieving more precise and accurate classification results. The experimentation and evaluation of benchmark HSI datasets underscore the importance of employing disjoint training, validation, and test samples. The results demonstrate the effectiveness of the fusion approach, showcasing its superiority over traditional methods and individual transformers. Incorporating disjoint samples enhances the robustness and reliability of the proposed methodology, emphasizing its potential for advancing hyperspectral image classification.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. M. Ahmad, S. Shabbir, S. K. Roy, D. Hong, X. Wu, J. Yao, A. M. Khan, M. Mazzara, S. Distefano, and J. Chanussot, “Hyperspectral image classification—traditional to deep models: A survey for future prospects,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021.
  2. V. Lodhi, D. Chakravarty, and P. Mitra, “Hyperspectral imaging for earth observation: Platforms and instruments,” Journal of the Indian Institute of Science, vol. 98, pp. 429–443, 2018.
  3. Y. Li, D. Hong, C. Li, J. Yao, and J. Chanussote, “Hd-net: High-resolution decoupled network for building footprint extraction via deeply supervised body and boundary decomposition,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 209, pp. 51–65, 2024.
  4. B. Lu, P. D. Dao, J. Liu, Y. He, and J. Shang, “Recent advances of hyperspectral imaging technology and applications in agriculture,” Remote Sensing, vol. 12, no. 16, p. 2659, 2020.
  5. T. Adão, J. Hruška, L. Pádua, J. Bessa, E. Peres, R. Morais, and J. J. Sousa, “Hyperspectral imaging: A review on uav-based sensors, data processing and applications for agriculture and forestry,” Remote sensing, vol. 9, no. 11, p. 1110, 2017.
  6. C. Li, B. Zhang, D. Hong, J. Yao, and J. Chanussot, “Lrr-net: An interpretable deep unfolding network for hyperspectral anomaly detection,” IEEE Transactions on Geoscience and Remote Sensing, 2023.
  7. E. Bedini, “The use of hyperspectral remote sensing for mineral exploration: A review,” Journal of Hyperspectral Remote Sensing, vol. 7, no. 4, pp. 189–211, 2017.
  8. C. Weber, R. Aguejdad, X. Briottet, J. Avala, S. Fabre, J. Demuynck, E. Zenou, Y. Deville, M. S. Karoui, F. Z. Benhalouche et al., “Hyperspectral imagery for environmental urban planning,” in IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium.   IEEE, 2018, pp. 1628–1631.
  9. M. B. Stuart, A. J. McGonigle, and J. R. Willmott, “Hyperspectral imaging in environmental monitoring: A review of recent developments and technological advances in compact field deployable systems,” Sensors, vol. 19, no. 14, p. 3071, 2019.
  10. C. B. Pande and K. N. Moharir, “Application of hyperspectral remote sensing role in precision farming and sustainable agriculture under climate change: A review,” Climate Change Impacts on Natural Resources, Ecosystems and Agricultural Systems, pp. 503–520, 2023.
  11. M. H. Khan, Z. Saleem, M. Ahmad, A. Sohaib, H. Ayaz, M. Mazzara, and R. A. Raza, “Hyperspectral imaging-based unsupervised adulterated red chili content transformation for classification: Identification of red chili adulterants,” Neural Computing and Applications, vol. 33, no. 21, pp. 14 507–14 521, 2021.
  12. M. H. Khan, Z. Saleem, M. Ahmad, A. Sohaib, H. Ayaz, and M. Mazzara, “Hyperspectral imaging for color adulteration detection in red chili,” Applied Sciences, vol. 10, no. 17, p. 5955, 2020.
  13. Z. Saleem, M. H. Khan, M. Ahmad, A. Sohaib, H. Ayaz, and M. Mazzara, “Prediction of microbial spoilage and shelf-life of bakery products through hyperspectral imaging,” IEEE Access, vol. 8, pp. 176 986–176 996, 2020.
  14. M. H. F. Butt, H. Ayaz, M. Ahmad, J. P. Li, and R. Kuleev, “A fast and compact hybrid cnn for hyperspectral imaging-based bloodstain classification,” in 2022 IEEE Congress on Evolutionary Computation (CEC).   IEEE, 2022, pp. 1–8.
  15. M. Zulfiqar, M. Ahmad, A. Sohaib, M. Mazzara, and S. Distefano, “Hyperspectral imaging for bloodstain identification,” Sensors, vol. 21, no. 9, p. 3045, 2021.
  16. H. Ayaz, M. Ahmad, M. Mazzara, and A. Sohaib, “Hyperspectral imaging for minced meat classification using nonlinear deep features,” Applied Sciences, vol. 10, no. 21, p. 7783, 2020.
  17. H. Ayaz, M. Ahmad, A. Sohaib, M. N. Yasir, M. A. Zaidan, M. Ali, M. H. Khan, and Z. Saleem, “Myoglobin-based classification of minced meat using hyperspectral imaging,” Applied Sciences, vol. 10, no. 19, p. 6862, 2020.
  18. D. Hong, B. Zhang, X. Li, Y. Li, C. Li, J. Yao, N. Yokoya, H. Li, P. Ghamisi, X. Jia, A. Plaza, P. Gamba, J. A. Benediktsson, and J. Chanussot, “Spectralgpt: Spectral remote sensing foundation model,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, dOI:10.1109/TPAMI.2024.3362475.
  19. M. Ahmad, A. M. Khan, M. Mazzara, S. Distefano, M. Ali, and M. S. Sarfraz, “A fast and compact 3-d cnn for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, 2020.
  20. U. Ghous, M. S. Sarfraz, M. Ahmad, C. Li, and D. Hong, “(2+1)d extreme xception net for hyperspectral image classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pp. 1–14, 2024.
  21. D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, “Graph convolutional networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 7, pp. 5966–5978, 2021.
  22. A. Jamali, S. K. Roy, D. Hong, P. M. Atkinson, and P. Ghamisi, “Attention graph convolutional network for disjoint hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, pp. 1–1, 2024.
  23. M. Ahmad and M. Mazzara, “Scsnet: Sharpened cosine similarity-based neural network for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 21, pp. 1–4, 2024.
  24. J. Yao, B. Zhang, C. Li, D. Hong, and J. Chanussot, “Extended vision transformer (exvit) for land use and land cover classification: A multimodal deep learning framework,” IEEE Transactions on Geoscience and Remote Sensing, 2023.
  25. M. Ahmad, U. Ghous, M. Usama, and M. Mazzara, “Waveformer: Spectral–spatial wavelet transformer for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 21, pp. 1–5, 2024.
  26. X. Huang, M. Dong, J. Li, and X. Guo, “A 3-d-swin transformer-based hierarchical contrastive learning method for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.
  27. B. Zu, T. Cao, Y. Li, J. Li, F. Ju, and H. Wang, “Swint-srnet: Swin transformer with image super-resolution reconstruction network for pollen images classification,” Engineering Applications of Artificial Intelligence, vol. 133, p. 108041, 2024.
  28. B. Liu, Y. Liu, W. Zhang, Y. Tian, and W. Kong, “Spectral swin transformer network for hyperspectral image classification,” Remote Sensing, vol. 15, no. 15, 2023.
  29. G. Farooque, Q. Liu, A. B. Sargano, and L. Xiao, “Swin transformer with multiscale 3d atrous convolution for hyperspectral image classification,” Engineering Applications of Artificial Intelligence, vol. 126, p. 107070, 2023.
  30. J. Xie, J. Hua, S. Chen, P. Wu, P. Gao, D. Sun, Z. Lyu, S. Lyu, X. Xue, and J. Lu, “Hypersformer: A transformer-based end-to-end hyperspectral image classification method for crop classification,” Remote Sensing, vol. 15, no. 14, 2023.
  31. S. Ayas and E. Tunc-Gormus, “Spectralswin: a spectral-swin transformer network for hyperspectral image classification,” International Journal of Remote Sensing, vol. 43, no. 11, pp. 4025–4044, 2022.
  32. Y. Peng, J. Ren, J. Wang, and M. Shi, “Spectral-swin transformer with spatial feature extraction enhancement for hyperspectral image classification,” Remote Sensing, vol. 15, no. 10, 2023.
  33. Y. Long, X. Wang, M. Xu, S. Zhang, S. Jiang, and S. Jia, “Dual self-attention swin transformer for hyperspectral image super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–12, 2023.
  34. L. Wang, Z. Zheng, N. Kumar, C. Wang, F. Guo, and P. Zhang, “Multilevel class token transformer with cross tokenmixer for hyperspectral images classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–13, 2024.
  35. X. He, Y. Chen, and Z. Lin, “Spatial-spectral transformer for hyperspectral image classification,” Remote Sensing, vol. 13, no. 3, 2021.
  36. G. Wang, Y. Wang, Z. Pan, X. Wang, J. Zhang, and J. Pan, “Vitfsl-baseline: A simple baseline of vision transformer network for few-shot image classification,” IEEE Access, vol. 12, pp. 11 836–11 849, 2024.
  37. Y. Ma, Y. Lan, Y. Xie, L. Yu, C. Chen, Y. Wu, and X. Dai, “A spatial-spectral transformer for hyperspectral image classification based on global dependencies of multi-scale features,” Remote Sensing, vol. 16, no. 2, 2024.
  38. J. Lian, L. Wang, H. Sun, and H. Huang, “Gt-had: Gated transformer for hyperspectral anomaly detection,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2024.
  39. S. Mei, Z. Han, M. Ma, F. Xu, and X. Li, “A novel center-boundary metric loss to learn discriminative features for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–16, 2024.
  40. A. Jamali, S. K. Roy, D. Hong, P. M. Atkinson, and P. Ghamisi, “Spatial-gated multilayer perceptron for land use and land cover mapping,” IEEE Geoscience and Remote Sensing Letters, vol. 21, pp. 1–5, 2024.
  41. Y. Xiao, Q. Yuan, K. Jiang, J. He, C.-W. Lin, and L. Zhang, “Ttst: A top-k token selective transformer for remote sensing image super-resolution,” IEEE Transactions on Image Processing, vol. 33, pp. 738–752, 2024.
  42. S. Mei, C. Song, M. Ma, and F. Xu, “Hyperspectral image classification using group-aware hierarchical transformer,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–14, 2022.
  43. D. Hong, Z. Han, J. Yao, L. Gao, B. Zhang, A. Plaza, and J. Chanussot, “Spectralformer: Rethinking hyperspectral image classification with transformers,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.
  44. J. Chen, C. Yang, L. Zhang, L. Yang, L. Bian, Z. Luo, and J. Wang, “Tccu-net: Transformer and cnn collaborative unmixing network for hyperspectral image,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pp. 1–20, 2024.
  45. J. Fang, J. Yang, A. Khader, and L. Xiao, “Mimo-sst: Multi-input multi-output spatial-spectral transformer for hyperspectral and multispectral image fusion,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1–1, 2024.
  46. M. Ye, J. Chen, F. Xiong, and Y. Qian, “Adaptive graph modeling with self-training for heterogeneous cross-scene hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–15, 2024.
  47. W. Huang, Y. Deng, S. Hui, Y. Wu, S. Zhou, and J. Wang, “Sparse self-attention transformer for image inpainting,” Pattern Recognition, vol. 145, p. 109897, 2024.
  48. Y. Sun, X. Zhi, S. Jiang, G. Fan, X. Yan, and W. Zhang, “Image fusion for the novelty rotating synthetic aperture system based on vision transformer,” Information Fusion, vol. 104, p. 102163, 2024.
  49. T. Kim, J. Kim, H. Oh, and J. Kang, “Deep transformer based video inpainting using fast fourier tokenization,” IEEE Access, vol. 12, pp. 21 723–21 736, 2024.
  50. Y. Shi, J. Xia, M. Zhou, and Z. Cao, “A dual-feature-based adaptive shared transformer network for image captioning,” IEEE Transactions on Instrumentation and Measurement, vol. 73, pp. 1–13, 2024.
  51. M. Ahmad, M. Mazzara, and S. Distifano, “Importance of disjoint sampling in conventional and transformer models for hyperspectral image classification,” arXiv preprint arXiv:2404.14944, 2024.
  52. M. Ahmad, M. H. F. Butt, M. Mazzara, and S. Distifano, “Pyramid hierarchical transformer for hyperspectral image classification,” arXiv preprint arXiv:2404.14945, 2024.
  53. M. H. F. Butt, J. P. Li, M. Ahmad, and M. A. F. Butt, “Graph-infused hybrid vision transformer: Advancing geoai for enhanced land cover classification,” International Journal of Applied Earth Observation and Geoinformation, vol. 129, p. 103773, 2024.
  54. M. Ahmad, S. Distifano, M. Mazzara, and A. M. Khan, “Traditional to transformers: A survey on current trends and future prospects for hyperspectral image classification,” arXiv preprint arXiv:2404.14955, 2024.
  55. X. Cao, Y. Lian, K. Wang, C. Ma, and X. Xu, “Unsupervised hybrid network of transformer and cnn for blind hyperspectral and multispectral image fusion,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–15, 2024.
  56. J. Li, Z. Zhang, R. Song, Y. Li, and Q. Du, “Scformer: Spectral coordinate transformer for cross-domain few-shot hyperspectral image classification,” IEEE Transactions on Image Processing, vol. 33, pp. 840–855, 2024.
  57. Z. Shu, Y. Wang, and Z. Yu, “Dual attention transformer network for hyperspectral image classification,” Engineering Applications of Artificial Intelligence, vol. 127, p. 107351, 2024.
  58. Q. Ma, J. Jiang, X. Liu, and J. Ma, “Reciprocal transformer for hyperspectral and multispectral image fusion,” Information Fusion, vol. 104, p. 102148, 2024.
  59. Y. Zhang, C. Lan, H. Zhang, G. Ma, and H. Li, “Multimodal remote sensing image matching via learning features and attention mechanism,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–20, 2024.
  60. B. Li, L. Fang, N. Chen, J. Kang, and J. Yue, “Enhancing hyperspectral image classification: Leveraging unsupervised information with guided group contrastive learning,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–17, 2024.
  61. S. Miao, Q. Xu, W. Li, C. Yang, B. Sheng, F. Liu, T. T. Bezabih, and X. Yu, “Mmtfn: Multi-modal multi-scale transformer fusion network for alzheimer’s disease diagnosis,” International Journal of Imaging Systems and Technology, vol. 34, no. 1, p. e22970, 2024.
  62. L. Qu, S. Liu, M. Wang, S. Li, S. Yin, and Z. Song, “Trans2fuse: Empowering image fusion through self-supervised learning and multi-modal transformations via transformer networks,” Expert Systems with Applications, vol. 236, p. 121363, 2024.
  63. M. Ahmad, A. M. Khan, M. Mazzara, S. Distefano, S. K. Roy, and X. Wu, “Hybrid dense network with attention mechanism for hyperspectral image classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 3948–3957, 2022.
  64. R. O. Green, M. L. Eastwood, C. M. Sarture, T. G. Chrien, M. Aronsson, B. J. Chippendale, J. A. Faust, B. E. Pavri, C. J. Chovit, M. Solis et al., “Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (aviris),” Remote sensing of environment, vol. 65, no. 3, pp. 227–248, 1998.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now