Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HyperKon: A Self-Supervised Contrastive Network for Hyperspectral Image Analysis (2311.15459v1)

Published 26 Nov 2023 in eess.IV

Abstract: The exceptional spectral resolution of hyperspectral imagery enables material insights that are not possible with RGB or multispectral images. Yet, the full potential of this data is often underutilized by deep learning techniques due to the scarcity of hyperspectral-native CNN backbones. To bridge this gap, we introduce HyperKon, a self-supervised contrastive learning network designed and trained on hyperspectral data from the EnMAP Hyperspectral Satellite\cite{kaufmann2012environmental}. HyperKon uniquely leverages the high spectral continuity, range, and resolution of hyperspectral data through a spectral attention mechanism and specialized convolutional layers. We also perform a thorough ablation study on different kinds of layers, showing their performance in understanding hyperspectral layers. It achieves an outstanding 98% Top-1 retrieval accuracy and outperforms traditional RGB-trained backbones in hyperspectral pan-sharpening tasks. Additionally, in hyperspectral image classification, HyperKon surpasses state-of-the-art methods, indicating a paradigm shift in hyperspectral image analysis and underscoring the importance of hyperspectral-native backbones.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. Hyperspectral imaging: A review on uav-based sensors, data processing and applications for agriculture and forestry. Remote sensing, 9(11):1110, 2017.
  2. Deep learning for biological image classification. Expert systems with applications, 85:114–122, 2017.
  3. Wele Gedara Chaminda Bandara and Vishal M Patel. Hypertransformer: A textural and spectral feature fusion transformer for pansharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1767–1777, 2022.
  4. Hyperspectral pansharpening based on improved deep image prior and residual reconstruction. IEEE Transactions on Geoscience and Remote Sensing, 60:1–16, 2021.
  5. Robustness assessment of hyperspectral image cnns using metamorphic testing. Information and Software Technology, page 107281, 2023.
  6. Precision agriculture and the role of remote sensing: a review. Canadian Journal of Remote Sensing, 24(3):315–327, 1998.
  7. Semi-supervised vision transformers at scale. Advances in Neural Information Processing Systems, 35:25697–25710, 2022.
  8. German Space Operation Center. Enmap ground segment. 2023.
  9. A survey on image fusion techniques and performance metrics. In 2018 Second International Conference on Electronics, Communication and Aerospace Technology (ICECA), pages 995–999. IEEE, 2018.
  10. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  11. Hyperspectral image super-resolution with self-supervised spectral-spatial residual network. Remote Sensing, 13(7):1260, 2021.
  12. Prospect of application of hyperspectral imaging technology in public security. In International Conference on Applications and Techniques in Cyber Security and Intelligence ATCI 2018: Applications and Techniques in Cyber Security and Intelligence, pages 299–304. Springer, 2019.
  13. Cnn-based salient object detection on hyperspectral images using extended morphology. IEEE Geoscience and Remote Sensing Letters, 19:1–5, 2022.
  14. Miguel Jorge da Lomba Magalhães. Hyperspectral image fusion–a comprehensive review. Master’s thesis, Itä-Suomen yliopisto, 2022.
  15. A comprehensive evaluation of spectral distance functions and metrics for hyperspectral image processing. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 8(6):3224–3234, 2015.
  16. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  17. Deep transformer network for hyperspectral image classification. Academic Journal of Computing & Information Science, 4(7):11–17, 2021.
  18. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019.
  19. Deep residual network of spectral and spatial fusion for hyperspectral image super-resolution. In 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), pages 266–270. IEEE, 2019.
  20. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  21. Hyperpnn: Hyperspectral pansharpening via spectrally predictive convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(8):3092–3100, 2019.
  22. Spatial-spectral transformer for hyperspectral image classification. Remote Sensing, 13(3):498, 2021.
  23. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11936–11945, 2021.
  24. Spectralformer: Rethinking hyperspectral image classification with transformers. IEEE Transactions on Geoscience and Remote Sensing, 60:1–15, 2021.
  25. Pan-sharpening via multiscale dynamic convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 59(3):2231–2244, 2020.
  26. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  27. Deep convolutional neural networks for hyperspectral image classification. Journal of Sensors, 2015:1–12, 2015.
  28. A 3-d-swin transformer-based hierarchical contrastive learning method for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 60:1–15, 2022.
  29. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv:1404.1869, 2014.
  30. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
  31. The environmental mapping and analysis program (enmap). In IEEE Int. Geoscience and Remote Sensing Symp, 2012.
  32. Feature extraction for hyperspectral image classification: A review. International Journal of Remote Sensing, 41(16):6248–6287, 2020.
  33. Sipsa-net: Shift-invariant pan sharpening with moving object alignment for satellite imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10166–10174, 2021.
  34. Deepunet: A deep fully convolutional network for pixel-level sea-land segmentation. IEEE journal of selected topics in applied earth observations and remote sensing, 11(11):3954–3962, 2018.
  35. Deep learning for hyperspectral image classification: An overview. IEEE Transactions on Geoscience and Remote Sensing, 57(9):6690–6709, 2019.
  36. Discriminant analysis-based dimension reduction for hyperspectral image classification: A survey of the most recent advances and an experimental comparison of different techniques. IEEE Geoscience and Remote Sensing Magazine, 6(1):15–34, 2018.
  37. Multi-dataset hyper-cnn for hyperspectral image segmentation of remote sensing images. Processes, 11(2):435, 2023.
  38. Hyperspectral pansharpening: A review. IEEE Geoscience and remote sensing magazine, 3(3):27–46, 2015.
  39. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 2021.
  40. Towards the copernicus hyperspectral imaging mission for the environment (chime). In Igarss 2018-2018 ieee international geoscience and remote sensing symposium, pages 157–159. IEEE, 2018.
  41. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  42. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724–732, 2016.
  43. Recent advances in techniques for hyperspectral image processing. Remote sensing of environment, 113:S110–S122, 2009.
  44. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. Advances in Neural Information Processing Systems, 33:3407–3418, 2020.
  45. Aneesh Prasad Rangnekar. Learning Representations in the Hyperspectral Domain in Aerial Imagery. Rochester Institute of Technology, 2022.
  46. Charles M Sarture Thomas G Chrien Mikael Aronsson Bruce J Chippendale Jessica A Faust Betina E Pavri Christopher J Chovit Manuel Solis Martin R Olah Robert O Green, Michael L Eastwood and Orlesa Williams. Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (aviris). Remote Sensing of Environment, 65(3):227–248, 1998.
  47. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
  48. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  49. Hyperspectral image classification based on spectral multiscale convolutional neural network. Remote Sensing, 14(8):1951, 2022.
  50. Deep learning meets hyperspectral image analysis: A multidisciplinary review. Journal of Imaging, 5(5):52, 2019.
  51. A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Transactions on Geoscience and Remote Sensing, 53(6):3373–3388, 2014.
  52. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  53. Quality metrics evaluation of hyperspectral images. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 40:1221–1226, 2014.
  54. Spectral–spatial attention network for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 58(5):3232–3245, 2019.
  55. Spectral–spatial feature tokenization transformer for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 60:1–14, 2022.
  56. Deep learning on image denoising: An overview. Neural Networks, 131:251–275, 2020.
  57. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018, 2018.
  58. Ssanet: An adaptive spectral–spatial attention autoencoder network for hyperspectral unmixing. Remote Sensing, 15(8):2070, 2023.
  59. Three-dimensional resnext network using feature fusion and label smoothing for hyperspectral image classification. Sensors, 20(6):1652, 2020.
  60. Deep gradient projection networks for pan-sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1366–1375, 2021.
  61. Pannet: A deep network architecture for pan-sharpening. In Proceedings of the IEEE international conference on computer vision, pages 5449–5457, 2017.
  62. Hyperspectral image transformer classification networks. IEEE Transactions on Geoscience and Remote Sensing, 60:1–15, 2022.
  63. Convolutional neural networks for hyperspectral image classification. Neurocomputing, 219:88–98, 2017.
  64. Quantum-inspired spectral-spatial pyramid network for hyperspectral image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9925–9934, 2023.
  65. Review on graph learning for dimensionality reduction of hyperspectral image. Geo-spatial Information Science, 23(1):98–106, 2020.
  66. A survey of hyperspectral image super-resolution technology. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pages 4476–4479. IEEE, 2021.
  67. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2998–3008, 2021.
  68. Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems, 30(11):3212–3232, 2019.
  69. Hyperspectral pansharpening using deep prior and dual attention residual network. IEEE transactions on geoscience and remote sensing, 58(11):8059–8076, 2020.
  70. Spectral–spatial residual network for hyperspectral image classification: A 3-d deep learning framework. IEEE Transactions on Geoscience and Remote Sensing, 56(2):847–858, 2017.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now