Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DF4LCZ: A SAM-Empowered Data Fusion Framework for Scene-Level Local Climate Zone Classification (2403.09367v1)

Published 14 Mar 2024 in cs.CV

Abstract: Recent advancements in remote sensing (RS) technologies have shown their potential in accurately classifying local climate zones (LCZs). However, traditional scene-level methods using convolutional neural networks (CNNs) often struggle to integrate prior knowledge of ground objects effectively. Moreover, commonly utilized data sources like Sentinel-2 encounter difficulties in capturing detailed ground object information. To tackle these challenges, we propose a data fusion method that integrates ground object priors extracted from high-resolution Google imagery with Sentinel-2 multispectral imagery. The proposed method introduces a novel Dual-stream Fusion framework for LCZ classification (DF4LCZ), integrating instance-based location features from Google imagery with the scene-level spatial-spectral features extracted from Sentinel-2 imagery. The framework incorporates a Graph Convolutional Network (GCN) module empowered by the Segment Anything Model (SAM) to enhance feature extraction from Google imagery. Simultaneously, the framework employs a 3D-CNN architecture to learn the spectral-spatial features of Sentinel-2 imagery. Experiments are conducted on a multi-source remote sensing image dataset specifically designed for LCZ classification, validating the effectiveness of the proposed DF4LCZ. The related code and dataset are available at https://github.com/ctrlovefly/DF4LCZ.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. I. D. Stewart, T. R. Oke, and E. S. Krayenhoff, “Evaluation of the ‘local climate zone’scheme using temperature observations and model simulations,” International journal of climatology, vol. 34, no. 4, pp. 1062–1080, 2014.
  2. I. D. Stewart and T. R. Oke, “Local climate zones for urban temperature studies,” Bulletin of the American Meteorological Society, vol. 93, no. 12, pp. 1879–1900, 2012.
  3. F. Leconte, J. Bouyer, R. Claverie, and M. Pétrissans, “Using local climate zone scheme for uhi assessment: Evaluation of the method using mobile measurements,” Building and Environment, vol. 83, pp. 39–49, 2015, special Issue: Climate adaptation in cities. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0360132314001413
  4. B. Bechtel, M. Demuzere, G. Mills, W. Zhan, P. Sismanidis, C. Small, and J. Voogt, “Suhi analysis using local climate zones—a comparison of 50 cities,” Urban Climate, vol. 28, p. 100451, 2019.
  5. F. Huang, S. Jiang, W. Zhan, B. Bechtel, Z. Liu, M. Demuzere, Y. Huang, Y. Xu, L. Ma, W. Xia et al., “Mapping local climate zones for cities: A large review,” Remote Sensing of Environment, vol. 292, p. 113573, 2023.
  6. S. Liu and Q. Shi, “Local climate zone mapping as remote sensing scene classification using deep learning: A case study of metropolitan china,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 164, pp. 229–242, 2020.
  7. Q. Chunping, M. Schmitt, M. Lichao, and Z. Xiaoxiang, “Urban local climate zone classification with a residual convolutional neural network and multi-seasonal sentinel-2 images,” in 2018 10th IAPR Workshop on Pattern Recognition in Remote Sensing (PRRS).   IEEE, 2018, pp. 1–5.
  8. M. Kim, D. Jeong, and Y. Kim, “Local climate zone classification using a multi-scale, multi-level attention network,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 181, pp. 345–366, 2021.
  9. J. Peng, X. Sun, H. Yu, Y. Tian, C. Deng, and F. Yao, “An instance-based multitask graph network for complex facility recognition in remote sensing imagery,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2021.
  10. J. Liang, Y. Deng, and D. Zeng, “A deep neural network combined cnn and gcn for remote sensing scene classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, pp. 4325–4338, 2020.
  11. Q. Yao, H. Li, P. Gao, H. Guo, and C. Zhong, “Mapping irregular local climate zones from sentinel-2 images using deep learning with sequential virtual scenes,” Remote Sensing, vol. 14, no. 21, p. 5564, 2022.
  12. C. Yoo, D. Han, J. Im, and B. Bechtel, “Comparison between convolutional neural networks and random forest for local climate zone classification in mega urban areas using landsat images,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 157, pp. 155–170, 2019.
  13. J. Rosentreter, R. Hagensieker, and B. Waske, “Towards large-scale mapping of local climate zones using multitemporal sentinel 2 data and convolutional neural networks,” Remote Sensing of Environment, vol. 237, p. 111472, 2020.
  14. P. Feng, Y. Lin, J. Guan, Y. Dong, G. He, Z. Xia, and H. Shi, “Embranchment cnn based local climate zone classification using sar and multispectral remote sensing data,” in IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium.   IEEE, 2019, pp. 6344–6347.
  15. A. Leichter, D. Wittich, F. Rottensteiner, M. Werner, and M. Sester, “Improved classification of satellite imagery using spatial feature maps extracted from social media,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences; 42-4, vol. 42, no. 4, pp. 403–410, 2018.
  16. R. Yang, Y. Zhang, P. Zhao, Z. Ji, and W. Deng, “Msppf-nets: a deep learning architecture for remote sensing image classification,” in IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium.   IEEE, 2019, pp. 3045–3048.
  17. P. Feng, Y. Lin, G. He, J. Guan, J. Wang, and H. Shi, “A dynamic end-to-end fusion filter for local climate zone classification using sar and multi-spectrum remote sensing data,” in IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium.   IEEE, 2020, pp. 4231–4234.
  18. C. Yoo, Y. Lee, D. Cho, J. Im, and D. Han, “Improving local climate zone classification using incomplete building data and sentinel 2 images based on convolutional neural networks,” Remote Sensing, vol. 12, no. 21, p. 3552, 2020.
  19. N. Zhao, Y. Zhong, and A. Ma, “Mapping local climate zones with circled similarity propagation based domain adaptation,” in IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium.   IEEE, 2020, pp. 1377–1380.
  20. X. Huang, A. Liu, and J. Li, “Mapping and analyzing the local climate zones in china’s 32 major cities using landsat imagery based on a novel convolutional neural network,” Geo-spatial Information Science, vol. 24, no. 4, pp. 528–557, 2021.
  21. C. Qiu, M. Schmitt, L. Mou, P. Ghamisi, and X. X. Zhu, “Feature importance analysis for local climate zone classification using a residual convolutional neural network with multi-source datasets,” Remote Sensing, vol. 10, no. 10, p. 1572, 2018.
  22. H. Jing, Y. Feng, W. Zhang, Y. Zhang, S. Wang, K. Fu, and K. Chen, “Effective classification of local climate zones based on multi-source remote sensing data,” in IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium.   IEEE, 2019, pp. 2666–2669.
  23. C. Qiu, L. Mou, M. Schmitt, and X. X. Zhu, “Local climate zone-based urban land cover classification from multi-seasonal sentinel-2 images with a recurrent residual network,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 154, pp. 151–162, 2019.
  24. C. Qiu, X. Tong, M. Schmitt, B. Bechtel, and X. X. Zhu, “Multilevel feature fusion-based cnn for local climate zone classification from sentinel-2 images: Benchmark results on the so2sat lcz42 dataset,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, pp. 2793–2806, 2020.
  25. L. Zhou, Z. Shao, S. Wang, and X. Huang, “Deep learning-based local climate zone classification using sentinel-1 sar and sentinel-2 multispectral imagery,” Geo-Spatial Information Science, vol. 25, no. 3, pp. 383–398, 2022.
  26. R. Wang, M. Wang, Z. Zhang, J. Xing, and X. Liu, “Llnet: Lightweight network with a channel and spatial attention mechanism for local climate zone classification from sentinel-2 image,” International Journal of Climatology, vol. 43, no. 3, pp. 1543–1560, 2023.
  27. Y. Shen, B. Zhou, X. Xiong, R. Gao, and Y. G. Wang, “How graph neural networks enhance convolutional neural networks towards mining the topological structures from histology,” in ICML Workshop on Computational Biology, vol. 8, 2022.
  28. A. Wang, J. Cai, J. Lu, and T.-J. Cham, “Modality and component aware feature fusion for rgb-d scene classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 5995–6004.
  29. X. Chen, X. Zheng, Y. Zhang, and X. Lu, “Remote sensing scene classification by local–global mutual learning,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022.
  30. J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” AI open, vol. 1, pp. 57–81, 2020.
  31. F. Peng, W. Lu, W. Tan, K. Qi, X. Zhang, and Q. Zhu, “Multi-output network combining gnn and cnn for remote sensing scene classification,” Remote Sensing, vol. 14, no. 6, p. 1478, 2022.
  32. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.
  33. X. Ma, Q. Wu, X. Zhao, X. Zhang, M.-O. Pun, and B. Huang, “Sam-assisted remote sensing imagery semantic segmentation with object and boundary constraints,” arXiv preprint arXiv:2312.02464, 2023.
  34. J. Zhang, Z. Zhou, G. Mai, L. Mu, M. Hu, and S. Li, “Text2seg: Remote sensing image semantic segmentation via text-guided visual foundation models,” arXiv preprint arXiv:2304.10597, 2023.
  35. D. Wang, J. Zhang, B. Du, M. Xu, L. Liu, D. Tao, and L. Zhang, “Samrs: Scaling-up remote sensing segmentation dataset with segment anything model,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  36. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
  37. J. Yang, Y.-Q. Zhao, J. C.-W. Chan, and L. Xiao, “A multi-scale wavelet 3d-cnn for hyperspectral image super-resolution,” Remote sensing, vol. 11, no. 13, p. 1557, 2019.
  38. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  39. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning.   pmlr, 2015, pp. 448–456.
  40. A. Roitberg, K. Peng, Z. Marinov, C. Seibold, D. Schneider, and R. Stiefelhagen, “A comparative analysis of decision-level fusion for multimodal driver behaviour understanding,” in 2022 IEEE Intelligent Vehicles Symposium (IV).   IEEE, 2022, pp. 1438–1444.
  41. X. X. Zhu, J. Hu, C. Qiu, Y. Shi, J. Kang, L. Mou, H. Bagheri, M. Haberle, Y. Hua, R. Huang et al., “So2sat lcz42: A benchmark data set for the classification of global local climate zones [software and data sets],” IEEE Geoscience and Remote Sensing Magazine, vol. 8, no. 3, pp. 76–89, 2020.
  42. J. Ching, G. Mills, B. Bechtel, L. See, J. Feddema, X. Wang, C. Ren, O. Brousse, A. Martilli, M. Neophytou et al., “Wudapt: An urban weather, climate, and environmental modeling infrastructure for the anthropocene,” Bulletin of the American Meteorological Society, vol. 99, no. 9, pp. 1907–1924, 2018.
  43. M. Demuzere, J. Kittner, and B. Bechtel, “Lcz generator: a web application to create local climate zone maps,” Frontiers in Environmental Science, vol. 9, p. 637455, 2021.
  44. M. Drusch, U. Del Bello, S. Carlier, O. Colin, V. Fernandez, F. Gascon, B. Hoersch, C. Isola, P. Laberinti, P. Martimort et al., “Sentinel-2: Esa’s optical high-resolution mission for gmes operational services,” Remote sensing of Environment, vol. 120, pp. 25–36, 2012.
  45. R. J. Lisle, “Google earth: a new geological resource,” Geology today, vol. 22, no. 1, pp. 29–32, 2006.
  46. J. M. Flenniken, S. Stuglik, and B. V. Iannone, “Quantum gis (qgis): An introduction to a free alternative to more costly gis platforms,” EDIS, vol. 2020, no. 2, pp. 7–7, 2020.
  47. C. Xu, P. Hystad, R. Chen, J. Van Den Hoek, R. A. Hutchinson, S. Hankey, and R. Kennedy, “Application of training data affects success in broad-scale local climate zone mapping,” International Journal of Applied Earth Observation and Geoinformation, vol. 103, p. 102482, 2021.
  48. D. Grattarola and C. Alippi, “Graph neural networks in tensorflow and keras with spektral [application notes],” IEEE Computational Intelligence Magazine, vol. 16, no. 1, pp. 99–106, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Qianqian Wu (4 papers)
  2. Xianping Ma (10 papers)
  3. Jialu Sui (6 papers)
  4. Man-On Pun (28 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now