Papers
Topics
Authors
Recent
Search
2000 character limit reached

Bidirectional recurrent imputation and abundance estimation of LULC classes with MODIS multispectral time series and geo-topographic and climatic data

Published 11 Oct 2023 in cs.CV, cs.LG, and eess.IV | (2310.07223v3)

Abstract: Remotely sensed data are dominated by mixed Land Use and Land Cover (LULC) types. Spectral unmixing (SU) is a key technique that disentangles mixed pixels into constituent LULC types and their abundance fractions. While existing studies on Deep Learning (DL) for SU typically focus on single time-step hyperspectral (HS) or multispectral (MS) data, our work pioneers SU using MODIS MS time series, addressing missing data with end-to-end DL models. Our approach enhances a Long-Short Term Memory (LSTM)-based model by incorporating geographic, topographic (geo-topographic), and climatic ancillary information. Notably, our method eliminates the need for explicit endmember extraction, instead learning the input-output relationship between mixed spectra and LULC abundances through supervised learning. Experimental results demonstrate that integrating spectral-temporal input data with geo-topographic and climatic information significantly improves the estimation of LULC abundances in mixed pixels. To facilitate this study, we curated a novel labeled dataset for Andalusia (Spain) with monthly MODIS multispectral time series at 460m resolution for 2013. Named Andalusia MultiSpectral MultiTemporal Unmixing (Andalusia-MSMTU), this dataset provides pixel-level annotations of LULC abundances along with ancillary information. The dataset (https://zenodo.org/records/7752348) and code (https://github.com/jrodriguezortega/MSMTU) are available to the public.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (89)
  1. Robert A Schowengerdt. Remote sensing: models and methods for image processing. Elsevier, 2006.
  2. Hyperspectral unmixing via deep convolutional neural networks. IEEE Geoscience and Remote Sensing Letters, 15(11):1755–1759, 2018.
  3. Spectral unmixing. IEEE signal processing magazine, 19(1):44–57, 2002.
  4. Deep learning in hyperspectral unmixing: A review. In IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium, pages 2189–2192, 2020.
  5. Spatio-temporal spectral unmixing of time-series images. Remote Sensing of Environment, 259:112407, 2021.
  6. Pixel unmixing in hyperspectral data by means of neural networks. IEEE transactions on Geoscience and remote sensing, 49(11):4163–4172, 2011.
  7. Real-time spatiotemporal spectral unmixing of modis images. IEEE Transactions on Geoscience and Remote Sensing, 60:1–16, 2021.
  8. Blind hyperspectral unmixing using autoencoders: A critical comparison. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:1340–1372, 2022.
  9. Hyperspectral unmixing using transformer network. IEEE Transactions on Geoscience and Remote Sensing, 60:1–16, 2022.
  10. Nonlinear spectral mixture modeling of lunar multispectral data: Implications for lateral transport. Journal of Geophysical Research: Planets, 103(E8):19419–19425, 1998.
  11. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2011–2018, 2014.
  12. Improving plankton image classification using context metadata. Limnology and Oceanography: Methods, 17(8):439–461, 2019.
  13. Timl: Task-informed meta-learning for agriculture. arXiv preprint arXiv:2202.02124, 2022.
  14. Siose andalusia. https://www.juntadeandalucia.es/medioambiente/portal/landing-page-%C3%ADndice/-/asset_publisher/zX2ouZa4r1Rf/content/sistema-de-informaci-c3-b3n-sobre-el-patrimonio-natural-de-andaluc-c3-ada-sipna-/20151, 2023. [Online; Accessed 07-2023].
  15. Deep learning in environmental remote sensing: Achievements and challenges. Remote Sensing of Environment, 241:111716, 2020.
  16. Deep learning for hyperspectral image classification: An overview. IEEE Transactions on Geoscience and Remote Sensing, 57(9):6690–6709, 2019.
  17. Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning. ISPRS journal of photogrammetry and remote sensing, 145:60–77, 2018.
  18. Deep learning-based tree classification using mobile lidar data. Remote Sensing Letters, 6(11):864–873, 2015.
  19. Urban land use and land cover classification using remotely sensed sar data through deep belief networks. Journal of Sensors, 2015, 2015.
  20. More diverse means better: Multimodal deep learning meets remote-sensing imagery classification. IEEE Transactions on Geoscience and Remote Sensing, 59(5):4340–4354, 2021.
  21. Multimodal hyperspectral unmixing: Insights from attention networks. IEEE Transactions on Geoscience and Remote Sensing, 60:1–13, 2022.
  22. Illumination invariant hyperspectral image unmixing based on a digital surface model. IEEE Transactions on Image Processing, 29:3652–3664, 2020.
  23. Cross-city matters: A multimodal remote sensing benchmark dataset for cross-city semantic segmentation using high-resolution domain adaptation networks. Remote Sensing of Environment, 299:113856, 2023.
  24. Spectralgpt: Spectral foundation model. arXiv preprint arXiv:2311.07113, 2023.
  25. Decoupled-and-coupled networks: Self-supervised hyperspectral image super-resolution with subpixel fusion. IEEE Transactions on Geoscience and Remote Sensing, 2023.
  26. A review of nonlinear hyperspectral unmixing methods. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(6):1844–1868, 2014.
  27. Daniel C Heinz et al. Fully constrained least squares linear spectral mixture analysis method for material quantification in hyperspectral imagery. IEEE transactions on geoscience and remote sensing, 39(3):529–545, 2001.
  28. Spectral unmixing. International Journal of Remote Sensing, 33(17):5307–5340, 2012.
  29. Hyperspectral unmixing overview: Geometrical, statistical, and sparse regression-based approaches. IEEE journal of selected topics in applied earth observations and remote sensing, 5(2):354–379, 2012.
  30. Reducing the effect of the endmembers’ spectral variability by selecting the optimal spectral bands. Remote Sensing, 9(9):884, 2017.
  31. A multitemporal linear spectral unmixing: An iterative approach accounting for abundance variations. In 2018 9th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), pages 1–5, 2018.
  32. Assessing the impact of endmember variability on linear spectral mixture analysis (lsma): A theoretical and simulation analysis. Remote Sensing of Environment, 235:111471, 2019.
  33. Mapping chaparral in the santa monica mountains using multiple endmember spectral mixture models. Remote sensing of environment, 65(3):267–279, 1998.
  34. Endmember bundles: A new approach to incorporating endmember variability into spectral mixture analysis. IEEE transactions on geoscience and remote sensing, 38(2):1083–1094, 2000.
  35. A geostatistical temporal mixture analysis approach to address endmember variability for estimating regional impervious surface distributions. GIScience & Remote Sensing, 53(1):102–121, 2016.
  36. Enhancing the performance of multiple endmember spectral mixture analysis (mesma) for urban land cover mapping using airborne lidar data and band selection. Remote sensing of environment, 221:260–273, 2019.
  37. Applications of independent component analysis in endmember extraction and abundance quantification for hyperspectral imagery. IEEE Transactions on Geoscience and Remote Sensing, 44(9):2601–2616, 2006.
  38. On the decomposition of mars hyperspectral data by ica and bayesian positive source separation. Neurocomputing, 71(10-12):2194–2208, 2008.
  39. Independent component analysis for blind unmixing of hyperspectral imagery with additional constraints. IEEE transactions on geoscience and remote sensing, 49(6):2165–2179, 2011.
  40. Endmember extraction from highly mixed data using minimum volume constrained nonnegative matrix factorization. IEEE Transactions on Geoscience and Remote Sensing, 45(3):765–777, 2007.
  41. Blind spectral unmixing based on sparse nonnegative matrix factorization. IEEE Transactions on Image Processing, 20(4):1112–1125, 2010.
  42. Structured sparse method for hyperspectral unmixing. ISPRS Journal of Photogrammetry and Remote Sensing, 88:101–118, 2014.
  43. Hyperspectral unmixing based on nonnegative matrix factorization: A comprehensive review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:4414–4436, 2022.
  44. Blind spectral unmixing based on sparse component analysis for hyperspectral remote sensing imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 119:49–63, 2016.
  45. A blind spectral unmixing in wavelet domain. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:10287–10302, 2021.
  46. Nonlinear unmixing of hyperspectral images using a generalized bilinear model. IEEE Transactions on Geoscience and Remote Sensing, 49(11):4153–4162, 2011.
  47. Nonlinear unmixing of hyperspectral images using radial basis functions and orthogonal least squares. In 2011 IEEE International Geoscience and Remote Sensing Symposium, pages 1151–1154. IEEE, 2011.
  48. Kernel fully constrained least squares abundance estimates. In 2007 IEEE international geoscience and remote sensing symposium, pages 4041–4044. IEEE, 2007.
  49. Using low-rank representation of abundance maps and nonnegative tensor factorization for hyperspectral nonlinear unmixing. IEEE Transactions on Geoscience and Remote Sensing, 60:1–17, 2022.
  50. Giles M Foody. Relating the land-cover composition of mixed pixels to artificial neural network classification output. Photogrammetric engineering and remote sensing, 62(5):491–498, 1996.
  51. Mapping sub-pixel proportional land cover with avhrr imagery. International Journal of Remote Sensing, 18(4):917–935, 1997.
  52. Multi-stage convolutional autoencoder network for hyperspectral unmixing. International Journal of Applied Earth Observation and Geoinformation, 113:102981, 2022.
  53. Cycu-net: Cycle-consistency unmixing network by learning cascaded autoencoders. IEEE Transactions on Geoscience and Remote Sensing, 60:1–14, 2022.
  54. Sparsity-enhanced convolutional decomposition: A novel tensor-based paradigm for blind hyperspectral unmixing. IEEE Transactions on Geoscience and Remote Sensing, 60:1–14, 2022.
  55. Deep half-siamese networks for hyperspectral unmixing. IEEE Geoscience and Remote Sensing Letters, 18(11):1996–2000, 2021.
  56. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  57. An attention-based 3d convolutional autoencoder for few-shot hyperspectral unmixing and classification. Remote Sensing, 15(2):451, 2023.
  58. A practical approach for hyperspectral unmixing using deep learning. IEEE Geoscience and Remote Sensing Letters, 19:1–5, 2022.
  59. Autonas: Automatic neural architecture search for hyperspectral unmixing. IEEE Transactions on Geoscience and Remote Sensing, 60:1–14, 2022.
  60. Lstm-dnn based autoencoder network for nonlinear hyperspectral image unmixing. IEEE Journal of Selected Topics in Signal Processing, 15(2):295–309, 2021.
  61. Cropland distributions from temporal unmixing of modis data. Remote Sensing of Environment, 93(3):412–422, 2004.
  62. Multitemporal unmixing of medium-spatial-resolution satellite images: A case study using meris images for land-cover mapping. IEEE Transactions on geoscience and remote sensing, 49(11):4308–4317, 2011.
  63. Temporal mixture analysis for estimating impervious surface area from multi-temporal modis ndvi data in japan. ISPRS journal of photogrammetry and remote sensing, 72:90–98, 2012.
  64. Continuous subpixel monitoring of urban impervious surface using landsat time series. Remote Sensing of Environment, 238:110929, 2020.
  65. Monitoring tropical forest degradation using spectral unmixing and landsat time series analysis. Remote sensing of Environment, 238:110968, 2020.
  66. Feiyun Zhu. Hyperspectral unmixing: ground truth labeling, datasets, benchmark performances and survey. arXiv preprint arXiv:1708.05125, 2017.
  67. Learning internal representations by error propagation. Technical report, 1985.
  68. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  69. Predicting missing values in spatio-temporal remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 56(5):2841–2853, 2018.
  70. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, 31, 2018.
  71. The shuttle radar topography mission. Reviews of geophysics, 45(2), 2007.
  72. Red de información ambiental de andalucía. https://www.juntadeandalucia.es/medioambiente/portal/acceso-rediam, 2023. [Online; Accessed 07-2023].
  73. Google earth engine: Planetary-scale geospatial analysis for everyone. Remote Sensing of Environment, 2017.
  74. Timespec4lulc: a global multispectral time series database for training lulc mapping models with machine learning. Earth System Science Data, 14(3):1377–1411, 2022.
  75. QGIS Development Team. QGIS Geographic Information System. QGIS Association, 2022.
  76. Andalusia-msmtu. March 2023.
  77. John Aitchison. The statistical analysis of compositional data. Journal of the Royal Statistical Society: Series B (Methodological), 44(2):139–160, 1982.
  78. The effect of imposing ‘fractional abundance constraints’ onto the multilayer perceptron for sub-pixel land cover classification. International Journal of Applied Earth Observation and Geoinformation, 44:226–238, 2016.
  79. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8):913–929, 2017.
  80. Leonardo Uieda. Verde: Processing and gridding spatial data using Green’s functions. Journal of Open Source Software, 3(29):957, 2018.
  81. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  82. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  83. An augmented linear mixing model to address spectral variability for hyperspectral unmixing. IEEE Transactions on Image Processing, 28(4):1923–1938, 2019.
  84. Orthogonal subspace unmixing to address spectral variability for hyperspectral image. IEEE Transactions on Geoscience and Remote Sensing, 61:1–13, 2023.
  85. Extending the unmixing methods to multispectral images. In Color and Imaging Conference, volume 2021, pages 311–316. Society for Imaging Science and Technology, 2021.
  86. Jakub Nalepa. Recent advances in multi-and hyperspectral image analysis. Sensors, 21(18):6002, 2021.
  87. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82–115, 2020.
  88. Physics-aware machine learning for geosciences and remote sensing. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pages 2086–2089, 2021.
  89. Chen Shi and Le Wang. Incorporating spatial information in spectral unmixing: A review. Remote Sensing of Environment, 149:70–87, 2014.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up