Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders (1802.02080v4)

Abstract: Earth observation (EO) sensors deliver data with daily or weekly temporal resolution. Most land use and land cover (LULC) approaches, however, expect cloud-free and mono-temporal observations. The increasing temporal capabilities of today's sensors enables the use of temporal, along with spectral and spatial features. Domains, such as speech recognition or neural machine translation, work with inherently temporal data and, today, achieve impressive results using sequential encoder-decoder structures. Inspired by these sequence-to-sequence models, we adapt an encoder structure with convolutional recurrent layers in order to approximate a phenological model for vegetation classes based on a temporal sequence of Sentinel 2 (S2) images. In our experiments, we visualize internal activations over a sequence of cloudy and non-cloudy images and find several recurrent cells, which reduce the input activity for cloudy observations. Hence, we assume that our network has learned cloud-filtering schemes solely from input data, which could alleviate the need for tedious cloud-filtering as a preprocessing step for many EO approaches. Moreover, using unfiltered temporal series of top-of-atmosphere (TOA) reflectance data, we achieved in our experiments state-of-the-art classification accuracies on a large number of crop classes with minimal preprocessing compared to other classification approaches.

• The paper introduces a novel sequence-to-sequence approach using convolutional recurrent models with GRU and LSTM to classify multi-temporal Earth observation data.
• It achieves state-of-the-art crop classification accuracy on Sentinel-2 data by efficiently filtering cloud cover without atmospheric correction.
• The work enables scalable remote sensing applications by automating preprocessing and reducing reliance on manual feature engineering.

Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders

The paper "Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders" by Rußwurm and Körner explores the application of sequential recurrent models for the classification of temporal remote sensing data, specifically for land use and land cover (LULC) tasks. It expands upon traditional approaches that primarily utilize mono-temporal, cloud-free datasets by implementing sequence-to-sequence architectures for extracting temporal features from the inherently sequential observations provided by Earth observation (EO) sensors.

Methodological Advances

This paper proposes the adaptation of sequence-to-sequence learning, traditionally used in domains like neural machine translation, to EO fields. It introduces convolutional recurrent neural networks (RNNs) enhanced with gated recurrent units (GRUs) and long short-term memory (LSTM) cells, allowing effective classification of multi-spectral, multi-temporal data. The innovative aspect lies in the model's ability to handle sequential inputs and internally filter out noisy observations like cloud cover without additional preprocessing. The authors visualize internal gate activations, highlighting that some recurrent cells learn to attenuate input activity for cloudy observations, thus miming cloud-filtering processes automatically.

Experimental Framework and Results

Using Sentinel 2 (S2) data, the research illustrates state-of-the-art accuracy in crop classification across two seasons. The methodology does not require atmospheric correction of input data, as it operates directly on top-of-atmosphere (TOA) reflectance values, which underscores the model's robustness against atmospheric noise and its low preprocessing requirement. In experiments, the model produced competitive classification accuracies across numerous crop classes with minimal manual intervention, showcasing the effectiveness of encoding temporal sequences for phenological modeling.

Implications and Future Directions

This work signifies a shift towards data-driven models in remote sensing applications, reducing reliance on manually designed, feature-specific techniques and leveraging the power of deep learning for capturing the complex temporal dynamics of phenological events. The implications of this are multifaceted:

1. Scalability: The methodology scales well, allowing for large-scale geographic deployments without needing extensive supervision or region-specific parameter tuning.
2. Automated Preprocessing: By internalizing cloud masking, it alleviates the need for external cloud-filtering algorithms—streamlining deployment in diverse and dynamic environments.
3. Theoretical Insight: Insights into recurrent architectures in EO provide pathways for further research into discriminative temporal feature learning, potentially influencing advancements in sequential data tasks beyond vegetation modeling.

The paper lays the foundation for further investigation into adaptive sequential learning techniques tailored for the unique challenges presented by EO data. Future research could explore the integration of additional data sources, refining temporal encoding methods, and addressing seasonally variant environmental influences that may impact classification robustness.

The authors have committed to sharing their TENSORFLOW implementation, facilitating reproducibility and encouraging adoption of these advanced techniques in related fields. With continuing advancements in machine learning and sensor capabilities, the role of temporal information in remotely sensed data applications is poised to expand, driving innovation and understanding of dynamic Earth systems.