Self-attention for raw optical Satellite Time Series Classification (1910.10536v3)

Abstract: The amount of available Earth observation data has increased dramatically in the recent years. Efficiently making use of the entire body information is a current challenge in remote sensing and demands for light-weight problem-agnostic models that do not require region- or problem-specific expert knowledge. End-to-end trained deep learning models can make use of raw sensory data by learning feature extraction and classification in one step solely from data. Still, many methods proposed in remote sensing research require implicit feature extraction through data preprocessing or explicit design of features. In this work, we compare recent deep learning models on crop type classification on raw and preprocessed Sentinel 2 data. We concentrate on the common neural network architectures for time series, i.e., 1D-convolutions, recurrence, a shallow random forest baseline, and focus on the novel self-attention architecture. Our central findings are that data preprocessing still increased the overall classification performance for all models while the choice of model was less crucial. Self-attention and recurrent neural networks, by their architecture, outperformed convolutional neural networks on raw satellite time series. We explore this by a feature importance analysis based on gradient back-propagation that exploits the differentiable nature of deep learning models. Further, we qualitatively show how self-attention scores focus selectively on few classification-relevant observations.

• The paper demonstrates that self-attention significantly enhances crop type classification from raw Sentinel-2 data.
• It compares neural architectures and finds that self-attention and RNNs outperform CNNs by capturing temporal features effectively.
• The study shows that reducing preprocessing needs and enhancing noise resilience can streamline remote sensing analysis.

Self-attention for Raw Optical Satellite Time Series Classification: An Overview

In the paper titled "Self-attention for raw optical Satellite Time Series Classification," Rußwurm and Körner address the challenge of effectively utilizing extensive Earth observation data from the Copernicus program's Sentinel satellites. Their work aims to advance remote sensing technologies by leveraging deep learning models, particularly focusing on crop type classification using Sentinel 2 data. The authors emphasize the power of end-to-end trained models that operate on raw sensory data, proposing an evaluation of self-attention—a mechanism that has revolutionized natural language processing—in the domain of satellite time series.

Key Findings and Methodologies

This paper investigates several neural network architectures—namely 1D-convolutions, recurrent neural networks (RNNs), and self-attention—to determine their respective efficacies in classifying crop types from raw and preprocessed satellite data. The authors conclude that while preprocessing enhances classification across all models, self-attention and RNNs outperform convolutional networks on raw data, owing to their architectural design capable of capturing temporal features more effectively.

Self-Attention Mechanism

The self-attention mechanism, a cornerstone of the Transformer architecture, enables models to focus on relevant parts of the input time series dynamically. By conducting a feature importance analysis based on gradient backpropagation, the authors demonstrate that self-attention selectively emphasizes classification-relevant observations, suggesting that self-attention can approximate preprocessing effects implicitly within the model's learning process. This mechanism assigns weights to different time steps, allowing the model to ignore irrelevant data (such as cloud cover) and focus on critical periods for crop classification.

Numerical Results and Model Evaluation

The paper provides comparative analyses using Kappa, overall accuracy, and class-mean F1 scores as metrics across various model architectures and data preprocessing states. Notably, LSTM-RNN and Transformer models showed superior performance on raw datasets compared to convolutional models. While preprocessing generally improved results across all models, the deep learning architectures based on self-attention and recurrence demonstrated robust resilience to noise and more effective handling of raw satellite time series data.

Implications and Future Directions

The implications of successfully applying self-attention to satellite time series classification are substantial. By reducing the dependency on extensive data preprocessing and expert knowledge, the approach has the potential to streamline the processing of vast volumes of geospatial data, facilitating more efficient land use and land cover monitoring. This research elucidates the adaptability of self-attention beyond traditional NLP domains and underscores its potential for enhancing temporal feature extraction in remote sensing tasks.

Looking ahead, further exploration into hybrid models that integrate diverse neural network architectures might yield even better results. Additionally, expanding the scope to include different pre-processing methods and satellite data types could validate the generalizability of the findings across various remote sensing applications.

In summary, this paper contributes to the field of remote sensing by demonstrating that self-attention provides a competitive edge in handling raw optical satellite time series. Their exploration into this emerging technique sets the stage for more refined, efficient, and scalable methods for processing Earth observation data.