Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding

Abstract: As spacecraft send back increasing amounts of telemetry data, improved anomaly detection systems are needed to lessen the monitoring burden placed on operations engineers and reduce operational risk. Current spacecraft monitoring systems only target a subset of anomaly types and often require costly expert knowledge to develop and maintain due to challenges involving scale and complexity. We demonstrate the effectiveness of Long Short-Term Memory (LSTMs) networks, a type of Recurrent Neural Network (RNN), in overcoming these issues using expert-labeled telemetry anomaly data from the Soil Moisture Active Passive (SMAP) satellite and the Mars Science Laboratory (MSL) rover, Curiosity. We also propose a complementary unsupervised and nonparametric anomaly thresholding approach developed during a pilot implementation of an anomaly detection system for SMAP, and offer false positive mitigation strategies along with other key improvements and lessons learned during development.

• The paper introduces an LSTM-based approach that learns long-term temporal patterns in spacecraft telemetry data for anomaly detection.
• It employs a nonparametric dynamic thresholding method to evaluate prediction errors and effectively prune false positives.
• Experimental results on NASA mission data demonstrate high precision (87.5%) and recall (80.0%), validating the method’s robustness.

Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding

The paper "Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding," presented at KDD '18, introduces an innovative approach for anomaly detection in spacecraft telemetry data leveraging Long Short-Term Memory (LSTM) networks. Authored by researchers from the NASA Jet Propulsion Laboratory, the work addresses the growing need for efficient and scalable anomaly detection systems due to the increasing volume and complexity of telemetry data sent by contemporary spacecraft.

Core Contributions

The paper outlines several key contributions to the anomaly detection domain, particularly in the context of aerospace applications:

1. LSTM-based Anomaly Detection: The authors employ LSTMs, a type of Recurrent Neural Network (RNN), to predict future telemetry values based on historical data. The objective is to capture both long-term and short-term temporal dependencies in multi-channel telemetry data streams.
2. Nonparametric Dynamic Thresholding: To evaluate the prediction errors generated by the LSTMs, the paper proposes a nonparametric dynamic thresholding method. This method eschews Gaussian assumptions, instead opting to dynamically set thresholds for anomaly detection based on historical error distributions.
3. False Positive Mitigation: The authors introduce strategies for pruning false positives by leveraging historical data and anomaly scores. This is crucial for ensuring operational efficiency and gaining the trust of operations engineers who must interpret the results.
4. Real-world Evaluation: Experimental results are provided using telemetry data from two NASA missions, the Soil Moisture Active Passive (SMAP) satellite and the Mars Science Laboratory (MSL) rover, Curiosity. The evaluation demonstrates the efficacy of their methods with specific emphasis on precision and recall metrics.

Methodology

The methodology involves training individual LSTM models for each telemetry channel to predict the next telemetry value. The model consumes a sequence of past telemetry values and encoded command information to generate predictions. Consequently, prediction errors are smoothed and evaluated against thresholds that are determined dynamically.

Dynamic Error Thresholds: The proposed thresholding mechanism calculates an optimal threshold from historical smoothed errors without relying on parametric assumptions. This approach identifies the threshold that maximizes the reduction in error mean and variance, balancing the precision and recall for anomaly detection.

Pruning False Positives: A critical aspect of the work is the adoption of pruning strategies to handle false positives. By evaluating the distribution and magnitudes of errors, the system reclassifies anomalies that do not exhibit significant deviations from historical norms. This step ensures that operational efficiency is maintained by reducing the number of superfluous alerts.

Experimental Evaluation

Experiments using labeled telemetry anomaly data from SMAP and MSL underscore the robustness of the proposed methods. Key findings include:

• Prediction Accuracy: The LSTM models achieved an average normalized absolute error of 5.9%, indicating reliable prediction performance across diverse telemetry channels.
• Precision and Recall: The nonparametric dynamic thresholding approach with pruning attained a balanced precision of 87.5% and a recall of 80.0%, outperforming a traditional Gaussian tail approach in anomaly detection tasks.
• Anomaly Type Analysis: The system effectively identified 90.3% of point anomalies and 69.0% of contextual anomalies, which are typically more challenging to detect using conventional methods.

Practical and Theoretical Implications

The practical implications of this research are substantial. By integrating LSTM-based predictions with nonparametric thresholding and false positive mitigation, spacecraft operations can achieve more reliable and scalable anomaly detection capabilities. This is particularly pertinent for missions with large volumes of telemetry data and dynamic operational profiles, such as future deep-space missions and Earth observation satellites.

Theoretically, the work extends the application of RNNs to the domain of spacecraft telemetry analysis, demonstrating that advanced machine learning techniques can be adapted and optimized for high-stakes applications. The nonparametric dynamic thresholding method also provides a novel approach to handle non-Gaussian error distributions, which is broadly applicable to other time-series anomaly detection problems.

Future Developments

The paper identifies several avenues for future work, including:

• Feature Engineering: Enhancing the input features with more granular command information and event records to improve prediction accuracy.
• Automated Training Data Selection: Developing techniques for automatic selection of relevant training data based on predicted spacecraft activities.
• Correlation Analysis: Implementing mathematical models to analyze the interactions between different telemetry channels, enabling better insights into complex system behaviors.

Conclusion

This research paper presents a comprehensive and innovative approach to spacecraft anomaly detection using LSTMs and nonparametric dynamic thresholding. Its contributions lie not only in the methodologies proposed but also in the practical demonstration of these techniques on real-world telemetry data from NASA missions. The potential for improved anomaly detection systems to enhance spacecraft operational safety and efficiency marks this work as a significant step forward in the field. Future efforts will further refine these methods and extend their applicability to a wider range of space missions and telemetry data types.