Collective Anomaly Detection based on Long Short Term Memory Recurrent Neural Network (1703.09752v1)

Abstract: Intrusion detection for computer network systems becomes one of the most critical tasks for network administrators today. It has an important role for organizations, governments and our society due to its valuable resources on computer networks. Traditional misuse detection strategies are unable to detect new and unknown intrusion. Besides, anomaly detection in network security is aim to distinguish between illegal or malicious events and normal behavior of network systems. Anomaly detection can be considered as a classification problem where it builds models of normal network behavior, which it uses to detect new patterns that significantly deviate from the model. Most of the cur- rent research on anomaly detection is based on the learning of normally and anomaly behaviors. They do not take into account the previous, re- cent events to detect the new incoming one. In this paper, we propose a real time collective anomaly detection model based on neural network learning and feature operating. Normally a Long Short Term Memory Recurrent Neural Network (LSTM RNN) is trained only on normal data and it is capable of predicting several time steps ahead of an input. In our approach, a LSTM RNN is trained with normal time series data before performing a live prediction for each time step. Instead of considering each time step separately, the observation of prediction errors from a certain number of time steps is now proposed as a new idea for detecting collective anomalies. The prediction errors from a number of the latest time steps above a threshold will indicate a collective anomaly. The model is built on a time series version of the KDD 1999 dataset. The experiments demonstrate that it is possible to offer reliable and efficient for collective anomaly detection.

• The paper presents a novel method that employs LSTM RNNs to track prediction errors over multiple time steps, enabling the detection of collective anomalies in network security.
• It introduces a circular array mechanism to monitor error metrics, including Relative Error, Danger Coefficient, and Average Relative Error, against set thresholds.
• Empirical results on the KDD 1999 dataset show a 100% detection rate for DoS attacks, highlighting the model’s potential despite a trade-off with higher false alarms.

Collective Anomaly Detection based on LSTM Recurrent Neural Network

This paper presents a novel framework for collective anomaly detection, specifically applied to network security, using Long Short-Term Memory Recurrent Neural Networks (LSTM RNNs). The focus lies in addressing limitations in traditional anomaly detection methods which often perceive anomalies as isolated events and fail to account for temporal dependencies. The proposed model leverages the predictive capabilities of LSTM RNNs, which are adept at incorporating historical data into predictions, to discern patterns indicative of collective anomalies, particularly in scenarios akin to denial-of-service (DoS) attacks.

Key Contributions

The primary innovation of this work is the employment of an LSTM RNN to monitor prediction errors over multiple time steps, rather than evaluating each point independently. This approach enables the detection of collective anomalies where anomalous behavior emerges only when considered in aggregation over a period.

1. LSTM RNN Architecture: Building on established success in time series prediction, LSTM RNNs are utilized for their ability to maintain state over time, thus effectively capturing temporal dependencies that are pivotal for identifying groups of anomalies.
2. Circular Array for Error Tracking: The paper introduces a mechanism to track and analyze prediction errors in a circular array, maintaining a moving window of error terms. Anomalies are flagged based on summary statistics of these errors against predetermined thresholds.
3. Metrics and Definitions: To operationalize collective anomaly detection, several metrics are defined, including Relative Error (RE), Danger Coefficient (DC), and Average Relative Error (ARE). These metrics assess, cumulatively and individually, deviations from expected network behavior.
4. Empirical Validation: Conducted on the time series version of the KDD 1999 dataset, the experiments validate the model’s capability. The results highlight the model's effectiveness in detecting anomalies that manifest not at single points, but across sequences of time steps.

Results

The experiments reveal a balance between detection accuracy and false alarm rates. By adjusting the danger coefficient and relative error thresholds, the model achieves a 100% detection rate at the cost of increased false alarms. This trade-off underscores a typical challenge in anomaly detection tasks: optimizing sensitivity and specificity.

Practical and Theoretical Implications

The implications of this research lie in enhancing intrusion detection systems by dynamically monitoring network data and identifying attacks like DoS in real-time. Theoretically, it extends the applicability of LSTM networks beyond typical sequence prediction to more complex temporal anomaly detection tasks.

Future Prospects

Looking forward, there is potential to improve the classifier's performance through more sophisticated feature engineering and by incorporating additional contextual data. Moreover, adapting the methodology to other domains (e.g., fraud detection, financial time series analysis) could be explored. Increasing the robustness of the model by integrating hybrid architectures that combine LSTM with other neural network models might offer further advancements in collective anomaly detection.

In conclusion, this paper innovatively adapts LSTM RNN frameworks for collective network anomaly detection, offering a novel methodology that is both empirically robust and adaptable for real-time network security monitoring.