Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks (1710.09207v1)

Abstract: We investigate anomaly detection in an unsupervised framework and introduce Long Short Term Memory (LSTM) neural network based algorithms. In particular, given variable length data sequences, we first pass these sequences through our LSTM based structure and obtain fixed length sequences. We then find a decision function for our anomaly detectors based on the One Class Support Vector Machines (OC-SVM) and Support Vector Data Description (SVDD) algorithms. As the first time in the literature, we jointly train and optimize the parameters of the LSTM architecture and the OC-SVM (or SVDD) algorithm using highly effective gradient and quadratic programming based training methods. To apply the gradient based training method, we modify the original objective criteria of the OC-SVM and SVDD algorithms, where we prove the convergence of the modified objective criteria to the original criteria. We also provide extensions of our unsupervised formulation to the semi-supervised and fully supervised frameworks. Thus, we obtain anomaly detection algorithms that can process variable length data sequences while providing high performance, especially for time series data. Our approach is generic so that we also apply this approach to the Gated Recurrent Unit (GRU) architecture by directly replacing our LSTM based structure with the GRU based structure. In our experiments, we illustrate significant performance gains achieved by our algorithms with respect to the conventional methods.

• The paper introduces a novel method integrating LSTM with OC-SVM and SVDD to transform variable-length data for effective anomaly detection.
• It employs both quadratic programming and gradient-based optimization to jointly train neural and detection components.
• Experimental results on diverse time-series datasets demonstrate significant performance gains over traditional anomaly detection methods.

Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks

This paper presents a comprehensive paper on anomaly detection using modern neural network architectures, focusing on unsupervised frameworks and leveraging Long Short-Term Memory (LSTM) networks. The research introduces novel algorithms that are capable of transforming variable-length data sequences into fixed-length representations, facilitating the use of anomaly detectors traditionally limited to fixed-length inputs, such as One-Class Support Vector Machines (OC-SVM) and Support Vector Data Description (SVDD).

Core Contributions

1. Integration of LSTM with OC-SVM and SVDD: The authors propose a method that combines LSTM networks with OC-SVM and SVDD algorithms, jointly optimizing them to improve anomaly detection performance. This integration is cited as a pioneering approach for the simultaneous training of LSTM architectures and anomaly detection algorithms, driven by gradient and quadratic programming-based methods.
2. Handling Variable-Length Sequences: A significant challenge in anomaly detection under an unsupervised framework is the handling of variable-length sequences. The paper tackles this by using LSTM networks to convert variable-length sequences into fixed-length feature vectors, thereby enabling the application of OC-SVM and SVDD.
3. Generic Approach: The framework is described as generic because it applies not only to LSTM-based structures but is also extended to include Gated Recurrent Unit (GRU) networks, presenting further versatility to the approach for various recurrent neural network architectures.
4. Demonstrated Improvement: The results indicate considerable performance improvements over traditional OC-SVM and SVDD methods, particularly in handling time-series data. This includes numerical evidence of significant gains in tasks requiring the detection of anomalies from datasets with complex temporal dependencies.

Technical Realization

Two main training strategies are proposed: quadratic programming and gradient-based optimization methods. Notably, the authors modify the original objective functions of the OC-SVM and SVDD to enable gradient-based training and prove convergence of these modified objectives to their original formulations.

Quadratic Programming Approach

This approach leverages the conversion of the anomaly detection problem into its dual form using Lagrangian principles to subsequently apply Sequential Minimal Optimization (SMO) for parameter updates.

Gradient-Based Approach

A differentiable approximation of the OC-SVM and SVDD formulations allows the use of standard stochastic gradient descent (SGD) techniques to jointly optimize both the LSTM and the anomaly detection parameters.

Experimental Evaluation

The paper validates its claims through extensive experimentation on varied datasets, including digit recognition, occupancy datasets, foreign exchange rates, and stock prices. A prominent aspect is the algorithm's success on datasets that display sequential and temporal characteristics, underscoring the efficacy of the memory-based LSTM in capturing long-range dependencies.

Implications and Speculation

The implications of this research are substantial in the context of tasks where labeled data for anomalies is scarce or costly. The ability to operate in a semi-supervised framework extends the utility of the proposed methods across domains such as network security, fraud detection, and more. Moving forward, the combination of different neural architectures, such as GRU, with advanced anomaly detection algorithms could further bolster the performance and applicability of these techniques.

The ability to effectively and efficiently detect anomalies in unsupervised or semi-supervised frameworks offers a path towards more intelligent and autonomous systems capable of dynamic learning and adaptation, crucial for real-time applications in ever-evolving data environments.

This contribution is positioned as a foundation for further exploration into integrated neural network and anomaly detection methods, promising advancements in the theoretical and practical frontiers of AI-driven anomaly detection.