Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection (1904.02639v2)

Abstract: Deep autoencoder has been extensively used for anomaly detection. Training on the normal data, the autoencoder is expected to produce higher reconstruction error for the abnormal inputs than the normal ones, which is adopted as a criterion for identifying anomalies. However, this assumption does not always hold in practice. It has been observed that sometimes the autoencoder "generalizes" so well that it can also reconstruct anomalies well, leading to the miss detection of anomalies. To mitigate this drawback for autoencoder based anomaly detector, we propose to augment the autoencoder with a memory module and develop an improved autoencoder called memory-augmented autoencoder, i.e. MemAE. Given an input, MemAE firstly obtains the encoding from the encoder and then uses it as a query to retrieve the most relevant memory items for reconstruction. At the training stage, the memory contents are updated and are encouraged to represent the prototypical elements of the normal data. At the test stage, the learned memory will be fixed, and the reconstruction is obtained from a few selected memory records of the normal data. The reconstruction will thus tend to be close to a normal sample. Thus the reconstructed errors on anomalies will be strengthened for anomaly detection. MemAE is free of assumptions on the data type and thus general to be applied to different tasks. Experiments on various datasets prove the excellent generalization and high effectiveness of the proposed MemAE.

• The paper introduces MemAE, a novel deep autoencoder that incorporates an external memory module to prevent accurate reconstruction of anomalies.
• It employs a content-addressable memory and a sparse addressing mechanism to store and retrieve prototypical normal patterns efficiently.
• Experimental results on image, video, and cybersecurity datasets demonstrate MemAE's superior performance over traditional anomaly detection methods.

Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection

Overview

The paper by Gong et al. presents a novel approach to anomaly detection using a memory-augmented deep autoencoder, referred to as MemAE. The paper capitalizes on the strengths of autoencoders (AEs) while addressing a significant drawback: the potential of AEs to reconstruct anomalies well, thereby leading to missed detections.

Memory-augmented Autoencoder (MemAE)

Problem Definition

Anomaly detection is an inherently challenging task, particularly in unsupervised settings where only normal data examples are available for training. Traditional AEs are trained to minimize reconstruction error on normal data, with the assumption that anomalies, being dissimilar to the training data, will yield higher reconstruction errors. However, AEs sometimes generalize too well and reconstruct anomalies accurately, undermining this assumption. The authors mitigate this issue by integrating an external memory module into the AE framework.

The MemAE Architecture

The proposed MemAE architecture incorporates an encoder, a memory module, and a decoder. The encoder generates a compressed encoding of the input, which then serves as a query to retrieve the most relevant items from the memory. These memory items, representing prototypical normal patterns, are aggregated and passed to the decoder for reconstruction.

1. Encoder and Decoder: The encoder transforms input data into a latent space, and the decoder reconstructs the original input from this representation. The structure of these components can be tailored to different applications.
2. Memory Module: A content-addressable memory is used to store prototypical normal patterns. During training, this memory is updated to efficiently represent normal data. In testing, the memory remains fixed.
3. Sparse Addressing: To ensure only essential memory items are used, an attention-based mechanism calculates the similarity between the encoded query and memory items. A hard shrinkage operator is applied to induce sparsity, encouraging the model to rely on fewer, more relevant memory items.
4. Training and Objective Function: The training objective combines the reconstruction error with an entropy-based regularizer to promote sparsity in memory addressing, ensuring that the memory accurately symbolizes normal patterns.

Results and Implications

Experimental Validation

The paper evaluated MemAE on multiple datasets, emphasizing its generality and applicability across different data types:

• Image Data: On MNIST and CIFAR-10 datasets, MemAE demonstrated superior performance in anomaly detection tasks compared to several existing methods, including OC-SVM, KDE, VAE, and DSEBM, showcasing the effectiveness of the memory module.
• Video Data: Extensive experiments were conducted on UCSD-Ped2, CUHK Avenue, and ShanghaiTech video datasets. MemAE consistently outperformed state-of-the-art methods such as TSC, StackRNN, and Frame-Pred. Importantly, MemAE's reconstruction focused on distinguishing normal patterns, leading to enhanced detection of anomalous events.
• Cybersecurity Data: On the KDDCUP99 dataset, MemAE achieved notable precision, recall, and F1 scores, reflecting its robustness in identifying anomalies in cybersecurity environments.

Theoretical and Practical Implications

The introduction of MemAE represents a significant methodological advancement in anomaly detection. The memory module's capability to store and utilize prototypical patterns of normal data enhances the robustness of AEs against false negatives. The sparse addressing mechanism is particularly impactful in ensuring that the model relies on relevant, concise memory items, thereby accentuating anomalies in the data.

Future Directions

The paper opens several avenues for future research:

• Integration with Advanced Models: The memory module's generality allows it to be seamlessly integrated into more sophisticated neural architectures, potentially broadening its applicability.
• Enhanced Anomaly Interpretation: Further investigation into using the memory addressing weights as a direct anomaly detection indicator could provide more intuitive and interpretable results.
• Broader Application Domains: Exploring MemAE's application in more complex or large-scale datasets can validate its scalability and effectiveness in diverse domains.

Conclusion

The paper by Gong et al. contributes a refined approach to unsupervised anomaly detection through the innovative use of a memory-augmented autoencoder. By addressing key limitations of traditional AEs, MemAE achieves improved detection performance across various datasets, underscoring its theoretical appeal and practical utility. The paper's insights and methodologies pave the way for continued advancements in the field of anomaly detection.