MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams (1911.04464v5)

Abstract: Given a stream of graph edges from a dynamic graph, how can we assign anomaly scores to edges in an online manner, for the purpose of detecting unusual behavior, using constant time and memory? Existing approaches aim to detect individually surprising edges. In this work, we propose MIDAS, which focuses on detecting microcluster anomalies, or suddenly arriving groups of suspiciously similar edges, such as lockstep behavior, including denial of service attacks in network traffic data. MIDAS has the following properties: (a) it detects microcluster anomalies while providing theoretical guarantees about its false positive probability; (b) it is online, thus processing each edge in constant time and constant memory, and also processes the data 162-644 times faster than state-of-the-art approaches; (c) it provides 42%-48% higher accuracy (in terms of AUC) than state-of-the-art approaches.

• The paper introduces MIDAS, a novel streaming method that leverages Count-Min Sketch to detect microcluster anomalies in dynamic graph edge streams.
• The approach provides theoretical guarantees with bounded false positive rates while achieving 42-48% higher AUC and 162-644 times faster processing than existing methods.
• The method is applicable to real-world scenarios like intrusion detection and fraud analytics, offering a reliable solution for timely anomaly identification.

MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams

The paper introduces MIDAS, a novel method for anomaly detection in dynamic graph edge streams, emphasizing its utility in identifying microcluster anomalies using a constant time and memory approach. The authors focus on detecting sudden groups of suspiciously similar edges, such as those evident in denial of service attacks. This method not only processes data significantly faster than existing techniques but also offers increased accuracy in anomaly detection.

Dynamic graphs are integral to various real-world applications, ranging from intrusion detection to financial fraud, requiring efficient and prompt anomaly detection mechanisms. Traditional approaches primarily target static graphs or aggregate edges into graph snapshots, which may overlook the temporal properties and emergent suspicious patterns within dynamic data streams. MIDAS aims to bridge this gap by monitoring for microcluster anomalies in real-time.

Contributions and Methodology

1. Microcluster Detection: MIDAS introduces a streaming approach that identifies microcluster anomalies consistently across the edge streams. Unlike traditional methods that focus on individual anomalous edges, MIDAS excels by detecting cohesive clusters of anomalous behavior.
2. Theoretical Guarantees: The paper provides a theoretical framework, assuring bounded false positive probabilities for anomaly detection, which is crucial for maintaining accuracy and reliability amid volatile data streams.
3. Efficiency and Performance: The authors demonstrate that MIDAS processes data 162 to 644 times faster than existing state-of-the-art approaches. Furthermore, the method achieves 42-48% higher accuracy in terms of the Area Under the Curve (AUC) metric, showcasing its effectiveness in real-world datasets.

Theoretical Framework and Algorithm Overview

MIDAS employs Count-Min Sketch (CMS) data structures to track and estimate edge counts in the stream, facilitating constant memory usage and processing time per edge. By analyzing the observed versus expected edge activity within a specified time frame, MIDAS effectively calculates anomaly scores, utilizing a hypothesis testing framework grounded in chi-squared distribution. This methodology allows MIDAS to achieve detections with quantifiable statistical reliability.

Additionally, the paper introduces MIDAS-R, an extension that considers temporal and spatial relations between edges. MIDAS-R leverages decayed counter updates to account for temporally proximate edges, enhancing detection capability by recognizing spatially correlated patterns of edge activity.

Experimental Evaluation

The experimental results confirm the robustness of MIDAS and MIDAS-R across various datasets, including DARPA network communications and Twitter datasets. The methods demonstrate superior speed and accuracy, markedly surpassing existing baseline methods like SedanSpot. Particularly in real-time detection scenarios, MIDAS exhibits enhanced adaptability and utility in identifying meaningful anomalies corresponding to real-world events, such as security threats and coordinated attacks.

Implications and Future Directions

The introduction of MIDAS has significant implications both practically and theoretically. By delivering real-time, accurate anomaly detection within edge streams, MIDAS positions itself as a valuable tool across industries reliant on dynamic network monitoring. The provision of false positive guarantees further reinforces its suitability for deployment in critical applications where precision is paramount.

Looking ahead, potential research could expand MIDAS's applicability to heterogeneous data types, such as multi-dimensional tensors, or explore integration with other machine learning models for enriched context-aware anomaly detection. The foundational principles outlined by MIDAS could inspire subsequent innovations in streaming anomaly detection, addressing evolving challenges posed by increasingly complex and voluminous data.