---
title: 'MIDAS: Microcluster Anomaly Detection'
url: https://www.emergentmind.com/topics/midas
type: topic
---

# MIDAS: Microcluster Anomaly Detection

MIDAS

MIDAS is an acronym employed across diverse disciplines to name distinct tools, models, and experiments in data science, machine learning, robotics, astronomy, computer vision, and sensing. Below, major MIDAS variants are organized as independent research contributions, each defined by its core technical achievements, methodologies, and empirical impacts, as found in primary arXiv sources.

## 1. Microcluster-Based Anomaly Detection in Edge Streams

MIDAS refers to the “Microcluster-Based Detector of Anomalies in Edge Streams”—an online algorithm for detecting anomalous groups of edges (microclusters) in massive, temporally evolving graphs. Unlike methods that solely identify individually rare edges, MIDAS focuses on sudden bursts of similar edge activity, as in lockstep or denial-of-service attacks [1911.04464].

### Problem Definition and Microcluster Anomalies

A microcluster is defined as a “suddenly arriving group of suspiciously similar edges,” for example, a burst of the same or related $(u,v)$ pairs within a short time tick. The detection objective is not the surprise of a single edge, but the collective deviation from historical behavior: for each $(u,v)$ (or around node $u$/$v$), MIDAS tests if the edge count at the current tick greatly exceeds the expectation under the historic per-tick rate.

### Algorithmic Structure

- **Data Structures**: MIDAS maintains two Count–Min Sketches (CMS): one for the total counts $s_{uv}$ over all ticks, and one for the current-tick counts $a_{uv}$ at time $t$. All CMSs operate with $O(1)$ time per edge (constant space per edge), with width $b$, depth $w$ fixed.
- **Online Anomaly Score**: The system executes a streaming, two-bin chi-squared test. At time $t$ for $(u,v)$, the expected count is $s_{uv}/t$. The score is
  $$
  \text{score}(u,v,t) = \left(\hat a_{uv}-\frac{\hat s_{uv}}{t}\right)^2 \frac{t^2}{\hat s_{uv}(t-1)}
  $$
  where $\hat a_{uv}$, $\hat s_{uv}$ are CMS estimates.
- **Extension – MIDAS-R**: Adds temporal decay ($\alpha$-discounting between ticks) and computes per-node CMS and scores for spatial relations, assigning to each edge the maximum anomaly score over edge and endpoint nodes.

### Theoretical Guarantees

MIDAS provides a provable upper bound $\epsilon$ on the probability of false positives. For bias-corrected count $\tilde a_{uv}$, the adjusted test statistic $\tilde X^2$ satisfies
$$
P(\tilde X^2 > \chi^2_{1-\epsilon/2}(1)) < \epsilon
$$
where $\chi^2_{1-\epsilon/2}(1)$ is the corresponding quantile of the chi-squared distribution.

### Empirical Results

Evaluations on real network and event-stream datasets, including DARPA Intrusion, TwitterSecurity, and TwitterWorldCup, demonstrate the following:

| Dataset          | AUC (SedanSpot) | AUC (MIDAS) | Time (SedanSpot, s) | Time (MIDAS, s) | AUC Gain | Speedup |
|------------------|----------------|------------|---------------------|-----------------|----------|---------|
| DARPA            | 0.64           | 0.91       |   84                |   0.13          | +42%     | 644×    |
| DARPA (MIDAS-R)  | -              | 0.95       |   -                 |   0.39          | +48%     | 215×    |
| TwitterWorldCup  | -              | -          |  27.58              |   0.06          | -        | 460×    |

Average precision improvements closely match AUC gains. MIDAS-R’s anomaly

Source: https://www.emergentmind.com/topics/midas