Anomaly Transformer

• The paper introduces a novel attention mechanism that uses association discrepancy to differentiate normal and anomalous time series patterns.
• It employs a dual-branch architecture combining series-association and prior-association to capture both global and local context in sequential data.
• Extensive benchmarks demonstrate the model’s superior performance through improved precision, recall, and F1 scores over multiple deep learning baselines.

Anomaly Transformer refers to a class of unsupervised, transformer-based models that leverage association discrepancy for detecting anomalies in sequential data, most notably time series. These models are characterized by novel attention mechanisms that contrast local and global attention patterns and introduce precise, theoretically grounded anomaly scoring functions. The principal representative is the "Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy" model (Xu et al., 2021), which established a foundation for subsequent advances.

1. Architectural Foundation and Attention Mechanism

Anomaly Transformer employs a stack of encoder-only transformer layers, each operating over embedded and optionally position-encoded time series data. For a time series $\mathcal X = \{x_1, ..., x_N\}$, where $x_t \in \mathbb{R}^d$, each input is linearly projected to a $d_\text{model}$-dimensional embedding. Positional encoding, typically sinusoidal, may be added.

The critical innovation is the two-branch "Anomaly-Attention" mechanism, formally comprising:

• Series-association branch: Computes conventional scaled dot-product self-attention,

$$S^l_{i,j} = \mathrm{softmax}_j \left( \frac{Q_i \cdot K_j}{\sqrt{d_\text{model}}} \right)$$

where $Q$, $K$, and $V$ are standard query, key, and value matrices, and $l$ is the current layer.

• Prior-association branch: Imposes an inductive bias toward temporal adjacency by generating a learnable, unimodal Gaussian over indices, parameterized by learnable per-index widths $\sigma_i$:

$$P^l_{i,j} = \frac{\exp \left( -\frac{|i-j|^2}{2 \sigma_i^2} \right) / (\sqrt{2\pi} \sigma_i)}{\sum_{k=1}^N \exp \left( -\frac{|i-k|^2}{2 \sigma_i^2} \right) / (\sqrt{2\pi} \sigma_i)}$$

In each encoder block, these two attention distributions, $x_t \in \mathbb{R}^d$0 and $x_t \in \mathbb{R}^d$1, are computed in parallel. The value outputs are propagated to subsequent layers via standard residual connections and a pointwise MLP.

2. Association Discrepancy and Anomaly Scoring

Central to the Anomaly Transformer is measuring the divergence between the series-association and prior-association distributions for each position $x_t \in \mathbb{R}^d$2. The association discrepancy metric quantifies this:

$x_t \in \mathbb{R}^d$3

where $x_t \in \mathbb{R}^d$4 is the number of Transformer layers.

Empirically, normal points develop support for non-adjacent associations, leading to a high discrepancy (large AssDis), while anomalies—unable to relate to typical long-range patterns—exhibit concentrated, local attention and thus lower AssDis. This property enables a theoretically motivated, point-wise discrimination criterion.

At test time, the model computes for each time step:

• $x_t \in \mathbb{R}^d$5: softmax-normalized association discrepancy (higher when AssDis is low)
• $x_t \in \mathbb{R}^d$6: per-point reconstruction error, $x_t \in \mathbb{R}^d$7
• Final anomaly score: $x_t \in \mathbb{R}^d$8

3. Minimax Training Objective

The optimization objective jointly balances two terms:

• Reconstruction loss: $x_t \in \mathbb{R}^d$9 enforces faithful representation of normal data.
• Discrepancy regularizer: Weighted sum of per-step association discrepancy, $d_\text{model}$0.

The overall loss function is

$d_\text{model}$1

A minimax strategy is used to avoid collapse ($d_\text{model}$2):

• Minimize-phase: Update prior branch parameters ($d_\text{model}$3) and decoder, minimizing $d_\text{model}$4, with gradients stopped on $d_\text{model}$5.
• Maximize-phase: Update $d_\text{model}$6 (series-attention branch) and decoder, maximizing $d_\text{model}$7 (i.e., increasing AssDis) while minimizing $d_\text{model}$8, with gradients stopped on $d_\text{model}$9.

This alternation amplifies the separation between normal and anomalous attention patterns.

4. Experimental Validation and Benchmarks

Anomaly Transformer has been rigorously benchmarked against deep learning baselines such as InterFusion, OmniAnomaly, and THOC. Evaluation spans multiple domains:

• Datasets: SMD (Server Machine Data), PSM (Pooled Server Metrics), MSL (Mars Science Laboratory), SMAP (Soil Moisture Satellite), SWaT (Water Treatment), and NeurIPS-TS (synthetic).
• Metrics: Precision, Recall, F1 score, ROC-AUC.

Representative F1 results extracted directly from the data:

Dataset	InterFusion	OmniAnomaly	THOC	Anomaly Transformer
SMD	86.22	85.22	84.99	92.33
MSL	86.62	87.67	89.69	93.59
SMAP	89.14	86.92	90.68	96.69
SWaT	83.01	82.83	85.13	94.07
PSM	83.52	80.83	89.54	97.89

The Anomaly Transformer achieves state-of-the-art performance on all listed benchmarks (Xu et al., 2021).

5. Implementation and Inference Details

Salient implementation specifications include:

• Architecture: 3 encoder layers, $$S^l_{i,j} = \mathrm{softmax}_j \left( \frac{Q_i \cdot K_j}{\sqrt{d_\text{model}}} \right)$$0, 8 attention heads
• Windowing: Non-overlapping input windows of length 100
• Optimization: Adam optimizer, learning rate $$S^l_{i,j} = \mathrm{softmax}_j \left( \frac{Q_i \cdot K_j}{\sqrt{d_\text{model}}} \right)$$1, batch size 32, early stopping within 10 epochs
• Thresholding: Calibrated on validation data; top $$S^l_{i,j} = \mathrm{softmax}_j \left( \frac{Q_i \cdot K_j}{\sqrt{d_\text{model}}} \right)$$2% of anomaly scores selected (e.g., $$S^l_{i,j} = \mathrm{softmax}_j \left( \frac{Q_i \cdot K_j}{\sqrt{d_\text{model}}} \right)$$3 for SWaT, $$S^l_{i,j} = \mathrm{softmax}_j \left( \frac{Q_i \cdot K_j}{\sqrt{d_\text{model}}} \right)$$4 for SMD)

During inference, the anomaly score for each point is computed using the hybrid criterion described above, allowing robust temporal anomaly delineation.

6. Related Directions and Extensions

Subsequent advances incorporate the core principles of the Anomaly Transformer, including:

• ADTR: Applies transformers for feature (rather than pixel) reconstruction, exploiting semantic embeddings from pretrained backbones and leveraging transformer attention to enhance anomaly–normal separation (You et al., 2022).
• AnomalyBERT: Introduces a self-supervised transformer trained using synthetic data degradations, with a relative position bias to capture flexible temporal dependencies in multivariate time series (Jeong et al., 2023).
• ISSTAD: Synergizes masked autoencoding with pixelwise self-supervision for image anomaly detection and localization, fusing per-pixel reconstruction errors with classifier outputs for precise detection (Jin et al., 2023).
• TiSAT: Proposes a sparse-attention transformer with a novel sequence-level evaluation criterion, further validating the efficacy of transformer architectures for timely anomaly detection in sequential data (Doshi et al., 2022).

7. Position in the Literature and Significance

The Anomaly Transformer established a conceptual and practical framework in which association discrepancy—reflecting the contrast between global and local attentional focus—serves as a robust indicator of anomalous structure in sequential data. The minimax training framework creates a sharp distinguishability criterion between normal and abnormal regimes, with empirical superiority across heterogeneous benchmarks. The mechanistic insights—particularly the use of attention distributions as intrinsic anomaly markers—have fundamentally shaped subsequent transformer-based anomaly detection research in both time series and high-dimensional vision settings (Xu et al., 2021, You et al., 2022, Jeong et al., 2023, Jin et al., 2023, Doshi et al., 2022).