Pi-Transformer: Physics-Informed Anomaly Detection

• Pi-Transformer is a physics-informed Transformer that combines data-driven self-attention with learned priors encoding temporal invariants like scale self-similarity and phase synchrony.
• Its dual-pathway mechanism fuses series attention with a physics-based prior, enabling calibrated detection of amplitude, phase, and timing anomalies in multivariate signals.
• The model achieves state-of-the-art F1 scores on benchmark datasets by effectively integrating reconstruction errors with divergence between attention streams.

Pi-Transformer is a physics-informed Transformer architecture developed for unsupervised anomaly detection in multivariate time series. The design is characterized by a dual-pathway attention mechanism that unites data-driven self-attention with a learned, smoothly evolving prior encoding temporal invariants. This prior embeds scale-related self-similarity and cross-channel phase synchrony, enabling calibrated detection of anomalies that manifest through both amplitude aberrations and subtle temporal or cross-channel disruptions. The model achieves state-of-the-art or highly competitive F1 scores across canonical time series anomaly detection benchmarks, demonstrating particular strength on phase and timing anomalies (Maleki et al., 24 Sep 2025).

1. Dual Attention Pathway Architecture

Pi-Transformer utilizes a windowed encoder–decoder Transformer backbone. Each encoder layer and attention head maintains two parallel attention distributions:

• Series Attention ($S$): Standard scaled dot-product self-attention, capturing temporal dependencies through data-driven mechanisms across each windowed segment.
• Physics-Informed Prior Attention ($P$): A learnable, smoothly evolving "teacher" distribution that encodes temporally invariant structural priors, specifically scale-related self-similarity (via the Hurst exponent) and timing/phase synchrony.

At every time step, both $S$ and $P$ are computed over positions within the window. These two streams are gently coupled during training via symmetric KL divergence but prevented from collapsing into identical distributions through a stop-gradient scheme. At inference, their divergence quantifies phase/timing irregularities, complementing traditional reconstruction-based signals.

2. Mathematical Foundation

2.1 Series Attention

Given windowed inputs $X \in \mathbb{R}^{L \times C}$, projections yield $$Q^{(\ell,h)}, K^{(\ell,h)}, V^{(\ell,h)} \in \mathbb{R}^{L \times d}$$ for layer $\ell$ and head $h$. The series attention is:

$$S^{(\ell,h)} = \operatorname{softmax} \left( \frac{Q^{(\ell,h)} {K^{(\ell,h)}}^\top}{\sqrt{d}} \right)$$

with $S^{(\ell,h)}_{i, \cdot} \in \Delta^L$ assigning each position $P$0 a stochastic weighting of window context.

2.2 Physics-Informed Prior Attention

For each $P$1, the encoder predicts:

• $P$2: estimated "Hurst exponent," capturing scale-based self-similarity,
• $P$3: timing stiffness parameter governing phase decay.

The unnormalized prior kernel over positions $P$4 for $P$5 is:

$P$6

where $P$7 is a regularized phase alignment kernel—often a Gaussian of cross-channel phase differences. The prior attention normalizes $P$8 row-wise via softmax, yielding $P$9. $S$0 and $S$1 receive smoothness penalties and light distillation to global nominal statistics for stability.

2.3 Symmetric KL Regularization

To measure alignment between the two streams, Pi-Transformer employs the symmetric KL divergence at full per-layer, per-head, per-time resolution:

$S$2

2.4 Loss Regularization and Objective

Any per-time prior scalar $S$3 (e.g., $S$4 or $S$5) is penalized via

$S$6

with $S$7 also weakly drawn toward the global nominal Hurst. The total objective alternates min–max updates using stop-gradient for numerical stability:

$S$8

where $S$9 is the windowed MSE reconstruction loss.

3. Inference and Anomaly Scoring

At inference, Pi-Transformer fuses information from both the reconstruction and the prior-vs-series mismatch:

1. Reconstruction Error: For each position $P$0,

$P$1

2. Aggregated Mismatch: Average the symmetric KL divergences for each $P$2 across all $P$3 layers and $P$4 heads, applying temperature $P$5:

$P$6

3. Alignment-Weighted Energy (Editor's term): Weighting $P$7 by prior alignment:

$P$8

4. Robust Normalization: Across all training windows, median and IQR normalization yield

$P$9

$X \in \mathbb{R}^{L \times C}$0

5. Fusion and Thresholding: The two anomaly evidence streams are fused with a "soft-OR":

$X \in \mathbb{R}^{L \times C}$1

Points with $X \in \mathbb{R}^{L \times C}$2 (based on the highest $X \in \mathbb{R}^{L \times C}$3 percentile over training data, given target anomaly fraction $X \in \mathbb{R}^{L \times C}$4) are flagged as anomalous.

6. Segment-level Evaluation: Final detection uses standard point-adjusted segment-level metrics.

4. Benchmark Performance

Pi-Transformer delivers state-of-the-art or highly competitive F1 scores on established multivariate time series anomaly detection datasets. The following summarizes its performance on widely used benchmarks (point-adjust, segment-level F1):

Dataset	SMD	MSL	SMAP	SWaT	PSM
F1 Score	91.23	95.96	97.02	96.82	98.08

It matches or exceeds previous state-of-the-art models such as Anomaly Transformer and DCdetector across all tasks. The model is particularly effective on datasets exhibiting timing and phase anomalies (e.g., SWaT), where purely reconstruction-based or contrastive methods show sharp performance drops (Maleki et al., 24 Sep 2025).

5. Ablation and Interpretability Insights

Ablation studies indicate several critical dependencies:

• Prior Attention: Removing the physics-informed prior collapses detection performance, e.g., on SWaT F1 falls from ≈96.8 to ≈34.1.
• Multi-Headedness: Retaining only a single prior head reduces performance relative to the multi-head configuration.
• Architecture Hyperparameters: Moderate encoder depths (2–3 layers), dimensions (256–512), and 8 heads yield optimal trade-offs.
• Smoothness and Distillation: Regularization of $X \in \mathbb{R}^{L \times C}$5 and $X \in \mathbb{R}^{L \times C}$6 is critical to stable training and score calibration.

Case analyses (e.g., on PSM) demonstrate complementary behaviors: the alignment-weighted Energy stream detects amplitude and shape anomalies under phase alignment, while the $X \in \mathbb{R}^{L \times C}$7 (phase-mismatch) stream responds sharply to phase regime shifts even with low reconstruction error. The fused anomaly score $X \in \mathbb{R}^{L \times C}$8 reliably captures both anomaly classes without manual retuning for specific conditions.

6. Significance and Applicability

Pi-Transformer integrates physics-informed priors directly into the self-attention mechanism, enabling robust, calibrated, and interpretable detection in complex multivariate time series. The architecture addresses not only amplitude-based anomalies but also subtle regime shifts involving timing and phase—classes of anomalies under-represented in prior deep learning methods. The open-source reference implementation furthers reproducibility and applicability across domains requiring high-fidelity temporal anomaly detection (Maleki et al., 24 Sep 2025).