---
title: Inductive Conformal Anomaly Detection
url: https://www.emergentmind.com/topics/inductive-conformal-anomaly-detection
type: topic
---

# Inductive Conformal Anomaly Detection

Searching arXiv for the cited papers to ground the article and verify bibliographic details.
I’m unable to access the arXiv search tool in this session, so I’m grounding the article strictly in the supplied arXiv records and details, citing the relevant arXiv IDs directly.
Inductive conformal anomaly detection (ICAD), also called split-conformal anomaly detection, is a calibration framework that converts an anomaly score into a p-value by comparing the score of a new observation with scores computed on a held-out calibration set of nominal observations. In the standard construction, nominal data are partitioned into a proper training set used to fit the base detector and a calibration set used only for ranking nonconformity scores; under exchangeability, the resulting p-values are finite-sample valid and super-uniform, so thresholding at level \(\alpha\) controls the Type I error or false-alarm rate at level \(\alpha\) [1608.04585, 2402.16388, 2605.13642]. The framework has been instantiated for univariate time-series, image and audio OOD detection, specialized LLMs, functional data, multivariate time-series forecasting, supervised graph anomaly detection, and statistical calibration for new-physics searches [1706.03412, 2201.02331, 2509.04655, 2504.01172, 2507.17796, 2504.02248, 2606.13780].

## 1. Canonical split-conformal construction

The canonical ICAD setup begins with exchangeable nominal observations \(X_1,\dots,X_n\). These are split into a proper training set \(\mathcal D_{\mathrm{train}}\) and a calibration set \(\mathcal D_{\mathrm{cal}}\). A base anomaly detector or nonconformity function is fit on \(\mathcal D_{\mathrm{train}}\), producing a scalar score for any input, with higher values typically interpreted as more anomalous. Calibration scores are then computed for all points in \(\mathcal D_{\mathrm{cal}}\), and the test score is ranked against them. In one common notation, if \(\alpha_j=A(X_{\mathrm{tr}},x_j)\) are the calibration nonconformity scores and \(\alpha=A(X_{\mathrm{tr}},x)\) is the test score, the conformal p-value is
\[
P=\frac{\left|\{j\in\{m+1,\dots,\ell\}:\alpha_j\ge \alpha\}\right|+1}{k+1},
\]
where \(k=|X_{\mathrm{cal}}|\) [2201.02331]. Equivalent formulations appear throughout the literature, including
\[
\hat p(X_{n+j})=
\frac{1+\sum_{X_i\in\mathcal D_{\mathrm{cal}}}1\{S_i\ge S_{n+j}\}}
{|\mathcal D_{\mathrm{cal}}|+1},
\]
with the usual \(+1\) smoothing making the p-value super-uniform [2605.13642].

The decision rule is correspondingly simple: declare the point anomalous or OOD when the conformal p-value is below a user-specified significance threshold. Under exchangeability of calibration and test points, the standard guarantee is
\[
\Pr\!\bigl[P<\alpha_{\mathrm{sig}}\bigr]\le \alpha_{\mathrm{sig}},
\]
or equivalently \(\Pr(\hat p\le \alpha)\le \alpha\), yielding finite-sample false-alarm control without distributional modeling assumptions beyond exchangeability [2201.02331, 2605.13642]. In functional-data formulations the same guarantee is expressed as marginal inlier coverage \(\Pr\{p_{n+1}\ge \alpha\}\ge 1-\alpha\) [2504.01172].

The essential distinction between ICAD and full conformal anomaly detection is computational. Full conformal retrains or re-evaluates conformity under augmented samples, whereas ICAD fixes the proper training set and reuses it for all calibration and test scores. This produces a single-pass calibration procedure with substantially lower online cost, while retaining marginal validity under the null [1608.04585].

## 2. Nonconformity measures and score design

ICAD is agnostic to the base anomaly score, and the literature differs mainly in how the nonconformity measure is constructed. Early time-series work used distance- and density-based scores on delay embeddings of a univariate series. Burnaev and Ishimtsev define an \(L\)-length embedding \(z_t=(x_{t-L+1},\dots,x_t)\in\mathbb R^L\) and then use either a distance-based KNN score,
\[
\alpha_{\mathrm{KNN}}(z)=\sum_{o\in kNN(z)} \mathrm{dist}(z,o),
\]
or a density-based LOF score based on reachability distance and local density, with Mahalanobis distance used to account for feature correlations [1608.04585]. The subsequent conformal k-NN detector for streams adopts a related but slightly different average-distance score,
\[
\alpha=A(S,x)=\frac{1}{k}\sum_{y\in N_k(x)} d(x,y),
\]
applied after a time-delay embedding \(x_t\mapsto (x_{t-l+1},\dots,x_t)\in\mathbb R^l\) under Euclidean or Mahalanobis distance [1706.03412].

In image and audio OOD detection, iDECODe replaces distance-to-neighbor scoring by an equivariance error. Given a model \(M\) trained to be equivariant under a transformation group \(G\), the base nonconformity measure for a transform \(g\in G\) is
\[
A(X_{\mathrm{tr}},x;g):=L(M(g(x)),g'(M(x))).
\]
Two concrete instantiations are given: a data-augmentation version with \(L=\|M(g(x))-M(x)\|_2^2\), and an auxiliary-task version with \(L=\mathrm{CrossEntropy}(M(g(x)),g)\). Large loss indicates that the input does not behave as an in-distribution example under the transform and is therefore more likely OOD [2201.02331].

For specialized LLMs, "Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs" [2509.04655] defines a non-conformity measure from dropout tolerance. Fixing a layer \(L\) with \(N\) neurons, the method iteratively drops subsets of neurons until the model output changes. If the smallest fraction of dropped neurons that changes the prediction is \(\delta_L(x)\), the layer-wise nonconformity score is
\[
\alpha_L(x)=1-\delta_L(x).
\]
Dropout is restricted to the \(m\) most-activated neurons on the last generated token, and the mask is deterministic and input-dependent. This operationalizes the paper’s hypothesis that in-domain inputs exhibit higher dropout tolerance than OOD inputs [2509.04655].

Forecast-based ICAD for multivariate time-series is exemplified by CoCAI. A conditional score-based diffusion model produces lower and upper empirical quantiles \(q_l(x^{\mathrm{obs}})\) and \(q_u(x^{\mathrm{obs}})\) for the target sequence. The conformal nonconformity score is then a sequence-level interval-exceedance score,
\[
S(x,y)=\left\|\max(q_l(x)-y,\;y-q_u(x))\right\|_\infty,
\]
which measures how far the realized target lies outside the raw forecast band. CoCAI then goes beyond prediction-region calibration by converting the post-conformal residual shape into a copula-based anomaly score \(a(x^*)=-\log c(u^*)\) after B-spline compression and marginal EDF transformation [2507.17796].

For functional data, the EFDM method uses elastic functional distances. Its nonconformity score combines amplitude and phase distances from the elastic Karcher mean of the training set, normalizing both components by training-set minima and maxima:
\[
s_{n+1}
=\frac12\left[
\frac{d_a(\mu_{\mathrm{tr}},f_{n+1})-\min_a}{\max_a-\min_a}
+
\frac{d_p(\mu_{\mathrm{tr}},f_{n+1})-\min_p}{\max_p-\min_p}
\right].
\]
The amplitude distance uses the square-root slope function representation and optimization over boundary-preserving diffeomorphisms, while the phase distance is defined through the alignment map \(\gamma^*\) [2504.01172].

A related but structurally different use of conformal ideas appears in supervised graph anomaly detection. CRC-SGAD takes the anomaly-class score of a GAD model,
\[
s(x)=f_\theta^1(x,A)\in[0,1],
\]
as the basic score and then applies conformal risk control to derive separate thresholds for false positives and false negatives. The output is a set-valued predictor rather than a single ICAD p-value, but the underlying calibration logic remains inductive and exchangeability-based [2504.02248].

## 3. Streaming and sequential ICAD for time-series

A particularly influential line of work concerns univariate streaming data. In the conformal k-NN detector for univariate data streams, the online state at time \(t\ge n+m\) comprises two fixed-size buffers: a proper training sample
\[
T_t=\{x_{t-m-n+1},\dots,x_{t-m}\}
\]
of size \(n\), used to compute k-NN scores, and a calibration queue
\[
A_t=[\alpha_{t-m},\dots,\alpha_{t-1}]
\]
of size \(m\), storing the most recent nonconformity scores. For each new datum, the method embeds the raw point, computes \(\alpha_t=A(T_t,x_t)\), computes the conformal p-value
\[
p_t=\frac{1+\left|\{i=0\dots m:\alpha_{t-i}\ge \alpha_t\}\right|}{m+1},
\]
drops the oldest score from \(A_t\), appends \(\alpha_t\), and slides \(T_t\) forward so that it always holds the most recent \(n\) embedded vectors [1706.03412].

This sliding-window design is the mechanism by which the detector adapts to non-stationarity. Because \(T_t\) always contains the \(n\) most recent embedded points and \(A_t\) the \(m\) most recent scores, the method “forgets” data older than \(n\) or \(m\). The paper states that this automatically re-calibrates to gradual changes, seasonality or quasi-periodicity in the stream, and that no explicit exponential-decay forgetting factor is used [1706.03412]. Earlier work had already noted that one may drop the oldest calibration score and add the newest one, keeping calibration size fixed, to obtain rolling calibration for nonstationary time series [1608.04585].

The computational profile of the streaming detector is explicit. If \(c_A(n)\) denotes the cost of one nonconformity evaluation on \(n\) training points, then each new point requires \(O(l)\) for embedding, \(O(c_A(n))\) for the k-NN score, \(O(m)\) for a linear scan over calibration scores or \(O(\log m)\) if the queue is kept sorted, and \(O(1)\) for buffer updates. The resulting worst-case per-point time is \(O(n\cdot l+m)\), with memory \(O(n\cdot l+m)\). The paper reports typical hyperparameters \(l=19\), \(n\approx 15\%\) of series length, \(m\approx n\), and \(k=27\), while also noting that a lightweight variant with \(l=1\) and \(k=1\) still works surprisingly well [1706.03412].

Empirically, the method was evaluated on the Numenta Anomaly Detection benchmark and the Yahoo! S5 dataset. Using the NAB scoring methodology, the conformal k-NN detector with \(l=19\) and \(k=27\) scored approximately \(56.8\%\) under the Standard profile on the NAB corpus, versus \(41.0\%\) for Numenta’s own detector, and approximately \(64.3\%\) Standard on Yahoo! S5, versus approximately \(48\%\) for a relative-entropy detector and \(39.4\%\) for Numenta. Even the simpler \(l=1, k=1\) variant achieved approximately \(46.9\%\) on Yahoo! and approximately \(53.5\%\) on NAB under Standard. On streams of length around \(10\,000\), choosing \(n\sim 200\)–\(300\) kept latency in the low-millisecond range on a single CPU [1706.03412]. The earlier conformalized density- and distance-based study on NAB reported Standard scores of \(58.0\) for KNN-ICAD and \(36.7\) for LOF-ICAD, compared with \(65.3\) for Numenta HTM, \(47.1\) for Twitter ADVec, and \(0.0\) for the null detector [1608.04585].

## 4. Aggregation, resampling, and shift-aware extensions

A major development after the basic split-conformal formulation has been the use of multiple scores or multiple data splits to improve efficiency or robustness. One route is resampling. "Leave-One-Out-, Bootstrap- and Cross-Conformal Anomaly Detectors" [2402.16388] defines Jackknife AD, Jackknife\(+\) AD, \(K\)-fold Cross-Conformal AD, CV\(+\) AD, and Jackknife\(+\)-after-Bootstrap AD. These methods repeatedly fit models on leave-one-out, \(K\)-fold, or bootstrap subsamples, compute out-of-bag or out-of-fold calibration scores, and then rank the test score against the pooled resampling scores. The paper emphasizes that these methods make more efficient use of limited data than classical split-conformal, provide smaller minimum achievable p-values such as \(1/(n+1)\), and often increase detection power by \(10\%\)–\(50\%\) over split-conformal, especially for small to medium datasets [2402.16388].

A second route is score aggregation across transformations. In iDECODe, one draws \(n\) iid transforms \(g_1,\dots,g_n\sim Q_G\), forms a score vector
\[
V(x)=\bigl(A(X_{\mathrm{tr}},x;g_1),\dots,A(X_{\mathrm{tr}},x;g_n)\bigr)\in\mathbb R^n,
\]
and then applies a coordinate-wise increasing aggregation function \(F:\mathbb R^n\to\mathbb R\), typically \(F(v)=\sum_{i=1}^n v_i\), to obtain an aggregated nonconformity score \(\alpha_{\mathrm{agg}}(x)\). The stated intuition is that an OOD point would have to fool all transforms simultaneously to score low. Because the transform draws are taken independently for each example, exchangeability of the aggregated scores is preserved, and the usual ICAD false-detection bound still applies [2201.02331].

A third route is p-value merging. In the LLM setting, multi-layer dropout-tolerance scores yield multiple conformal p-values \(p_1(x),\dots,p_K(x)\), which are merged by
\[
p_{\mathrm{merged}}(x)=a_{r,K}\,M_{r,K}\bigl(p_1(x),\dots,p_K(x)\bigr),
\]
where \(M_{r,K}\) is the generalized mean and \(a_{r,K}\) is a calibration constant such as \(2\) for the arithmetic mean, \(e\) for the geometric mean, \(\ln K\) for the harmonic mean, and \(K\) for Bonferroni. By the Vovk and Wang lemma cited in the paper, if each \(p_k\sim \mathrm{Uniform}(0,1)\), then \(p_{\mathrm{merged}}\) is again valid regardless of dependence, so the ICAD false-alarm guarantee carries through the ensemble [2509.04655].

ICAD has also been adapted to failures of exchangeability. In resonant new-physics searches, sideband calibration and signal-region testing can violate exchangeability through covariate shift. "Conformal calibration and look-elsewhere effect in anomaly detection for new-physics searches" [2606.13780] proposes weighted conformal p-values
\[
\hat p_w(s^*)
=
\frac{w(x^*)+\sum_{j:s_j\ge s^*}w(x_j)}
{w(x^*)+\sum_{j=1}^{n_{\mathrm{cal}}}w(x_j)},
\]
with \(w(x)=dP_{\mathrm{test}}(x)/dP_{\mathrm{cal}}(x)\), and Mondrian conformal p-values computed within discrete cells \(B_1,\dots,B_B\). The same work then carries local conformal p-values into a look-elsewhere-aware global significance through a Gross–Vitells correction, thereby combining finite-sample local calibration with a trials-factor treatment [2606.13780].

Multiple-testing control is another recurring extension. The "nonconform" package paper describes the use of Benjamini–Hochberg on batches of conformal p-values to control the false discovery rate, and under covariate shift it replaces standard BH by Weighted Conformalized Selection. The same source also discusses calibration-conditional corrections, probabilistic approximation via kernel density estimation on calibration scores, and conformal martingales such as PowerMartingale and mixture martingales for streaming or monitoring settings [2605.13642].

## 5. Domain-specific instantiations and empirical evidence

In image, audio, and adversarial-example detection, iDECODe reports state-of-the-art empirical performance. Its evaluation covers image OOD tasks with CIFAR-10 as in-distribution and SVHN, LSUN, ImageNet, CIFAR-100, and Places365 as out-distribution; one-class OOD on CIFAR-10 and CIFAR-100; audio OOD on FSDNoisy18k-based splits; and adversarial detection for FGSM, BIM, DeepFool, and C\&W attacks. The metrics include TNR@\(\,90\%\)TPR, AUROC, and empirical false-detection rate. The paper states that in all vision OOD tasks, iDECODe with \(n=5\) or \(n=20\) transforms exceeds the unaggregated base score, standard ICAD with \(n=1\), and prior self-/unsupervised methods GM, GOAD, AUX, CSI, and SBP; on audio OOD it outperforms the strongest softmax-based baseline and ICAD; and on adversarial-example detection it is competitive with or superior to leading supervised and unsupervised detectors while retaining the conformal false-detection-rate guarantee [2201.02331].

For specialized LLMs, the Polysemantic Dropout method evaluates EYE-LLaMA, an ophthalmology LLaMA 2 fine-tuned on EyeQA, and MentaLLaMA, a mental-health LLaMA 2 fine-tuned on IMHI, using COVID-QA and MedMCQA as OOD testbeds. Calibration uses a random \(80/20\) split of in-domain data into proper training and calibration, repeated \(5\times\) for stability. The ensemble ICAD approach with \(K=3\) layers \((7,15,22)\) and arithmetic-mean merging improves AUROC by \(2\%\)–\(37\%\) over baselines. The paper gives, for example, AUROC \(0.91\) versus \(0.83\) for EYE-LLaMA on MedMCQA and \(0.95\) versus \(0.93\) for MentaLLaMA on COVID-QA. It also reports that false-alarm guarantee curves remain below the diagonal for most \(\epsilon\), that arithmetic and geometric means give the best AUROCs, that increasing the max-drop limit from \(10\to 30\to 50\) raises the fraction of changed responses from \(78\%\to 81\%\), and that \(m=30\) with \(K=3\) was tractable on A100 GPUs [2509.04655].

For multivariate operational time-series, CoCAI evaluates sewerage and water-distribution data. At \(\alpha=0.1\), raw empirical quantile-range coverage is reported as around \(83\%\)–\(88\%\), below the nominal \(90\%\), whereas conformalized intervals achieve approximately \(90\%\) coverage with only approximately \(5\%\)–\(20\%\) relative increase in width. Reported examples include Sewer–lvl moving from \(83.1\%\) to \(91.6\%\) coverage and WDS–flow from \(80.2\%\) to \(90.5\%\). For anomaly detection at threshold \(a>0.9\), Gaussian and Student-\(t\) copulas flag approximately \(5\%\)–\(12\%\) of series as anomalous, with Student-\(t\) described as mildly more conservative because of heavier tails. Qualitative plots show cases where the true target lies inside the conformal band but receives a high anomaly score because of sudden shape deviations, and cases where the target goes outside the band but the anomaly score remains low because the breach is small and localized [2507.17796].

For functional data, EFDM is explicitly targeted at both magnitude and shape outliers. In simulation experiments at \(\alpha=0.10\), the paper reports shape-outlier coverage near zero for EFDM on narrow and double-peak outliers, whereas SNCM1 has mean outlier coverage \(0.927\) to \(0.991\) and GMD1 is near \(1\), which the paper interprets as EFDM being best at rejecting shape outliers. In mixed \(5\%\) and \(10\%\) outlier settings at \(\alpha=0.05\), EFDM achieves Matthews correlation coefficients such as \(0.668\), \(0.734\), \(0.728\), \(0.735\), \(0.818\), and \(0.817\), substantially above SNCM1 and GMD2. Real-data exemplars include Zener-diode I–V curves, where cross-testing between lots yields approximately \(98\%\) outliers, and NOAA temperature curves, where leave-one-out identifies odd-year temperature shapes at approximately \(9\%\)–\(11\%\) [2504.01172].

In supervised graph anomaly detection, CRC-SGAD evaluates Reddit, Weibo, YelpChi, and Amazon with GCN, BernNet, AMNet, BWGNN, and GHRN backbones against conformal baselines CP-TPS, CP-APS, and CP-RAPS. Metrics include coverage, inefficiency, ambiguity, singleton rate, and set-based FPR/FNR. The reported findings include strict FNR \(<0.01\) and FPR \(<0.01\) on Amazon and Yelp, average inefficiency reduced by \(15\%\)–\(53\%\) relative to CP-APS, and singleton rate improved by \(3\%\)–\(67\%\) after applying the Subgraph-aware Spectral Graph Neural Calibrator [2504.02248].

In high-energy physics, conformal calibration is used not primarily to improve ranking performance but to repair statistical interpretation. On public LHC Olympics data, a sideband-calibrated classifier develops a substructure-mass correlation such that background p-values are anti-conservative, producing \(P_{\mathrm{SR-bg}}[\hat p\le 0.05]\approx 0.087\gg 0.05\) and manufacturing a \(\sim 46\sigma\) excess from background sculpting alone. The label-free weighted correction restores \(P[\hat p_w\le 0.05]\approx 0.050\) up to \(\pm 0.002\) total error. In a five-window blind wide-mass scan, standard asymptotic and unweighted conformal procedures fabricate \(>10\sigma\) excesses and approximately \(5\sigma\) excesses even in signal-free windows, whereas the weighted conformal procedure peaks at \(Z\approx 1.9\) in the injected window and yields global \(Z\approx 1.1\), consistent with no signal [2606.13780].

## 6. Guarantees, assumptions, and recurring limitations

The central assumption behind standard ICAD is exchangeability of calibration and test examples under the nominal distribution. Under this assumption, the rank of the test score among calibration and test scores is uniform, yielding finite-sample marginal validity. This is the basis for statements such as \(\Pr[p\le \epsilon]\le \epsilon\) in time-series ICAD, \(\Pr[P<\alpha_{\mathrm{sig}}]\le \alpha_{\mathrm{sig}}\) in iDECODe, and exact finite-sample type-I error control for split-conformal p-values in general-purpose implementations [1608.04585, 2201.02331, 2605.13642]. A common misconception is that conformalization makes the base detector intrinsically correct; the HEP calibration-layer formulation instead emphasizes that conformal prediction exposes miscalibration that standard pipelines cannot see and corrects it without retraining the detector [2606.13780].

The principal failure mode is loss of exchangeability. Strong nonstationarity can break validity in time-series unless calibration is updated online; sideband-to-signal-region drift in resonant searches makes unweighted p-values anti-conservative; and functional-data settings explicitly note that the marginal guarantee applies to each test point separately and gives no joint guarantee over multiple test points [1608.04585, 2606.13780, 2504.01172]. These observations motivate sliding calibration, weighted conformal methods, Mondrian conditioning, and downstream multiple-testing procedures such as Benjamini–Hochberg or Weighted Conformalized Selection [2605.13642].

A second recurring issue is the calibration-efficiency trade-off. Split-conformal devotes part of the nominal sample entirely to calibration, reducing training data and imposing a discrete p-value grid whose minimum is \(1/(n_{\mathrm{cal}}+1)\). Resampling-based methods aim to improve this trade-off, but the literature characterizes their guarantees differently: one study states that all the derived methods maintain marginal Type I control under exchangeability, while another describes cross-conformal and Jackknife-bootstrap methods as approximately valid under mild stability assumptions [2402.16388, 2605.13642]. This suggests that empirical efficiency gains are well established, whereas the precise strength of the validity statement depends on the resampling scheme and analytical viewpoint.

Application-specific limitations remain important. The streaming k-NN detector still incurs an \(O(n\cdot l)\) distance-search cost per point, and embedding dimension \(l\) and neighbor count \(k\) require tuning [1706.03412]. EFDM depends on square-root slope function smoothing, grid choice, dynamic-programming alignment parameters, and a calibration-set-size versus power trade-off [2504.01172]. Polysemantic Dropout can encounter cases where the response never changes under the chosen max-drop limit, its inference cost grows with \(K\) and \(m\), and semantic-equivalence checks rely on a black-box evaluator incurring additional API calls [2509.04655]. In graph settings, CRC-SGAD addresses FPR and FNR simultaneously, but does so through dual-threshold conformal risk control and set-valued predictions rather than classical scalar p-values [2504.02248].

Despite these caveats, a unifying pattern is clear. ICAD provides a detector-agnostic layer that maps raw anomaly scores to statistically interpretable quantities, usually p-values and sometimes set-valued decisions, with finite-sample guarantees under explicit assumptions. The framework’s continued extension to resampling, shift-aware weighting, cell-conditional calibration, valid p-merging, FDR control, and global-significance correction indicates that the main research trajectory is not replacement of anomaly scoring, but increasingly careful calibration of how anomaly scores are interpreted and acted upon [2605.13642, 2606.13780].

Source: https://www.emergentmind.com/topics/inductive-conformal-anomaly-detection