Inductive Conformal Anomaly Detection
- Inductive conformal anomaly detection is a calibration framework that transforms anomaly scores into p-values with finite-sample validity under exchangeability.
- It partitions data into proper training and calibration sets, enabling efficient and statistically controlled detection across time-series, images, audio, and specialized LLMs.
- Recent advancements include resampling, aggregation, and weighted methods to boost robustness, efficiency, and adaptability to nonstationarity and covariate shifts.
Searching arXiv for the cited papers to ground the article and verify bibliographic details. I’m unable to access the arXiv search tool in this session, so I’m grounding the article strictly in the supplied arXiv records and details, citing the relevant arXiv IDs directly. Inductive conformal anomaly detection (ICAD), also called split-conformal anomaly detection, is a calibration framework that converts an anomaly score into a p-value by comparing the score of a new observation with scores computed on a held-out calibration set of nominal observations. In the standard construction, nominal data are partitioned into a proper training set used to fit the base detector and a calibration set used only for ranking nonconformity scores; under exchangeability, the resulting p-values are finite-sample valid and super-uniform, so thresholding at level controls the Type I error or false-alarm rate at level (Burnaev et al., 2016, Hennhöfer et al., 2024, Hennhöfer et al., 13 May 2026). The framework has been instantiated for univariate time-series, image and audio OOD detection, specialized LLMs, functional data, multivariate time-series forecasting, supervised graph anomaly detection, and statistical calibration for new-physics searches (Ishimtsev et al., 2017, Kaur et al., 2022, Gupta et al., 4 Sep 2025, Adams et al., 1 Apr 2025, Pearson et al., 23 Jul 2025, Bai et al., 3 Apr 2025, Araz et al., 11 Jun 2026).
1. Canonical split-conformal construction
The canonical ICAD setup begins with exchangeable nominal observations . These are split into a proper training set and a calibration set . A base anomaly detector or nonconformity function is fit on , producing a scalar score for any input, with higher values typically interpreted as more anomalous. Calibration scores are then computed for all points in , and the test score is ranked against them. In one common notation, if are the calibration nonconformity scores and is the test score, the conformal p-value is
where 0 (Kaur et al., 2022). Equivalent formulations appear throughout the literature, including
1
with the usual 2 smoothing making the p-value super-uniform (Hennhöfer et al., 13 May 2026).
The decision rule is correspondingly simple: declare the point anomalous or OOD when the conformal p-value is below a user-specified significance threshold. Under exchangeability of calibration and test points, the standard guarantee is
3
or equivalently 4, yielding finite-sample false-alarm control without distributional modeling assumptions beyond exchangeability (Kaur et al., 2022, Hennhöfer et al., 13 May 2026). In functional-data formulations the same guarantee is expressed as marginal inlier coverage 5 (Adams et al., 1 Apr 2025).
The essential distinction between ICAD and full conformal anomaly detection is computational. Full conformal retrains or re-evaluates conformity under augmented samples, whereas ICAD fixes the proper training set and reuses it for all calibration and test scores. This produces a single-pass calibration procedure with substantially lower online cost, while retaining marginal validity under the null (Burnaev et al., 2016).
2. Nonconformity measures and score design
ICAD is agnostic to the base anomaly score, and the literature differs mainly in how the nonconformity measure is constructed. Early time-series work used distance- and density-based scores on delay embeddings of a univariate series. Burnaev and Ishimtsev define an 6-length embedding 7 and then use either a distance-based KNN score,
8
or a density-based LOF score based on reachability distance and local density, with Mahalanobis distance used to account for feature correlations (Burnaev et al., 2016). The subsequent conformal k-NN detector for streams adopts a related but slightly different average-distance score,
9
applied after a time-delay embedding 0 under Euclidean or Mahalanobis distance (Ishimtsev et al., 2017).
In image and audio OOD detection, iDECODe replaces distance-to-neighbor scoring by an equivariance error. Given a model 1 trained to be equivariant under a transformation group 2, the base nonconformity measure for a transform 3 is
4
Two concrete instantiations are given: a data-augmentation version with 5, and an auxiliary-task version with 6. Large loss indicates that the input does not behave as an in-distribution example under the transform and is therefore more likely OOD (Kaur et al., 2022).
For specialized LLMs, "Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs" (Gupta et al., 4 Sep 2025) defines a non-conformity measure from dropout tolerance. Fixing a layer 7 with 8 neurons, the method iteratively drops subsets of neurons until the model output changes. If the smallest fraction of dropped neurons that changes the prediction is 9, the layer-wise nonconformity score is
0
Dropout is restricted to the 1 most-activated neurons on the last generated token, and the mask is deterministic and input-dependent. This operationalizes the paper’s hypothesis that in-domain inputs exhibit higher dropout tolerance than OOD inputs (Gupta et al., 4 Sep 2025).
Forecast-based ICAD for multivariate time-series is exemplified by CoCAI. A conditional score-based diffusion model produces lower and upper empirical quantiles 2 and 3 for the target sequence. The conformal nonconformity score is then a sequence-level interval-exceedance score,
4
which measures how far the realized target lies outside the raw forecast band. CoCAI then goes beyond prediction-region calibration by converting the post-conformal residual shape into a copula-based anomaly score 5 after B-spline compression and marginal EDF transformation (Pearson et al., 23 Jul 2025).
For functional data, the EFDM method uses elastic functional distances. Its nonconformity score combines amplitude and phase distances from the elastic Karcher mean of the training set, normalizing both components by training-set minima and maxima: 6 The amplitude distance uses the square-root slope function representation and optimization over boundary-preserving diffeomorphisms, while the phase distance is defined through the alignment map 7 (Adams et al., 1 Apr 2025).
A related but structurally different use of conformal ideas appears in supervised graph anomaly detection. CRC-SGAD takes the anomaly-class score of a GAD model,
8
as the basic score and then applies conformal risk control to derive separate thresholds for false positives and false negatives. The output is a set-valued predictor rather than a single ICAD p-value, but the underlying calibration logic remains inductive and exchangeability-based (Bai et al., 3 Apr 2025).
3. Streaming and sequential ICAD for time-series
A particularly influential line of work concerns univariate streaming data. In the conformal k-NN detector for univariate data streams, the online state at time 9 comprises two fixed-size buffers: a proper training sample
0
of size 1, used to compute k-NN scores, and a calibration queue
2
of size 3, storing the most recent nonconformity scores. For each new datum, the method embeds the raw point, computes 4, computes the conformal p-value
5
drops the oldest score from 6, appends 7, and slides 8 forward so that it always holds the most recent 9 embedded vectors (Ishimtsev et al., 2017).
This sliding-window design is the mechanism by which the detector adapts to non-stationarity. Because 0 always contains the 1 most recent embedded points and 2 the 3 most recent scores, the method “forgets” data older than 4 or 5. The paper states that this automatically re-calibrates to gradual changes, seasonality or quasi-periodicity in the stream, and that no explicit exponential-decay forgetting factor is used (Ishimtsev et al., 2017). Earlier work had already noted that one may drop the oldest calibration score and add the newest one, keeping calibration size fixed, to obtain rolling calibration for nonstationary time series (Burnaev et al., 2016).
The computational profile of the streaming detector is explicit. If 6 denotes the cost of one nonconformity evaluation on 7 training points, then each new point requires 8 for embedding, 9 for the k-NN score, 0 for a linear scan over calibration scores or 1 if the queue is kept sorted, and 2 for buffer updates. The resulting worst-case per-point time is 3, with memory 4. The paper reports typical hyperparameters 5, 6 of series length, 7, and 8, while also noting that a lightweight variant with 9 and 0 still works surprisingly well (Ishimtsev et al., 2017).
Empirically, the method was evaluated on the Numenta Anomaly Detection benchmark and the Yahoo! S5 dataset. Using the NAB scoring methodology, the conformal k-NN detector with 1 and 2 scored approximately 3 under the Standard profile on the NAB corpus, versus 4 for Numenta’s own detector, and approximately 5 Standard on Yahoo! S5, versus approximately 6 for a relative-entropy detector and 7 for Numenta. Even the simpler 8 variant achieved approximately 9 on Yahoo! and approximately 0 on NAB under Standard. On streams of length around 1, choosing 2–3 kept latency in the low-millisecond range on a single CPU (Ishimtsev et al., 2017). The earlier conformalized density- and distance-based study on NAB reported Standard scores of 4 for KNN-ICAD and 5 for LOF-ICAD, compared with 6 for Numenta HTM, 7 for Twitter ADVec, and 8 for the null detector (Burnaev et al., 2016).
4. Aggregation, resampling, and shift-aware extensions
A major development after the basic split-conformal formulation has been the use of multiple scores or multiple data splits to improve efficiency or robustness. One route is resampling. "Leave-One-Out-, Bootstrap- and Cross-Conformal Anomaly Detectors" (Hennhöfer et al., 2024) defines Jackknife AD, Jackknife9 AD, 0-fold Cross-Conformal AD, CV1 AD, and Jackknife2-after-Bootstrap AD. These methods repeatedly fit models on leave-one-out, 3-fold, or bootstrap subsamples, compute out-of-bag or out-of-fold calibration scores, and then rank the test score against the pooled resampling scores. The paper emphasizes that these methods make more efficient use of limited data than classical split-conformal, provide smaller minimum achievable p-values such as 4, and often increase detection power by 5–6 over split-conformal, especially for small to medium datasets (Hennhöfer et al., 2024).
A second route is score aggregation across transformations. In iDECODe, one draws 7 iid transforms 8, forms a score vector
9
and then applies a coordinate-wise increasing aggregation function 00, typically 01, to obtain an aggregated nonconformity score 02. The stated intuition is that an OOD point would have to fool all transforms simultaneously to score low. Because the transform draws are taken independently for each example, exchangeability of the aggregated scores is preserved, and the usual ICAD false-detection bound still applies (Kaur et al., 2022).
A third route is p-value merging. In the LLM setting, multi-layer dropout-tolerance scores yield multiple conformal p-values 03, which are merged by
04
where 05 is the generalized mean and 06 is a calibration constant such as 07 for the arithmetic mean, 08 for the geometric mean, 09 for the harmonic mean, and 10 for Bonferroni. By the Vovk and Wang lemma cited in the paper, if each 11, then 12 is again valid regardless of dependence, so the ICAD false-alarm guarantee carries through the ensemble (Gupta et al., 4 Sep 2025).
ICAD has also been adapted to failures of exchangeability. In resonant new-physics searches, sideband calibration and signal-region testing can violate exchangeability through covariate shift. "Conformal calibration and look-elsewhere effect in anomaly detection for new-physics searches" (Araz et al., 11 Jun 2026) proposes weighted conformal p-values
13
with 14, and Mondrian conformal p-values computed within discrete cells 15. The same work then carries local conformal p-values into a look-elsewhere-aware global significance through a Gross–Vitells correction, thereby combining finite-sample local calibration with a trials-factor treatment (Araz et al., 11 Jun 2026).
Multiple-testing control is another recurring extension. The "nonconform" package paper describes the use of Benjamini–Hochberg on batches of conformal p-values to control the false discovery rate, and under covariate shift it replaces standard BH by Weighted Conformalized Selection. The same source also discusses calibration-conditional corrections, probabilistic approximation via kernel density estimation on calibration scores, and conformal martingales such as PowerMartingale and mixture martingales for streaming or monitoring settings (Hennhöfer et al., 13 May 2026).
5. Domain-specific instantiations and empirical evidence
In image, audio, and adversarial-example detection, iDECODe reports state-of-the-art empirical performance. Its evaluation covers image OOD tasks with CIFAR-10 as in-distribution and SVHN, LSUN, ImageNet, CIFAR-100, and Places365 as out-distribution; one-class OOD on CIFAR-10 and CIFAR-100; audio OOD on FSDNoisy18k-based splits; and adversarial detection for FGSM, BIM, DeepFool, and C&W attacks. The metrics include TNR@16TPR, AUROC, and empirical false-detection rate. The paper states that in all vision OOD tasks, iDECODe with 17 or 18 transforms exceeds the unaggregated base score, standard ICAD with 19, and prior self-/unsupervised methods GM, GOAD, AUX, CSI, and SBP; on audio OOD it outperforms the strongest softmax-based baseline and ICAD; and on adversarial-example detection it is competitive with or superior to leading supervised and unsupervised detectors while retaining the conformal false-detection-rate guarantee (Kaur et al., 2022).
For specialized LLMs, the Polysemantic Dropout method evaluates EYE-LLaMA, an ophthalmology LLaMA 2 fine-tuned on EyeQA, and MentaLLaMA, a mental-health LLaMA 2 fine-tuned on IMHI, using COVID-QA and MedMCQA as OOD testbeds. Calibration uses a random 20 split of in-domain data into proper training and calibration, repeated 21 for stability. The ensemble ICAD approach with 22 layers 23 and arithmetic-mean merging improves AUROC by 24–25 over baselines. The paper gives, for example, AUROC 26 versus 27 for EYE-LLaMA on MedMCQA and 28 versus 29 for MentaLLaMA on COVID-QA. It also reports that false-alarm guarantee curves remain below the diagonal for most 30, that arithmetic and geometric means give the best AUROCs, that increasing the max-drop limit from 31 raises the fraction of changed responses from 32, and that 33 with 34 was tractable on A100 GPUs (Gupta et al., 4 Sep 2025).
For multivariate operational time-series, CoCAI evaluates sewerage and water-distribution data. At 35, raw empirical quantile-range coverage is reported as around 36–37, below the nominal 38, whereas conformalized intervals achieve approximately 39 coverage with only approximately 40–41 relative increase in width. Reported examples include Sewer–lvl moving from 42 to 43 coverage and WDS–flow from 44 to 45. For anomaly detection at threshold 46, Gaussian and Student-47 copulas flag approximately 48–49 of series as anomalous, with Student-50 described as mildly more conservative because of heavier tails. Qualitative plots show cases where the true target lies inside the conformal band but receives a high anomaly score because of sudden shape deviations, and cases where the target goes outside the band but the anomaly score remains low because the breach is small and localized (Pearson et al., 23 Jul 2025).
For functional data, EFDM is explicitly targeted at both magnitude and shape outliers. In simulation experiments at 51, the paper reports shape-outlier coverage near zero for EFDM on narrow and double-peak outliers, whereas SNCM1 has mean outlier coverage 52 to 53 and GMD1 is near 54, which the paper interprets as EFDM being best at rejecting shape outliers. In mixed 55 and 56 outlier settings at 57, EFDM achieves Matthews correlation coefficients such as 58, 59, 60, 61, 62, and 63, substantially above SNCM1 and GMD2. Real-data exemplars include Zener-diode I–V curves, where cross-testing between lots yields approximately 64 outliers, and NOAA temperature curves, where leave-one-out identifies odd-year temperature shapes at approximately 65–66 (Adams et al., 1 Apr 2025).
In supervised graph anomaly detection, CRC-SGAD evaluates Reddit, Weibo, YelpChi, and Amazon with GCN, BernNet, AMNet, BWGNN, and GHRN backbones against conformal baselines CP-TPS, CP-APS, and CP-RAPS. Metrics include coverage, inefficiency, ambiguity, singleton rate, and set-based FPR/FNR. The reported findings include strict FNR 67 and FPR 68 on Amazon and Yelp, average inefficiency reduced by 69–70 relative to CP-APS, and singleton rate improved by 71–72 after applying the Subgraph-aware Spectral Graph Neural Calibrator (Bai et al., 3 Apr 2025).
In high-energy physics, conformal calibration is used not primarily to improve ranking performance but to repair statistical interpretation. On public LHC Olympics data, a sideband-calibrated classifier develops a substructure-mass correlation such that background p-values are anti-conservative, producing 73 and manufacturing a 74 excess from background sculpting alone. The label-free weighted correction restores 75 up to 76 total error. In a five-window blind wide-mass scan, standard asymptotic and unweighted conformal procedures fabricate 77 excesses and approximately 78 excesses even in signal-free windows, whereas the weighted conformal procedure peaks at 79 in the injected window and yields global 80, consistent with no signal (Araz et al., 11 Jun 2026).
6. Guarantees, assumptions, and recurring limitations
The central assumption behind standard ICAD is exchangeability of calibration and test examples under the nominal distribution. Under this assumption, the rank of the test score among calibration and test scores is uniform, yielding finite-sample marginal validity. This is the basis for statements such as 81 in time-series ICAD, 82 in iDECODe, and exact finite-sample type-I error control for split-conformal p-values in general-purpose implementations (Burnaev et al., 2016, Kaur et al., 2022, Hennhöfer et al., 13 May 2026). A common misconception is that conformalization makes the base detector intrinsically correct; the HEP calibration-layer formulation instead emphasizes that conformal prediction exposes miscalibration that standard pipelines cannot see and corrects it without retraining the detector (Araz et al., 11 Jun 2026).
The principal failure mode is loss of exchangeability. Strong nonstationarity can break validity in time-series unless calibration is updated online; sideband-to-signal-region drift in resonant searches makes unweighted p-values anti-conservative; and functional-data settings explicitly note that the marginal guarantee applies to each test point separately and gives no joint guarantee over multiple test points (Burnaev et al., 2016, Araz et al., 11 Jun 2026, Adams et al., 1 Apr 2025). These observations motivate sliding calibration, weighted conformal methods, Mondrian conditioning, and downstream multiple-testing procedures such as Benjamini–Hochberg or Weighted Conformalized Selection (Hennhöfer et al., 13 May 2026).
A second recurring issue is the calibration-efficiency trade-off. Split-conformal devotes part of the nominal sample entirely to calibration, reducing training data and imposing a discrete p-value grid whose minimum is 83. Resampling-based methods aim to improve this trade-off, but the literature characterizes their guarantees differently: one study states that all the derived methods maintain marginal Type I control under exchangeability, while another describes cross-conformal and Jackknife-bootstrap methods as approximately valid under mild stability assumptions (Hennhöfer et al., 2024, Hennhöfer et al., 13 May 2026). This suggests that empirical efficiency gains are well established, whereas the precise strength of the validity statement depends on the resampling scheme and analytical viewpoint.
Application-specific limitations remain important. The streaming k-NN detector still incurs an 84 distance-search cost per point, and embedding dimension 85 and neighbor count 86 require tuning (Ishimtsev et al., 2017). EFDM depends on square-root slope function smoothing, grid choice, dynamic-programming alignment parameters, and a calibration-set-size versus power trade-off (Adams et al., 1 Apr 2025). Polysemantic Dropout can encounter cases where the response never changes under the chosen max-drop limit, its inference cost grows with 87 and 88, and semantic-equivalence checks rely on a black-box evaluator incurring additional API calls (Gupta et al., 4 Sep 2025). In graph settings, CRC-SGAD addresses FPR and FNR simultaneously, but does so through dual-threshold conformal risk control and set-valued predictions rather than classical scalar p-values (Bai et al., 3 Apr 2025).
Despite these caveats, a unifying pattern is clear. ICAD provides a detector-agnostic layer that maps raw anomaly scores to statistically interpretable quantities, usually p-values and sometimes set-valued decisions, with finite-sample guarantees under explicit assumptions. The framework’s continued extension to resampling, shift-aware weighting, cell-conditional calibration, valid p-merging, FDR control, and global-significance correction indicates that the main research trajectory is not replacement of anomaly scoring, but increasingly careful calibration of how anomaly scores are interpreted and acted upon (Hennhöfer et al., 13 May 2026, Araz et al., 11 Jun 2026).