Data Deviation Kernel Methods

• Data Deviation Kernels are kernel-based constructions that measure deviations from baseline models such as empirical support, Bayesian priors, or temporal regimes.
• They include methods like posterior second-moment kernels in regression and kernelized residual scores for anomaly detection, enabling precise quantification of nonconformity.
• These approaches are applied in change-point testing, structural change detection, and streaming anomaly detection, providing actionable insights through data-dependent geometry.

“Data Deviation Kernel” (Editor’s term) is a useful umbrella label for kernel constructions in which the kernel itself, or a kernel-induced score, operator, or subspace, is organized around deviation from a reference object: a prior over functions, an empirical data cloud, a nominal distribution, a learned InD subspace, or a current streaming window. The phrase is not used as a single standard formalism across the cited literature. Instead, the literature develops several technically distinct objects: posterior second-moment kernels in kernel regression, kernelized residual scores for outlier detection, MMD-related discrepancy functionals for two-sample and change-point testing, corrected kernel covariance operators for structural-change subspaces, and distributional kernel mean embeddings for streaming anomaly detection (Simon, 2022, Askari et al., 2018, Cheng et al., 2021, Yu et al., 2023, Xu et al., 5 Dec 2025).

1. Terminological scope and canonical formulations

A plausible unifying interpretation is that a data deviation kernel is any kernel-based construction whose central role is to encode how observed data depart from a baseline. The baseline may be prior uncertainty, empirical support, an InD manifold, or a nominal temporal regime. In some papers the kernel itself is data-dependent; in others the kernel is fixed but the deviation quantity is a kernel-induced residual, witness function, or corrected covariance operator. This suggests that “deviation” is not tied to one mathematical object, but to a family of roles kernels can play in measuring nonconformity, discrepancy, or structure change (Simon, 2022, Askari et al., 2018, Cheng et al., 2021, Yu et al., 2023).

Interpretation	Representative expression	Source
Posterior data-dependent kernel	$$\mathcal K_{\mathrm{post}}(x,x')=\mathbb E[f(x)f(x')\mid \mathcal D]$$	(Simon, 2022)
Kernelized residual from empirical support	$q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$	(Askari et al., 2018)
Distributional discrepancy on a manifold	$$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$	(Cheng et al., 2021)
Quantile witness discrepancy under partial overlap	$$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$	(Das et al., 2021)
Corrected deviation operator in RKHS	$$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$	(Yu et al., 2023)
Non-linear subspace residual for OoD	$S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$	(Fang et al., 21 May 2025)
Streaming distributional similarity score	$$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$	(Xu et al., 5 Dec 2025)

The common thread is structural rather than terminological. A kernel either defines the geometry in which deviation is measured or is itself updated to reflect the deviation information revealed by the sample.

2. Data-dependent kernels in supervised regression

The most explicit data-dependent kernel construction appears in kernel regression with a Bayesian prior over target functions. In the fixed-kernel setting, one observes $\mathcal D=(\mathbf X,\mathbf Y)$ with $\mathbf Y=\{f(x_i)\}_{i=1}^n$, uses the predictor

$$\hat f(x)=\mathbf k_{x\mathbf X}\mathbf K_{\mathbf X\mathbf X}^{-1}\mathbf Y,$$

and minimizes Bayes risk

$q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$0

Before seeing data, the optimal fixed kernel is the prior second-moment kernel

$q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$1

or the prior covariance in the centered case. After allowing the kernel to depend on the full observed dataset, including labels, the optimal updated choice becomes the posterior second-moment kernel

$q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$2

and kernel regression with this kernel yields exactly the posterior mean

$q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$3

Under squared loss, that posterior mean is Bayes-optimal among all predictors depending arbitrarily on the observed data (Simon, 2022).

This construction gives a precise meaning to “deviation” in the Bayesian regression setting. The kernel does not react to residuals heuristically. It is updated by conditioning the law of $q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$4 on $q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$5, so the observed sample changes the kernel from prior uncertainty to posterior uncertainty or posterior second moment. On the training set, the posterior kernel becomes especially simple: $q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$6 because $q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$7 almost surely under the posterior. Off the training set, however, the kernel can have nontrivial structure. The result is therefore label-dependent and fully dataset-dependent, not merely input-dependent.

A further clarification concerns Gaussian process priors. If $q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$8 is a centered GP with kernel $q_\psi(x)=K(x,x)-K_x^\top(\rho I+K_{XX})^{-1}K_x$9, then ordinary kernel regression with the fixed prior kernel already returns the posterior mean. In that case, data-dependent kernel adaptation is unnecessary for Bayes optimality. The posterior-kernel perspective becomes most informative when the target prior is not Gaussian. The same paper connects this observation to neural-network viewpoints in which training can be interpreted, approximately, as learning a data-dependent kernel and then performing kernel regression with it; the posterior kernel then functions as an ideal benchmark rather than a practical algorithm.

3. Deviation from empirical support and local geometry

A second major interpretation treats deviation as failure of a query point to be explained by the empirical support or by the span generated by the training sample. The clearest example is the kernelized inverse Christoffel construction. Starting from polynomial moment geometry, the kernelized score is

$$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$0

or, in kernel form,

$$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$1

This is the regularized residual energy of $$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$2 after projection onto the feature-space span of the training data. Low score means that the point is well represented by the data cloud; high score means geometrical inconsistency with that cloud and hence outlierness (Askari et al., 2018).

That support-based interpretation extends to other anomaly detectors but with different geometry. Kernel Outlier Detection begins from a PSD kernel matrix, centers it, constructs feature coordinates from the eigendecomposition $$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$3, and then measures outlyingness by projection pursuit in the induced feature space. The final score is

$$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$4

where the direction families include One Point, Two Point, Basis Vector, and Random directions. In this formulation the kernel does not itself return the deviation value; it creates the non-linear geometry in which directional deviation from the bulk becomes visible (Dağıdır et al., 28 Jun 2025).

A different route modifies the similarity function itself to match non-Gaussian nominal structure. The Generalized Hyperbolic construction defines

$$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$5

with $$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$6 the GH density. The stated motivation is sensitivity to skewness, heavy tails, and kurtosis, so deviation is measured relative to a similarity model that is not Gaussian-centric. The paper proves PSD by the standard squared-integral argument and uses the kernel inside KDE and OCSVM anomaly detectors (Bourigault et al., 25 Jan 2025).

The same literature also warns that “kernel based” is not always used in the strict Mercer-RKHS sense. In sequential business-process anomaly detection, each trace is mapped to a symbol sequence, similarity is defined by normalized longest common subsequence,

$$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$7

and anomaly score is the inverse similarity to the $$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$8-th nearest neighbor,

$$T=\int_M\int_M K_\gamma(x,y)(p-q)(x)(p-q)(y)\,dV(x)\,dV(y)$$9

The method is operationally kernel-like and deviation-oriented, but the paper itself does not establish PSD, feature maps, or Mercer conditions (Sureka, 2015).

4. Distributional deviation, partial overlap, and temporal change

Another large class of constructions measures deviation between distributions rather than between a point and a data cloud. For manifold-supported data, the kernel two-sample statistic

$$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$0

estimates the population discrepancy

$$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$1

The key deviation quantity is the intrinsic squared $$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$2-difference

$$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$3

For small bandwidth, the paper proves

$$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$4

with an explicit bias term, and shows that when $$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$5, deviations as small as $$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$6 are detectable up to constants and logarithmic factors. The leading rates depend on the intrinsic dimension $$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$7, not the ambient dimension $$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$8 (Cheng et al., 2021).

In temporal settings, change-point detection can be cast as repeated local two-sample testing. KL-CPD compares a left window and a right window, uses MMD as the discrepancy,

$$\mathbf{kdiff}(\mu_1,\mu_2;\alpha)=\big(T(\mu_1-\mu_2)\big)^\#(\alpha)$$9

and learns a deep kernel by maximizing a lower bound on test power against an auxiliary distribution $$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$0: $$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$1 The learned kernel is

$$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$2

with $$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$3 parameterized by an RNN encoder. The conceptual point is that local data deviation in a time series is treated as a distributional shift between adjacent windows, but kernel learning is stabilized by generating realistic surrogate alternatives instead of fitting directly to very scarce abnormal-side samples (Chang et al., 2019).

For structured data with only partial support overlap, the witness-function approach of kdiff replaces mean aggregation by a lower quantile. With

$$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$4

the distance is

$$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$5

This is explicitly presented as a more general form of MMD for partial-support matching: MMD averages the squared witness function, whereas kdiff takes a lower $$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$6-quantile. As a result, kdiff can stay small when two structured objects share a meaningful local motif or foreground on only part of their support, even if they differ elsewhere (Das et al., 2021).

5. Deviation subspaces, OoD residuals, and streaming embeddings

A further development shifts attention from scalar discrepancies to subspaces that preserve deviation structure. In model structural-change detection, Corrected Kernel PCA begins from the RKHS embedding $$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$7 and defines the central distribution deviation subspace

$$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$8

The expected kernel covariance decomposes as

$$\Delta_n^{\mathrm{kernel}}=M_n^{\mathrm{kernel}}-\Sigma_{\mathrm{pooled},n}^{\mathrm{kernel}}$$9

where

$S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$0

CKPCA estimates the deviation operator by

$S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$1

and the paper proves that the range of $S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$2 is exactly the central distribution deviation subspace. Classical KPCA fails here because it diagonalizes total centered variance rather than the corrected between-segment deviation operator (Yu et al., 2023).

Out-of-distribution detection via KPCA uses a closely related residual-subspace logic, but now with InD training data defining the reference. After mapping deep penultimate-layer features $S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$3 through a kernel approximation $S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$4, one learns a non-linear InD principal subspace and scores a test input by reconstruction error: $S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$5 The kernel is chosen to reflect two specific InD–OoD disparities: feature-norm imbalance and useful $S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$6 geometry after normalization. That leads to the Cosine-Gaussian construction, implemented through RFF or Nyström approximations, with Nyström landmarks selected from low-energy InD samples (Fang et al., 21 May 2025).

Streaming anomaly detection via IDK-S replaces subspace residuals by similarity to a dynamically maintained distributional centroid. A point-level data-dependent feature map $S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$7 is built from Isolation-Kernel hypersphere partitions, and the current window is embedded by

$S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$8

The normality score is

$S(\hat{\mathbf x})=-e^\Phi(\hat{\mathbf x})$9

Low similarity to that evolving kernel mean embedding indicates deviation from the current stream distribution. The main methodological point is incremental maintenance of a data-dependent feature map whose sampling distribution is claimed to be statistically equivalent to full retraining (Xu et al., 5 Dec 2025).

6. Stochastic deviation, generative discrepancy, and methodological limits

In asymptotic theory, “deviation” often refers not to anomaly or shift but to stochastic fluctuation of a kernel estimator around its expectation. For kernel copula estimators, the central object is

$$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$0

and the paper proves a uniform-in-bandwidth law of the iterated logarithm for local linear, mirror-reflection, and transformation estimators. With

$$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$1

the maximal deviation over $$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$2 and $$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$3 satisfies an almost-sure $$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$4 bound by $$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$5, yielding the stochastic half of strong uniform consistency and confidence-band arguments (Ba et al., 2016).

The same asymptotic use of “deviation” appears in kernel density estimation under dependence or recursion. For bifurcating Markov chains, the pointwise estimator of the invariant density obeys a moderate deviation principle for

$$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$6

with quadratic rate function

$$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$7

For recursive kernel density estimators defined by stochastic approximation, the variance-minimizing stepsize

$$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$8

produces the same pointwise LDP and MDP as the Rosenblatt estimator (Penda, 2021, Slaoui, 2013).

A modern generative-model interpretation pushes the deviation idea further. Kernel-gradient drifting defines

$$\mathrm{score}_i(x)=\frac1t\langle \Phi_i(x),\widehat\Phi(\mathcal P_{\mathbf X_i})\rangle$$9

Here the deviation object is a score difference between kernel-smoothed data and model distributions, not an RKHS norm. The paper derives identifiability for characteristic kernels and interprets the dynamics as steepest infinitesimal descent of a smoothed KL divergence (Esteban-Casadevall et al., 11 May 2026).

These literatures also delimit the concept’s scope. The posterior kernel in regression is an ideal Bayesian benchmark requiring knowledge of the true posterior, not a practical training recipe. CKPCA corrects a covariance operator rather than inventing a new scalar kernel. Some anomaly-detection papers use “kernel based” in a loose similarity sense rather than in the strict PSD-RKHS sense. Accordingly, “Data Deviation Kernel” is best treated as a technically useful umbrella term for multiple kernel-centered mechanisms of deviation, not as a single canonical object or universally standardized definition (Simon, 2022, Yu et al., 2023, Sureka, 2015).