Leave-One-Out EigenScore (LOOE)

• LOOE is a family of sensitivity measures that use eigenspectral and curvature-based diagnostics to assess leave-one-out instability in models.
• It encompasses both exact formulations in kernel regression and approximate variants in high-dimensional convex learning, leveraging self-influence corrections.
• LOOE highlights the impact of small eigenvalues, regularization, and double descent phenomena to guide reliable model selection and performance assessment.

"Leave-One-Out EigenScore" ("LOOE", Editor's term) denotes a family of leave-one-out sensitivity quantities whose controlling terms are eigenspectral or curvature-based objects that measure how strongly predictions depend on an individual sample, feature, or latent mode. In the cited literature, no paper introduces LOOE as a formal named method. Instead, the term is most naturally reconstructed from two lines of work: exact leave-one-out formulas for kernel least squares and kernel-regime deep networks, where instability is governed by spectral filters and inverse eigenvalues, and approximate leave-one-out formulas for high-dimensional convex learning, where the analogous objects are generalized hat-matrix diagonals, inverse-curvature resolvents, and Jacobians of proximal or projection maps (Bachmann et al., 2022, Wang et al., 2018).

1. Conceptual status and scope

LOOE is not a standardized method in the available sources. The closest direct foundation appears in the study of leave-one-out error for kernel regression, multiclass targets encoded as one-hot vectors, classification treated as regression under squared loss, and deep neural networks in the kernel regime, especially via the Neural Tangent Kernel and random feature models. In that setting, leave-one-out is the standard retrain-with-one-point-removed estimator: for each $i$, one forms $\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$, trains on $\mathcal S_{-i}$, and averages the resulting loss or accuracy on the omitted point. The same sources also motivate a feature-removal notion of leave-one-out in Transformers, where $\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$ measures the output change after removing one input feature (Bachmann et al., 2022, You et al., 21 Oct 2025).

These two uses of leave-one-out operate at different levels. The kernel-regression formulation is sample-centric and tied to retraining under squared loss. The Transformer formulation is feature-centric and instantiated by masking one token at a time in text or zeroing one pixel at a time in images, with the measured quantity taken to be predicted class logit change. A plausible implication is that LOOE is best understood not as a single algorithm, but as a spectral or curvature-based summary of leave-one-out instability whose exact form depends on whether the omitted object is a training example or an input feature.

2. Exact kernel leave-one-out formulas

In kernel least squares with ridge parameter $\lambda\ge 0$, the predictor is

$$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$

and in the unregularized limit,

$\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$

Here $K\in\mathbb R^{n\times n}$ is the Gram matrix, $Y\in\mathbb R^{n\times C}$ stores the targets row-wise, and $K_x\in\mathbb R^n$ has entries $\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$0. The key object is

$\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$1

With this notation, the leave-one-out residual is

$\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$2

and the paper gives the exact shortcut

$\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$3

This avoids retraining $\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$4 separate models (Bachmann et al., 2022).

The decomposition of $\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$5 is the immediate kernel-level precursor of LOOE. The numerator $\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$6 is the training residual, while the denominator $\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$7 is a self-influence correction analogous to the classical hat-matrix denominator in linear regression. In binary classification with $\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$8, the same structure appears as

$\mathcal S_{-i}=\mathcal S\setminus\{(x_i,y_i)\}$9

with

$\mathcal S_{-i}$0

Any LOOE derived from these formulas is therefore anchored in the interaction between residual magnitude and self-influence.

3. Eigenspectral structure

The eigenspectral content becomes explicit once the kernel is written as

$\mathcal S_{-i}$1

with eigenvalues $\mathcal S_{-i}$2 and rank $\mathcal S_{-i}$3. The shrinkage matrix then has the spectral form

$\mathcal S_{-i}$4

Hence each spectral mode is filtered by

$\mathcal S_{-i}$5

and the diagonal self-influence term becomes

$\mathcal S_{-i}$6

The synthesis explicitly interprets this denominator as a local spectral defect or leave-one-out vulnerability term (Bachmann et al., 2022).

The corresponding corollary for the leave-one-out residual is

$\mathcal S_{-i}$7

In the rank-deficient ridgeless case $\mathcal S_{-i}$8,

$\mathcal S_{-i}$9

so only null-space or zero-eigenvalue directions contribute. In the full-rank ridgeless case $\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$0,

$\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$1

This establishes the central spectral fact behind LOOE: small eigenvalues dominate leave-one-out instability, and a sample is unstable when it loads heavily on small-$\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$2 eigenvectors.

4. Candidate score constructions

Because no canonical LOOE is defined, the most defensible formulations are explicitly synthetic restatements of the kernel formulas. The first and simplest candidate is a per-sample spectral sensitivity score,

$\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$3

This uses only the sample-specific denominator. Large values indicate that point $\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$4 is spectrally fragile under leave-one-out; small values indicate stability. This is the cleanest direct translation of the self-influence term into a scalar score.

A second candidate is a residual-based score,

$\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$5

The global aggregate is exactly the paper’s leave-one-out loss, but it can be interpreted as an eigen-sensitive score because $\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$6 already has the spectral form above. This suggests that LOOE may be either local, with one score per sample or sample-output pair, or global, with one score per model.

A third candidate emerges in the full-rank ridgeless limit: $\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$7 The synthesis presents this as a plausible leave-one-out eigen-instability score rather than a named quantity from the source paper. Its meaning is immediate: a sample receives a high score when its energy is concentrated in poorly conditioned eigendirections.

A fourth candidate is global rather than samplewise. The double-descent analysis uses

$\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$8

which is nearly a ready-made global LOOE. A plausible implication is that practical LOOE families separate into local scores built from $\mathrm{LOO}_i=f(x)-f(x_{\setminus i})$9 or $\lambda\ge 0$0, and global scores built from the summed reciprocal vulnerability $\lambda\ge 0$1 (Bachmann et al., 2022).

5. Generalization, double descent, and empirical behavior

The kernel-regime work uses leave-one-out as a tractable proxy for generalization, but it does not prove a new full theorem that $\lambda\ge 0$2 converges to test error for NTK models. Instead it relies on prior statistical motivation via stability, prior consistency results for ridge regression, and extensive empirical evidence for NTK, random-feature, and deep transfer settings. Empirically, for fully connected NTK models on MNIST and CIFAR10, $\lambda\ge 0$3 and $\lambda\ge 0$4 closely follow test loss and accuracy, and the variance decreases as $\lambda\ge 0$5 increases. Under increasing label corruption, leave-one-out tracks test performance almost perfectly. In transfer learning with pretrained ResNet18, AlexNet, VGG16, and DenseNet161 and only top-layer retraining, leave-one-out accuracy and test accuracy closely match, with the same behavior reported for loss in the appendix (Bachmann et al., 2022).

The most important qualitative theorem-level connection is double descent. At the interpolation threshold $\lambda\ge 0$6, for large $\lambda\ge 0$7,

$\lambda\ge 0$8

so $\lambda\ge 0$9 almost surely as $$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$0. In the specific case $$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$1,

$$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$2

This shows that blow-up depends on the smallest or vanishing mode’s eigenvector entries and on label alignment with that mode. For LOOE, the significance is direct: the score is not merely a measure of residual size, but a detector of interpolation instability driven by near-null eigendirections, weak regularization, and mode-specific label alignment.

Regularization enters through the spectral factors

$$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$3

Increasing $$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$4 damps instability by preventing denominators $$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$5 from collapsing. This suggests that any LOOE intended as a model-selection or reliability diagnostic must be read jointly with the regularization level.

6. High-dimensional convex analogues

A broader and more abstract precursor to LOOE appears in approximate leave-one-out for convex penalized learning problems of the form

$$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$6

Here the exact leave-one-out prediction is $$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$7, and the paper develops three equivalent approximation frameworks: primal, dual, and proximal. In smooth settings, the approximate leave-one-out correction reduces to

$$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$8

with generalized hat matrix

$$\hat f^\lambda_{\mathcal S}(x)=K_x^\top (K+\lambda I_n)^{-1}Y,$$9

The diagonal $\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$0 is the high-dimensional convex analogue of $\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$1: a leverage-like self-sensitivity term that is already an eigen-score in all but name (Wang et al., 2018).

The same paper shows that this structure persists in dual and proximal language. In a least-squares dual formulation,

$\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$2

where $\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$3 is the Jacobian of the relevant proximal map. In constrained problems, the Jacobian of the projection $\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$4 plays the same role; for polyhedral constraints, if $\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$5 spans the active-face tangent space, then

$\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$6

These are explicit subspace objects. The paper therefore supplies a general blueprint for LOOE: define leave-one-out corrections through local curvature operators, then summarize sample influence by diagonal entries of inverse-Hessian, projector, or proximal-Jacobian resolvents.

The equivalence of the primal, dual, and proximal approaches under smoothness conditions reinforces the interpretation. In the smooth regime, the paper also imports an accuracy statement of the form

$\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$7

with high probability when $\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$8. This does not define LOOE, but it places eigen- or leverage-based leave-one-out scoring on a broader asymptotic footing.

7. Transformer benchmarking, faithfulness criteria, and open limits

The Transformer attribution literature contributes evaluation criteria rather than a spectral LOOE construction. In that setting, leave-one-out is treated as an intuitive and implementation-invariant reference for feature importance,

$\hat f_{\mathcal S}(x)=K_x^\top K^\dagger Y.$9

computed by masking each token individually in text or zeroing each pixel one at a time in images, and measuring predicted class logit change. Exact leave-one-out requires one forward pass per feature, so it is computationally prohibitive at scale. For benchmarking approximations, attribution maps are normalized to sum to one per example and compared to leave-one-out by Pearson correlation (You et al., 21 Oct 2025).

This benchmarking perspective is important for LOOE because it identifies two failure modes for any fast proxy based on internal model structure rather than explicit removal. First, implementation invariance may fail: in a controlled linear-attention setting, AttnLRP has $K\in\mathbb R^{n\times n}$0 correlation between left- and right-factorized implementations, while LOO and Integrated Gradients are both $K\in\mathbb R^{n\times n}$1. Second, correlation with exact leave-one-out may remain poor even for axiomatic methods: in the same experiment, AttnLRP has $K\in\mathbb R^{n\times n}$2 and $K\in\mathbb R^{n\times n}$3 correlation with LOO under the two factorizations, while Integrated Gradients is implementation invariant but negatively correlated with LOO at $K\in\mathbb R^{n\times n}$4. On SST, CP-LRP improves correlation with LOO from $K\in\mathbb R^{n\times n}$5 to $K\in\mathbb R^{n\times n}$6; on IMDB, the improvement is from $K\in\mathbb R^{n\times n}$7 to $K\in\mathbb R^{n\times n}$8. The paper attributes much of the mismatch to bilinear propagation error and softmax propagation error, with the strongest benefits from bypassing softmax in middle-to-late Transformer layers.

For LOOE, these results imply a clear distinction between mathematical ingredients and validated approximations. The kernel and convex-learning papers provide unusually strong ingredients for a leave-one-out eigen-score: $K\in\mathbb R^{n\times n}$9, $Y\in\mathbb R^{n\times C}$0-weighted residuals, $Y\in\mathbb R^{n\times C}$1, $Y\in\mathbb R^{n\times C}$2, and active-subspace projectors. The Transformer paper shows how any approximation to exact leave-one-out should be judged: by exact ablation, implementation invariance, and direct correlation with leave-one-out. What remains missing across the cited literature is an explicit definition of LOOE, a theorem establishing an optimal spectral score, a mode-wise additive decomposition of total leave-one-out loss, and large-scale approximation algorithms specialized to such a score. The present state of the subject is therefore best described as a mathematically rich precursor stage: the constituent operators are explicit, but the canonical Leave-One-Out EigenScore has not yet been formalized.