---
title: Leave-One-Out Influence in ML
url: https://www.emergentmind.com/topics/leave-one-out-influence
type: topic
---

# Leave-One-Out Influence in ML

Leave-one-out influence is a deletion-based notion of sensitivity: one omits a single unit—most commonly a training example, but in some settings a covariate, an agent, or a specific training event—and measures the resulting change in a fitted model, a prediction, a loss, a risk estimate, or an output distribution. In the literature, exact leave-one-out (LOO) is the canonical reference because it answers a direct counterfactual question: what would the procedure have produced had that unit been absent? The same literature also treats LOO as a computational bottleneck, a stability concept, an information-theoretic observable, and a target for approximation in high-dimensional and nonconvex regimes [2205.00359] [2005.03694] [2412.09538] [2505.22192].

## 1. Core concept and principal objects

A standard supervised-learning formulation measures the influence of a training example \(z_i=(x_i,y_i)\) on a target example \(z_e=(x_e,y_e)\) through the change in target loss after retraining without \(z_i\):
\[
\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).
\]
This definition makes LOO influence a retraining-based quantity in loss space rather than a direct measure of parameter displacement or raw prediction displacement. Positive values indicate that \(z_i\) reduces the target loss, while negative values indicate that removing \(z_i\) improves the target prediction [2205.00359].

A distinct individual-level formulation is leave-one-out unfairness, which asks whether the inclusion or removal of a single other person in the training set can change a fixed individual’s predictive distribution. For a learning rule \(h\), training sample \(S\), and query point \(x\), the quantity is
\[
\mathrm{LUF}(h,S,x)=\max_{i,k}\left|\Pr[h_S(x)=k]-\Pr[h_{S^{(\setminus i)}}(x)=k]\right|.
\]
This shifts attention from aggregate error to the consistency of a specific person’s outcome under unit changes to the training data [2107.10171].

The deleted unit need not be a training example. In high-dimensional regression, leave-one-covariate-out removes a predictor \(X_j\) and measures how much the entire LASSO solution path changes:
\[
T_j(s,t)=\|\hat \beta-\hat \beta^{(-j)}\|_{s,t}.
\]
In stochastic training, trajectory-specific leave-one-out removes a datum from the specific iteration at which it appeared and measures the resulting change in final validation loss:
\[
TSLOO(z^*; (val)) := \ell(\theta_T', (val)) - \ell(\theta_T, (val)).
\]
These variants preserve the deletion logic while changing the omitted object and the measured response [2005.03694] [2412.09538].

| Deleted unit | Representative quantity | Measured change |
|---|---|---|
| Training example \(z_i\) | \(\mathcal I_{LOO}(z_i,z_e)\) | Target loss after retraining |
| Other person in training set | \(\mathrm{LUF}(h,S,x)\) | Predictive distribution for one individual |
| Covariate \(X_j\) | \(T_j(s,t)\) | Whole LASSO path discrepancy |
| Time-specific training event \(z^*,t_s\) | \(TSLOO(z^*;(val))\) | Final validation loss on a fixed trajectory |

These definitions are not interchangeable. Some quantify self-influence, some cross-point influence, some target prediction error, and some target the learned representation or optimization trajectory itself. That distinction is central to the modern literature.

## 2. Leave-one-out error, predictive stability, and generalization

One classical object is the leave-one-out error estimate
\[
\widehat R_1(A,D) = \frac{1}{n}\sum_{i=1}^n c(A(\tau_i(D),X_i),Y_i),
\]
where \(\tau_i(D)\) is the dataset with observation \(i\) removed. A general non-asymptotic analysis introduces \(L^q\) stability through
\[
\mathcal S_q(A,n)^q = \mathbb E\!\left[ \left| c(A(D,X),Y)-c(A(\tau_j(D),X),Y) \right|^q \right],
\]
which measures the \(L^q\) magnitude of the one-point deletion effect on loss evaluated at an independent test point. This framework yields moment bounds and PAC-style exponential bounds for the discrepancy between \(\widehat R_1(A,D)\) and the true prediction error; for ridge regression it gives explicit \(O(n^{-1/2})\)-scale LOO generalization bounds under bounded or sub-Gaussian assumptions [1608.06412].

In kernel methods and deep models in the kernel regime, LOO becomes explicitly computable. For kernel ridge regression with fitted-value operator \(\bm A=\bm K(\bm K+\lambda \bm 1_n)^{-1}\), the deleted-case residual is
\[
\Delta_{ik}^{\lambda} = \frac{Y_{ik}-\hat f_k^\lambda(\bm x_i)}{1-A_{ii}}.
\]
This converts LOO retraining into a single kernel solve plus access to the diagonal of \(\bm A\), and the resulting LOO loss and accuracy were shown to track double descent, random-label degradation, and transfer-learning behavior in the kernel regime [2203.03443].

LOO residuals also support predictive inference. In high-dimensional linear regression, with
\[
\tilde u_i = y_i - x_i'\hat\beta_{(i)},
\]
the empirical quantiles of \(\tilde u_i\) can be used to form prediction intervals
\[
PI_\alpha^{(L1O)}(Y,X,x_0) = \Big[x_0'\hat\beta+\tilde q_{n,\alpha/2},\; x_0'\hat\beta+\tilde q_{n,1-\alpha/2}\Big].
\]
Uniform asymptotic validity follows under a case-deletion stability condition requiring that deleting one observation has asymptotically negligible effect on the fitted predictor in the prediction-relevant norm. In that sense, the paper treats LOO influence as asymptotic negligibility of one-case deletion, sufficient to transfer the distribution of LOO residuals to the future prediction error [1602.05801].

The same deletion geometry can also distort evaluation. In LOOCV for binary labels,
\[
\bar y_{-i} = \frac{N\bar y - y_i}{N-1},
\]
so the training-fold mean is a decreasing affine function of the left-out label. This produces a negative correlation between the omitted label and the training-fold label distribution, termed distributional bias. Because many models regress toward the training mean, aggregated LOOCV metrics such as auROC and auPR can be biased, and the bias can favor weaker regularization. A rebalanced variant removes one opposite-label training point so that all training folds have identical label composition [2406.01652].

## 3. Information-theoretic and distributional formulations

A distinct line of work formalizes leave-one-out influence information-theoretically. In supervised learning with supersample
\[
Z=(Z_i)_{i\in[n+1]},\qquad U\sim \mathrm{Unif}([n+1]),\qquad S_n=Z_{-U},
\]
and loss vector
\[
L=(L_i)_{i\in[n+1]},\qquad L_i=\ell(\mathcal A_n(S_n),Z_i),
\]
the central quantity is the leave-one-out evaluated conditional mutual information
\[
\mathrm{LOO}^{e}\mathrm{CMI}_{\mathcal A_n}(D) \triangleq I(L;U\mid Z).
\]
Conditioned on the full supersample \(Z\), this measures how much the losses reveal which point was omitted. The same paper places it in the hierarchy
\[
I(L;U) \le I(L;U\mid Z) \le I(\hat Y;U\mid Z) \le I(\mathcal A_n(S_n);U\mid Z) \le I(\mathcal A_n(S_n);S_n).
\]
For interpolating \(0\)–\(1\) learners it proves
\[
R_n\le \frac{I(L;U\mid Z)}{\log(n+1)},
\qquad
R_n\log(n+1)\le I(L;U\mid Z)\le H(R_n)+R_n\log(n+1),
\]
and for the unconditional root quantity,
\[
R_n=\frac{I(L;U)}{\log(n+1)}.
\]
This identifies a precise bridge from omit-one behavior to population risk, and in the realizable VC setting it recovers optimal rates for the one-inclusion graph algorithm [2206.14800].

A complementary framework defines leave-one-out distinguishability by comparing entire output distributions under inclusion versus exclusion of a record:
\[
\text{LOOD}(Q; D, D') \;:=\; \mathrm{dist}\big(f_D(Q), f_{D'}(Q)\big).
\]
Two important instantiations are mean-distance LOOD,
\[
M(Q; D, D') = \left\|\mu_{D,\sigma^2}(Q) - \mu_{D',\sigma^2}(Q)\right\|_2^2,
\]
and KL-based LOOD for Gaussian predictive distributions. In the latter case the quantity depends on both mean and covariance changes, which makes it suitable for analyzing leakage and membership inference. Under a GP or NNGP model of training randomness, LOOD unifies leave-one-out influence, memorization, and privacy risk; the literature further shows that the differing point is a stationary query for KL-LOOD under mild kernel conditions, and that optimized queries can reconstruct training examples [2309.17310].

These two directions differ in observable. Leave-one-out CMI measures information revealed by loss patterns about the omitted index, whereas LOOD measures statistical distance between output distributions under neighboring training sets. Both, however, replace purely parametric sensitivity with distributional change induced by omission.

## 4. Approximate leave-one-out and high-dimensional asymptotics

Exact LOO retraining is often replaced by influence-style approximations. For regularized empirical risk minimization,
\[
\hat b = \argmin_{b\in \mathbb R^p} \sum_{i=1}^n L_{y_i}(x_i^Tb) + R(b),
\qquad
b^i = \argmin_{b\in \mathbb R^p} \sum_{l\neq i} L_{y_l}(x_l^Tb)+R(b),
\]
approximate leave-one-out cross-validation replaces \(b^i\) by a Newton correction formed from the full-data solution \(\hat b\). In the smooth case,
\[
\alo = \frac1n\sum_{i=1}^n g\Bigl( x_i^T\hat b + L_{y_i}'(x_i^T\hat b) \frac{x_i^T\hat A x_i}{1-x_i^T\hat A x_i\,L_{y_i}''(x_i^T\hat b)}, \,y_i \Bigr),
\]
with
\[
\hat A= \Bigl(\sum_{i=1}^n x_iL_{y_i}''(x_i^T\hat b)x_i^T+\nabla^2R(\hat b)\Bigr)^{-1}.
\]
A recent high-dimensional analysis proves that, under Gaussian covariates and strong convexity, these ALO weights are close to their mean-field counterparts even for non-differentiable regularizers, so ALO approximates the exact leave-one-out quantity up to negligible error terms [2501.02624].

The same idea now appears inside conformal prediction. For Jackknife+ and Jackknife-minmax, the paper defines
\[
\tilde{\theta}_{/i} := \hat{\theta} + \frac{\mathbf{J}^{-1} \boldsymbol{x}_i \, \dot{\ell}_i( \hat{\theta})} {1 - \boldsymbol{x}_i^\top \mathbf{J}^{-1} \boldsymbol{x}_i \, \ddot{\ell}_i(\hat{\theta})},
\]
and proves the new uniform bound
\[
\max_{1\leq i\leq n}|\boldsymbol{x}_{n+1}^\top\tilde{\theta}_{/i}-\boldsymbol{x}_{n+1}^\top\hat{\theta}_{/i}|\leq\frac{C_0}{\sqrt{p}}
\]
with high probability. This is the conformal-specific analogue of an influence approximation, but at the test covariate \(x_{n+1}\) rather than at the held-out training covariate \(x_i\). As a consequence, ALO-based Jackknife+ and Jackknife-minmax retain asymptotic coverage and efficiency while substantially reducing runtime [2606.31915].

For non-decomposable losses, the standard influence-function recipe fails because a single training object can participate in multiple coupled loss terms. The Versatile Influence Function (VIF) addresses this by aligning the finite-difference direction with actual deletion:
\[
VIF(\hat{\theta}(\mathbf{1}); i) := - \left[\frac{1}{n}\nabla_{\theta}^2 L(\hat{\theta}(\mathbf{1}), \mathbf{1}) \right]^{-1}
 \nabla_{\theta}\left(L(\hat{\theta}(\mathbf{1}), \mathbf{1}) - L(\hat{\theta}(\mathbf{1}), \mathbf{1}_{-i}) \right).
\]
In decomposable ERM this reduces exactly to the classical influence formula; for Cox regression, node embedding, and listwise ranking it serves as a first-order deletion proxy that empirically tracks brute-force LOO while being up to \(10^3\) times faster [2412.01335].

At the asymptotic end of the spectrum, precise high-dimensional results characterize the entire distribution of leave-one-out influences in convex ridge-regularized M-estimation. With
\[
IF_i = n\Big(\mathcal E_{\rm gen}[\hat w]-\mathcal E_{\rm gen}[\hat w_{(i)}]\Big),
\qquad
DFBETA_i = n\|\hat w-\hat w_{(i)}\|^2,
\]
the empirical distribution of \(DFBETA_i\) and the marginal law of \(IF_i\) converge to explicit limiting measures determined by resolvent fixed-point equations. The same analysis gives evidence that influential samples tend to lie close to the decision boundary, making contact with a standard active-learning heuristic [2607.09250].

## 5. Variants beyond deleting one training example

Removing one predictor rather than one observation yields leave-one-covariate-out influence. For the LASSO path
\[
\hat{\beta}(\lambda) = \operatorname*{argmin}_{\beta \in \mathbb{R}^p} \left(\|Y - \mathbf{X}\beta \|_{2}^{2} + \lambda\|\beta\|_{1} \right),
\]
the LOCO path fixes \(\beta_j=0\) and compares the full and constrained paths through
\[
T_j(s,t)= \| \hat \beta - \hat \beta^{(-j)} \|_{s,t}.
\]
Because the discrepancy aggregates over all coordinates and all \(\lambda\), it captures both the direct disappearance of variable \(j\) and the indirect reorganization of the sparse model under competition and correlation. The same statistic supports variable screening and residual-bootstrap testing for single or multiple coefficients [2005.03694].

In order-sensitive optimization, the deleted unit is a specific training event on a realized trajectory. Standard SGD updates
\[
\theta_{t+1} = \theta_t - \eta_t \sum_{z \in B_t} \nabla \ell(\theta_t, z)
\]
are replaced at the deletion time \(t_s\) by
\[
\theta_{t_s+1}' = \theta_{t_s} - \eta_{t_s} \sum_{z \in B_{t_s} \setminus \{z^*\}} \nabla \ell(\theta_{t_s}, z),
\]
and the effect on a validation point is
\[
TSLOO(z^*; (val)) := \ell(\theta_T', (val)) - \ell(\theta_T, (val)).
\]
A first-order unrolling gives
\[
TSLOO(z^*;(val)) \approx \nabla \ell(\theta_T,(val))^\top DVEmb(z^*),
\]
where the data value embedding stores the cumulative interaction between the removed point and the subsequent trajectory. This makes LOO time-specific and order-sensitive, and reveals distinct phases in which early and late training examples exert greater impact than middle-stage examples [2412.09538].

In multi-agent LLM debate, exact LOO removes one agent from the beginning and reruns the debate among the remaining \(N-1\) agents. To approximate this cheaply, IntrospecLOO adds one extra post-debate round in which each remaining agent is asked to ignore one designated agent’s responses. Exact re-debate has token complexity
\[
O(RTN^2),
\]
whereas IntrospecLOO reduces this to
\[
O(RN).
\]
The measured quantity is the omission-induced change in per-agent or majority-vote performance, so the deleted unit is a participant in a social reasoning process rather than a datum in an empirical risk [2505.22192].

A different but related use appears in quickest change detection. There, a leave-one-out density estimator
\[
\widehat p^{n,k}_{-i}(x_i) = \frac{1}{(n-k)h}\sum_{\substack{j=k\\ j\neq i}}^n K\!\left(\frac{x_i-x_j}{h}\right)
\]
is used inside a sequential likelihood-ratio detector so that the observation being scored does not appear in its own estimated post-change density. The key advantage is independence between \(\widehat p^{n,k}_{-i}\) and \(X_i\), which yields identities such as
\[
\mathbb{E}_\infty\!\left[\frac{\widehat p^{n,k}_{-i}(X_i)}{p_0(X_i)}\right]=1
\]
and supports false-alarm control and first-order asymptotic optimality. Here LOO serves as self-influence removal rather than as a diagnostic of training-point importance [2211.00223].

## 6. Interpretation, limitations, and controversies

Exact LOO is often treated as a gold-standard single-deletion definition, but its meaning depends sharply on the question. In gradient-boosted decision trees, exact retraining after removing one training example is the reference notion of influence and reliably identifies the single most influential example for a target prediction. Yet the same study finds that LOO performs poorly when one ranks points individually and then removes the top \(k\) as a set. The reason is non-additivity: the impact of removing one example depends on which others are removed, and small deletions can change tree structure, leaf assignments, and downstream residuals. The resulting ranking instability makes exact single-point LOO a poor proxy for group influence [2205.00359].

A related ambiguity concerns what influence estimators actually approximate. In nonlinear networks, practical influence-function estimates often match exact cold-start LOO poorly. A five-term decomposition attributes the discrepancy to warm-start gap, proximity gap, non-convergence gap, linearization error, and solver error. The main empirical conclusion is that practical methods often approximate the proximal Bregman response function (PBRF) much better than true LOO retraining. Thus influence-function algorithms may still be informative, but they frequently answer a different local-response question than “what would happen if this point were removed and the model retrained from scratch?” [2209.05364].

The fairness literature makes another point of separation: low generalization error does not imply low individual-level leave-one-out sensitivity. Leave-one-out unfairness shows that a model can have small aggregate error while still changing a specific person’s outcome because a single other person is added to or removed from the training set. The same paper reports that adversarial training and randomized smoothing have opposite empirical effects on leave-one-out fairness, which links deletion sensitivity to robustness and memorization without collapsing them into the same object [2107.10171].

Across these literatures, the central interpretive caution is that “leave-one-out influence” is not a single invariant quantity. It can mean deletion effect on target loss, on predictive distribution, on the full regularization path, on the output distribution at a query, on a validation loss along a realized optimization trajectory, or on a collective decision process. Exact retraining remains the clearest counterfactual definition when computationally feasible, but approximation quality, causal interpretation, and even the identity of the deleted unit are all domain-dependent.

Source: https://www.emergentmind.com/topics/leave-one-out-influence