Leave-One-Out Influence in ML

• Leave-one-out influence is a deletion-based sensitivity measure that quantifies how removing a single data point, covariate, or training event changes a model’s loss or output distribution.
• It is widely used to evaluate predictive stability and generalization by comparing outcomes with and without specific components, addressing computational bottlenecks in large or high-dimensional models.
• Variants such as leave-one-covariate-out and approximations like ALO extend its application to diverse contexts, informing model fairness, robustness, and active learning strategies.

Leave-one-out influence is a deletion-based notion of sensitivity: one omits a single unit—most commonly a training example, but in some settings a covariate, an agent, or a specific training event—and measures the resulting change in a fitted model, a prediction, a loss, a risk estimate, or an output distribution. In the literature, exact leave-one-out (LOO) is the canonical reference because it answers a direct counterfactual question: what would the procedure have produced had that unit been absent? The same literature also treats LOO as a computational bottleneck, a stability concept, an information-theoretic observable, and a target for approximation in high-dimensional and nonconvex regimes (Brophy et al., 2022, Cao et al., 2020, Wang et al., 2024, Cui et al., 28 May 2025).

1. Core concept and principal objects

A standard supervised-learning formulation measures the influence of a training example $z_i=(x_i,y_i)$ on a target example $z_e=(x_e,y_e)$ through the change in target loss after retraining without $z_i$: $$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$ This definition makes LOO influence a retraining-based quantity in loss space rather than a direct measure of parameter displacement or raw prediction displacement. Positive values indicate that $z_i$ reduces the target loss, while negative values indicate that removing $z_i$ improves the target prediction (Brophy et al., 2022).

A distinct individual-level formulation is leave-one-out unfairness, which asks whether the inclusion or removal of a single other person in the training set can change a fixed individual’s predictive distribution. For a learning rule $h$, training sample $S$, and query point $x$, the quantity is

$$\mathrm{LUF}(h,S,x)=\max_{i,k}\left|\Pr[h_S(x)=k]-\Pr[h_{S^{(\setminus i)}}(x)=k]\right|.$$

This shifts attention from aggregate error to the consistency of a specific person’s outcome under unit changes to the training data (Black et al., 2021).

The deleted unit need not be a training example. In high-dimensional regression, leave-one-covariate-out removes a predictor $z_e=(x_e,y_e)$0 and measures how much the entire LASSO solution path changes: $z_e=(x_e,y_e)$1 In stochastic training, trajectory-specific leave-one-out removes a datum from the specific iteration at which it appeared and measures the resulting change in final validation loss: $z_e=(x_e,y_e)$2 These variants preserve the deletion logic while changing the omitted object and the measured response (Cao et al., 2020, Wang et al., 2024).

Deleted unit	Representative quantity	Measured change
Training example $z_e=(x_e,y_e)$3	$z_e=(x_e,y_e)$4	Target loss after retraining
Other person in training set	$z_e=(x_e,y_e)$5	Predictive distribution for one individual
Covariate $z_e=(x_e,y_e)$6	$z_e=(x_e,y_e)$7	Whole LASSO path discrepancy
Time-specific training event $z_e=(x_e,y_e)$8	$z_e=(x_e,y_e)$9	Final validation loss on a fixed trajectory

These definitions are not interchangeable. Some quantify self-influence, some cross-point influence, some target prediction error, and some target the learned representation or optimization trajectory itself. That distinction is central to the modern literature.

2. Leave-one-out error, predictive stability, and generalization

One classical object is the leave-one-out error estimate

$z_i$0

where $z_i$1 is the dataset with observation $z_i$2 removed. A general non-asymptotic analysis introduces $z_i$3 stability through

$z_i$4

which measures the $z_i$5 magnitude of the one-point deletion effect on loss evaluated at an independent test point. This framework yields moment bounds and PAC-style exponential bounds for the discrepancy between $z_i$6 and the true prediction error; for ridge regression it gives explicit $z_i$7-scale LOO generalization bounds under bounded or sub-Gaussian assumptions (Celisse et al., 2016).

In kernel methods and deep models in the kernel regime, LOO becomes explicitly computable. For kernel ridge regression with fitted-value operator $z_i$8, the deleted-case residual is

$z_i$9

This converts LOO retraining into a single kernel solve plus access to the diagonal of $$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$0, and the resulting LOO loss and accuracy were shown to track double descent, random-label degradation, and transfer-learning behavior in the kernel regime (Bachmann et al., 2022).

LOO residuals also support predictive inference. In high-dimensional linear regression, with

$$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$1

the empirical quantiles of $$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$2 can be used to form prediction intervals

$$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$3

Uniform asymptotic validity follows under a case-deletion stability condition requiring that deleting one observation has asymptotically negligible effect on the fitted predictor in the prediction-relevant norm. In that sense, the paper treats LOO influence as asymptotic negligibility of one-case deletion, sufficient to transfer the distribution of LOO residuals to the future prediction error (Steinberger et al., 2016).

The same deletion geometry can also distort evaluation. In LOOCV for binary labels,

$$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$4

so the training-fold mean is a decreasing affine function of the left-out label. This produces a negative correlation between the omitted label and the training-fold label distribution, termed distributional bias. Because many models regress toward the training mean, aggregated LOOCV metrics such as auROC and auPR can be biased, and the bias can favor weaker regularization. A rebalanced variant removes one opposite-label training point so that all training folds have identical label composition (Austin et al., 2024).

3. Information-theoretic and distributional formulations

A distinct line of work formalizes leave-one-out influence information-theoretically. In supervised learning with supersample

$$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$5

and loss vector

$$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$6

the central quantity is the leave-one-out evaluated conditional mutual information

$$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$7

Conditioned on the full supersample $$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$8, this measures how much the losses reveal which point was omitted. The same paper places it in the hierarchy

$$\mathcal{I}_{LOO}(z_i, z_e) = \ell(y_e, A(D \setminus \{z_i\})(x_e)) - \ell(y_e, A(D)(x_e)).$$9

For interpolating $z_i$0–$z_i$1 learners it proves

$z_i$2

and for the unconditional root quantity,

$z_i$3

This identifies a precise bridge from omit-one behavior to population risk, and in the realizable VC setting it recovers optimal rates for the one-inclusion graph algorithm (Haghifam et al., 2022).

A complementary framework defines leave-one-out distinguishability by comparing entire output distributions under inclusion versus exclusion of a record: $z_i$4 Two important instantiations are mean-distance LOOD,

$z_i$5

and KL-based LOOD for Gaussian predictive distributions. In the latter case the quantity depends on both mean and covariance changes, which makes it suitable for analyzing leakage and membership inference. Under a GP or NNGP model of training randomness, LOOD unifies leave-one-out influence, memorization, and privacy risk; the literature further shows that the differing point is a stationary query for KL-LOOD under mild kernel conditions, and that optimized queries can reconstruct training examples (Ye et al., 2023).

These two directions differ in observable. Leave-one-out CMI measures information revealed by loss patterns about the omitted index, whereas LOOD measures statistical distance between output distributions under neighboring training sets. Both, however, replace purely parametric sensitivity with distributional change induced by omission.

4. Approximate leave-one-out and high-dimensional asymptotics

Exact LOO retraining is often replaced by influence-style approximations. For regularized empirical risk minimization,

$z_i$6

approximate leave-one-out cross-validation replaces $z_i$7 by a Newton correction formed from the full-data solution $z_i$8. In the smooth case,

$z_i$9

with

$z_i$0

A recent high-dimensional analysis proves that, under Gaussian covariates and strong convexity, these ALO weights are close to their mean-field counterparts even for non-differentiable regularizers, so ALO approximates the exact leave-one-out quantity up to negligible error terms (Bellec, 5 Jan 2025).

The same idea now appears inside conformal prediction. For Jackknife+ and Jackknife-minmax, the paper defines

$z_i$1

and proves the new uniform bound

$z_i$2

with high probability. This is the conformal-specific analogue of an influence approximation, but at the test covariate $z_i$3 rather than at the held-out training covariate $z_i$4. As a consequence, ALO-based Jackknife+ and Jackknife-minmax retain asymptotic coverage and efficiency while substantially reducing runtime (Cong et al., 30 Jun 2026).

For non-decomposable losses, the standard influence-function recipe fails because a single training object can participate in multiple coupled loss terms. The Versatile Influence Function (VIF) addresses this by aligning the finite-difference direction with actual deletion: $z_i$5 In decomposable ERM this reduces exactly to the classical influence formula; for Cox regression, node embedding, and listwise ranking it serves as a first-order deletion proxy that empirically tracks brute-force LOO while being up to $z_i$6 times faster (Deng et al., 2024).

At the asymptotic end of the spectrum, precise high-dimensional results characterize the entire distribution of leave-one-out influences in convex ridge-regularized M-estimation. With

$z_i$7

the empirical distribution of $z_i$8 and the marginal law of $z_i$9 converge to explicit limiting measures determined by resolvent fixed-point equations. The same analysis gives evidence that influential samples tend to lie close to the decision boundary, making contact with a standard active-learning heuristic (Cui, 10 Jul 2026).

5. Variants beyond deleting one training example

Removing one predictor rather than one observation yields leave-one-covariate-out influence. For the LASSO path

$h$0

the LOCO path fixes $h$1 and compares the full and constrained paths through

$h$2

Because the discrepancy aggregates over all coordinates and all $h$3, it captures both the direct disappearance of variable $h$4 and the indirect reorganization of the sparse model under competition and correlation. The same statistic supports variable screening and residual-bootstrap testing for single or multiple coefficients (Cao et al., 2020).

In order-sensitive optimization, the deleted unit is a specific training event on a realized trajectory. Standard SGD updates

$h$5

are replaced at the deletion time $h$6 by

$h$7

and the effect on a validation point is

$h$8

A first-order unrolling gives

$h$9

where the data value embedding stores the cumulative interaction between the removed point and the subsequent trajectory. This makes LOO time-specific and order-sensitive, and reveals distinct phases in which early and late training examples exert greater impact than middle-stage examples (Wang et al., 2024).

In multi-agent LLM debate, exact LOO removes one agent from the beginning and reruns the debate among the remaining $S$0 agents. To approximate this cheaply, IntrospecLOO adds one extra post-debate round in which each remaining agent is asked to ignore one designated agent’s responses. Exact re-debate has token complexity

$S$1

whereas IntrospecLOO reduces this to

$S$2

The measured quantity is the omission-induced change in per-agent or majority-vote performance, so the deleted unit is a participant in a social reasoning process rather than a datum in an empirical risk (Cui et al., 28 May 2025).

A different but related use appears in quickest change detection. There, a leave-one-out density estimator

$S$3

is used inside a sequential likelihood-ratio detector so that the observation being scored does not appear in its own estimated post-change density. The key advantage is independence between $S$4 and $S$5, which yields identities such as

$S$6

and supports false-alarm control and first-order asymptotic optimality. Here LOO serves as self-influence removal rather than as a diagnostic of training-point importance (Liang et al., 2022).

6. Interpretation, limitations, and controversies

Exact LOO is often treated as a gold-standard single-deletion definition, but its meaning depends sharply on the question. In gradient-boosted decision trees, exact retraining after removing one training example is the reference notion of influence and reliably identifies the single most influential example for a target prediction. Yet the same study finds that LOO performs poorly when one ranks points individually and then removes the top $S$7 as a set. The reason is non-additivity: the impact of removing one example depends on which others are removed, and small deletions can change tree structure, leaf assignments, and downstream residuals. The resulting ranking instability makes exact single-point LOO a poor proxy for group influence (Brophy et al., 2022).

A related ambiguity concerns what influence estimators actually approximate. In nonlinear networks, practical influence-function estimates often match exact cold-start LOO poorly. A five-term decomposition attributes the discrepancy to warm-start gap, proximity gap, non-convergence gap, linearization error, and solver error. The main empirical conclusion is that practical methods often approximate the proximal Bregman response function (PBRF) much better than true LOO retraining. Thus influence-function algorithms may still be informative, but they frequently answer a different local-response question than “what would happen if this point were removed and the model retrained from scratch?” (Bae et al., 2022).

The fairness literature makes another point of separation: low generalization error does not imply low individual-level leave-one-out sensitivity. Leave-one-out unfairness shows that a model can have small aggregate error while still changing a specific person’s outcome because a single other person is added to or removed from the training set. The same paper reports that adversarial training and randomized smoothing have opposite empirical effects on leave-one-out fairness, which links deletion sensitivity to robustness and memorization without collapsing them into the same object (Black et al., 2021).

Across these literatures, the central interpretive caution is that “leave-one-out influence” is not a single invariant quantity. It can mean deletion effect on target loss, on predictive distribution, on the full regularization path, on the output distribution at a query, on a validation loss along a realized optimization trajectory, or on a collective decision process. Exact retraining remains the clearest counterfactual definition when computationally feasible, but approximation quality, causal interpretation, and even the identity of the deleted unit are all domain-dependent.