Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics

Abstract: The impact of a given training point on a statistical model is classically measured through its leave-one-out influence, which quantifies the effect of its removal from the training set on the model accuracy. While the statistics of leave-one-out influences are well understood in the low-dimensional, large sample limit $n\to \infty, d=O(1)$, they become more intricate in high dimensions, as the influence of a given sample develops non-trivial dependencies on all other training samples. For convex M-estimation under Gaussian design, in the high-dimensional limit $n\asymp d$, we show that the distribution of the influences across the training set converges to a limiting measure which we sharply characterize. Building on these results, we provide evidence that influential samples tend to lie close to the decision boundary, thereby making contact with a standard data selection heuristic in active learning.

• The paper rigorously derives precise asymptotic distributions for leave-one-out influence and DFBETA in high-dimensional convex M-estimation.
• It employs resolvent-based methods to map Gaussian inputs through nonlinear functionals, linking influence measures to the model's decision boundary.
• Empirical validations on datasets like MNIST and chest X-rays confirm the theoretical predictions, informing data pruning and active learning strategies.

Precise High-Dimensional Influence Diagnostics in Convex M-estimation

Introduction and Motivation

The statistical influence of individual data points on model estimators is a central concept for both robust statistics and explainability in machine learning. Classic influence metrics, such as leave-one-out (LOO) influence functions, Cook's distance, and DFBETA, are well-understood in low-dimensional, large-$n$ regimes. However, as contemporary ML systems routinely operate in “high-dimensional” settings where the number of features $d$ is comparable to, or exceeds, the number of samples $n$, these classical insights cease to hold. In such proportional regimes ($n \asymp d$), LOO influences develop pronounced stochastic dependencies across the dataset, reflecting non-trivial geometric and statistical correlations that classical asymptotics erase.

This paper rigorously analyzes influence metrics—particularly the test error influence and DFBETA—in the regime of convex M-estimation under isotropic Gaussian design, assuming $n,d \rightarrow \infty$ with $\alpha = n/d = \Theta(1)$. The primary contributions are: (i) explicit characterization of the limiting distributions of LOO influences; (ii) concentration results for influence-based diagnostics across the dataset; (iii) geometric analysis relating influence to the decision boundary, including empirical validation on real-world data.

Asymptotic Theory for Leave-One-Out Influences

The Proportional Regime

Given a training set $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$ with $x_i \sim \mathcal{N}(0,I_d)$ and $y_i = \phi(\langle x_i, \beta\rangle)$ for some ground truth unit vector $\beta$, the ERM with ridge regularization is: $d$0 LOO estimators $d$1 are analogously defined, and influence metrics such as the normalized test error influence and DFBETA are expressed as: $d$2

Breakdown of Classical Approximations

In contrast to the low-dimensional setting ($d$3 fixed, $d$4), where $d$5 converges to a deterministic function of $d$6 and decouples from the dataset, high-dimensionality induces a complex, random dependence of $d$7 on the global sample configuration. This non-triviality is a direct consequence of the subtle behavior of high-dimensional random matrices and the geometry of M-estimators, particularly as captured by the Hessian at the solution.

Limiting Distributions: Analytical Characterization

The main technical result shows that as $d$8 with $d$9, the empirical LOO influence distribution converges weakly to the pushforward of a $n$0-dimensional Gaussian through an explicit nonlinear map $n$1. Specifically,

$n$2

where $n$3 is a deterministic covariance matrix determined by resolvent-based summary statistics of the Hessian and alignment with the underlying signal direction (see main theorem for precise details).

Figure 1: Empirical distribution $n$4 of leave-one-out test error influences in simulated and real data, contrasted with the theoretically predicted pushforward distribution.

A similar (though stronger) convergence holds for DFBETA: its empirical distribution concentrates (in $n$5) to the pushforward of a $n$6-dimensional Gaussian by a related functional $n$7. The explicit forms are tractable, and for simple models (e.g., ridge regression), closed-form limits can be derived (e.g., scaled $n$8).

Figure 2: Empirical distribution $n$9 compared to theoretical limiting density under high-dimensional ridge regression.

Key Numerical Results and Implications

High-Fidelity Between Finite and Asymptotic Distributions

Empirical histograms from both synthetic and real datasets (e.g., CT slice location, MNIST, chest X-rays) show excellent agreement with the limiting distributions derived from the resolvent equations, even at moderate $n \asymp d$0 ($n \asymp d$1 thousands). The shape of the influence distribution is highly non-Gaussian: sharp peak at zero (most data points are low influence), asymmetric tails (with a fatter left tail, indicating a persistent fraction of highly "helpful" points), and a long but thin right tail (harmful points).

Dependence on Sample Complexity and Noise

Numerical studies confirm that as label noise increases, the influence distribution flattens and contracts towards zero, consistent with intuition that increased stochasticity dilutes the impact of individual samples. The mean influence as a function of $n \asymp d$2 exhibits a non-monotonic dependence: it is maximized (in magnitude) at intermediate $n \asymp d$3, rather than in the over- or underdetermined regimes, aligning with theory for high-dimensional double descent phenomena.

Figure 3: Test error influence distribution and conditional moments as a function of sample complexity and noise for logistic loss classification.

Relation to the Decision Boundary

By conditioning on the sample margin (distance to the empirical decision boundary), the authors demonstrate that highly influential points are overwhelmingly concentrated near the boundary and especially in disagreement regions between the empirical and population separators. This provides a rigorous foundation for classical heuristic data selection rules used in active learning, such as uncertainty sampling and margin-based selection.

Figure 4: Conditional influence distributions as a function of distance to the estimated and oracle decision boundaries: highest influence at the disagreement region.

Empirical Validation on Real Data

Iterative experiments on MNIST and chest X-ray datasets with neural network feature maps show that the structure and location of highly influential samples closely mirror the theoretical predictions for Gaussian models, reinforcing the applicability of these asymptotic results in overparametrized deep learning pipelines.

Figure 5: Test error influence maps on MNIST and chest X-ray data, visualizing maximal impact in the margin overlap region of estimators with neural features.

Theoretical and Practical Implications

These results have substantial impact on both theoretical statistics and practical machine learning:

• Theoretical: The work generalizes the classical influence function machinery to the high-dimensional limit, unveiling latent dependencies erased by traditional asymptotics. The exact characterization of influence distributions via random matrix theory–inspired resolvent systems and their closure under nonlinear functionals is novel and widely extensible.
• Practical: The explicit link between influence and proximity to the decision boundary formalizes longstanding empirical data curation techniques. Reliable estimation of influence enables principled data pruning, robust subsampling, and active learning strategies in modern high-dimensional regimes. The results also highlight intrinsic differences between beneficial and detrimental datapoints, suggesting new approaches for adversarial example detection and data valuation.
• Extension: While the analysis specifically addresses convex M-estimation under Gaussian/elliptical designs, the approach is structurally amenable to broader settings, including heavy-tailed, dependence-laden data via universality results. The generalization to group-wise or subset influence is technically challenging but presents a promising direction.

Outlook for Future Work

A key open problem is extending these precise asymptotic analyses beyond convex losses to non-convex ERMs (e.g., deep networks), where recent empirical work has observed the brittleness of influence measures. Additionally, the field of subset influence—quantifying the marginal impact of removing structured groups of datapoints in high dimension—remains largely undeveloped.

Finally, understanding the interplay between high-dimensional geometry, stochasticity, model overparametrization, and influence not only deepens our understanding of sample efficiency, but also offers a foundation for principled dataset design, reliable model audits, and enhanced robustness under distribution shifts.

Conclusion

This paper provides a comprehensive asymptotic analysis of influence diagnostics in high-dimensional convex M-estimation, establishing exact characterizations for the distribution of influence metrics, their dependence on geometric factors, and their empirical manifestation. These findings systematically bridge classical statistics and modern machine learning, clarifying the role of influential samples in regimes central to current AI practice, and opening new avenues for theoretically justified data-centric methodologies.

Reference: "Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics" (2607.09250)