Single-Model Predictive Data Attribution

• The paper introduces methods using influence functions and its refinements to quantify training sample impacts without the need for expensive retraining.
• Single-model predictive data attribution is a framework that leverages analytical and algorithmic techniques to trace model predictions back to individual training data points.
• Efficient algorithms such as matrix-free Hessian-vector products and advanced solvers enable scalable attribution in high-dimensional and non-convex deep learning models.

Single-model predictive data attribution refers to methods that, for a fixed trained model, estimate the impact or influence of individual training datapoints on the model’s predictions or loss function. Unlike ensemble or retraining-based attribution, single-model attribution leverages analytic or algorithmic techniques—derived from robust statistics, optimization, and machine learning theory—to compute these effects without necessitating expensive retraining or stochastic averaging. This paradigm encompasses classic influence functions, their modern refinements, path-integral approaches, stagewise and Bayesian variants, and extensions for non-decomposable objectives.

1. Formal Foundations: Influence Functions and Beyond

The foundational concept underlying most single-model data attribution methods is the influence function (IF) from robust statistics. For a supervised learning scenario with empirical risk $R(\theta) = \frac{1}{n} \sum_{i=1}^n L(z_i,\theta)$ and fitted parameter $\theta^* = \arg\min_\theta R(\theta)$, the influence of infinitesimally upweighting a training sample $z$ on the parameters is given by the implicit differentiation:

$$\hat\theta_{\varepsilon,z} \approx \theta^* - \varepsilon H_{\theta^*}^{-1}\nabla_\theta L(z,\theta^*)$$

where $H_{\theta^*} = \nabla^2_\theta R(\theta^*)$ is the Hessian at the minimizer (Zhu et al., 10 Aug 2025). The impact on a test loss $L(z_{\text{test}},\theta^*)$ is then

$$I_{\mathrm{up,loss}}(z, z_{\mathrm{test}}) = -\nabla_\theta L(z_{\text{test}}, \theta^*)^\top H_{\theta^*}^{-1} \nabla_\theta L(z, \theta^*)$$

This first-order approximation forms the basis for most single-model data attribution frameworks, facilitating identification of both "helpful" and "harmful" training examples.

2. Algorithmic Techniques: Matrix-Free Curvature and Efficient Solvers

Influence computation for overparameterized models ($p \gg 10^6$) is dominated by the challenge of inverting or applying the Hessian $H_{\theta^*}$. Direct computation is infeasible, so matrix-free methods are standard:

• Pearlmutter’s trick computes Hessian-vector products (HVPs) $v \mapsto Hv$ in $O($one gradient$)$ via reverse-mode autodiff.
• Stochastic Neumann/LiSSA (Agarwal et al. 2017) and Lanczos/Conjugate Gradient (CG) solvers approximate inverse-Hessian-vector products (IHVP) $(H + \lambda I)^{-1} v$ with early stopping and mini-batch stochasticity, where damping $\lambda I$ ensures positive definiteness.
• Subsampling and curvature approximations (e.g., substituting $H$ by Gauss–Newton or Fisher information matrices) further reduce runtime, at some loss in fidelity (Zhu et al., 10 Aug 2025).

In non-convex deep models, all matrix-free solvers require explicit damping and carefully controlled vector operations, with empirical studies reporting convergence in $O(100-1000)$ HVPs per test point for high precision.

3. Refinements: High-dimensional, Non-convex, and Integral-based Attribution

3.1. Rescaled Influence Functions (RIF)

In high-dimensional regimes ($d \sim n$ or $d > n$), classic IF systematically underestimates the impact of point-removal due to failing to update the Hessian appropriately. The rescaled influence function (RIF) corrects for this by using the leave-one-out Hessian via Sherman–Morrison identities:

$\text{RIF}_i = (1-h_i)^{-1}\text{IF}_i$

with $h_i$ the leverage score for $z_i$. RIF is a strictly additive, drop-in replacement for classical IF, maintaining accuracy in extreme overparameterization ($n \approx O(d)$) and robust to vanishing regularization (Rubinstein et al., 7 Jun 2025).

3.2. Integrated Influence with Baseline

Integrated Influence (IIF) generalizes IF by defining a continuous path from a non-informative baseline dataset $D_0$ to the original data $D_1$, accumulating influence along an interpolated sequence:

$$\Phi_i = -\int_0^1 G(t)^\top [H_t]^{-1} J_i(t) dt \cdot (y_i - y_i^0)$$

where $G(t)$ is the test gradient and $J_i(t)$ the model’s cross-derivative with respect to training target. Discrete approximations (Euler sum) yield practical estimation, and both IF and TracIn emerge as limiting/special cases depending on choice of baseline and discretization (Yang et al., 7 Aug 2025).

3.3. Stagewise and Bayesian Influence

Classical IF methods are static and miss the dynamic, stagewise patterns in neural network training. The Bayesian Influence Function (BIF), evaluated via local SGLD posterior samples at different SGD checkpoints, computes the covariance matrix:

$$I_{ij}(t) = -\operatorname{Cov}_{\theta \sim p_\beta(\cdot|D_t)}[\ell_i(\theta), \ell_j(\theta)]$$

BIF reveals phase transitions, influence sign-flips, and hierarchical learning phases, enabling detailed developmental analysis of data impact at every epoch (Lee et al., 14 Oct 2025).

4. Practical Protocols and Empirical Metrics

Standard empirical workflows involve ranking all training points by their influence score with respect to each test instance. For evaluation:

• Linear Data-modeling Score (LDS): Spearman correlation between predicted and true leave-set loss shifts over randomly subsampled retraining subsets (Zhu et al., 10 Aug 2025).
• Mislabeled-data detection: Points with extreme negative self-influence $\text{SelfInf}(z) = I_{\mathrm{up,loss}}(z, z)$ are prioritized for inspection, recovering 70–80% of synthetic label errors in the top 10–20% of candidates on MNIST.
• Group and stagewise analysis: Dynamic BIF traces, KNN over top-influential tokens, and clustering of phase transitions validate structural encoding and semantic neighborhood properties (Lee et al., 14 Oct 2025).

Typical LDS benchmarks for deep vision models with IF or IIF are 0.10–0.16; with RIF or MAGIC on moderate- to large-scale tasks, LDS reaches 0.80–0.97, far exceeding kernel or gradient approximation baselines (Rubinstein et al., 7 Jun 2025, Ilyas et al., 23 Apr 2025).

5. Limitations and Research Directions

Open Problems

• Non-convexity and local minima: All Taylor-based methods assume a local quadratic model; in deep overparameterized settings, $H_{\theta^*}$ is often indefinite and mini-batch Hessian estimation injects noise. Large perturbations undermine first-order accuracy.
• Computational bottlenecks: Even matrix-free IF or IIF demand thousands of HVPs and $O(np)$ dot products per test point; scaling to LLMs or massive vision models is challenging (Zhu et al., 10 Aug 2025).
• Ultra high-dimensional degeneracy: IF accuracy deteriorates as $n/d \to O(1)$ or regularization $\lambda \to 0$, but RIF and metagradient unrolling approaches remain stable in these regimes (Rubinstein et al., 7 Jun 2025, Ilyas et al., 23 Apr 2025).
• Stagewise and singular learning: Static-attribution methods entirely miss non-monotonic influence and developmental phase transitions (Lee et al., 14 Oct 2025).

Promising Directions

• Hybrid and unrolling methods: Combining implicit gradients and forward unrolling, e.g., SOURCE (Bae et al. 2024), drastically reduces bias under non-convex conditions.
• Kernel-based and projection shortcuts: TRAK and checkpoint approaches yield tractable approximations for transformers and vision models.
• Parameter-efficient attribution: DataInf and related frameworks provide closed-form influences under LoRA or fine-tuning.
• Machine unlearning: Rapid one-step parameter corrections for data removal or label repair are natural extensions (Zhu et al., 10 Aug 2025).
• Improvements in curvature approximation: Techniques like EK-FAC and low-rank curvature capture more eigenspectrum for robust inverse-Hessian estimation.
• Rigorous evaluation: Uniform LDS and out-of-sample LDS (after model update) measure calibration and predictive validity.

6. Applications and Extensions

Single-model predictive data attribution is instrumental in:

• Debugging and accountability: Pinpointing harmful or helpful examples for targeted data curation or correcting mislabeled data (Zhu et al., 10 Aug 2025).
• Interpretability: Providing first-order explanations of model behavior and counterfactual predictions without ensemble retraining.
• Curriculum design and risk monitoring: Stagewise influence analysis uncovers implicit learning pathways and critical samples for adversarial or curriculum manipulation (Lee et al., 14 Oct 2025).
• Robustness and poisoning detection: RIF and IIF are sensitive to subtle shifts and can robustly flag poisoned or outlier samples missed by classical IF (Rubinstein et al., 7 Jun 2025, Yang et al., 7 Aug 2025).
• Large-scale industrial deployment: Production architectures, e.g., LiDDA at LinkedIn, employ single-layer self-attention for per-sample credit attribution at web scale (Bencina et al., 14 May 2025).

Extensions for non-decomposable losses, as exemplified by the Versatile Influence Function (VIF), generalize IF to settings such as survival analysis, listwise ranking, and contrastive learning using finite differencing over "presence" indicators with auto-diff and CG-based Hessian inversion, without closed-form derivations for each task (Deng et al., 2 Dec 2024).

7. Summary Table: Core Methods and Their Properties

Method	Loss Type	Non-convex/High-Dim	Stagewise/Temporal	Requires Retraining	Main Limitation
Classic IF	Decomposable	No	No	No	Underestimates in high-d
Rescaled IF (RIF)	Decomposable	Yes	No	No	Needs leverage computation
Integrated Influence	Decomposable	Yes	No	No	Heavier computation (path)
Bayesian IF (BIF)	Decomposable	Yes	Yes	No	SGLD sampling overhead
MAGIC/Metagradient	Decomposable	Yes	No	No	Replay per test point
Versatile IF (VIF)	Any	Yes	No	No	Needs L(θ,b) definition

These methods, through first-order theory, efficient algorithmics, and careful empirical validation, enable tracing the predictive logic of complex learned models back to their training data at scale. Ongoing advancements continue to push data attribution toward greater scalability, fidelity, and interpretability across domains (Zhu et al., 10 Aug 2025, Rubinstein et al., 7 Jun 2025, Ilyas et al., 23 Apr 2025, Lee et al., 14 Oct 2025, Deng et al., 2 Dec 2024, Yang et al., 7 Aug 2025).