Influence Function Attribution

• Influence function attribution is a methodology that quantifies the impact of each training sample on model predictions using first-order approximations.
• It employs Hessian-vector products and scalable algorithms like LiSSA and EK-FAC to efficiently approximate influence scores for data debugging and model accountability.
• Empirical results show its effectiveness in identifying influential training points, detecting mislabeled data, and benchmarking data-model correlations via metrics like LDS.

Influence function attribution is a principled methodology for quantifying how individual training points affect a model's predictions by tracing sensitivity through the learning process. Originating in robust statistics, influence functions provide a first-order, computationally scalable approximation to the effect of upweighting (or removing) a training sample on learned parameters and downstream predictions. Influence function attribution has become foundational across interpretability, data debugging, model accountability, and data-centric algorithmic research due to these properties and its generalizability to a wide variety of domains and learning paradigms.

1. Mathematical Definition and Core Formulation

Let $Z = \{z_1, \ldots, z_n\}$ denote the training set, with each $z_i = (x_i, y_i)$, and let $L(z, \theta)$ be the per-sample loss function. The empirical risk minimizer is

$$\hat \theta = \arg \min_\theta \frac{1}{n} \sum_{i=1}^n L(z_i, \theta)$$

and the Hessian at $\hat\theta$ is $$H_{\hat\theta} = \frac{1}{n} \sum_i \nabla^2_\theta L(z_i, \hat\theta)$$.

The influence of upweighting a training sample $z$ by $\epsilon$ is given by

$$\hat\theta_{\epsilon, z} = \arg\min_\theta \left[ \frac{1}{n} \sum_i L(z_i, \theta) + \epsilon L(z, \theta) \right]$$

Differentiating at $\epsilon = 0$,

$z_i = (x_i, y_i)$0

(equation (1)), which represents the first-order change in parameters.

The influence of $z_i = (x_i, y_i)$1 on the loss at a test point $z_i = (x_i, y_i)$2 is

$z_i = (x_i, y_i)$3

(equation (2)). Removing $z_i = (x_i, y_i)$4 corresponds to $z_i = (x_i, y_i)$5, leading to $z_i = (x_i, y_i)$6.

For non-convex or early-stopped models, $z_i = (x_i, y_i)$7 is replaced by a damped Gauss–Newton matrix $z_i = (x_i, y_i)$8 (guaranteeing positive-definiteness): the influence becomes

$z_i = (x_i, y_i)$9

(equation (3)) (Zhu et al., 10 Aug 2025).

2. Theoretical Foundations and Assumptions

The influence function methodology rests on perturbation analysis of M-estimators in robust statistics (Cook, 1982; Hampel, 1974). Key steps involve:

• First-order Taylor expansion of the stationarity condition around the ERM solution, ignoring higher-order terms in $L(z, \theta)$0 and $L(z, \theta)$1.
• The main mathematical assumptions are:
  • Twice differentiable empirical risk, strictly convex in $L(z, \theta)$2 near $L(z, \theta)$3 (positive-definite $L(z, \theta)$4).
  • Model reaches a global minimum.
  • Negligible higher-order terms (validity of first-order approximation).

Limitations and caveats:

• Deep neural networks are typically non-convex, and $L(z, \theta)$5 can be indefinite; $L(z, \theta)$6 may not be a global optimum.
• The approximation can be inaccurate if the sample has very large influence.
• Stochastic optimization dynamics (SGD, multi-stage fine-tuning) introduce biases not reflected in the classical influence formalism (Zhu et al., 10 Aug 2025).

3. Scalable Computation of Influence Scores

Direct computation of $L(z, \theta)$7 is prohibitive; scalable algorithms are essential:

• LiSSA (Stochastic Neumann Series): Iteratively estimates $L(z, \theta)$8 using Hessian-vector products (HVPs) from randomly-sampled mini-batches. Cost per step is equivalent to a gradient evaluation, but total compute scales with the number of iterations and HVPs needed for convergence.
• EK-FAC (Eigendecomposed Kronecker-Factored Approximation): Exploits layer-wise structure in neural networks, approximating the Gauss–Newton block as a Kronecker product of input activations and unitwise gradient factors, then eigendecomposing these small matrices. This allows blockwise inverse computations with significant memory and runtime gains, especially for large deep models (Zhu et al., 10 Aug 2025).

Both methods avoid explicit formation of the parameter Hessian and can be applied with damping to guarantee stability.

4. Empirical Validation and Applications

Influential-Point Identification

Influence function attribution correctly identifies training examples with strong impact on specific test predictions. On MNIST, FashionMNIST (simple CNN), Flowers102 (ResNet50), and Food101 (ViT-B/16), top positive/negative influential points exhibit clear human-interpretable visual similarity or inter-class confusion with the query (Zhu et al., 10 Aug 2025).

Mislabeled Example Detection

Self-influence, $L(z, \theta)$9, is highly effective for flagging mislabeled or ambiguous data. In simulated experiments with 10% random label corruption, self-influence sharply outperforms random baselines in retrieving truly mismatched labels for inspection budgets between 10%–50%. On raw MNIST, 16 of the top-20 high self-influence points were genuinely mislabeled or ambiguous (Zhu et al., 10 Aug 2025).

Linear Data-Modeling Score (LDS)

LDS, the Spearman correlation between the actual loss changes under leave-one-out retraining and those predicted via influence summation, serves as a quantitative benchmark. For MNIST, FashionMNIST, Flowers102, and Food101, LDS values are 0.50, 0.47, 0.46, 0.43, respectively (Zhu et al., 10 Aug 2025).

5. Practical and Theoretical Challenges

Influence function attribution in large-scale deep networks confronts several algorithmic and theoretical obstacles:

• Computational Cost: Even approximate IHVPs (Hessian or Gauss–Newton) become prohibitive when deployed across thousands or millions of samples.
• Non-Convexity and Indefinite Hessians: Naïve influence scores can be unstable or misleading due to violated positive-definiteness.
• First-Order Approximation Error: Neglects higher-order and interaction terms; truncation (as in LiSSA) or blockwise (as in K-FAC/EK-FAC) approximations may deviate from true responses.
• Optimizer Bias and Multi-Stage Training: Optimizer dynamics (SGD, early stop, fine-tuning) are not captured by the standard IF formula; attribution accuracy can degrade under these regimes (Zhu et al., 10 Aug 2025).

6. Emerging Directions and Future Work

Advances are addressing the fundamental and practical issues in influence-function attribution:

• Hybrid Unrolling and Implicit Differentiation (e.g., SOURCE): Combines unrolling-based TDA with influence-function formalisms to reduce optimizer bias and improve accuracy in non-convex settings.
• Kernel Methods (TRAK, DataInf): Neural tangent kernel and Fisher-kernel approaches provide scalable, checkpoint-efficient approximations for influence estimation.
• Machine Unlearning: Influence scores serve for approximate parameter updates in sample removal and label-repair without retraining, using the formula

$$\hat \theta = \arg \min_\theta \frac{1}{n} \sum_{i=1}^n L(z_i, \theta)$$0

and analogous formulas for label repair (Zhu et al., 10 Aug 2025).

• Curvature Approximations: More robust inverse-Hessian surrogates (damped GNH, EK-FAC, diagonal-plus-low-rank) are under development.
• Extension to Generative and Diffusion Models: Recent work brings scalable influence-function-based attribution to architectures such as diffusion models and LLMs (Zhu et al., 10 Aug 2025).

• "Revisiting Data Attribution for Influence Functions" (Zhu et al., 10 Aug 2025) (comprehensive review of the theory, scalable algorithms, and empirical evaluation).
• The EK-FAC and LiSSA algorithms and mislabel detection benchmarks reported in (Zhu et al., 10 Aug 2025).