---
title: Influence Function Attribution
url: https://www.emergentmind.com/topics/influence-function-attribution
type: topic
---

# Influence Function Attribution

Influence function attribution is a principled methodology for quantifying how individual training points affect a model's predictions by tracing sensitivity through the learning process. Originating in robust statistics, influence functions provide a first-order, computationally scalable approximation to the effect of upweighting (or removing) a training sample on learned parameters and downstream predictions. Influence function attribution has become foundational across interpretability, data debugging, model accountability, and data-centric algorithmic research due to these properties and its generalizability to a wide variety of domains and learning paradigms.

## 1. Mathematical Definition and Core Formulation

Let $Z = \{z_1, \ldots, z_n\}$ denote the training set, with each $z_i = (x_i, y_i)$, and let $L(z, \theta)$ be the per-sample loss function. The empirical risk minimizer is
\[
\hat \theta = \arg \min_\theta \frac{1}{n} \sum_{i=1}^n L(z_i, \theta)
\]
and the Hessian at $\hat\theta$ is $H_{\hat\theta} = \frac{1}{n} \sum_i \nabla^2_\theta L(z_i, \hat\theta)$.

The influence of upweighting a training sample $z$ by $\epsilon$ is given by
\[
\hat\theta_{\epsilon, z} = \arg\min_\theta \left[ \frac{1}{n} \sum_i L(z_i, \theta) + \epsilon L(z, \theta) \right]
\]
Differentiating at $\epsilon = 0$,
\[
I_{\text{params}}(z) = \frac{d\hat\theta_{\epsilon, z}}{d\epsilon}\Big|_{\epsilon=0} = - H_{\hat\theta}^{-1} \nabla_\theta L(z, \hat\theta)
\]
(equation (1)), which represents the first-order change in parameters.

The influence of $z$ on the loss at a test point $z_{\text{test}}$ is
\[
I_{\text{loss}}(z, z_{\text{test}}) = \frac{d}{d\epsilon} L(z_{\text{test}}, \hat\theta_{\epsilon, z}) \Big|_{\epsilon=0} = - \nabla_\theta L(z_{\text{test}}, \hat\theta)^\top H_{\hat\theta}^{-1} \nabla_\theta L(z, \hat\theta)
\]
(equation (2)). Removing $z$ corresponds to $\epsilon = -1/n$, leading to $\hat\theta_{-z} - \hat\theta \approx -\frac{1}{n} I_{\text{params}}(z)$.

For non-convex or early-stopped models, $H$ is replaced by a damped Gauss–Newton matrix $G + \lambda I$ (guaranteeing positive-definiteness): the influence becomes
\[
I_{\text{loss}}(z, z_{\text{test}}) \approx - \nabla_\theta L(z_{\text{test}}, \theta)^\top (G + \lambda I)^{-1} \nabla_\theta L(z, \theta)
\]
(equation (3)) [2508.07297].

## 2. Theoretical Foundations and Assumptions

The influence function methodology rests on perturbation analysis of M-estimators in robust statistics (Cook, 1982; Hampel, 1974). Key steps involve:
- First-order Taylor expansion of the stationarity condition around the ERM solution, ignoring higher-order terms in $\epsilon$ and $\Delta\theta$.
- The main mathematical assumptions are:
  - Twice differentiable empirical risk, strictly convex in $\theta$ near $\hat\theta$ (positive-definite $H$).
  - Model reaches a global minimum.
  - Negligible higher-order terms (validity of first-order approximation).

Limitations and caveats:
- Deep neural networks are typically non-convex, and $H$ can be indefinite; $\hat\theta$ may not be a global optimum.
- The approximation can be inaccurate if the sample has very large influence.
- Stochastic optimization dynamics (SGD, multi-stage fine-tuning) introduce biases not reflected in the classical influence formalism [2508.07297].

## 3. Scalable Computation of Influence Scores

Direct computation of $H^{-1}v$ is prohibitive; scalable algorithms are essential:

- **LiSSA (Stochastic Neumann Series):** Iteratively estimates $(G+\lambda I)^{-1} v$ using Hessian-vector products (HVPs) from randomly-sampled mini-batches. Cost per step is equivalent to a gradient evaluation, but total compute scales with the number of iterations and HVPs needed for convergence.

- **EK-FAC (Eigendecomposed Kronecker-Factored Approximation):** Exploits layer-wise structure in neural networks, approximating the Gauss–Newton block as a Kronecker product of input activations and unitwise gradient factors, then eigendecomposing these small matrices. This allows blockwise inverse computations with significant memory and runtime gains, especially for large deep models [2508.07297].

Both methods avoid explicit formation of the parameter Hessian and can be applied with damping to guarantee stability.

## 4. Empirical Validation and Applications

### Influential-Point Identification

Influence function attribution correctly identifies training examples with strong impact on specific test predictions. On MNIST, FashionMNIST (simple CNN), Flowers102 (ResNet50), and Food101 (ViT-B/16), top positive/negative influential points exhibit clear human-interpretable visual similarity or inter-class confusion with the query [2508.07297].

### Mislabeled Example Detection

Self-influence, $SI(z) = I_{\text{loss}}(z, z)$, is highly effective for flagging mislabeled or ambiguous data. In simulated experiments with 10% random label corruption, self-influence sharply outperforms random baselines in retrieving truly mismatched labels for inspection budgets between 10%–50%. On raw MNIST, 16 of the top-20 high self-influence points were genuinely mislabeled or ambiguous [2508.07297].

### Linear Data-Modeling Score (LDS)

LDS, the Spearman correlation between the actual loss changes under leave-one-out retraining and those predicted via influence summation, serves as a quantitative benchmark. For MNIST, FashionMNIST, Flowers102, and Food101, LDS values are 0.50, 0.47, 0.46, 0.43, respectively [2508.07297].

## 5. Practical and Theoretical Challenges

Influence function attribution in large-scale deep networks confronts several algorithmic and theoretical obstacles:

- **Computational Cost:** Even approximate IHVPs (Hessian or Gauss–Newton) become prohibitive when deployed across thousands or millions of samples.
- **Non-Convexity and Indefinite Hessians:** Naïve influence scores can be unstable or misleading due to violated positive-definiteness.
- **First-Order Approximation Error:** Neglects higher-order and interaction terms; truncation (as in LiSSA) or blockwise (as in K-FAC/EK-FAC) approximations may deviate from true responses.
- **Optimizer Bias and Multi-Stage Training:** Optimizer dynamics (SGD, early stop, fine-tuning) are not captured by the standard IF formula; attribution accuracy can degrade under these regimes [2508.07297].

## 6. Emerging Directions and Future Work

Advances are addressing the fundamental and practical issues in influence-function attribution:

- **Hybrid Unrolling and Implicit Differentiation (e.g., SOURCE):** Combines unrolling-based TDA with influence-function formalisms to reduce optimizer bias and improve accuracy in non-convex settings.
- **Kernel Methods (TRAK, DataInf):** Neural tangent kernel and Fisher-kernel approaches provide scalable, checkpoint-efficient approximations for influence estimation.
- **Machine Unlearning:** Influence scores serve for approximate parameter updates in sample removal and label-repair without retraining, using the formula
  \[
  \theta_{\text{new}} = \theta + \frac{1}{n} \sum_{z \in D_{\text{forget}}} H_\theta^{-1} \nabla_\theta L(z, \theta)
  \]
  and analogous formulas for label repair [2508.07297].
- **Curvature Approximations:** More robust inverse-Hessian surrogates (damped GNH, EK-FAC, diagonal-plus-low-rank) are under development.
- **Extension to Generative and Diffusion Models:** Recent work brings scalable influence-function-based attribution to architectures such as diffusion models and large language models [2508.07297].

## References

- "Revisiting Data Attribution for Influence Functions" [2508.07297] (comprehensive review of the theory, scalable algorithms, and empirical evaluation).
- The EK-FAC and LiSSA algorithms and mislabel detection benchmarks reported in [2508.07297].

Source: https://www.emergentmind.com/topics/influence-function-attribution