---
title: Influence-Based Data Valuation
url: https://www.emergentmind.com/topics/influence-based-data-valuation
type: topic
---

# Influence-Based Data Valuation

Influence-based data valuation quantifies the utility or impact of each data instance in supervised learning, typically by measuring the marginal effect of individual data on a model’s predictions, generalization, parameter trajectory, or other outcomes of interest. This paradigm synthesizes tools from statistical influence functions, cooperative game theory (Shapley value), information geometry, and modern large-scale optimization, aiming to assign per-example (or per-batch/client) value scores. These scores underlie core tasks such as pruning, noise detection, data market pricing, federated aggregation, debugging, and privacy-preserving data exchange.

## 1. Conceptual Foundations of Influence-Based Data Valuation

Central to influence-based data valuation is the principle that datasets are not homogeneous in their contribution to model training or downstream utility. Data instances vary in their informativeness, redundancy, noisiness, adversarial impact, or even adversarial potential. Influence quantifies this heterogeneity using a variety of mathematical and operational definitions.

Classic influence functions, grounded in robust statistics, measure the effect of upweighting or removing a data point on the learned parameter $\theta^*$ or the downstream test loss. For empirical risk minimization $\theta^* = \arg\min_\theta \frac{1}{n}\sum_{i=1}^n L(z_i, \theta)$, the influence of training point $z$ on a held-out point $z_{\text{test}}$ is
\[
I(z, z_{\text{test}}) = -\nabla_\theta L(z_{\text{test}}, \theta^*)^\top H^{-1}_{\theta^*} \nabla_\theta L(z, \theta^*)
\]
where $H_{\theta^*}$ is the Hessian of the empirical loss at $\theta^*$. This “first-order” influence can be seen as a local linearization of the leave-one-out retraining effect, and is central to much of the recent literature [2405.13954], [2512.06033], [2503.01052], [2502.09969], [2012.06430].

Extensions and alternatives include leave-one-out complexity measures (e.g., the complexity-gap score [2301.00930]), geometric measures such as leverage scores [2511.02100], linearized future influence kernels [2503.01052], and distributional influence via MMD-based functionals [2506.23799].

## 2. Core Methodological Variants and Mathematical Formulation

Influence-based data valuation frameworks diverge in their choice of utility function, level of model access/assumptions, and computational tractability. Key methods include:

- **Classic Influence Functions**: Rely on the Taylor expansion of the parameter optimum and test loss under infinitesimal data perturbation [2405.13954], [2012.06430], [2502.09969]. These typically require (approximate) Hessian inversion.

- **Gradient-dot-product Variants**: Replace costly second-order terms by first-order approximations; e.g., TracIn, LinFiK, and the LoGra projection use
\[
\mathcal{I}(z, z_{\mathrm{te}}) \approx \nabla_\theta \ell(z_{\mathrm{te}})^\top \nabla_\theta \ell(z)
\]
with projections to reduce computational overhead [2503.01052], [2508.10180], [2405.13954].

- **Complexity-based Scores**: The complexity-gap (CG) score compares leave-one-out NTK-based complexity [2301.00930]:
\[
\mathrm{CG}(i) = y^\top \left(H^{(0)}_{-i}\right)^{-1} y_{-i} - y^\top \left(H^{(0)}\right)^{-1} y
\]
where $H^{(0)}$ is the NTK Gram matrix.

- **Kernel-based Distributional Influence**: KAIROS computes a distributional influence functional for each $x_i$ using the MMD between noisy training and a clean reference set [2506.23799]:
\[
\hat{I}(x_i) = \frac{1}{M}\sum_{j=1}^M k(x_j^\text{val}, x_i) - \frac{1}{N-1}\sum_{j \neq i} k(x_j, x_i)
\]

- **Federated/Client Influence**: In cross-silo or cross-device FL, FedIF and related methods quantify the contribution of each client’s update by its alignment with a public validation gradient [2509.25560].

A summary table of core paradigms:

| Method               | Valuation Principle                        | Main Computation                                       |
|----------------------|--------------------------------------------|--------------------------------------------------------|
| Influence Function   | First-order test loss change               | $\nabla L^\top H^{-1} \nabla L$                        |
| LoGra/TracIn/LinFiK  | Gradient inner product, projected space    | $(Pg_{te})^\top (PH P^\top)^{-1} Pg$                   |
| Complexity-Gap (CG)  | Leave-one-out NTK complexity increase      | Closed-form Schur complement formula                   |
| KAIROS               | Kernel-mean discrepancy (distributional)   | Kernel mean differences (no models needed)             |
| Leverage Score       | Span increase in feature space             | $x_i^\top (X^T X + \lambda I)^{-1} x_i$                |
| For-Value            | Hidden-representation + error alignment    | Forward pass only, inner products over embeddings      |

## 3. Computational Strategies and Scaling Solutions

Influence-based approaches historically suffered from prohibitive cost: either full Hessian inversion for $d \gg 10^6$, $n \gg 10^6$, or repeated retraining (leave-one-out, Shapley value) for combinatorial $n$.

Recent solutions address these bottlenecks:
- **Low-rank/Structured Projection**: LoGra [2405.13954], TIP [2512.06033], and other scalable variants exploit model structure, storing only projected per-example gradients (e.g., via LoRA adapters). This reduces both memory footprint and compute from $O(nd)$ to $O(nk)$ ($k \ll d$), with empirical throughput gains up to 6,500$\times$ at LLM scale.
- **Forward-only Algorithms**: For-Value [2508.10180] eliminates backprop entirely by re-expressing influence as weighted hidden-state inner products and prediction-error covariation, broadening feasibility to models with restricted access or frozen weights.
- **Neural Approximation/Distillation**: Tiny neural networks (NN-CIFT [2502.09969], ALinFiK [2503.01052]) are trained to regress or classify influence values over large pools, using a small sampling of true influence values for supervision, yielding $10^2$--$10^3\times$ speedups at minimal accuracy cost.
- **Closed-form/Streaming Updates**: KAIROS [2506.23799] achieves $O(N^2)$ batch computation but supports $O(mN)$ online increments when new data arrive, with $\mathcal{O}(1/N^2)$ ranking errors.
- **Encrypted/Private Valuation**: The Trustworthy Influence Protocol (TIP) [2512.06033], as well as MPC-based forward influence [2012.06430], implement the full first-order pipeline under homomorphic encryption or secret sharing, enabling secure third-party appraisal and data sale.

## 4. Empirical Performance, Use Cases, and Limitations

A diverse set of empirical results uniformly demonstrates that influence-based scores strongly correlate with the true impact of data on test loss, robustness, or adversarial outcomes:

- **Pruning and Compression**: CG and influence-based methods enable removal of up to 40% of training samples (CIFAR-10) with $<1\%$ performance drop; inverse pruning of high-influence samples severely degrades performance [2301.00930], [2405.13954].
- **Noise/Mislabeled Data Detection**: CG, For-Value, Diff-In, and KAIROS excel at identifying and ranking corrupted or anomalous points; precision of mislabeled point detection approaches 100% in several vision and NLP benchmarks [2301.00930], [2508.10180], [2508.14648], [2506.23799].
- **Data Market and Pricing**: Influence scores reveal heavy-tailed distributions of value (e.g., top 5% of books contributing most positive utility in GPT-2 pretraining [2512.06033]), challenging flat-rate compensation in data markets.
- **Federated Aggregation**: Influence-based scores enable robust aggregation in FL under noise/adversary conditions, outperforming Shapley-based schemes (FedIF achieves 450$\times$ aggregation speedup [2509.25560]).
- **Active Learning and Subset Selection**: Leverage scores, KAIROS, and influence-based coreset methods consistently exceed random or heuristic batch selection in downstream accuracy, both in convex and neural settings [2511.02100], [2506.23799].

Limitations arise in non-convexity (some methods assume convex losses or two-layer NTK structure, e.g., [2301.00930]), approximation error under high-order or long-horizon dynamics (addressed by Diff-In [2508.14648]), and cost of full-gradient or kernel computation in extremely high-dimensional or massive data regimes.

## 5. Model-Agnostic and Geometric Alternatives

A growing body of work emphasizes valuation without model gradients or even explicit model fitting:

- **Geometric Leverage Scores**: These assign each sample a share of the dataset’s span or effective dimension in feature space, satisfying dummy, efficiency, and symmetry axioms of Shapley valuation. Ridge-leverage sampling provides $O(\varepsilon)$ guarantees on parameter and risk closeness to full retraining [2511.02100].
- **Kernel-MMD and Conditional MMD**: KAIROS leverages the MMD (and MCMD) functional between a noisy distribution and a clean reference set, yielding closed-form influence scores without model training or gradients. These admit strong theoretical guarantees (symmetry, density separation) and extend naturally to label-conditional shifts [2506.23799].
- **Nearest-Neighbor Shapley**: For KNN models, soft-label KNN-SV computes data Shapley values efficiently with closed-form recursions, and locality-sensitive hashing enables sublinear approximate solutions at scale [2304.04258].

## 6. Privacy-Preserving and Federated Settings

Influence-based valuation methods are increasingly deployed under privacy constraints:

- **Homomorphic Encryption and MPC**: TIP [2512.06033] and forward-influence MPC [2012.06430] show that high-fidelity influence scores can be computed across buyer-seller boundaries in encrypted form, maintaining privacy while enabling trusted transactions.
- **Federated Influence**: In federated learning, client contributions are measured by the alignment of local update directions and a public reference gradient, combined with normalization and smoothing to resist adversarial and noisy participant updates [2509.25560].

## 7. Theoretical Guarantees, Axiomatic Properties, and Open Directions

Recent research grounds influence-based scores in formal bounds or axiomatic guarantees:

- **Approximation to Leave-One-Out Utility**: KAIROS and geometric leverage scores approximate the true leave-one-out ranking within $\mathcal{O}(1/N^2)$, and achieve fair data pricing via axioms inherited from Shapley theory [2506.23799], [2511.02100].
- **Tight Loss Change Bounds**: Influence-weighted federated averaging yields provably tighter upper bounds on one-step global loss change compared to uniform aggregation (FedAvg), under Lipschitz and bounded-dissimilarity assumptions [2509.25560].
- **Consistency and Robustness**: Diff-In provides a second-order influence approximation with bounded, polynomial error in the number of steps, outperforming first-order dynamic approximations under non-convex training dynamics [2508.14648].
- **Open Challenges**: Strong theoretical guarantees in deep, highly non-convex regimes (beyond the two-layer NTK or locally-linear settings) remain a frontier, as do extensions to batch/group influence, continual model updates, robust valuation under distribution shift, and complex privacy/adversary models [2301.00930], [2512.06033].

In summary, influence-based data valuation now encompasses a spectrum of theoretically grounded, computationally efficient, and empirically validated methods tailored for the demands of large-scale, privacy-sensitive, and heterogeneous ML and data markets.

Source: https://www.emergentmind.com/topics/influence-based-data-valuation