Loss-Relative Influence Overview

• Loss-Relative Influence is a set of methods that quantify sample impact by measuring how training perturbations alter a model’s loss relative to a baseline.
• It encompasses techniques using gradients, Hessians, and loss trajectories to evaluate local sensitivity and distinguish between global and local influence.
• Applications include data cleaning, model auditing, and targeted fine-tuning, while challenges remain in handling non-convexity, noise, and computational complexity.

Loss-Relative Influence is a family of technical notions that quantify influence through changes in loss, or through a normalization relative to a loss-derived baseline. In contemporary data-attribution work, it most directly means how the presence, removal, or infinitesimal reweighting of a training example changes the model’s loss on a target example; in other strands of the literature, related constructions measure the benefit of side information as Bayes-risk reduction, the fraction of fine-scale information eliminated by a deterministic map, or the excess loss of an online estimator relative to an offline optimum (Kokhlikyan et al., 13 Oct 2025, Jiao et al., 2014, Geiger et al., 2012, Azoury et al., 2013). Across the cited literature, the phrase is therefore not a single canonical formula but a perspective: influence is expressed relative to loss, self-influence, global model effects, or a divergence-controlled loss baseline.

1. Loss-change as the core attribution quantity

In the influence-function tradition, the central object is the first-order change in a target loss induced by perturbing the training weight of a data point. With empirical risk minimizer $\hat{\theta}$, Hessian $H_{\hat{\theta}}$, training point $z_p$, and target point $z_t$, the canonical loss influence is

$$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$

The associated parameter perturbation under upweighting is

$$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$

and approximate deletion satisfies

$$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$

This gives the usual sign interpretation: if $I_{loss}>0$, removing $z_p$ decreases the loss at $z_t$; if $H_{\hat{\theta}}$0, removing $H_{\hat{\theta}}$1 increases it (Kounavis et al., 2023).

This loss-change viewpoint is also the basis for later formulations of memorization and generalization. One line of work defines a point as $H_{\hat{\theta}}$2-memorizable when $H_{\hat{\theta}}$3, and $H_{\hat{\theta}}$4-generalizable when the same quantity is at most $H_{\hat{\theta}}$5. In a simple single-layer logistic setting with an “almost positive” inverse Hessian and nonnegative inputs, the sign of the loss influence aligns with label match or mismatch: same-label points tend to have negative influence and opposite-label points tend to have positive influence. This situates loss-relative influence as both a local explanation of prediction sensitivity and a diagnostic of how strongly a sample is encoded by the trained model (Kounavis et al., 2023).

2. Relative normalization, explanatory selection, and influence-aware training

A prominent use of the adjective “relative” appears in RelatIF. There, the abstract states that the method separates global versus local influence, introduces “a new class of criteria for choosing relevant training examples by way of an optimization objective that places a constraint on global influence,” and considers “the local influence that an explanatory example has on a prediction relative to its global effects on the model.” The reported empirical outcome is that the returned examples are “more intuitive” than those found using influence functions (Barshan et al., 2020).

Several later formulations make this relativity explicit in different ways.

Setting	Relative quantity	Role
RelatIF	local influence relative to global effects	explanatory example selection
Subset-Hessian relative influence	replace $H_{\hat{\theta}}$6 by $H_{\hat{\theta}}$7	scalable approximation
DeepRx	divide by $H_{\hat{\theta}}$8	comparable scores across points
IB-Loss	divide loss by an influence proxy	down-weight high-influence samples

In the subset-Hessian formulation, the relative Hessian for a subset $H_{\hat{\theta}}$9 is

$z_p$0

and the corresponding Loss-Relative Influence is

$z_p$1

Under the paper’s Loss Estimate Preserving condition, $z_p$2 is a uniform scaling of the full $z_p$3, so ranking and sign are preserved even though the Hessian is computed on a subset (Kounavis et al., 2023).

A distinct normalization is used for targeted adaptation of DNN-based wireless receivers. For classical matched-loss influence, the paper defines

$z_p$4

and gives the analogous cross-loss form when $z_p$5. Negative values indicate beneficial training points, while positive values indicate harmful points. This normalization is used to rank samples for targeted fine-tuning of DeepRx, where capacity-like binary cross-entropy and first-order updates on beneficial samples were reported to reduce the relative BER gap more consistently than random fine-tuning in single-target scenarios (Tuononen et al., 19 Sep 2025).

Influence can also enter the training objective itself. Influence-Balanced Loss starts from the classical influence-function precursor

$z_p$6

drops $z_p$7, and uses the proxy

$z_p$8

The resulting loss is

$z_p$9

optionally combined with class-wise weights $z_t$0. In the paper’s interpretation, this reduces the contribution of boundary-overfitting samples, especially majority-class hard examples in imbalanced classification (Park et al., 2021).

3. Loss-only and trajectory-based estimators

A more recent development replaces gradients and inverse Hessians by checkpointed loss trajectories. Z0-Inf defines the ideal loss-relative influence of a training point $z_t$1 on a target example $z_t$2 as

$z_t$3

This is a remove-and-retrain quantity: it measures how including $z_t$4 changes loss at $z_t$5. The paper then estimates influence from losses observed at intermediate checkpoints, without backpropagation or Hessian inversion. For checkpoint $z_t$6 and neighborhood $z_t$7,

$z_t$8

and train-test influence is aggregated as

$z_t$9

Self-influence is the special case $$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$0, with a practical proxy

$$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$1

The method assumes nearby checkpoints support a local linear approximation, emphasizes early checkpoints for LLMs, and reports higher Spearman correlations with SSRT than TracIn and Logra for fine-tuned LLM self-influence, together with order-of-magnitude savings in time and memory (Kokhlikyan et al., 13 Oct 2025).

Diff-In also uses the training trajectory, but through accumulated differences in influence between consecutive learning steps. Its practical loss-relative score for target set $$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$2 is

$$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$3

with

$$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$4

Hessian-vector products are approximated by finite differences of first-order gradients, so the method is second-order but remains comparable in cost to first-order baselines. The paper states that Diff-In avoids the convexity assumptions used by classical influence approximations, achieves lower theoretical approximation error than existing estimators, and performs strongly on data cleaning, data deletion, coreset selection, and large-scale vision-language pre-training (Tan et al., 20 Aug 2025).

Taken together, these methods shift loss-relative influence from a static optimum-based perturbation to a trajectory-grounded estimator. This suggests a broad methodological split: one branch linearizes around $$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$5, while the other treats the training path itself as the empirical substrate on which influence is observed.

4. Cross-loss, non-decomposable objectives, and sequential influence representations

Loss-relative influence is not restricted to matched supervised losses. Cross-loss influence replaces the test loss in the classical formula by an arbitrary differentiable evaluation objective. With training point $$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$6, test point $$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$7, training objective $$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$8, and evaluation loss $$I_{loss}(\hat{\theta}, z_t, z_p) = - \nabla_\theta \mathcal{L}(z_t,\hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_\theta \mathcal{L}(z_p,\hat{\theta}).$$9,

$$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$0

and for a total evaluation objective $$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$1,

$$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$2

This makes influence available in unsupervised and semi-supervised settings, and the paper uses it to explain cluster memberships, nearest-neighbor structure in embeddings, and WEAT bias in LLMs (Silva et al., 2020).

For objectives that are non-decomposable, the Versatile Influence Function introduces per-object weights $$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$3 into the coupled loss $$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$4 and differentiates the stationarity condition. The parameter influence is

$$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$5

and for any differentiable target functional $$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$6,

$$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$7

A finite-difference VIF is also given, based on the gradient difference between the full objective and the objective with object $$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$8 removed. The paper applies this machinery to Cox regression, node embedding, and listwise learning-to-rank, and reports close agreement with brute-force leave-one-out retraining while being up to $$I_{up}(\hat{\theta}, z_p) = - H_{\hat{\theta}}^{-1}\nabla_\theta \mathcal{L}(z_p,\hat{\theta}),$$9 times faster (Deng et al., 2024).

In sequential decision making, the term acquires yet another form. Influence-based abstraction summarizes the rest of the system through conditional distributions $$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$0 that affect the local model. The resulting value-loss bounds quantify how approximation error in influence propagates to control performance:

$$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$1

and, via Pinsker,

$$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$2

Because the approximation $$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$3 is trained with cross-entropy, the paper argues that minimizing expected KL divergence aligns with minimizing an upper bound on value loss (Congeduti et al., 2020).

5. Risk reduction, relative information loss, and online relative loss bounds

Outside data attribution, loss-relative influence appears as a measure of how side information changes optimal predictive risk. For a loss $$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$4 and side information $$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$5, the relevance measure is

$$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$6

Under logarithmic loss,

$$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$7

so

$$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$8

The paper’s main theorem states that when $$\mathcal{L}(z_t,\hat{\theta}_{-1/n,z_p})-\mathcal{L}(z_t,\hat{\theta}) = -\frac{1}{n}I_{loss}(\hat{\theta}, z_t, z_p).$$9 and the Data Processing Axiom holds, the relevance measure is uniquely determined by mutual information, up to a multiplicative factor. In the binary case, admissible forms are instead characterized by symmetric convex envelopes on $I_{loss}>0$0 (Jiao et al., 2014).

A different but related construction is relative information loss for deterministic systems. With $I_{loss}>0$1,

$I_{loss}>0$2

The central theorem gives

$I_{loss}>0$3

where $I_{loss}>0$4 is Rényi information dimension. This makes $I_{loss}>0$5 a fraction of fine-scale uncertainty removed by the map $I_{loss}>0$6. For continuous inputs passed through quantizers, the paper states $I_{loss}>0$7; for mappings that are constant on a set $I_{loss}>0$8 of positive measure and piecewise bijective elsewhere, $I_{loss}>0$9. A Fano-type bound further gives

$z_p$0

linking relative information loss to unavoidable reconstruction error (Geiger et al., 2012).

In online density estimation for exponential families, “relative loss” refers to the excess cumulative negative log-likelihood of the online algorithm over an offline optimum. With expectation parameter $z_p$1, the update is

$z_p$2

and the per-round regret decomposes through the Bregman divergence generated by $z_p$3. The key identity is

$z_p$4

which telescopes across rounds. With $z_p$5, the paper derives logarithmic relative loss bounds for arbitrary sequences, including explicit Bernoulli and Gaussian bounds (Azoury et al., 2013).

These formulations are not identical, but they share a common template: influence is measured by how loss, risk, or recoverable information changes under a constrained perturbation.

6. Empirical roles, practical uses, and recurrent limitations

Loss-relative influence is used operationally for explanatory example retrieval, data selection and pruning, outlier and mislabel detection, memorization diagnostics, attribution and auditing, bias analysis, coreset construction, and targeted adaptation. Z0-Inf recommends the variance proxy for fast self-influence and uses train-test ranking for auditing; Diff-In is evaluated on data cleaning, data deletion, and coreset selection; and the subset-Hessian relative-influence paper explicitly proposes using sign and magnitude to keep memorizable points, sample among generalizable ones, and remove positively influential points for targeted test examples (Kokhlikyan et al., 13 Oct 2025, Tan et al., 20 Aug 2025, Kounavis et al., 2023).

Cross-loss influence broadens these uses to mismatched objectives. The cited applications include explaining cluster memberships in unsupervised training, identifying semantically reinforcing training summaries for embedding neighborhoods, surfacing training sentences that amplify or mitigate WEAT bias, and then debiasing by fine-tuning on mitigating samples or counteracting amplifying ones (Silva et al., 2020). In imbalanced visual classification, influence enters directly into the training loss through inverse gradient-magnitude weighting, and the reported effect is a smoother, less biased decision boundary for minority classes (Park et al., 2021). In DNN-based receivers, loss-relative ranking of beneficial samples supports targeted fine-tuning of poorly performing cases and improves BER toward genie-aided performance in single-target scenarios, although multi-target adaptation is reported as less effective (Tuononen et al., 19 Sep 2025).

Several limitations recur. Classical influence methods require differentiability, an invertible or regularized Hessian, and local linearization around the optimum; non-convexity, indefinite curvature, and damping sensitivity remain practical issues. Train-test influence is repeatedly reported as noisier than self-influence. Trajectory-based methods depend on checkpoint density and proximity: if checkpoints are too far apart, local linearity degrades. Group effects can dilute individual influence when many similar training examples exist. Influence estimates can also become stale after adaptation steps, as observed in receiver fine-tuning, and early-stage influence estimates can be noisy in influence-balanced training (Kokhlikyan et al., 13 Oct 2025, Tan et al., 20 Aug 2025, Tuononen et al., 19 Sep 2025, Park et al., 2021).

The cumulative picture is technically cohesive despite definitional variety. In all of its major arXiv forms, Loss-Relative Influence treats a perturbation as meaningful only through its effect on a loss-like object: a prediction loss, a self-loss baseline, a global-versus-local explanation criterion, a value function, a Bayes risk, a fine-scale information measure, or an online excess-loss decomposition. That shared structure explains why the term appears across robust statistics, interpretable machine learning, information theory, online learning, and sequential decision making without collapsing to a single universal formula.