Influence Function Projection Approach

• Influence Function Projection Approach is a framework that reduces inverse-curvature computation via methods like blockwise subspace, dual reformulation, Kronecker factorization, or random sketching.
• It employs various paradigms to approximate the inverse Hessian efficiently, thereby mitigating high computational and memory costs in influence analysis.
• These techniques enhance practical influence estimation across domains such as language models, diffusion models, and data selection tasks by preserving key attribution directions.

As used here, “Influence Function Projection Approach” functions as an umbrella description for methods that preserve the classical influence-function objective while replacing the full inverse-curvature computation by a reduced operator, such as a blockwise subspace, a Jacobian-span dual representation, a Kronecker factorization, or a random sketch. In its standard form, influence evaluation couples a query gradient with an inverse Hessian or related curvature inverse; the projection step changes how that inverse action is represented, not the underlying attribution objective itself (Zhou et al., 2024). The literature also uses the word projection in other, non-equivalent senses—directional projection in robust regression, line-of-sight projection in cosmology, or orthogonal projection of nuisance directions in nonparametric testing—so the phrase denotes a family resemblance rather than a single canonical algorithm (Zuo, 2018).

1. Classical influence formulation and the source of the bottleneck

Influence functions start from an empirical-risk minimizer

$$R(\theta)\coloneqq \frac{1}{n}\sum_{i=1}^n \ell(y_i,f_\theta(x_i)),\qquad \theta^\star=\arg\min_{\theta} R(\theta).$$

If the $k$-th training point is infinitesimally upweighted, the parameter perturbation is

$$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$

with

$H(\theta)=\nabla_\theta^2 R(\theta).$

For a validation or test query, the standard bilinear score is

$$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$

where $v$ is an average validation gradient (Zhou et al., 2024).

The same inverse-curvature structure appears in deletion and retraining approximations. In a dual linearized formulation, removing a subset $\mathcal D_f$ yields

$$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$

and predicted changes in outputs or losses follow by first-order Taylor expansion around $\theta^\star$ (Sun et al., 11 May 2026). In diffusion models, the generic influence form is likewise

$$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$

with $k$0 a query-specific measurement functional (Mlodozeniec et al., 2024).

The computational obstruction is therefore stable across domains: one needs either a full inverse Hessian or a sequence of inverse-Hessian-vector products for a matrix whose dimension is the parameter count. This motivates projection-like reductions. One abstracted line of work explicitly speeds up inverse-Hessian calculation by Arnoldi iteration and reports scalability to full-size language and vision Transformer models with several hundreds of millions of parameters; this suggests a Krylov-subspace interpretation, although the supplied material does not include the paper’s method section needed for a full reconstruction of that reduction (Schioppa et al., 2021).

2. Principal reduction paradigms

One prominent family replaces the flattened Hessian by a structured blockwise subspace. HyperINF assumes the Hessian is approximately block diagonal across model blocks and then, for a matrix-shaped gradient $k$1, uses a generalized Fisher information matrix

$k$2

together with the Kronecker surrogate

$k$3

This reduces the inverse action on an $k$4-dimensional flattened gradient to a $k$5 inverse shared across the $k$6 LoRA directions, yielding the practical score

$k$7

The corresponding memory drops from $k$8 for the flattened Hessian to $k$9 for the GFIM, and the Hessian-gradient product cost drops from $$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$0 to $$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$1; for LoRA rank $$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$2, the paper states that HyperINF needs only $$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$3 of the memory and $$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$4 of the computations of the original flattened-form Hessian-vector product (Zhou et al., 2024).

A second family performs a dual reformulation in the Jacobian span. For linearized models, deletion-induced parameter change is shown to lie in the affine space

$$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$5

so one writes

$$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$6

and solves the influence problem entirely in the dual coefficients $$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$7, equivalently through an NTK/Gram matrix. This shifts complexity from model size to the product of dataset size and output dimension. The representation is exact for the explicitly linearized model, but the paper states that it is limited to linearizable models and requires materializing a matrix whose size grows with the product of model output dimension and dataset size (Sun et al., 11 May 2026).

A third family uses random sketching before inversion. Here the true regularized influence bilinear form

$$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$8

is approximated by

$$\mathcal I_{\theta^\star}(x_k,y_k)= -H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$9

where $H(\theta)=\nabla_\theta^2 R(\theta).$0 is a sketch. The theoretical result is sharper than a Johnson–Lindenstrauss argument: in the unregularized case, exact preservation on $H(\theta)=\nabla_\theta^2 R(\theta).$1 holds iff $H(\theta)=\nabla_\theta^2 R(\theta).$2 is injective on $H(\theta)=\nabla_\theta^2 R(\theta).$3, which requires

$H(\theta)=\nabla_\theta^2 R(\theta).$4

With ridge regularization, the barrier changes to the effective dimension

$H(\theta)=\nabla_\theta^2 R(\theta).$5

and approximation guarantees scale with $H(\theta)=\nabla_\theta^2 R(\theta).$6 rather than $H(\theta)=\nabla_\theta^2 R(\theta).$7. The same perspective extends to Kronecker-factored curvatures $H(\theta)=\nabla_\theta^2 R(\theta).$8 with factorized sketches $H(\theta)=\nabla_\theta^2 R(\theta).$9, and the paper also quantifies a leakage term when test gradients have components in $$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$0 (Hu et al., 11 Feb 2026).

A fourth family keeps full parameter gradients but projects the curvature geometry into a structured Kronecker form. For diffusion models, influence evaluation is built on a diffusion-specific generalized Gauss–Newton matrix, then approximated by K-FAC. The practical score becomes

$$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$1

This is not a low-rank projector in the narrow sense, but it is a projection-like restriction of inverse-curvature action to block-diagonal, Kronecker-factored geometry (Mlodozeniec et al., 2024).

3. Algorithmic mechanics of projected influence computation

Once curvature has been compressed, the inverse is usually computed by iterative or low-rank linear algebra rather than exact elimination. HyperINF uses the hyperpower family, specialized to Schulz iteration. For $$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$2, the general iteration

$$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$3

is instantiated with $$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$4, giving

$$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$5

The paper reports practical convergence from $$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$6 and states that starting from a small identity matrix or random Gaussian initialization could converge to a desirable error rate in finite steps $$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$7. In its complexity table, HyperINF has inverse computation cost $$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$8, Hessian-inverse-vector cost $$\mathcal I(x_k,y_k)= -v^\top H(\theta^\star)^{-1}\nabla_\theta \ell_k,$$9, and memory $v$0, whereas exact Gaussian elimination and LiSSA retain $v$1-scale memory in LoRA-tuned settings (Zhou et al., 2024).

Low-rank eigenspace methods provide a more literal projection strategy. In a convex M-estimation setting, if

$v$2

the rank-$v$3 inverse approximation keeps only the top eigendirections: $v$4 The paper derives

$v$5

and further shows polynomial or exponential decay when the Hessian spectrum itself decays polynomially or exponentially. Empirically, low-rank Arnoldi had the smallest error for question answering when $v$6 and matched other methods at larger $v$7 (Fisher et al., 2022).

Krylov methods occupy an intermediate position between exact inversion and explicit projection. Conjugate gradient solves the inverse-Hessian system in the nested Krylov spaces generated by $v$8 and the residual. The paper gives the standard rate

$v$9

so CG is best understood as an implicit subspace-projection method whose subspace is not fixed a priori but grows adaptively with iterations (Fisher et al., 2022).

The random-sketch theory turns these algorithmic choices into design rules. Without regularization, sketch dimension must reach the rank barrier. With regularization, one should choose sketch size according to $\mathcal D_f$0; in the factorized case, the relevant sizes are governed by $\mathcal D_f$1 and $\mathcal D_f$2, not by ambient dimensions alone (Hu et al., 11 Feb 2026).

4. Projection across model structure and across training stages

Influence projection is not confined to single-stage Hessian compression. In multi-stage influence functions, the perturbation induced by a pretraining datum is first computed in the full pretraining parameter space $\mathcal D_f$3, then restricted to the shared downstream block $\mathcal D_f$4, then propagated into finetuning parameters through a cross-Hessian transfer operator. With $\mathcal D_f$5 fixed during finetuning, the downstream response is

$\mathcal D_f$6

This is not presented as a projector in the narrow linear-algebraic sense, but it does implement a coordinate restriction onto the transferable parameter subspace $\mathcal D_f$7, followed by cross-stage transport into the finetuned model (Chen et al., 2020).

In sequential recommendation attacks, projection appears as a target-direction readout rather than a curvature reduction. INFAttack scores the replacement of a training sequence $\mathcal D_f$8 by a polluted sequence $\mathcal D_f$9 through

$$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$0

The paper states that its method is not explicitly framed as a projection approach, but the score is naturally read as the inner product between a target-item attack direction and a curvature-preconditioned candidate-induced parameter displacement (Du et al., 2024).

In LLM behavior correction, LANCET combines influence-based recall with a Bregman-regularized post-training objective. Its scalable LinFAC recall stage approximates a Transformer sublayer by a surrogate linear module and uses Kronecker factors

$$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$1

to approximate the local curvature. The subsequent Influence-driven Bregman Optimization minimizes

$$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$2

anchoring the corrected model to the reference policy while increasing or decreasing the probability of recalled training examples according to their signed influence. This is best described as a soft, reference-centered projection in output-probability and parameter space, rather than as an exact projector (Zhang et al., 2024).

5. Empirical roles and application domains

The projection family is motivated by scale, but its empirical role is broader than acceleration alone. HyperINF evaluates projected influence scores on mislabeled-data detection and data selection for LLM and VLM fine-tuning. On LoRA-tuned GLUE mislabeled-data detection with rank $$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$3, average recall at inspection $$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$4 is $$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$5 for HyperINF, compared with $$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$6 for DataInf, $$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$7 for LiSSA, and $$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$8 for TracIN; at $$\theta_r^\star \approx \theta^\star + \frac{|\mathcal D_f|}{|\mathcal D|}\, H_{\theta^\star}^{-1}\nabla_\theta \mathcal L_{\mathcal D_f}(\theta^\star),$$9, the corresponding numbers are $\theta^\star$0, $\theta^\star$1, $\theta^\star$2, and $\theta^\star$3. On LLM LoRA fine-tuning data selection with $\theta^\star$4 selected data, HyperINF’s average accuracy is $\theta^\star$5, above Random $\theta^\star$6, DataInf $\theta^\star$7, LiSSA $\theta^\star$8, and TracIN $\theta^\star$9 (Zhou et al., 2024).

In diffusion models, the structured-curvature route is tied to attribution fidelity rather than just runtime. K-FAC Influence with MC-Fisher / $$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$0 is reported to outperform TRAK in all settings considered, to outperform D-TRAK on the evaluated LDS settings using the loss measurement, and to identify training examples whose removal produces larger retraining effects in “retraining without top influences” experiments. The paper also reports that K-FAC-expand outperforms K-FAC-reduce and that the method is more robust to damping choice than TRAK and D-TRAK (Mlodozeniec et al., 2024).

Across training stages, multi-stage influence enables attribution from downstream predictions back to pretraining data. In the fixed-embedding case, the paper reports Pearson correlation $$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$1 on CIFAR-10 and $$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$2 on MNIST between predicted pretraining influence and actual downstream loss difference under retraining; in the embedding-updated case on CIFAR-10, the correlation is $$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$3. In a data-cleansing application, removing the top $$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$4 highest positive-influence pretraining examples improved downstream CIFAR-10 accuracy from $$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$5 to $$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$6, while random removal reduced it to $$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$7 (Chen et al., 2020).

In behavior correction, LANCET’s empirical message is that influence-guided recall and Bregman-regularized correction are complementary. The paper reports that LANCET effectively and efficiently corrects inappropriate behaviors of LLMs, can outperform methods that rely on collecting human preferences, and that the compatibility study favors LinFAC + IBO over EK-FAC + PBO and related combinations, indicating that both better influence approximation and the correction geometry matter (Zhang et al., 2024).

6. Alternative meanings of “projection,” common misconceptions, and limitations

A first source of confusion is terminological. In cluster cosmology, projection refers to line-of-sight superposition of halos, not to curvature reduction. There the central modeling move is a clean-plus-blended Gaussian mixture for the mass–observable relation,

$$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$8

introduced to capture the non-Gaussian high-observable tail created by blended systems (Erickson et al., 2011). In robust projection regression, projection means directional reduction of a multivariate regressor to one-dimensional projected regressions, leading to projection regression depth

$$\mathrm{Infl}(x_j\!\to\! x')=\nabla_\theta^\top m(\theta^\star,x')\,H^{-1}\nabla_\theta \ell(\theta^\star,x_j),$$9

and, for the deepest projection regression functional, a bounded influence function under the stated assumptions (Zuo, 2018). In nonparametric testing, projection means orthogonalizing the ICM weight $k$00 against a nuisance direction generated by $k$01, yielding a root-$k$02 asymptotically linear projected process rather than a scalable inverse-Hessian method (Song et al., 17 Feb 2026).

A second misconception is to equate practical influence estimation with exact leave-one-out retraining. The proximal Bregman response analysis shows that, in nonlinear neural networks, practical influence estimates may align poorly with cold-start leave-one-out retraining while still being good approximations to a different local object, the proximal Bregman response function. The paper’s decomposition attributes much of the mismatch to warm-start gap, proximity gap, and non-convergence gap, while linearization error and solver error are often much smaller (Bae et al., 2022).

A third misconception is that every projection method is a literal low-rank orthogonal projector. The supplied literature includes coordinate restriction to a shared parameter block, dual Jacobian-span reformulation, random sketches before inversion, block-diagonal Kronecker curvature, and nuisance orthogonalization. These are structurally related but mathematically distinct. The semiparametric literature makes this especially clear: influence functions are derived as Gateaux derivatives, and the efficient influence function is tied to an explicit projection operation in restricted semiparametric models, but that projection acts on score or tangent-space structure rather than on Hessian inverses (Ichimura et al., 2015).

The main limitations are method-specific. HyperINF’s theoretical support relies on negative log-likelihood loss, Fisher–Hessian equivalence in expectation, a blockwise diagonal Hessian approximation, and, in its central GFIM lemma, the assumption that columns of the matrix-form gradient are i.i.d. zero-mean under the model distribution (Zhou et al., 2024). The dual NTK formulation is limited to linearizable models and requires materializing a matrix whose size grows with output dimension times dataset size (Sun et al., 11 May 2026). Random-sketch theory identifies a leakage term when test gradients have components in $k$03, so even a regularized sketch can create artifacts for out-of-range queries (Hu et al., 11 Feb 2026). Diffusion attribution still lacks a universally satisfactory measurement functional for “probability of generating this sample,” even when curvature approximation is substantially improved (Mlodozeniec et al., 2024).

Taken together, these results support a broad but precise interpretation. An Influence Function Projection Approach is not a single algorithm but a class of constructions that keep the bilinear influence objective intact while replacing the full inverse-curvature problem by a reduced geometry. The reduction may be spectral, Jacobian-dual, Kronecker-factorized, randomly sketched, blockwise, or orthogonalized against nuisance directions. The unifying principle is that attribution is preserved only to the extent that the chosen reduced geometry preserves the inverse-sensitive directions that actually govern the influence score.