Influence Approximation Unlearning (IAU)

• Influence Approximation Unlearning (IAU) is a set of techniques that remove specific training data influences using first-order and low-rank curvature approximations.
• IAU methods avoid full retraining by employing efficient gradient-based updates, achieving computational savings and speed-ups in various applications.
• IAU is applied in recommendation systems, computer vision, graph neural networks, and LLMs to ensure regulatory compliance and data integrity.

Influence Approximation Unlearning (IAU) refers to a body of algorithmic techniques that use (approximations to) influence functions to achieve efficient, accurate, and scalable machine unlearning—removing the effect of specific training data from learned models. IAU leverages first-order or low-rank curvature approximations to avoid the cost of full retraining, drawing on developments from classical statistical influence theory, scalable machine learning, and domain-specific sensitivity analysis. IAU methods are widely deployed in recommendation systems, computer vision, graph neural networks, and LLMs for regulatory compliance, privacy, data integrity, and preference alignment.

1. Influence Function Foundations

Influence functions quantify the sensitivity of a model's parameters $\theta^*$ to infinitesimal up-weightings of particular data points in the training set $D = \{z_i\}_{i=1}^N$. For empirical risk minimization problems,

$$\theta^* = \arg\min_\theta \frac{1}{N}\sum_{i=1}^N \ell(z_i, \theta)$$

the classical score measures (for a function $L$ and Hessian $H_{\theta^*}$ at the optimum) are

$$I(z_i, z_t) = \nabla_\theta L(z_t, \theta^*)^\top H_{\theta^*}^{-1} \nabla_\theta L(z_i, \theta^*)$$

as formulated for both parameter and prediction influence (Li et al., 2024, Wu et al., 2023). Removing a point $z_j$, or a subset $S$, effects a parameter shift that is, to first order in $\varepsilon = -1/N$,

$$\Delta\theta \approx -H_{\theta^*}^{-1} \sum_{i \in S} \nabla_\theta \ell(z_i, \theta^*)\,.$$

In IAU, computing or approximating this shift is the core operation; exact Hessian inversion is rarely tractable for deep models.

2. Scalable Influence Approximation Mechanisms

Two major threads define the evolution of IAU:

• First-order or Hessian-free gradient-based unlearning: Instead of directly solving $D = \{z_i\}_{i=1}^N$0, recent frameworks propose single or few steps of parameter update in the gradient direction of the forget set (Liu et al., 31 Jul 2025). The key observation is that, for most parameter regimes, a single gradient ascent step on the forget set closely mimics the exact influence update:

$D = \{z_i\}_{i=1}^N$1

For batch deletions, a gradient-correction is often employed (Liu et al., 31 Jul 2025).

• Efficient influence estimation and pruning: Several works exploit the empirical finding that many data points have negligible influence. By computing influence scores—via Hessian-approximation, random-projection datastores (LESS), or checkpointed gradient norms—one may safely omit low-influence points from both the forget and retain sets, reducing downstream unlearning cost by 20–60% without statistical degradation (Kleiman et al., 4 Dec 2025). In graph and recommendation contexts, importance-based structural pruning and neighbor propagation further shrink the effective parameter subsets to update (Zhang et al., 2023, Wu et al., 2023).

3. Domain-Specific and Structural Extensions

The application of IAU in different model classes and data modalities has yielded specialized formulations:

• Graph neural networks: The Graph Influence Function (GIF) augments the perturbation objective to include not only direct losses on the deleted nodes/edges but also the spillover influence on their k-hop neighbors, capturing the recursively distributed impact of deletions on the graph's computational structure. The closed form becomes

$D = \{z_i\}_{i=1}^N$2

where $D = \{z_i\}_{i=1}^N$3 aggregates both removal and neighbor-gradient terms (Wu et al., 2023). Recent frameworks introduce influence-aware negative preference optimization (INPO), with explicit influence-weighted message passing to mitigate topological coupling and preserve model utility during aggressive unlearning (Chen et al., 22 Oct 2025).

• Recommendation unlearning: IAU distinguishes between direct influence (erased interactions) and spillover (other interactions whose graph context is affected). The IFRU method decomposes unlearning updates into these two terms, aggregates their gradients, and uses parameter pruning on user/item embeddings to yield efficient subproblem solutions (Zhang et al., 2023).
• LLMs: For LLM unlearning, scalable influence proxies (e.g., normalized token-wise gradient alignment) are constructed between forget and retain batches, resulting in per-example adaptive reweighting of parameter updates. Example: RapidUn computes four directional influence matrices (FF, FR, RF, RR), robustly normalizes them, and maps to bounded sample weights for selective (LoRA-only) fine-tuning. This achieves forgetting performance close to full retrain at two orders of magnitude higher efficiency (Zhao et al., 4 Dec 2025).
• Preference alignment: In LLM alignment, IAU is adapted as a bi-level optimization to select negative examples whose removal maximally improves a downstream task metric. Implicit differentiation using the influence approximation enables efficient greedy set construction and weighting (U2A framework) (Feng et al., 9 Apr 2025).

4. Empirical Outcomes and Comparative Efficiency

IAU methods consistently demonstrate substantial gains in computational efficiency over retrain- and Hessian-based baselines, without compromising utility or forgetting effectiveness:

• Wall-clock acceleration: IFRU in recommendation unlearning achieves >250× speed-up compared to full retraining, with test AUC and MIA indistinguishability close to re-training baselines (Zhang et al., 2023, Liu et al., 31 Jul 2025).
• Graph and GNN settings: GIF matches or slightly surpasses retrained models’ F1 in node/edge/feature unlearning while running 10–100× faster (Wu et al., 2023). INPO achieves up to 0.98 AUC on forget sets with minimal performance loss on retain sets (Chen et al., 22 Oct 2025).
• LLMs: RapidUn attains up to 100× speedup over full retraining and achieves retention–forgetting trade-offs competitive with LoReUn, GA, and Fisher baselines, consistently outperforming them on seen and OOD attack success rate reductions (Zhao et al., 4 Dec 2025).
• Generic unlearning problems: Filtering out low-influence points prior to unlearning—up to 50% of the total—has negligible effect on accuracy but halves total computation for high-performing NeurIPS unlearning competition entries (Kleiman et al., 4 Dec 2025).
• Corrective unlearning: Delta-Influence identifies and excises minimal poison sets in vision benchmarks via “influence collapse,” restoring clean performance with negligible accuracy loss and outperforming all other state-of-the-art detectors (Li et al., 2024).

5. Limitations, Assumptions, and Open Problems

IAU performance and guarantees are constrained by several recurrent factors noted throughout the literature:

• First-order approximation limitations: Most IAU frameworks (gradient-based, influence-function, or iterative variants) use first-order Taylor expansions, neglecting higher-order curvature and potential nonlinearity in deep nets. Their efficacy drops for ill-conditioned or highly non-convex settings (Liu et al., 31 Jul 2025, Kleiman et al., 4 Dec 2025).
• Hessian invertibility and convexity: All methods assume local convexity and invertibility of the empirical Hessian at $D = \{z_i\}_{i=1}^N$4, which may not hold for deep or nonconvex models; justifications hinge on empirical or local properties (Zhang et al., 2023, Kleiman et al., 4 Dec 2025).
• Parameter pruning and subset selection: Aggressive pruning, while effective for scale and speed, may trade off completeness; tuning sparsity levels is empirical, and exact deletion guarantees are usually not provided (Zhang et al., 2023).
• Approximate influence estimation: Domain-specific adaptations (e.g., for GCNs or LLMs) rely on proxy metrics or heuristics for scalable influence computation. Their accuracy varies with model architecture, data distribution, and training regime (Zhao et al., 4 Dec 2025, Wu et al., 2023).
• No formal privacy or certified deletion: With the exception of certain theoretical extensions, IAU methods rarely produce certified unlearning bounds or formal differential privacy guarantees (Liu et al., 31 Jul 2025, Kleiman et al., 4 Dec 2025).

6. Extensions, Directions, and Open Research

Several future avenues and extensions are being actively explored:

• Certified and privacy-preserving unlearning: Incorporating formal verification, differential privacy, or adaptive thresholds to guarantee safety and privacy of unlearning (Liu et al., 31 Jul 2025, Kleiman et al., 4 Dec 2025).
• Group and correlated influence: Extending influence estimation to correlated deletions, streaming or continual unlearning, and structure-aware group selection (Kleiman et al., 4 Dec 2025).
• Scalability to large-scale deep learning: Block-Hessian or randomized sketching for models with billions of parameters, as in transformer-based LLMs or graph-based recommendation (Zhang et al., 2023, Feng et al., 9 Apr 2025).
• Adaptive optimizer coupling: Aligning single-shot gradient updates with the optimizer statistics and momentum/Adam variants to reduce residual utility gap (Liu et al., 31 Jul 2025).
• Universal and cross-domain IAU pipelines: Integrating influence ranking with all downstream unlearning algorithms as a preprocessing stage, establishing orthogonality and portability across datasets and domains (Kleiman et al., 4 Dec 2025).

Emerging results suggest that IAU enables practical, effective, and interpretable machine unlearning at scales necessary for both modern regulation and production deployment. However, open questions remain concerning formal guarantees, higher-order behavior, robustness to model drift, and universal influence computations in highly non-convex landscapes.