Efficient machine unlearning with minimax optimality

Abstract: There is a growing demand for efficient data removal to comply with regulations like the GDPR and to mitigate the influence of biased or corrupted data. This has motivated the field of machine unlearning, which aims to eliminate the influence of specific data subsets without the cost of full retraining. In this work, we propose a statistical framework for machine unlearning with generic loss functions and establish theoretical guarantees. For squared loss, especially, we develop Unlearning Least Squares (ULS) and establish its minimax optimality for estimating the model parameter of remaining data when only the pre-trained estimator, forget samples, and a small subsample of the remaining data are available. Our results reveal that the estimation error decomposes into an oracle term and an unlearning cost determined by the forget proportion and the forget model bias. We further establish asymptotically valid inference procedures without requiring full retraining. Numerical experiments and real-data applications demonstrate that the proposed method achieves performance close to retraining while requiring substantially less data access.

• The paper introduces a gradient-based estimator that debiases pre-trained models by accounting for the influence of the forget set.
• It employs a Hessian-weighted correction for squared loss, attaining minimax optimal error rates comparable to full retraining.
• Extensive simulations and real-world experiments demonstrate the method’s efficiency, reliability, and compliance with privacy regulations.

Efficient Machine Unlearning with Minimax Optimality

Introduction and Motivation

Machine unlearning addresses the challenge of efficiently excising the influence of specified "forget" data from trained models, as increasingly demanded by privacy regulations (such as GDPR), user data withdrawal, and the need to mitigate corrupted or biased data. The foundational setup is one where, instead of full retraining—which is often computationally or logistically prohibitive—one seeks accurate unlearning using only the pre-trained model, the forget set, and a limited subsample of the retained data.

This work presents a novel statistical framework for machine unlearning supporting general loss functions, with a detailed focus on the squared loss case. The authors introduce a gradient-based estimator supporting minimax-optimal error guarantees and demonstrate how the method both debiases the pre-trained estimator and supports valid statistical inference without access to the entire retained dataset.

Problem Formulation

The machine unlearning problem is formalized as the estimation of the model that would have emerged had the forget set $D_f$ never been used for training, resulting in the retained data set $D_r$. In practice, only the pre-trained model $\hat{\theta}_p$, the forget data $D_f$, and a small subsample $\widetilde{D}_r \subseteq D_r$ are available, with typically $|\tilde{D}_r| \ll |D_r|$. The loss function $\ell(\theta; x, y)$ is assumed differentiable, with the main focus in the analysis being the squared loss.

The statistical target is the minimizer of the expected loss over $D_r$:

$$\theta_r^* = \arg\min_{\theta} \mathbb{E}\left[\ell(\theta; D_r)\right].$$

The oracle solution is the retrained estimator $\mathring{\theta}_r^{(\text{rtr})}$, which is practically unattainable in the unlearning context due to restrictions on $D_r$0.

The key question is: How accurately can $D_r$1 be estimated given only $D_r$2, $D_r$3, and $D_r$4?

Proposed Methodology

Generic Losses

Through a Taylor expansion of the loss gradients, the paper introduces a principled update to the pre-trained estimator, mediated by gradients from the forget and retained subsample: $D_r$5 This leads to an estimator performing a Hessian-weighted correction to the pre-trained parameters, capturing the (estimated) bias induced by $D_r$6. Crucially, the computation does not require outcome labels from the retained data in the squared and cross-entropy loss cases.

A gradient descent realization is developed, which iteratively refines the parameter vector, with proven global convergence under standard strong convexity and smoothness conditions.

Unlearning Least Squares (ULS) for Squared Loss

Under squared loss, the approach yields a closed-form estimator (ULS): $D_r$7 This update debiases $D_r$8 by projecting out the influence of $D_r$9, weighted by the estimated Hessian from the (small) retained subsample.

A robustified version (ULS+) adds an explicit loss term from $\hat{\theta}_p$0, interpolating between unlearning and pure retrained OLS, which is essential when the magnitude of the forget-bias is large.

Theoretical Guarantees

Finite Sample Convergence and Minimax Optimality

The estimation error of the ULS estimator is shown to decompose as: $\hat{\theta}_p$1 where $\hat{\theta}_p$2 is the forget proportion, and $\hat{\theta}_p$3 is the parameter shift between $\hat{\theta}_p$4 and $\hat{\theta}_p$5. The first term matches the error from retraining on $\hat{\theta}_p$6 (oracle rate), while the second is the "unlearning cost" from the limited access to retained data and the potential bias induced by $\hat{\theta}_p$7.

A matching minimax lower bound is established, showing that any estimator using only $\hat{\theta}_p$8, $\hat{\theta}_p$9, and $D_f$0 must incur an unlearning risk at least of this order—thus, ULS is minimax optimal.

When the shift is large, the robustified ULS+ estimator attains the optimal rate across all regimes.

Statistical Inference

The authors derive asymptotic normality for linear functionals of the unlearned parameter, and they show how to construct valid confidence intervals based solely on subsampled retained data, the pre-trained estimator, and the forget set. The variance expressions include additional terms reflecting the uncertainty induced by limited retained data access and explicit bias correction, which are shown to be consistently estimable from accessible quantities.

Empirical Evaluation

Simulations

Extensive simulations confirm the theoretical claims:

• ULS matches the performance of full retraining whenever the forget-bias is not overwhelming.
• Competing transfer learning, gradient ascent, and subsampled OLS estimators fail to match the oracle rate, either suffering large bias or excessive variance, especially with small retained subsamples.
• The robustified estimator ULS+ demonstrates superior stability when the distributional drift is severe.

Real-World Applications

Experiments on the Yelp review dataset (text regression, large bag-of-words feature spaces) and UK Biobank clinical data (outlier removal in length-of-stay prediction) show that ULS and ULS+ achieve predictive accuracies equivalent to retraining, with orders of magnitude less computational cost and data access than previously proposed projective update methods. The method also yields statistically efficient confidence intervals for downstream inference.

Comparison to Baselines and Existing Protocols

Contrasted with gradient ascent (standard in LLM unlearning) and the "GradDiff" heuristic, ULS is shown to be superior both conceptually and theoretically—gradient ascent and GradDiff can only match subsampled OLS in the minimax regime and completely fail to borrow strength from the pre-trained estimator in high-bias regimes.

Transfer learning estimators, by ignoring $D_f$1, are strictly sub-optimal for unlearning under distribution shift.

Implications and Future Directions

The framework introduced in this work prioritizes rigorous statistical guarantees for machine unlearning, enabling minimax-optimal estimation and valid inference. In practical terms, it allows compliance with privacy and data removal requests in real deployments of LLMs, tree-based models, and regression systems, with minimal retraining and computation.

From a theoretical perspective, the work exposes the fundamental trade-off between retained sample size and forget-bias, and establishes that minimax-optimal unlearning is achievable without access to the full data or expensive second-order information.

Addressing the extension to non-smooth models (e.g., Lasso, quantile regression), nonparametric estimators, or deep networks remains open; the main challenge lies in bias correction in nonsmooth, high-dimensional settings where Hessian-based correction is not applicable.

Conclusion

This paper provides a complete statistical treatment of machine unlearning, deriving minimax-optimal estimators and valid inference procedures for generic losses and, in particular, least squares regression. The proposed update yields a computationally efficient, theoretically justified solution for the unlearning problem, obviating the need for full retraining. The work sets a high standard for future research on unlearning in both theory and practice, with significant implications for legal compliance, bias mitigation, and ethical AI.