Certified Unlearning in ML

• Certified Unlearning is a data deletion process defined by (ε,δ)-guarantees that ensures a model’s output after unlearning is statistically indistinguishable from that of a retrained model.
• The methodology employs influence functions, Newton-style updates, and calibrated Gaussian noise to efficiently approximate retraining while maintaining privacy and accuracy.
• Extended frameworks address nonconvex and decentralized settings through trust-region methods and surrogate data approaches to manage distribution shifts and scalability challenges.

Certified Unlearning refers to algorithmic methodologies for provably removing the influence of specified training data from learned models, such that the resulting model is, to within rigorously defined (ε,δ) statistical indistinguishability, equivalent to a retrained model that never encountered the forgotten data. This is motivated by privacy regulations such as the "right to be forgotten" and constitutes a formalization of post-hoc data erasure guarantees in machine learning systems, often using frameworks and analytical tools from differential privacy. Certified unlearning guarantees are typically quantified by output-distribution proximity between unlearning mechanisms and retraining, enforced via analytical sensitivity analysis combined with carefully calibrated stochastic post-processing.

1. Formal Definitions and Statistical Guarantees

Certified unlearning is operationalized through probabilistic indistinguishability of model outputs under deletion requests, typically using the following criterion: for any (randomized) learning algorithm $\mathcal{A}(\mathcal{D})$ and unlearning procedure $\mathcal{U}(\cdot)$, for all possible forget sets $\mathcal{D}_f\subseteq\mathcal{D}$ and measurable model-parameter sets $S\subseteq\mathbb{R}^d$,

$$\Pr[\mathcal{U}(\mathcal{A}(\mathcal{D}),\mathcal{D},\mathcal{D}_f)\in S] \leq e^{\epsilon}\Pr[\bar{\mathcal{A}}(\mathcal{D}\setminus\mathcal{D}_f)\in S] + \delta,$$

and symmetrically, where $\bar{\mathcal{A}}$ is the retraining algorithm (potentially with randomized initialization). This definition, parallel to (ε,δ)-differential privacy, ensures the statistical closeness of the unlearned and retrained model distributions, with the parameters ε,δ quantifying the certification budget (Koloskova et al., 8 Jun 2025, Mu et al., 2024, Zhang et al., 2024, Wu et al., 10 Jan 2026, Dolgova et al., 8 Jan 2026).

In decentralized scenarios, the definition is extended to model view-based events on distributed model aggregates, and in graph and minimax optimization settings certification is expressed with respect to the relevant model output or parameter structures (Wu et al., 10 Jan 2026, Liu et al., 2023, Chien et al., 2022).

2. Algorithmic Frameworks for Certified Unlearning

Certified unlearning procedures are predominantly based on influence-based model updates, Newton-style second-order approximations, and post-hoc stochastic perturbations:

• Single-step Newton/Influence updates: For convex and strongly convex models, the post-deletion model is approximated by a second-order Taylor (Newton) update derived from the empirical risk's gradient and Hessian with respect to the removed data points. The update is given by

$$w^- = w^* + H^{-1}\left(\nabla L(w^*;\mathcal{D})-\nabla L(w^*;\mathcal{D}_r)\right),$$

where $w^*$ is the pre-unlearning minimizer, $H$ is the (empirical or Fisher) Hessian, and $\mathcal{D}_r$ is the retained set (Chien et al., 2022, Wu et al., 10 Jan 2026, Zhang et al., 2024, He et al., 10 Nov 2025).

• Gaussian Mechanism for certification: To ensure the output distribution matches the retrained model, calibrated Gaussian noise is added to the update; noise magnitude is analytically tied to a provable upper bound $\Delta$ on the sensitivity (i.e., $L_2$-distance) between the approximate unlearned and retrained model. Explicitly,

$$\sigma \ge \frac{\Delta}{\epsilon}\sqrt{2\ln(1.25/\delta)}$$

(Mahadevan et al., 2021, Koloskova et al., 8 Jun 2025, Zhang et al., 2024, Liu et al., 2023).

• Noisy Fine-Tuning and Post-Processing: For deep and nonconvex models, or in cases where the original data is unavailable, noisy fine-tuning procedures or projected SGD with stochastic post-processing can be used, distributing the noise budget across multiple update steps or orthogonal parameter subspaces to preserve both privacy and utility (Koloskova et al., 8 Jun 2025, Dolgova et al., 8 Jan 2026, Chien et al., 2024).
• Distribution-Aware Trust Region Methods: In the presence of non-i.i.d. or biased deletion requests leading to distribution shift, iterative trust-region-constrained Newton updates using local Lipschitz constants are employed to control approximation error and avoid exploding certification bounds (Guo et al., 11 Jan 2026).
• Decentralized and Federated Unlearning: Certified unlearning for decentralized federated learning entails constructing local Newton-style corrective steps, broadcasting with calibrated noise using network topology-aware analysis, and employing Fisher information matrix approximations for scalability (Wu et al., 10 Jan 2026, Lamri et al., 9 Dec 2025).
• Surrogate Data and Source-Free Unlearning: When deletion must occur without access to the source data, surrogate datasets can be used to estimate statistics; the certification noise is then further scaled according to the total variation or KL divergence between the surrogate and source distributions (Basaran et al., 6 Jun 2025).
• Graph and Structured Data: Certified graph unlearning adapts the above techniques to GNNs, approximating the effect of node/edge/feature deletions via Hessian-inverse or influence-function updates, with additional graph-propagation sensitivity analysis to control the effect of topological perturbations (Chien et al., 2022, Yi et al., 2024, Dong et al., 2024, Zhao et al., 18 Nov 2025, Kose et al., 20 May 2025).

3. Theoretical Guarantees and Utility Bounds

Analytical guarantees for certified unlearning typically consist of the following elements:

• (ε,δ)-certification via Gaussian mechanism, conditional on a tight bound Δ between the (approximate) unlearned and retrained parameters.
• Utility bounds quantify the excess empirical or population risk of the unlearned model, often of the form:

$$\mathbb{E}[f(\widetilde x)-\min_x f(x)] \leq O\left(\frac{m L^2}{(n-m)\lambda}+\frac{\sqrt{d\log(1/\delta)}M L^3 m^2}{\lambda^3 n^2\epsilon}\right)$$

for strongly convex smooth objectives (Wu et al., 10 Jan 2026, Liu et al., 2023).

• Deletion capacity—the maximal number of deletions such that utility or generalization loss remains under a target—is characterized in terms of data/parameter dimension ratios, privacy budget, and distribution shift (Liu et al., 2023, Guo et al., 11 Jan 2026).
• For graph and structured data unlearning, error bounds propagate through feature mixing, graph diffusion, and are controlled by node degree, propagation depth, and graph-laplacian statistics (Chien et al., 2022, Yi et al., 2024, Kose et al., 20 May 2025).

4. Extensions: Nonconvexity, Distribution Shift, Surrogate Data, and Decentralization

Recent work has extended certified unlearning to address nonconvex learning (deep neural networks), non-i.i.d. deletion requests, unlearning in the absence of the original data, and fully decentralized architectures:

• Nonconvex models: Certified unlearning for DNNs employs local convexification (e.g., adding $\lambda\|w\|^2$), Hessian-inverse estimators (LiSSA), projection onto norm-bounded domains, and group privacy composition for sequential requests (Zhang et al., 2024, Zhang et al., 2024, Mu et al., 20 Nov 2025, Qiao et al., 2024).
• Distribution shift: Trust-region Newton methods adapt step-size and local curvature to the empirical geometry of the retained set, delivering tight sensitivity bounds even for cluster-wise or biased deletions (Guo et al., 11 Jan 2026).
• Surrogate data: Certification noise is inflated proportional to KL divergence or total variation between surrogate and original data distributions (Basaran et al., 6 Jun 2025).
• Decentralized FL and GNNs: Unlearning protocols employ peer-to-peer correction propagation, token-passing gradient walks, and network-DP amplification for distributed privacy certification (Lamri et al., 9 Dec 2025, Wu et al., 10 Jan 2026, Dong et al., 2024).

5. Practical Trade-Offs and Empirical Evaluation

Certified unlearning protocols are subject to inherent trade-offs among privacy, accuracy, and computational efficiency:

Approach	Speedup vs Retrain	Accuracy Drop	Certification Guarantee
Single-step Newton	10–100×	<1–3% (typical)	(ε,δ) via sensitivity + Gaussian noise
Noisy Fine-Tuning	2–10×	Data-dependent	Closed-form (ε,δ)-unlearning
Trust-Region	Moderate	Tighter under shift	Locally controlled sensitivity
Decentralized FL	Up to 97% faster	<2% (typical)	Network-view (ε,δ)-unlearning
Scalable GNN	100–200×	<0.5%	Approximation + propagation error

Empirical findings, across benchmarks such as MNIST, CIFAR-10, ogbn-arxiv, and billion-edge graphs, consistently demonstrate that certified unlearning matches retraining in effectiveness (test accuracy, membership-inference attack resistance) at orders-of-magnitude lower computational cost, provided approximation error and certification noise are tightly controlled (Zhang et al., 2024, Lamri et al., 9 Dec 2025, Yi et al., 2024, Chien et al., 2022). Data value-weighted schemes, trust-region adaptation, and subspace noise scheduling mitigate accuracy collapse and improve utility-privacy trade-off in practice (He et al., 10 Nov 2025, Dolgova et al., 8 Jan 2026). Online and batch unlearning, surrogate datasets, and error-thresholding triggers further support practical deployment at scale (Basaran et al., 6 Jun 2025, Qiao et al., 2024, Mahadevan et al., 2021).

6. Certified Unlearning for Structured Data: Graphs and Signed Graphs

Certified unlearning in graph-structured domains adapts foundational techniques to address propagation effects, structural correlation, and sign information:

• Graph GNNs: Certified Graph Unlearning leverages local propagation bounds (SGC, GPR), node/edge/feature removal analysis, and efficient Newton-style updates with propagation-aware gradient and Hessian estimation, maintaining (ε,δ)-guarantees while scaling to large graphs (Chien et al., 2022, Dong et al., 2024, Yi et al., 2024).
• Signed graphs: CSGU integrates triadic structure identification, sociological centrality (balance/status) for importance weighting, and tailored privacy budget allocation in influence-aware Newton updates, achieving stricter privacy-utility trade-offs for SGNNs (Zhao et al., 18 Nov 2025).
• Bias mitigation: Certified unlearning, applied selectively to fairness-critical graph elements, provably reduces statistical parity gaps and group fairness risks without retraining costs (Kose et al., 20 May 2025).

7. Limitations and Open Directions

Certified unlearning currently faces several research frontiers:

• Extending certification guarantees and algorithmic efficiency to general nonconvex, adversarial, or large-scale deep learning settings.
• Minimizing the utility loss and certification noise under distribution shift and data valuation heterogeneity.
• Achieving tight bounds and efficient computation for sequential, batch, and combinatorial deletion requests.
• Surrogate-based and source-free unlearning with tractable statistical-distance estimation.
• Further optimizing decentralized and federated protocols for heterogeneity and network-level certification.

The scope of certified unlearning continues to expand, with theoretical, algorithmic, and system-level innovations aiming to balance strong privacy erasure guarantees with scalable, practical machine learning deployment (Wu et al., 10 Jan 2026, Zhang et al., 2024, He et al., 10 Nov 2025, Guo et al., 11 Jan 2026, Dong et al., 2024, Koloskova et al., 8 Jun 2025, Zhao et al., 18 Nov 2025).