Distribution-Aware Certified Unlearning

• The paper presents a framework that certifies the removal of distribution signals using statistical divergence measures such as KL-divergence.
• It employs methods like divergence optimization, trust-region Newton updates, and surrogate data techniques to balance forgetting and model utility.
• Empirical evaluations demonstrate significant improvements in deletion efficiency and retention performance across synthetic and real-world datasets.

A distribution-aware certified unlearning framework is a methodology for machine unlearning that rigorously quantifies and certifies the removal of distributional signals, rather than merely deleting individual samples or records. Such frameworks offer guarantees—quantified in formal divergence or indistinguishability terms—about the extent to which both sub-populations and specific data distributions are forgotten, while preserving utility on retained data. These frameworks encompass a range of settings, including sample-level, sub-population, non-i.i.d. deletion, source-free unlearning, and LLM applications. The defining property is certification under statistical distances (e.g., Kullback–Leibler divergence, Total Variation, differential privacy metrics) that account for entire distributions, with formal bounds on model behavior post-unlearning.

1. Formalization and General Principles

Distribution-aware certified unlearning departs from traditional pointwise removal by addressing the unlearning of distributions. The formal problem is as follows: given an observed mixture of samples from a to-forget distribution ($p_1$) and a to-retain distribution ($p_2$), the goal is to identify a deletion set $S$ such that the resulting empirical distribution $p$ is statistically far from $p_1$ (removal) and close to $p_2$ (preservation). The principal certification tool is the Kullback–Leibler (KL) divergence:

• Removal: $D_{\mathrm{KL}}(p_1 \Vert p) \geq \alpha$
• Preservation: $D_{\mathrm{KL}}(p_2 \Vert p) \leq \varepsilon$

This establishes an optimization problem over data subsets and provides a Pareto frontier $(\alpha,\varepsilon)$ that quantifies the optimal trade-off between forgetting unwanted information and maintaining utility (Allouah et al., 20 Jul 2025).

In a broader sense, distribution-aware certification requires explicit modeling of (i) the statistical effect of removal (distribution shift) and (ii) its impact on downstream training and inference. This principle underlies frameworks for non-i.i.d. deletion, source-free settings relying on surrogate distributions, and unlearning paradigms for large models constrained by differential privacy (Guo et al., 11 Jan 2026, Basaran et al., 6 Jun 2025, Mahmud et al., 18 Apr 2025).

2. Methodologies and Algorithmic Techniques

Several classes of algorithmic frameworks instantiate the above principles:

Distributional Unlearning via Divergence Optimization

Approaches such as "Distributional Unlearning" (Allouah et al., 20 Jul 2025) use KL-divergence to define deletion strategies. The method constructs an explicit Pareto frontier for Gaussians with shared covariance: $$\mathrm{PF}(p_1, p_2; \mathcal{P}) = \{ (\alpha,\varepsilon) : \varepsilon = (\sqrt{\alpha} - \sqrt{D_{\mathrm{KL}}(p_1\Vert p_2)})^2, \ \alpha \geq D_{\mathrm{KL}}(p_1\Vert p_2) \}$$

A "distance-based" selection algorithm is employed: samples from $p_1$ are scored by their distance from the $p_2$ mean, and those farthest away are deleted to maximize distributional divergence from $p_1$ while minimizing divergence from $p_2$. Empirically, this leads to a quadratic reduction in deletion budget compared to random removal.

Newton and Trust-Region Updates under Distribution Shift

Certified unlearning based on Newton’s method has traditionally assumed i.i.d. deletion. Under non-i.i.d. (biased) deletion, standard Newton-based updates produce vacuous bounds and unlearning becomes ineffective due to inflated remainder terms in Taylor expansions. Distribution-aware certified unlearning now adopts iterative trust-region constrained Newton updates:

At each iteration $t$,

• Minimize a quadratic model $m_t(p)$ around the current $w_t$ on a trust-region ball with adaptive radius;
• Accept updates based on model agreement ratios;
• Exponentially tighten the parameter error bound $\Delta$ via local strong convexity, yielding a much smaller sensitivity parameter for certifying (ε, δ)-unlearning via the Gaussian mechanism.

This delivers robust certification under substantial distribution shift and empirically reduces accuracy drop (ΔF1) by 3× compared to single-step Newton (Guo et al., 11 Jan 2026).

Unlearning without Source Data via Surrogate Distributions

In scenarios where the original training set is inaccessible, certified unlearning can be achieved using a surrogate dataset $\mathcal{D}_s$ approximating the empirical source distribution. The statistical distance between the true and surrogate distributions, measured via Total Variation or KL divergence, calibrates the added noise to maintain (ε, δ)-certified unlearning:

• Replace the unavailable retained-data Hessian in the Newton unlearning update with an estimator derived from the surrogate set
• Quantify the error introduced using bounds dependent on TV or KL
• Adjust the noise scale to reflect this mismatch, ensuring that the released model is statistically indistinguishable (to within $(\epsilon, \delta)$) from exact unlearning (Basaran et al., 6 Jun 2025)

Distribution-Aware Differential Privacy for LLM Unlearning

For LLMs, "DP2Unlearning" utilizes DP mechanisms during training on sensitive subsets, enabling efficient unlearning by fine-tuning a DP-protected model exclusively on retained data:

• Employ DP-MLM (exponential mechanism with distributional embedding–based utility) or DP-SGD (per-step gradient clipping and Gaussian noise)
• Use post-processing immunity of DP to certify that the model after unlearning is (ε,δ)-DP with respect to the forgotten subset
• Empirically verify, via Jensen-Shannon divergence, Wasserstein distance, and entropy metrics, that output distributions of the unlearned model on both forget and retain sets match the retrained-from-scratch gold standard (Mahmud et al., 18 Apr 2025)

3. Certification Guarantees

Distribution-aware frameworks provide multiple forms of certification:

• Pareto frontier bounds: Quantitative trade-offs between amount-of-forget (removal KL) and preservation (retained KL), with closed-form solutions in Gaussian (and some exponential-family) settings (Allouah et al., 20 Jul 2025).
• Log-loss shift guarantees: For any retrained classifier $h$ on the edited dataset, the increase in log-loss on the forgotten distribution is at least $\alpha-\delta_1$, and the log-loss on the retained data increases by at most $\varepsilon-\delta_2$ (Allouah et al., 20 Jul 2025).
• (ε, δ)-Certifiability: In trust-region and source-free frameworks, sensitivity bounds on model parameters translate directly into the noise scale for the Gaussian mechanism, ensuring indistinguishability from retraining to the specified $(\epsilon,\delta)$ level (Guo et al., 11 Jan 2026, Basaran et al., 6 Jun 2025).
• Distribution-level indistinguishability for LLMs: Empirical verification that output distributions on forgotten data, after certified unlearning, are statistically matched to those of models retrained on retained data only (Mahmud et al., 18 Apr 2025).

The following table summarizes key certification mechanisms:

Framework	Metric/Certainty	Guarantee Type
Distributional Unlearning	KL-divergence, Pareto frontier	Divergence bounds on edit
Trust-region Newton	Model param. norm ($\Delta$)	(ε, δ)-certified (via Gaussian)
Source-free Unlearning	TV/KL dist. + Hessian bounds	(ε, δ)-certified
DP2Unlearning (LLMs)	Differential privacy ($\epsilon$)	Distributional indistinguish.

4. Empirical Evaluation and Comparative Performance

Empirical results demonstrate the practical efficiency and efficacy of distribution-aware certified unlearning frameworks:

• Distributional unlearning achieves 15–72% reduction in number of deletions required relative to random removal, with negligible loss on retained class performance across synthetic (Gaussian), text (Jigsaw Toxic Comments, SMS spam), and vision (CIFAR-10) benchmarks. For example, reducing f (fraction of samples to delete) from 65% (random) to 18% (selective) achieves 72% savings in the low-divergence Gaussian setting (Allouah et al., 20 Jul 2025).
• Robust (trust-region) certified unlearning under non-i.i.d. deletions halves or triples the reduction in F1 utility gap compared to prior approaches using the single-step Newton method. On MNIST at KL ≈ 0.104, the utility loss ΔF1 drops from ≈4% (Zhang 2024, Guo 2020) to 1.37% (trust-region). The match in utility to retraining is substantially closer across tested KL ranges (Guo et al., 11 Jan 2026).
• Source-free certified unlearning delivers model accuracy and membership inference indistinguishability within 1% of the gold standard over varying surrogate-source divergence levels, validated on synthetic and real datasets (CIFAR-10, Caltech-256, Stanford Dogs). Noise calibrates to KL/TV mismatch, preserving privacy and utility (Basaran et al., 6 Jun 2025).
• DP2Unlearning for LLMs achieves formal $\epsilon$-DP (or $(\epsilon,\delta)$-DP) forgetting at roughly half the retraining cost, with nearly identical utility on the retained set and "forget quality" (FQ) closely tracking true retraining (FQ > 0.9 vs. 1.0 for RFS). Distribution-level metrics (JSD, Wasserstein, Entropy) confirm indistinguishability to within 0.02, 0.11, and 0.042 respectively at 10% forget ratio (Mahmud et al., 18 Apr 2025).

5. Limitations, Assumptions, and Extensions

Distribution-aware certified unlearning frameworks are constrained by several structural and practical considerations:

• Divergence type: Most formulations employ forward KL divergence; alternatives (e.g., backward KL, Wasserstein, or $f$-divergences) may emphasize different aspects of the distribution shift.
• Distributional assumptions: The derivation of Pareto frontiers and sample-complexity proofs typically require Gaussian or exponential-family models with shared parameters; application to arbitrary distributions is heuristic.
• Surrogate accuracy: Source-free methods depend on the fidelity of the surrogate distribution, with theoretical guarantees scaling with statistical distance from the original; practical TV/KL estimates may be noisy.
• Model and computational costs: Trust-region Newton methods are iterative but may still be costlier than heuristic unlearning; DP2Unlearning substantially reduces LLM retraining cost, but training DP-MLMs or DP-SGD models carries a one-time overhead.
• Robustness to high-dimensional, non i.i.d., or adversarial deletions remains limited by the tightness of sensitivity bounds and the quality of local curvature estimation.
• Current extension directions include distributional unlearning in latent or representation spaces, adversarial/concept-erasure integration, and fairness/causal unlearning beyond pure divergence objectives (Allouah et al., 20 Jul 2025).

6. Impact and Future Directions

Distribution-aware certified unlearning marks a transition in privacy and compliance technology—from per-sample editing to principled, data-centric optimization guaranteeing that models cannot be used to reconstruct or exploit sub-populations that should be forgotten. Deployment relevance is underscored by regulatory frameworks (e.g., GDPR) demanding enforceable, certifiable removal of personal or proprietary data.

Ongoing research tackles the integration of these frameworks with large-scale foundation models, causal reasoning for sub-group fairness in unlearning, exploitation of finer statistical divergences, and closing utility gaps in high-dimensional, high-shift scenarios. Empirical advances have already demonstrated both practical and theoretical reductions in resource requirements and improved privacy-utility trade-offs in both vision and language domains (Allouah et al., 20 Jul 2025, Guo et al., 11 Jan 2026, Basaran et al., 6 Jun 2025, Mahmud et al., 18 Apr 2025).