Reliable Machine Unlearning

• Reliable Machine Unlearning (RMU) is a framework that erases a model's memory with statistical rigor, ensuring verifiable removal of training data influence.
• The SMIA method within RMU uses mixture modeling and distance measures like MMD and Wasserstein to accurately estimate forgetting rates with bootstrapped confidence intervals.
• Empirical assessments show that RMU outperforms traditional MIA-based auditing in terms of audit fidelity, computational efficiency, and reliability for compliance and privacy.

Reliable Machine Unlearning (RMU) refers to the principled, auditable removal of specified training data’s influence from a machine learning model, accompanied by verifiable guarantees—audited with statistical rigor—that such forgetting has indeed occurred. Unlike naive or heuristic approaches, RMU centers both on the construction of reliable unlearning procedures and on the development of trustworthy, quantifiable auditing mechanisms. Key advances situate RMU at the intersection of statistical auditing (notably, the Statistical Membership Inference Attack, or SMIA), distributional testing, and confidence-aware evaluation, with broad relevance for compliance, privacy, and trustworthy AI.

1. Statistical Limitations of Traditional MIA-Based Unlearning Auditing

The standard approach to auditing unlearning has relied on Membership Inference Attacks (MIAs) that cast the query “Was $x$ in the training set?” as a binary classification problem, treating attack failures as evidence of true forgetting: $$\text{MIA}(x;w) = \mathbf{1}\{\mathrm{score}_{\mathrm{MIA}}(x;w) \geq \beta\}.$$ However, this MIA-based auditing is fundamentally unreliable due to two unobservable sources of statistical error:

• Generalization error across shifted distributions: As shown by PAC-Bayes analysis (Theorem 2.1), if the auditing attack model is trained on a distribution $\mathcal{P}$ (e.g., shadow-model outputs) but evaluated on a testing distribution $\mathcal{Q}$ (the audited model post-unlearning), the true risk is bounded by the empirical risk plus a term scaling with $\chi^2(\mathcal{Q}\|\mathcal{P})$. For unlearning, an irreducible “auditing error” related to $D_\infty(\mathcal{D}_t\|\mathcal{D}_f)$ arises (Corollary 2.2), preventing accurate error estimation at test time. This makes it impossible to observe failures in auditing reliability as the shift between data distributions increases.
• Unobservable error due to class imbalance and statistical shift: Even at high attack accuracy and true forgetting, systematic overestimation of forgetting rates emerges solely due to member/non-member imbalance (as in Figure 1 of the source). The indistinguishability between genuine forgetting and attacker failure results in systematic, undetectable overestimation of the efficacy of unlearning (Sun et al., 1 Feb 2026).

In summary, a failed MIA decision is ambiguous—it may indicate either true forgetting or simply attack model failure. This impurity in the auditing signal invalidates MIA as a reliable tool for RMU.

2. The Statistical Membership Inference Attack (SMIA) Framework

The SMIA method reframes unlearning auditing: from per-sample, binary classification to a statistical test of full-feature distribution alignment. The approach is distinguished by its mixture-modeling and explicit confidence quantification.

Mixture model for audit data: $$\mathcal{D}_f = \alpha\,\mathcal{D}_t^v + (1-\alpha)\,\mathcal{D}_t^t$$ where:

• $\mathcal{D}_t^t$ = features from retained training (members)
• $\mathcal{D}_t^v$ = features from non-member/reference set
• $\mathcal{D}_f$ = features from the purportedly forgotten set
• $\alpha$ = estimated forgetting rate

Estimation procedure:

• Statistical distance matching: Find $\alpha$ minimizing the distance between the empirical distribution of $\mathcal{D}_f$ and the synthetic mixture. SMIA implements several instantiations:
  • SMIA-0: Matches means and covariances in feature space.
  • SMIA-M: Minimizes the Maximum Mean Discrepancy (MMD) within a reproducing kernel Hilbert space (RKHS).
  • SMIA-W: Minimizes the entropic regularized $p$-Wasserstein distance between empirical distributions.

Bootstrap confidence intervals: To provide quantifiable reliability, SMIA bootstraps the data multiple times, computes forgetting-rate estimates $\{\alpha_i\}$ for each, and reports both the median point estimate and a coverage-calibrated confidence interval $[\alpha^{5\%}, \alpha^{95\%}]$. Under mild moment conditions, this interval is statistically valid as the number of bootstraps grows (Sun et al., 1 Feb 2026).

3. Theoretical Guarantees and Reliability Properties

SMIA’s advantages are underpinned by several theoretical properties:

• Auditing error quantification: By bypassing per-sample classification and instead matching distributions, SMIA eliminates irreducible error terms tied to $D_\infty(\mathcal{D}_t\|\mathcal{D}_f)$ that frustrate MIA-based audits.
• Consistent coverage: The percentile-bootstrap yields confidence intervals for the forgetting rate that are asymptotically valid under finite fourth-moment conditions.
• Computational guarantees: Complexity is sharply reduced. For $d$-dimensional features and $n$ samples, SMIA-0 runs in $O(n d^2 + K d^3)$ and SMIA-M in $O(n^2 K)$ for $K$ bootstraps, vastly outpacing traditional MIA’s $O(M T_\mathrm{train})$ for $M$ shadow models, as shown by observed 150 ms (SMIA-0) versus 40 min (ResNet-18) on CIFAR-100 (Sun et al., 1 Feb 2026).

SMIA’s statistical formulation provides RMU with the first framework in which forgetting-rate estimates can be supplied with explicit, non heuristic, uncertainty quantification.

4. Empirical Assessment and Audit Efficacy

Table: Experimental performance of SMIA vs. MIA-based baselines on CIFAR-10 (with 5% random unlearning) (Sun et al., 1 Feb 2026):

Method	Non-member Acc. (%)	Notes
EMIA	61.1	MIA-ensemble
LiRA	52.5	MIA-reference
SMIA-0	86.6	200 bootstraps
SMIA-M	87.4	200 bootstraps
SMIA-W	72.5	Wasserstein-based

• SMIA variants consistently show higher audit fidelity and stability to sampling noise, requiring as few as 5–10 samples per class for peak small-sample performance.
• Bootstrap interval width and median stabilize beyond 100–200 bootstraps, providing operational guidance for practical audit protocol design.

Thus, SMIA’s efficiency, scalability, and coverage properties enable verifiable, robust estimation of forgetting rates, overcoming statistical pitfalls endemic to MIA.

5. Paradigm Shift and Open Challenges for Reliable Machine Unlearning

SMIA reframes the RMU audit problem. Instead of the ill-posed samplewise query, “Was this example forgotten?”, RMU becomes a question of whether the distributions over retained, reference, and forget-labeled data align under true memory erasure—a fundamentally statistical test.

Future directions emerging from the SMIA paradigm include:

• Individual-level auditing: As sample size drops, the confidence interval width inflates, limiting precision on per-sample guarantees in low-data or streaming scenarios.
• Extension to complex models: Applying SMIA to large-scale or non-standard extractors, such as transformer LLMs or vision-foundation models, requires feature representation engineering and scalability improvements.
• Type I/II error control: Designing adaptive, finite-sample audit metrics that allow direct specification of acceptable false positive/negative rates.
• Protocol integration: Directly embedding SMIA-based audits within end-to-end unlearning protocols, enabling automatic verification and tight policy compliance.

In summary, SMIA provides the first statistically principled, computation-efficient, and coverage-guaranteed mechanism for RMU auditing—enabling trustworthy unlearning outcomes essential for regulatory compliance and privacy guarantees in modern learning systems (Sun et al., 1 Feb 2026).