Mitigating Privacy Risk via Forget Set-Free Unlearning

Abstract: Training machine learning models requires the storage of large datasets, which often contain sensitive or private data. Storing data is associated with a number of potential risks which increase over time, such as database breaches and malicious adversaries. Machine unlearning is the study of methods to efficiently remove the influence of training data subsets from previously-trained models. Existing unlearning methods typically require direct access to the "forget set" -- the data to be forgotten-and organisations must retain this data for unlearning rather than deleting it immediately upon request, increasing risks associated with the forget set. We introduce partially-blind unlearning -- utilizing auxiliary information to unlearn without explicit access to the forget set. We also propose a practical framework Reload, a partially-blind method based on gradient optimization and structured weight sparsification to operationalize partially-blind unlearning. We show that Reload efficiently unlearns, approximating models retrained from scratch, and outperforms several forget set-dependent approaches. On LLMs, Reload unlearns entities using <0.025% of the retain set and <7% of model weights in <8 minutes on Llama2-7B. In the corrective case, Reload achieves unlearning even when only 10% of corrupted data is identified.

• The paper introduces partially-blind unlearning via the Reload algorithm, eliminating targeted training data without retaining the forget set.
• Reload integrates proxy gradient ascent, selective weight reinitialization, and fine-tuning on the retain set to effectively erase data influence.
• Empirical results on vision and language models show Reload matches or outperforms traditional methods while reducing privacy and compliance risks.

Forget Set-Free Unlearning for Privacy Risk Mitigation: The Reload Algorithm

Introduction

The ongoing increase in regulatory demands for user data privacy—exemplified by legislation such as the GDPR—has highlighted the need to remove specific training data from deployed machine learning models quickly and completely. This requirement motivates the “machine unlearning” problem, which seeks to efficiently eliminate the effect of designated datapoints (the forget set) from a trained model, ideally without costly retraining. However, existing unlearning methods often force organizations to retain the forget set after deletion requests, thereby perpetuating privacy risks for extended periods. "Mitigating Privacy Risk via Forget Set-Free Unlearning" (2604.10636) addresses this issue by introducing partially-blind unlearning (PBU), where unlearning can proceed without direct access to the forget set, and by proposing Reload, an algorithmic framework that leverages cached gradients as auxiliary data for effective, forget set-free unlearning.

Problem Formulation and Partially-Blind Unlearning

Traditional unlearning algorithms, whether gradient ascent on the forget set, weight saliency methods, or distillation-based approaches, require explicit access to the data to be forgotten. This requirement is fundamentally at odds with the intent of “right to be forgotten” regulations, as user data requests cannot be executed immediately, and instead the data must be retained until unlearning is complete, compounding privacy risk (dataset and model risk).

The paper rigorously defines the partially-blind unlearning setting, in which only the retain set, the trained model, and a privacy-preserving auxiliary object (e.g., summed gradients over all training data) are available—not the forget set itself. By reducing the data held post-deletion request to aggregated statistics from training, the approach achieves strict reductions in breach, compliance, and adversarial risk relative to conventional methods that require the raw forget set. The authors formalize the requirements for such auxiliary data, emphasizing the low recoverability of individual datapoints to minimize information leakage.

The Reload Algorithm

Reload operationalizes PBU by combining elements from gradient-based, structured weight sparsification, and fine-tuning methods into a three-stage process harnessing cached gradients (Figure 1):

Figure 1: Overview of the Reload algorithm for partially-blind approximate unlearning, integrating a proxy gradient ascent, weight reinitialization by knowledge value, and fine-tuning.

1. Proxy Gradient Ascent:

Reload uses cached final training gradients $\nabla_\theta \mathcal{L}(\mathcal{D})$ and fresh gradients on the retain set $$\nabla_\theta \mathcal{L}(\mathcal{D}_{\mathrm{retain}})$$ to compute a proxy for the (now-deleted) forget set gradient:

$$\nabla_\theta \mathcal{L}(\mathcal{D}_{\mathrm{forget}}) = \nabla_\theta \mathcal{L}(\mathcal{D}) - \nabla_\theta \mathcal{L}(\mathcal{D}_{\mathrm{retain}})$$

The model is perturbed away from the forget set optimum by taking a single ascent step in this direction.

2. Selective Weight Reinitialization:

Inspired by neural modularity, Reload estimates “knowledge values” for each parameter, quantifying its relative specialization for the forget set via a ratio of gradient magnitudes. Weights below a quantile threshold are reinitialized (e.g., Xavier or Kaiming schemes), erasing targeted memory traces.

3. Fine-Tuning on the Retain Set:

The model is fine-tuned on $\mathcal{D}_{\mathrm{retain}}$ to recover utility lost in the prior destructive steps.

This methodology is agnostic to model architecture and loss type, and it achieves strong privacy by design: the only persistent auxiliary object required is the final summed gradient. Adaptations for LMs use embedding-corrupted prompts and layer selection to address scale constraints.

Empirical Analysis

Unlearning Individual and Correlated Samples:

Across a spectrum of vision datasets (CIFAR-10, CIFAR-100, SVHN) and architectures (ResNet-18, VGG16-BN, Vision Transformer), Reload matches or outperforms not only other PBU baselines but also many methods that directly employ the forget set. In 10% random forgetting settings, Reload achieves the best or near-best performance on critical metrics, including minimal accuracy gap on the forget set (ΔFA), reduced forget-set membership inference risk (ΔFMIA), and close predictive equivalence to a retrained-from-scratch reference model.

Internal Mechanistic Verification:

Feature map and t-SNE visualizations of layer activations during each Reload step (Figure 2) demonstrate that ascent affects later-layer representations but is insufficient alone. Only after selective reinitialization do the target class responses collapse, confirming the necessity of both stages for effective influence erasure.

Figure 2: Feature map visualization and representational dynamics of a ResNet-18 on SVHN during class-unlearning, evidencing the distinct effects of the reinitialization step.

Entity Unlearning in LMs:

Using TOFU synthetic author datasets on Llama2-7B and Phi-1.5, Reload efficiently unlearns small numbers of entities (1% forget) with strong empirical KS-test similarity to fully retrained models and improved model utility. Forgetting larger cohorts degrades utility, highlighting limitations tied to repair set size and data-parameter ratios.

Corrective Unlearning:

In scenarios involving only partial identification of corrupted or poisoned data, Reload maintains high correction accuracy (Acc_corr) even at low recall rates (γ ≪ 1), often outperforming methods such as BadTeacher or SCRUB, with favorable compute profile for realistic adversarial threat models.

Ablation and Robustness:

The authors provide extensive ablations (gradient quantization, LayerNorm, varying fine-tuning data proportions, out-of-distribution fine-tuning) demonstrating Reload’s stability across storage constraints, model types, and practical operational variations.

Theoretical and Practical Implications

The paper formalizes the trade-offs between privacy, retraining cost, and effectiveness under regulatory constraints, showing that forget set-free unlearning can be realized with only minimal auxiliary information while achieving empirical closeness to “gold standard” retraining. From a theoretical perspective, they prove that gradient linearity enables proxy estimation in the PBU regime. Empirically, the modular combination of global gradient ascent and local weight reset capitalizes on distributed and localized memorization properties in deep networks, with the knowledge value mechanism targeting disproportionately “specialist” weights.

On the practical front, Reload enables immediate deletion of user data and unlearning without holding raw data, directly addressing a core bottleneck in privacy-focused ML deployments. The LLM adaptation demonstrates scalability and efficiency, with unlearning steps executing orders of magnitude faster than retraining or zeroing approaches under realistic hardware profiles, provided auxiliary information is available.

Limitations and Future Directions

Reload’s reliance on cached gradients intrinsically imposes a storage overhead, albeit mitigated by quantization and small-fraction storage for large LMs. The algorithm’s effectiveness declines sharply when the repair/retain set is much smaller than the forget set, particularly in LLMs, suggesting future work on scalable representation repair or architectural priors for large generative networks. Furthermore, the approach is best suited to models trained with additive decomposable losses; its efficacy for self-supervised or contrastive learning remains open and merits evaluation via representation-level metrics such as CKA. Open theoretical questions include quantifying the optimality gap between Reload-generated and retrained models as a function of knowledge value distribution and auxiliary information compressibility.

Conclusion

By introducing the partially-blind unlearning paradigm and operationalizing it via Reload, this work enables effective, efficient, and privacy-preserving post hoc influence removal without the persistence of sensitive forget set data. Reload achieves empirical parity or superiority with existing unlearning algorithms—even those that violate privacy desiderata—across vision and language domains, and sets a new standard for what is achievable in practical machine unlearning under realistic privacy constraints. The results provide technical justification for stricter data retention policies and open avenues for architectural and algorithmic refinement in privacy-centric deep learning systems (2604.10636).