---
title: Centralized Unlearning Methods
url: https://www.emergentmind.com/topics/centralized-unlearning
type: topic
---

# Centralized Unlearning Methods

Centralized unlearning refers to the class of methods and frameworks designed to remove the influence of specific data points or subsets from machine learning models trained in a centrally coordinated setting. The objective is to render the model statistically indistinguishable from one trained on the retained data only, aligning with privacy requirements such as the right to be forgotten. Centralized approaches contrast with distributed/federated unlearning; all operations and guarantees are coordinated by a central algorithm or server, enabling strong theoretical guarantees and practical performance trade-offs relevant to large-scale machine learning deployments. 

## 1. Formal Frameworks and Guarantees

In the centralized paradigm, let $D \subseteq \mathcal{Z}^n$ denote the full training set, $D_e \subseteq D$ the erased subset, and $D_r = D \setminus D_e$ the retained data. For learning algorithm $\mathcal{A} : \mathcal{Z}^* \to \mathcal{H}$ and hypothesis (model parameter) space $\mathcal{H}$, a centralized unlearning process $\mathcal{U}(D, D_e, M)$ (where $M = \mathcal{A}(D)$) must output $M_{\mathrm{unlearn}}$ such that:

\[
M_{\mathrm{unlearn}} \approx \mathcal{A}(D_r)
\]

### Exact Unlearning
Exact unlearning requires statistical indistinguishability:

\[
\Pr[\mathcal{U}(D, D_e, \mathcal{A}(D)) \in S] = \Pr[\mathcal{A}(D_r) \in S]
\]
for every measurable $S \subseteq \mathcal{H}$. This reflects perfect removal as if retrained from scratch on $D_r$ [2405.07406].

### Approximate Unlearning
Approximate (ε-) unlearning relaxes this to an $\varepsilon$-indistinguishability guarantee, typically formulated as:

\[
e^{-\varepsilon} \leq \frac{\Pr[\mathcal{U}(D, \{z\}, \mathcal{A}(D)) \in S]}{\Pr[\mathcal{A}(D \setminus \{z\}) \in S]} \leq e^{\varepsilon}
\]
which can be extended to $(\varepsilon, \delta)$ guarantees in analogy to differential privacy [2405.07406].

## 2. Algorithmic Taxonomy

### Exact Methods

- **Full Retraining:** The baseline, running $\mathcal{A}$ on $D_r$; cost is proportional to the original dataset size [2403.08254].
- **Data Partitioning / SISA (Sharded, Isolated, Sliced, Aggregated):** $D$ is split into $k$ shards $D^{(i)}$, each with submodel $M^{(i)}$. Erasure in $D_e \subseteq D^{(j)}$ triggers retraining only of the affected submodel, greatly reducing cost [2405.07406][2403.08254].
- **Mergeable Encodings / Ticketed Unlearning:** Compress $D$ into small mergeable encodings (central state) and issue per-sample tickets, enabling exact recovery of the model on $D_r$. Efficiency depends on concept class compressibility [2306.15744].
- **Coded Unlearning:** Use linear encodings to mix shards before training weak learners. On data deletion, update only affected coded shards and retrain corresponding learners, achieving perfect unlearning at minimal retrain cost [2012.15721].

### Approximate Methods

- **Certified Data Removal (Newton Step):** Remove influence using one-step Newton updates, adding noise/calibrated perturbation for $\varepsilon$-certified removal [2405.07406][2403.08254].
- **Influence Function / Fisher Information Based:** Apply $H^{-1}\nabla L(w; D_e)$ or $F^{-1}\nabla L(w; D_e)$ corrections, with optional noise for privacy guarantees [2403.08254].
- **Gradient Tracking (DeltaGrad):** Maintain and adjust cached gradients along training trajectory for efficient update [2405.07406].
- **Bayesian Posterior Unlearning:** Update posteriors to match $p(\theta | D_r)$ via reverse-KL and variational optimization [2405.07406].

## 3. Representative Algorithms

### Split-Shard (SISA) Unlearning

1. Partition dataset: $D \to \{D^{(1)}, ..., D^{(k)}\}$.
2. Train submodels independently, aggregate predictions.
3. On unlearning, retrain only affected submodel, update aggregate [2405.07406][2403.08254].

### Certified Newton Step (Guo et al.)

For regularized loss $L(\theta; D)$:
\[
\theta^- = \theta^* - H^{-1} \Delta + \text{noise}(\varepsilon)
\]
where $H$ is the Hessian on $D_r$ and $\Delta$ the gradient on $D_e$ [2405.07406].

### Coded Protocol

On each unlearning request, subtract erased sample’s contributions from coded shards, retrain only involved weak learners, and average outputs [2012.15721].

### Ticketed Scheme

Encode the data in compact central state and tickets, merge encodings using tickets of survivors to reconstruct unlearned model—enabling polylogarithmic space [2306.15744].

## 4. Theoretical Analysis and Complexity

| Method         | Computational Cost            | Guarantee                               |
|----------------|------------------------------|------------------------------------------|
| Full Retrain   | $O(|D|)$                     | Exact, statistical indistinguishability  |
| SISA Partition | $O(|D|/k)$                   | Exact on retained/affected shards        |
| Certified Newton| $O(d^3)$ (Hessian invert)   | $\varepsilon$-certified removal          |
| Influence/Fisher| $O(m)$ in erased set size   | Parameter closeness bound                |
| Coded Protocol | $O$(#affected weak learners) | Exact on convex models                   |
| Ticketed Unlearn| $O(\operatorname{polylog} n)$| Exact for mergeable classes             |

Capacity under DP unlearning is $m = \frac{n}{d^{1/4}}$ deletions before utility degradation [2405.07406].

## 5. Metrics and Verification

- **Anamnesis Index (AIN):** Ratio of relearning time for unlearned vs. retrained model ($\approx 1$ denotes complete unlearning) [2403.08254].
- **Activation/Weight Distance:** $\|\sigma(w̄^T x)-\sigma(w_r^T x)\|_1$, $\|w̄-w_r\|_2$.
- **Output Distribution KL/JS Divergence**
- **Test Accuracy:** Measured on $D_r$, $D_e$, and held-out data.
- **Representation Diagnostics:** PCA-similarity, PCA-shift, centered kernel alignment, Fisher information spectrum to detect reversible vs. irreversible forgetting, particularly for large models [2505.16831].
- **Verification:** Watermark-based triggers, membership inference, cryptographic proofs [2403.08254].

## 6. Trade-Offs, Case Studies, and Open Questions

### Practical Trade-offs

| Aspect              | Exact (SISA/Coded)                       | Approximate (Newton, Influence, Bayesian)   |
|---------------------|------------------------------------------|---------------------------------------------|
| Retrain Cost        | Sublinear in $|D|$ (per-shard/coded)     | Linear in erased set or matrix computation  |
| Storage             | Shard/encoding memory required           | Gradient/Hessian/VI state, typically smaller|
| Utility Loss        | None (sharded/coded), minimal (encode)   | $\varepsilon$-residual error possible       |
| Complexity          | Replay/merge, modular                    | Hessian/influence/VI machinery              |
| Verification        | Direct discriminative comparison         | Requires DP or statistical analysis         |

#### Empirical Benchmarks

- SISA yields retraining speedup up to $10\times$ with $2\times$ storage overhead; test accuracy drop $<0.5\%$ [2405.07406][2403.08254].
- Amnesiac training on MNIST erases $1\%$ of data in $2$ sec; full retrain is $\sim10$ min [2403.08254].
- Coded protocols on regression problems achieve $>20\%$ lower MSE at fixed cost compared to uncoded ensembles [2012.15721].

### Reversibility in LLM Unlearning

Shallow perturbations (e.g., to output layer weights) result in "reversible catastrophic forgetting," where original behavior is rapidly restorable, suggesting token-level metrics can misdiagnose true erasure. Only deep, multi-layer perturbations induce "irreversible catastrophic forgetting," detectable via representation-level metrics (PCA, CKA, Fisher spectra) [2505.16831]. This distinction is central to trustworthy unlearning in large models.

### Open Problems

- Characterization of concept classes admitting polylog-space ticketed unlearning [2306.15744].
- Design of irreversible, formally certified unlearning for deep models [2505.16831][2405.07406].
- Integration with privacy constraints and adversarial robustness.
- Streaming and multi-round unlearning extension for practical deployments.

## 7. Architecture and Implementation Considerations

- **Centralization Requirements:** Data partitioning, ticket issuance, code generation, and state management are orchestrated on the server; all computational branches (shard retrain, encoding merge, gradient update) are centrally supervised [2405.07406][2012.15721][2306.15744].
- **Communication Costs:** Partitioned/coded and ticketed approaches minimize retraining and communication overhead by only updating affected segments of the model/state, not global model parameters.
- **Warm-Start Optimization:** Valid for convex learners (ridge regression) but full retrain needed for deep networks to guarantee perfect unlearning [2012.15721][2405.07406].
- **Deployment:** Exact methods preferred for high-stakes privacy contexts; approximate methods beneficial for routine, low-latency, or high-frequency unlearning requests.

Centralized unlearning remains a rapidly evolving area with ongoing improvements in algorithms, complexity bounds, verification techniques, and large-model deployment strategies. It is instrumental for compliance, privacy, and control in contemporary machine learning infrastructures.

Source: https://www.emergentmind.com/topics/centralized-unlearning