---
title: Analytic Continual Unlearning (ACU)
url: https://www.emergentmind.com/topics/analytic-continual-unlearning-acu
type: topic
---

# Analytic Continual Unlearning (ACU)

Analytic Continual Unlearning (ACU) is a class of gradient-free, mathematically grounded procedures for sequentially, efficiently, and exactly forgetting specific data or knowledge from machine learning models without violating data privacy or incurring performance degradation on retained information. ACU methods have emerged to address the limitations of prior "machine unlearning" approaches in continual learning (CL) regimes, particularly under constraints where historical data cannot be revisited after ingestion, and unlearning requests may be frequent, adversarial, or cumulative. Core works in this domain provide fully analytic, closed-form solutions to continual unlearning—rooted in linear algebra, least-squares regression, and controlled knowledge distillation, and are applicable to neural, analytic, and modular architectures [2505.12239][2408.11374][2505.15178][2407.10223].

## 1. Problem Formulation and Privacy Constraints

Continual Learning (CL) aims to incrementally incorporate knowledge from a stream of disjoint tasks by sequentially updating the model. In standard CL, once the training data of a task is ingested, it is discarded and cannot be revisited due to privacy, storage, or policy constraints. Continual Unlearning (CU) formalizes the challenge of allowing post-hoc deletion of specific data—at the granularity of individual samples, classes, or distributions—while maintaining fidelity on all remaining knowledge.

Key requirements that distinguish ACU from prior “single-shot” or retrain-based unlearning approaches:
- **No Access to Retained Data**: At each step, only the current forget set is available; all other data are irrecoverably discarded.
- **Efficient, Cumulative Operation**: The unlearning algorithm must remain computationally efficient regardless of the number or frequency of requests, and accuracy on retained data must not degrade under repeated unlearning.
- **Exactness**: The resulting model should match the (infeasible) retrained-from-scratch baseline on the remaining data set, up to numerical precision.
- **Data Privacy**: Internal data structures cannot leak or permit reconstruction of the original data; privacy against membership inference and data extraction is required [2505.12239].

## 2. Core Analytic Frameworks

### 2.1 Analytic Ridge Regression Unlearning

The “ACU” architecture introduced in [2505.12239] is grounded in ridge-regularized least-squares classifiers operating over frozen features. Training proceeds by extracting features $f_j$ and labels $y_j$, then solving
$$
W_0 = \left( \sum_{j \in D} f_j^\top f_j + \gamma I \right)^{-1} \sum_{j \in D} f_j^\top y_j
$$
where $W_0$ is the starting classifier after the CL phase.

When a batch $\mathcal{D}_i$ is to be forgotten, the analytic knowledge-tracker matrix $T_{i-1}$ and weights $W_{i-1}$ are updated via Woodbury identities:
$$
\begin{aligned}
T_i & = T_{i-1} + T_{i-1} \hat{F}_i^\top (I - \hat{F}_i T_{i-1} \hat{F}_i^\top )^{-1} \hat{F}_i T_{i-1} \\
W_i & = \left[ I + T_i \left( \sum_{j \in \mathcal{D}_i} f_j^\top f_j \right) \right] W_{i-1} - T_i \left( \sum_{j \in \mathcal{D}_i} f_j^\top y_j \right)
\end{aligned}
$$
with $\hat{F}_i$ the to-be-forgotten feature matrix. This update provably yields $W_i$ equivalent to retraining on the retained set, without ever revisiting or storing it.

### 2.2 Controlled Distillation in Deep Networks

In [2408.11374], analytic continual unlearning is formalized for deep models using a sequence of teacher–student distillation losses. 
- A dual-teacher mechanism is used: a CL-teacher representing retained knowledge, and a “bad” unlearning teacher (randomly initialized) representing the desired state for forgotten data.
- The loss for unlearning requests mixes KL-divergence terms that both “pull” the model away from data being forgotten and "preserve" responses on retained data, formulated as:
$$
L_{\rm cu} =
(1 - l_u)\, \mathrm{KL}(f_{\Theta_T,\Phi_T}(x)\,||\,f_{\Theta_s,\Phi_s}(x))
+ l_u\, \mathrm{KL}(f_{\Theta_b,\Phi_b}(x)\,||\,f_{\Theta_s,\Phi_s}(x))
$$
where $l_u$ indicates whether $x$ is in the forget set or buffer.
- The system leverages a fixed-size memory buffer with reservoir sampling for continual learning requests and explicit purging for unlearning, allowing bounded, analytically characterized trade-offs between utility and unlearning efficacy.

### 2.3 Unified Optimization Theory for CL-Unlearning

[2505.15178] presents ACU as a unified Kullback-Leibler divergence minimization:
$$
\theta_{k+1} = \arg\min_\theta \; D_{\mathrm{KL}}\!\bigl(p_z(\theta_*) \;\|\;p_z(\theta)\bigr) + \frac{1}{\alpha_k}\rho(\theta_k, \theta)
$$
with explicit decomposition into learning, unlearning, replay (retention), and saliency modulation terms using second-order expansions. The step is:
$$
\Delta\theta_k \approx -\alpha_k \left[ \nabla_\theta \mathcal{L}^R(\theta_k)\,p^R + \frac{1}{2}(H^L_k\,p^L + H^U_k\,p^U)(H^R_*)^{-1} ( \nabla_\theta \mathcal{L}^L + \nabla_\theta \mathcal{L}^U ) \right]
$$
with Hessian and saliency-based adjustments for stable and plastic unlearning.

### 2.4 Modular Orthogonal Adapter Unlearning

For large language models, [2407.10223] describes the OOO (Orthogonal LoRA + OOD detector) framework. Each unlearning request instantiates a rank-$K$ LoRA module per block, trained with:
$$
\mathcal{L}_{\mathrm{LoRA}}^t = \mathcal{L}_{\mathrm{CE}}^t + \lambda \mathcal{L}_{\mathrm{Orth}}^t
$$
where $\mathcal{L}_{\mathrm{Orth}}^t$ enforces strict orthogonality between adapters for different unlearning requests, thus preserving prior removals. A glocal-aware out-of-distribution detector (contrastive entropy + Mahalanobis/cosine-layer scoring) determines activation of unlearning adapters during inference. No retained data are required; operations are data-disjoint and privacy-compliant.

## 3. Algorithmic Procedures and Theoretical Guarantees

### 3.1 Pseudocode Summary

A typical ACU step (ridge-based, [2505.12239]):

```python
Input:  W0, T0, sequence of forget sets {D̆₁,…,D̆_K}
Output: WK

for i in 1…K:
    Extract features f_j, labels y_j for (x_j,y_j) in D̆_i
    ĤF_i = row-stack of f_j
    // Update knowledge tracker
    M = I – ĤF_i T_{i−1} ĤF_iᵀ
    T_i = T_{i−1} + T_{i−1} ĤF_iᵀ (M⁻¹) ĤF_i T_{i−1}
    // Exact unlearning update
    S_f = sum of f_jᵀ f_j; S_y = sum of f_jᵀ y_j over D̆_i
    W_i = [I + T_i S_f] W_{i−1} − T_i S_y
return WK
```

Empirically, each unlearning request is handled in linear time with respect to the forget set size and feature dimension. The underlying matrices $T_i$ and $W_i$ are of minimal dimension, and do not leak information about past data.

### 3.2 Theoretical Properties

- **Exactness**: $W_i$ matches the sub-sampled ridge regression solution, identical to full retraining on the retained set [2505.12239].
- **No Historical Data Access**: Only needs current forget set plus current ($T_{i-1}$, $W_{i-1}$).
- **Privacy**: The knowledge-tracker $T_i$ cannot be inverted to reconstruct individual features or samples.
- **Interpretable Decomposition**: Each update is explicitly split into “amplification” of remaining knowledge and erasure of forgotten knowledge.
- **Trade-off Characterization**: Methods provide an analytic characterization of the trade-off between buffer size, utility retention, and unlearning sharpness, e.g. $P_{\rm tot}(N) = \alpha\log N(1 - \frac{\beta}{N})$ for buffer size $N$ [2408.11374].

## 4. Empirical Evaluation and Results

### 4.1 Experimental Protocols

Representative studies [2505.12239][2408.11374] use:
- Datasets: CIFAR-10, CIFAR-100, ciFAIR-10.
- Evaluation over 5 to 25 sequential unlearning events.
- Metrics:
    - Parameter gap: $\Delta_{\text{Params}} = \frac{\|W_K - \hat{W}_K\|_2}{\|\hat{W}_K\|_2}$.
    - Accuracy gaps: $\Delta_{\text{Retain}}$, $\Delta_{\text{Forget}}$, $\Delta_{\text{Test}}$.
    - Membership inference attack robustness.
    - Cumulative runtime for all unlearning requests.

### 4.2 Key Quantitative Findings

| Method        | Δ_Params | Δ_Retain | Δ_Forget | Δ_Test | Δ_MIA |
|---------------|----------|----------|----------|--------|-------|
| Finetune      | 35.4     | 6.1      | 4.2      | 4.2    | 0.04  |
| L1-Sparsity   | 25.4     | 15.2     | 10.3     | 10.5   | 0.10  |
| RandomLabel   | 26.8     | 3.3      | 2.5      | 3.9    | 0.03  |
| ACU           | **0.00** | **0.00** | **0.00** | **0.00**| **0.00** |

ACU achieves zero gap on all metrics, matching the retrained-from-scratch baseline, while running 50–125× faster than state-of-the-art matrix-influence or distillation baselines, and over 10,000× faster than full retrain. On CIFAR-10 and CIFAR-100, accuracy on unlearned classes drops to exactly zero, with >97% retention on preserved classes with typical buffer sizes and minimal collateral utility loss [2505.12239][2408.11374].

## 5. Privacy, Limitations, and Future Research

ACU methods guarantee privacy by design: no retained raw data or reconstructible statistics are ever stored beyond the analytic matrices or memory buffers, and membership inference attack rates match the retrained baseline. However, several limitations persist:
- **Frozen Backbones**: In [2505.12239], only the analytic classifier weights are subject to unlearning; representation backbones remain unchanged. Unlearning at the feature learning level remains an open research direction.
- **Analytic Layer Scope**: Current ACU approaches focus on single-layer or shallow analytic models; extending to kernelized or deep analytic networks is an open problem.
- **Buffer-Utility Trade-off**: In methods employing replay buffers, larger buffers improve retention but can degrade unlearning sharpness; this log–$1/N$ trade-off is analytically described [2408.11374].
- **Generalization to Large Models**: Recent advances extend ACU principles to modular architectures (e.g., orthogonal LoRAs for continual LLM unlearning), but full, end-to-end ACU in high-capacity models presents significant challenges [2407.10223].

Ongoing work aims to support end-to-end unlearning, introduce certified privacy (e.g., via differential privacy), and leverage higher-order analytic approximations for more expressive unlearning [2505.12239][2408.11374][2505.15178].

## 6. Broader Applicability and Methodological Variants

ACU’s analytic framework has been instantiated in several practical and theoretical forms:
- **Gradient-Free Ridge Solutions**: Suited for settings where features are fixed and historical data must remain private.
- **Dual-Teacher Distillation**: Enables interpretable, exact unlearning for deep architectures with minimal retained data, supporting a spectrum of tasks and operational scenarios [2408.11374].
- **Unified KL-Minimization**: Provides a principled optimization-theoretic base for simultaneously handling continual learning, unlearning, and retention in a unified descent loop [2505.15178].
- **Orthogonal Subspace Allocation in Modular Models**: Applied in LLMs, this approach allows each sequence of continual unlearning operations to be cleanly separated, supporting non-interfering, data-private, and composable forgetting [2407.10223].

In all cases, analytic continual unlearning provides an interpretable and efficiently computable solution for privacy-preserving, sequential, and exact knowledge erasure, advancing the state of the art in dynamic, compliant, and modular machine learning systems.

Source: https://www.emergentmind.com/topics/analytic-continual-unlearning-acu