---
title: 'ZS-PAG: Zero-Shot Machine Unlearning'
url: https://www.emergentmind.com/topics/zs-pag
type: topic
---

# ZS-PAG: Zero-Shot Machine Unlearning

ZS-PAG is an abbreviation for Zero-Shot Machine Unlearning with Proxy Adversarial Data Generation, a framework for data removal from trained classifiers when only the unlearning samples and the trained model are accessible, with no access to the remaining dataset. ZS-PAG circumvents the reliance on remaining data by using adversarially generated proxy samples, subspace-constrained updates, and an influence-based pseudo-labeling procedure. It forms the first known method to explicitly address the zero-shot unlearning setting, providing both theoretical guarantees for risk preservation and empirical superiority over prior unlearning baselines [2507.21738].

## 1. Problem Setting and Objectives

ZS-PAG addresses the machine unlearning scenario where, after model deployment, it becomes necessary to eliminate the influence of a specific subset of training data, $\mathcal{D}_u$, from a trained model $f(\cdot;\theta_0)$. The classic paradigm assumes access to the remaining subset $\mathcal{D}^r$ so as to regulate model updates and avoid the phenomenon of over-unlearning, where model utility collapses. In ZS-PAG, by contrast, only the model $f(\cdot;\theta_0)$ and the unlearning data $\mathcal{D}_u$ are available, with $\mathcal{D}^r$ being entirely inaccessible.

The optimization goal is to obtain parameters $\theta_u$ such that: (a) the model's predictive accuracy on the unlearning data is driven to chance, i.e., the class to be unlearned is “forgotten”; and (b) the risk $L^r(\theta)$ over the unseen remaining data is preserved near its original value $L^r(\theta_0)$.

## 2. Proxy Adversarial Data Generation

ZS-PAG generates synthetic samples to mimic the distributional role of unavailable remaining data. For each $(x_i, y_i)\in\mathcal{D}_u$, a proxy adversarial example $x_i+\delta^*_i$ is constructed by PGD (Projected Gradient Descent), targeting the model's second highest logit:
\[
y_\text{target} = \arg\max_{k \neq y_i} f_k(x_i; \theta_0).
\]
The optimization is:
\[
\delta^*_i = \arg\min_{\|\delta\|_p \leq \epsilon} \ell(f(x_i+\delta; \theta_0), y_\text{target}),
\]
yielding an adversarial proxy set $\mathcal{D}_\text{adv} = \{(x_i+\delta^*_i, y_\text{target})\}$. These “proxies” are empirically shown to cluster with the remaining data classes, enabling estimation of the effect of unlearning updates.

## 3. Subspace-Constrained Unlearning

To mitigate over-unlearning, ZS-PAG constrains the gradient-based updates during parameter refinement to be orthogonal to the estimated subspace occupied by the features of the remaining classes, as reconstructed from the adversarial proxy data. For each layer $\ell$ and each class $k \neq c$ (where $c$ is the unlearning class), the activation matrix $R_\ell^k$ is formed and subjected to SVD to yield $U_\ell^k$, the orthonormal basis of class-$k$ features. All such bases are concatenated and re-orthonormalized to form $S_\ell^{\bar{c}}$, the overall retained-class subspace. The orthogonal projector $P_\ell = I - S_\ell^{\bar{c}} (S_\ell^{\bar{c}})^T$ is then applied:
\[
w_{t+1}^\ell = w_t^\ell - \eta P_\ell \nabla_{w^\ell} \ell(f(x), y),
\]
ensuring parameter adjustment does not disrupt directions informative to the retained data classes.

## 4. Influence-Based Pseudo-Labeling

Within the projected (orthogonal complement) subspace, ZS-PAG determines optimal pseudo-labels $y_{pl}$ for proxy adversarial samples to minimize first-order risk shifts for the (unseen) remaining data. Influence functions as described by Koh and Liang are used:
\[
\mathcal{I}_z = - H_{\theta_0}^{-1} \nabla_\theta \ell(z;\theta_0).
\]
The total first-order effect on $L^r$ by removing $x_i$ with soft label $y$ is:
\[
\Delta L^r \approx \frac{1}{N}[{\nabla_\theta L^r(\theta_0)}]^T H_{\theta_0}^{-1} \nabla_\theta \ell(x_i, y; \theta_0).
\]
The pseudo-label is optimized to minimize this risk increment subject to a KL-penalty with respect to the original label, using projected gradient steps over the $(C-1)$-dimensional probability simplex.

## 5. Combined Algorithm

ZS-PAG proceeds in four sequential stages:

1. **Proxy Generation:** Adversarial examples corresponding to the unlearning samples, targeting the nearest non-original logit.
2. **Subspace Estimation:** SVD-based local subspaces per class and layer, aggregated into a global retained-class subspace.
3. **Pseudo-Label Optimization:** Influence-based risk minimization with KL-penalty.
4. **Projected Unlearning:** PGD updates, per-layer projection using $P_\ell$, and pseudo-label supervision.

Empirically, these techniques yield near-complete forgetting ($\text{Acc}_u\rightarrow 0$) and preservation or slight enhancement of retained-class accuracy ($\text{Acc}_r$). For example, on CIFAR-100/ResNet: original $\text{Acc}_r$ 73.31%, ZS-PAG $\text{Acc}_r$ 75.54% [2507.21738].

## 6. Theoretical Guarantee

ZS-PAG possesses a convergence guarantee under the Polyak–Łojasiewicz condition for the “remaining risk” functional. If $L^r$ is $L$-smooth and admits a PL constant $\mu>0$, then with learning rate $\eta\in (0,2/L)$ and projected-GD unlearning updates:
\[
L^r(\theta_u) - L^r(\theta^*) \leq (1-\eta\mu)^{T_u} [L^r(\theta_0) - L^r(\theta^*)],
\]
implying exponential convergence to the original performance as $T_u \rightarrow \infty$. The subspace-projection step is critical to the preservation of the retained-data risk [2507.21738].

## 7. Experimental Evaluation and Impact

ZS-PAG has been benchmarked on multiple vision datasets and network architectures, including Facescrub/AlexNet, SVHN/VGG, CIFAR-10/ViT, and CIFAR-100/ResNet. The key metrics reported are accuracy on unlearned and retained classes and membership inference attack (MIA) robustness. ZS-PAG consistently outperforms zero-shot and data-free baselines, both achieving effective forgetting and maintaining or improving accuracy on retained data ($\text{Acc}_u \approx 0\%$, $\text{Acc}_r \gtrsim$ original).

Component ablations demonstrate that adversarial proxies are adequate for approximating the remaining distribution, and that the combination of subspace projection and influence pseudo-labeling achieves maximal retained-risk preservation. Robustness is reported for different adversarial strengths and attack methodologies. The framework is also extensible to multiclass and multiattribute unlearning.

ZS-PAG’s principled approach to zero-shot unlearning expands the applicability of unlearning procedures and provides a template for risk-preserving, data-minimal interventions in model lifecycle management [2507.21738].

Source: https://www.emergentmind.com/topics/zs-pag