--- title: Zero-Shot Unlearning url: https://www.emergentmind.com/topics/zero-shot-unlearning type: topic --- # Zero-Shot Unlearning Zero-shot unlearning is a family of techniques for selectively erasing data-induced knowledge from trained machine learning models, under the stringent constraint that no access to the original training data (except potentially the forget set) is permitted during the unlearning process. This paradigm addresses critical requirements for privacy, regulatory compliance (e.g., GDPR “right to be forgotten”), model decontamination, and downstream risk mitigation in both foundation models and domain-specific neural architectures. Zero-shot unlearning encompasses a variety of computational strategies including closed-form feature-space projections, adversarial proxy synthesis, gradient or subspace manipulations, and architectural design for post hoc instance removal. The core technical challenge is to precisely “forget” specific information while maintaining model fidelity on all unrelated tasks, often in highly overparameterized, multimodal, or structured prediction settings. ## 1. Foundational Principles and Problem Settings Zero-shot unlearning is formalized as the process of transforming a pretrained model $f_\theta$ into a new model $f_{\theta'}$ such that the influence of a designated forget set $\mathcal{D}_f$ (typically a class, individual, or statistical subset) is effectively erased, while utility on the retain set $\mathcal{D}_r$ is preserved. Critically, access to $\mathcal{D}_r$—which in conventional machine unlearning is required for re-optimization or influence estimation—is explicitly disallowed. In the strictest versions, only model weights and metadata about $\mathcal{D}_f$ (e.g., class labels or prompt text) are available [2201.05629], [2507.21738]. Contemporary frameworks extend zero-shot unlearning across multiple modalities and deployment settings: - **Multimodal vision–language models (e.g. CLIP):** Erasure of classes or domains via subspace projections in embedding space [2512.14113], [2512.14137], [2407.07485], [2506.03117]. - **Text-to-speech and biometrics:** Preventing unauthorized voice synthesis by zero-shot steering of hidden activations or randomized teacher–student protocols [2601.20481], [2507.20140]. - **Structured or codebook-based architectures in LLMs:** Deletion of discrete activation codes tied to unwanted concepts [2410.10866], [2311.15268]. - **Federated and personalized models:** Edge-device or client-specific zero-shot updates with verifiable proofs [2512.09953], [2604.04030]. - **Domain adaptation:** Erasure of source-exclusive knowledge in deployed source-free models [2604.08238]. The zero-shot constraint is distinct from “data-free” settings, as the latter may permit some retained proxy samples or distillation from intermediate representations. ## 2. Methodological Taxonomy of Zero-Shot Unlearning Zero-shot unlearning strategies divide into several major classes: ### 2.1 Closed-form Feature-Space Projections - **Nullspace/Orthogonal projections:** By constructing an orthonormal basis for the subspace spanned by the forget set (e.g., text and/or visual prototypes in CLIP), unlearning is achieved by projecting features orthogonally to this span [2512.14113], [2512.14137]. The core operator is $P = I - UU^\top$ where $U$'s columns form the forget subspace basis. This method enables efficient, data-free erasure at test time, without retraining. - **Partial (soft) projection:** Linear transform $W$ is selected to minimize a joint functional penalizing projections along forget directions while preserving retain-class structure, yielding a tunable tradeoff [2512.14137]. ### 2.2 Adversarial Proxy and Subspace Modeling - **Proxy data generation:** When $\mathcal{D}_r$ is inaccessible, adversarial perturbations of the forget set are optimized to cross the decision boundary into surrogate classes, forming a proxy for the retained distribution [2507.21738]. Singular value decomposition (SVD) is then used to identify the retained-feature subspace, enabling subspace-constrained gradient updates that prevent over-unlearning on $\mathcal{D}_r$. - **Statistical estimation of Hessians or Fisher matrices:** For parameter-centric models with convex losses, source-free unlearning can estimate the Hessian of the remaining data via second-order Taylor expansions using only small perturbations and loss differences on $\mathcal{D}_f$ [2508.15127]. Newton-type updates are then computed in closed-form. ### 2.3 Data Synthesis, Distillation, and Decoupled Representations - **Synthetic sample and generator-based approaches:** Generative Feedback Networks (GFNs) synthesize optimal erasure samples (OES) to maximize loss on forget classes, driving aggressive forgetting while a secondary “recovery” phase restores utility on the tiny available retained set [2511.13116]. Lipschitz regularization on both input and text embeddings in CLIP achieves similar effects [2407.07485], as does direct local smoothing [2402.01401]. - **Discrete key–value bottleneck and codebook intervention:** Discrete representational codes activated by the forget set are masked at inference, ensuring immediate zero-shot unlearning without retraining [2311.15268], [2410.10866]. Similar logics underpin “key deletion” architectures designed for instant memory erasure [2603.15033]. ### 2.4 Network Path Disruption and Relevance Analysis - **Layer-wise relevance analysis (LRA):** Highly relevant neurons for the forget class are detected via backward relevance propagation using only auxiliary proxies, and their outgoing weights are dropped or re-randomized (neuronal path perturbation, NPP). This severs classification paths while preserving utility [2410.23693]. ### 2.5 Inference-Time and Structural Approaches - **Inference-time steering:** In TTS, zero-shot unlearning occurs via dynamic, layer-selective subtraction of speaker-specific components from hidden activations at inference, suppressing identity with no retraining [2601.20481]. - **Unlearning by design:** Models such as MUNKEY are architected from the start for key-based instant forgetting, sidestepping gradient updates altogether [2603.15033]. ## 3. Formal Guarantees, Theoretical Results, and Metrics Zero-shot unlearning research addresses both empirical effectiveness and formal guarantees: - **Differential privacy and indistinguishability:** Some frameworks define success via output or parameter distributional closeness to an oracle retrained model (e.g., $P[f_{\theta'}\in W] \approx_{(\epsilon,\delta)} P[f_{\theta_{-\mathcal{D}_f}}\in W]$) [2402.01401], [2201.05629]. - **Bounding over-unlearning:** Approaches like ZS-PAG prove that subspace-projected gradient updates retain performance on $\mathcal{D}_r$ under Polyak–Łojasiewicz conditions [2507.21738]. - **Safety from attacks:** Membership-inference (MIA) and model inversion attacks are standard evaluation metrics; state-of-the-art zero-shot techniques (e.g., LRA+NPP, proxy-adversarial methods, nullspace projections) achieve post-unlearning MIA scores near random-guess on the forget set, matching retrained baselines [2512.14113], [2410.23693], [2311.15268]. - **Novel metrics:** Speaker-zero retrain forgetting (spk-ZRF) quantifies speaker identity randomness in TTS; Anamnesis Index (AIN) measures recoverability of forget-set performance under retraining [2507.20140], [2201.05629]. ## 4. Practical Implementations and Empirical Outcomes Empirical work on zero-shot unlearning spans datasets including CIFAR-10/100, SVHN, ImageNet-1K, PACS, DomainNet, and foundation models such as CLIP and T5-based LLMs. Notable implementation and outcome patterns: - **CLIP-specific frameworks:** Closed-form nullspace projections drop forget-class accuracy from $>98\%$ to $\sim 0.4$–$3\%$, with retain accuracy drops $<5\%$ and MIA improvements $20$–$40$ points versus strong synthetic-data or iterative baselines [2512.14113], [2512.14137], [2407.07485]. - **Vision classifiers:** Subspace-constrained (ZS-PAG) and proxy-based methods achieve forget-class test acc $\sim0\%$ and strong retention ($>75\%$ on CIFAR-100), outperforming data-free or random-label baselines [2507.21738], [2508.15127]. - **Discrete codebook (DKVB, CodeUnlearn):** Masking approximately $15$–$18\%$ of codes achieves $0\%$ accuracy on forget classes with no more than $0.2\%$ loss on retain classes—at near-zero computational cost [2311.15268], [2410.10866]. - **TTS and speaker unlearning:** Inference-time steering (TruS) reduces speaker similarity measure SIM-SO from $0.657$ to $0.477$ on opt-out speakers with word error rates preserved, matching the performance of high-cost retraining methods [2601.20481]. - **Federated and personalized settings:** Jellyfish achieves full erasure on $D_f$ and recovers within $3\%$ of original $D_r$ accuracy using only proxy data, while ZK-APEX enables verifiable unlearning proofs with $10^7 \times$ speedup over retraining-based verification [2604.04030], [2512.09953]. ## 5. Limitations, Trade-Offs, and Extensions Zero-shot unlearning methods, despite their efficiency and privacy alignment, face recurring challenges: - **Retain data unavailability:** Proxy generation for $\mathcal{D}_r$ or subspace estimation may degrade if the adversarial (or synthetic) proxy diverges from the real remaining data distribution [2507.21738], [2508.15127]. - **Over-unlearning:** Excessive smoothing, aggressive projections, or overbroad key masking can erode utility on non-target classes or tasks [2402.01401], [2512.14137]. - **Scalability and architectural assumptions:** SDP-based Hessian estimation scales poorly beyond $d \sim 10^5$, while neural collapse or codebook approaches are architecture-dependent and may require end-to-end retraining for capacity adaptation [2601.21564], [2311.15268]. - **Theoretical guarantees:** Most methods guarantee only local distributional indistinguishability or surrogate optimality (e.g., via influence estimates), with formal certified unlearning guarantees still limited [2512.09953], [2601.21564]. - **Task granularity and generalization:** Instance-level forgetting is more challenging than class-level, and extensions beyond classification (e.g., detection, segmentation, open-vocabulary generative tasks) are an open arena [2410.10866], [2604.08238]. ## 6. Emerging Directions and Applications Research is progressively advancing toward: - **Continual and streaming zero-shot unlearning:** Sequential handling of multiple, potentially overlapping forget requests, with utility decay management [2512.09953], [2201.05629]. - **Federated, edge, and verifiable protocols:** Cryptographically sound proofs of unlearning with low communication and compute overhead fit privacy-oriented deployments [2512.09953], [2604.04030]. - **Hybrid strategies and cross-modal extensions:** Combining proxy-based, projection, and codebook interventions with hybrid access to minimal retained data or synthetic anchors for robust utility [2511.13116], [2512.14137]. - **Broader impact and regulatory utility:** Deployment for right-to-be-forgotten compliance, model decontamination, bias mitigation, and remediation of unwanted correlations or offense content [2512.14137], [2604.08238]. - **Privacy and attack resistance:** Systematic evaluation against membership inference, model inversion, and reactivation attacks is standard in state-of-the-art baselines [2512.14113], [2311.15268], [2410.23693]. ## 7. Representative Methods and Empirical Results | Method/Domain | Mechanism | Retain Acc Impact | Forget Acc Drop | MIA/Privacy | |---------------------------------------------------|-----------------------------------------|-------------------|-----------------|-------------------| | CLIP Nullspace Proj. [2512.14113][2512.14137] | $I - UU^\top$ feature projection | $<5\%$ | $>90\%$ | MIA $+20$ points | | ZS-PAG [2507.21738] | Proxy subspace/PGD + projected update | $+2.2\%$ | $\sim0\%$ | matches retrain | | DKVB (sparse code) [2311.15268] | Mask codebook entries | $\le0.2\%$ | to $0\%$ | matches SCRUB | | TruS (TTS) [2601.20481] | Inference-time identity steering | no loss | SIM drops | Spk-ZRF: $0.87$ | | Jellyfish (Fed) [2604.04030] | Noise proxies + channel disent. | $\lesssim4\%$ | to $0\%$ | MIA $\approx$ retr | | ZK-APEX [2512.09953] | Mask+group-OBS + ZK proof | $>99\%$ recovery | $>39\%$ drop | Verifiably safe | These empirical results indicate that, when properly designed, zero-shot unlearning mechanisms can deliver targeted erasure and strong retention with minimal computational and data overhead, underpinning a rapidly maturing set of deployable solutions across the machine learning landscape. Source: https://www.emergentmind.com/topics/zero-shot-unlearning