On the Robustness of Machine Unlearning for Vision-Language Models

Abstract: Vision-LLMs (VLMs) may memorize undesirable information from training data, motivating growing interest in machine unlearning. In this work, we present the first systematic survey and robustness analysis of VLM unlearning. We provide a comprehensive taxonomy and review of existing VLM unlearning methods, together with unified evaluations under multiple prompt settings. We then propose three attack paradigms to examine whether forgotten multimodal knowledge can be reactivated through contextual prompting or downstream retraining. Extensive experiments show that many existing methods remain vulnerable under these attacks, indicating that current approaches often hide rather than fully remove target knowledge. Our study provides new insights into the robustness and limitations of current VLM unlearning methods and highlights the need for more reliable multimodal unlearning strategies. Code is available at https://github.com/XMUDeepLIT/VLM-UnL-Attack.

• The paper shows that standard prompt-based evaluations overestimate forgetting, as contextual prompts and retraining can reactivate supposedly erased visual-language associations.
• The paper finds that retraining on just 1% of forget-set data reduces SLUG’s forgetting score from 95.67 to 13.00, while HFRU provides the strongest balance of forgetting, retention, and utility.
• The paper demonstrates that out-of-distribution retraining generally fails to restore forgotten identities but severely damages retention, highlighting the need for robustness testing beyond direct queries.

Motivation and scope

Machine unlearning for vision-LLMs (VLMs) has largely been evaluated under benign, direct prompting: a model is deemed to have forgotten an entity if it fails to name that entity when asked directly. The paper "On the Robustness of Machine Unlearning for Vision-LLMs" (2605.26992) argues that this evaluation protocol systematically overestimates forgetting. Because VLMs distribute knowledge across vision encoders, text encoders, and multimodal decoders, suppressing one output pathway may leave residual visual, textual, or cross-modal associations intact. The authors provide three contributions: a taxonomy and review of existing VLM unlearning methods, unified evaluations of representative methods under multiple prompt settings, and — the central contribution — the first systematic robustness study of VLM unlearning via three attack paradigms.

A taxonomy of VLM unlearning methods

The paper decomposes a VLM as $$f_{\theta}(x,t) = G_{\theta_d}(H_{\theta_v}(x); E_{\theta_t}(t))$$ and categorizes methods by which components are modified:

• Full-parameter finetuning: direct adaptations of LLM unlearning (GA, GD, NPO, SimNPO, RMU, UNDIAL) plus VLM-specific extensions such as FTTP, which applies gradient ascent only to image-grounded tokens. These offer expressiveness but risk catastrophic degradation of general capability.
• Vision-encoder finetuning: updates only $\theta_v$; e.g., HFRU uses supervised finetuning on incorrect labels or GRPO-style optimization penalizing keyword generation. The paper notes these tend to be more effective on discriminative queries than full-parameter approaches.
• Selective parameter finetuning (Mmunlearner, SLUG): optimizes a subset $\theta_s \subset \theta$ chosen via attribution or importance scores. Effectiveness depends critically on selection accuracy.
• Inference-time intervention (SAUCE, MANU, R-MUSE, CAGUL, MLLMEraser, MIP-Editor): feature masking, activation steering, or neuron pruning without parameter updates. Excluded from the attack experiments because retraining attacks require a trainable base model.

The formulation identifies three coupled objectives — forgetting on $\mathcal{D}_f$, retention on $\mathcal{D}_r$, and preservation of general multimodal capability — and the empirical results show most methods satisfy at most two.

Three attack paradigms

The attacks probe whether unlearning erases knowledge or merely hides it:

• In-context Attack: prepends semantically related but non-identifying context $C$ to the query ($t' = [C;t]$), testing whether latent associations can be reactivated without parameter changes.
• In-distribution (InD) Attack: fine-tunes the unlearned model on proportions $p \in \{1\%, 5\%, 10\%, 25\%, 50\%, 75\%, 100\%\}$ of forget-distribution data, using labels generated by the original pretrained model.
• Out-of-distribution (OOD) Attack: fine-tunes on semantically unrelated auxiliary data (PACS), testing whether residual representations resurface through distributional transfer.

All experiments use Qwen2.5-VL-3B-Instruct on a face-identity QA task built from VGGFace2 (10 celebrity identities: 3 forget, 7 retain), with utility measured on MMStar, OCRBench, MMMU, and RealWorldQA.

Baseline evaluation reveals prompt-dependent failures

Under original, paraphrased, and discriminative prompts, full-parameter methods achieve near-perfect forgetting on generative queries but destroy retention (e.g., GA: For. 100.00, Ret. 0.00). More importantly, several methods that appear successful generatively fail on discriminative prompts — RMU scores only 23.33 forgetting and SatImp 0.00 there — implying visual associations remain partially preserved even when textual generation is suppressed. HFRU is the standout, achieving 99.67/99.14 average forgetting/retention with utility preserved (60.47), supporting the claim that directly modifying visual representations removes identity concepts more thoroughly than decoder-focused suppression.

In-context attacks reactivate suppressed knowledge

Contextual cues partially reactivate "forgotten" identities across paradigms: RMU's forgetting drops from 54.00 to 20.67, SatImp from 20.67 to 15.67, WGA from 39.67 to 36.00, RAZOR from 94.00 to 85.00, and SLUG from 95.67 to 67.33 under attack. Notably, the attack succeeds against vision-encoder and selective-finetuning methods as well as full-parameter ones, indicating the vulnerability is paradigm-general. The paper also reports a counterintuitive side effect: retain-set performance often improves under contextual attack, attributed to shared background attributes (professions, political roles) between forget and retain identities providing incidental semantic cues. The mechanistic explanation offered is that contextual priors in language-model hidden states align with residual or weakened visual features during cross-modal interaction.

InD retraining rapidly restores forgotten knowledge

This is the paper's strongest quantitative result: retraining on only 1% of the forget set reduces SLUG's unlearning effectiveness by over 80% (forgetting drops from 95.67 to 13.00 on original prompts). WGA, RAZOR, and RMU show similarly rapid recovery at low attack ratios. Mmunlearner is the most robust method, degrading significantly only after retraining on 50–75% of the forget set. Retention is comparatively stable up to 10% retraining data before declining. Under discriminative prompts, trends are inconsistent — some methods appear to improve forgetting, but the authors attribute this artifact to general capability collapse after aggressive retraining rather than genuine robustness. The implication is that many methods perform shallow suppression of token-level behavior while leaving underlying representations recoverable.

OOD retraining harms retention, not forgetting

Contrary to what a residual-representation hypothesis might predict, OOD retraining does not restore forgotten knowledge; forgetting performance remains stable or even improves across all prompt settings. Instead, retention degrades consistently and severely as OOD data increases — for instance, at 1000 PACS samples, nearly all methods drop to near-zero retain accuracy on discriminative prompts. The authors conclude that gradients from unrelated distributions fail to reactivate target-specific associations but do damage retained capabilities. This finding bounds the threat model: distributional transfer alone does not resurrect forgotten identities.

Limitations and open questions

The paper concedes several constraints plainly. The unified evaluation protocol simplifies implementation details per method, which may affect absolute performance figures. Contextual attack prompts are manually constructed; automated or more diverse prompt-generation strategies remain unexplored. The study covers a single backbone (Qwen2.5-VL-3B-Instruct), a single unlearning domain (face identity), and excludes inference-time intervention methods from retraining attacks. Open questions include whether HFRU-style vision-encoder interventions resist stronger contextual attacks at scale, and whether any current method achieves removal deep enough to survive InD retraining at moderate attack ratios.

Conclusion

This work establishes that standard prompting-based evaluation substantially overstates the completeness of VLM unlearning. Across thirteen representative methods spanning three paradigms, in-context prompting and small-scale in-distribution retraining reliably reactivate supposedly forgotten identity knowledge, while out-of-distribution retraining primarily damages retention instead. Vision-encoder finetuning — particularly HFRU — currently offers the best forgetting-retention-utility balance, though it too is not immune to contextual reactivation. The results argue that future VLM unlearning methods should be validated against reactivation and retraining attacks rather than direct-query suppression alone.