Can VLMs Truly Forget? Benchmarking Training-Free Visual Concept Unlearning

Abstract: VLMs trained on web-scale data retain sensitive and copyrighted visual concepts that deployment may require removing. Training-based unlearning methods share a structural flaw: fine-tuning on a narrow forget set degrades general capabilities before unlearning begins, making it impossible to attribute subsequent performance drops to the unlearning procedure itself. Training-free approaches sidestep this by suppressing concepts through prompts or system instructions, but no rigorous benchmark exists for evaluating them on visual tasks. We introduce VLM-UnBench, the first benchmark for training-free visual concept unlearning in VLMs. It covers four forgetting levels, 7 source datasets, and 11 concept axes, and pairs a three-level probe taxonomy with five evaluation conditions to separate genuine forgetting from instruction compliance. Across 8 evaluation settings and 13 VLM configurations, realistic unlearning prompts leave forget accuracy near the no-instruction baseline; meaningful reductions appear only under oracle conditions that disclose the target concept to the model. Object and scene concepts are the most resistant to suppression, and stronger instruction-tuned models remain capable despite explicit forget instructions. These results expose a clear gap between prompt-level suppression and true visual concept erasure.

• The paper’s main contribution is the development of VLM-UnBench, a benchmark assessing training-free visual concept unlearning in VLMs.
• It employs multi-level VQA protocols across four forgetting levels to disentangle instruction compliance from true knowledge erasure.
• Empirical results indicate that prompt-based unlearning only alters output behavior without effectively erasing underlying visual representations.

Benchmarking Training-Free Visual Concept Unlearning in VLMs

Motivation and Problem Setting

The increasing deployment of vision-LLMs (VLMs) has heightened scrutiny around their ability to retain or erase sensitive, private, or copyrighted visual concepts. Conventionally, machine unlearning in deep models has relied on parameter updates, e.g., via fine-tuning, gradient ascent, or distillation, to suppress target concepts. However, these training-based protocols suffer from a fundamental confound: the very act of fine-tuning on narrow forget sets significantly erodes model capabilities prior to applying unlearning itself, making accurate attribution of resulting performance degradation challenging. For instance, application of standard gradient-based unlearning can reduce a model’s core performance dramatically independent of actual unlearning efficacy.

Training-free unlearning—where concepts are suppressed via system-level prompts or instructions in the frozen model—offers an attractive alternative, especially in settings where model weights are inaccessible (e.g., API endpoints). Yet, visual concept unlearning in VLMs via prompt manipulation lacked a rigorous, systematic evaluation framework. Existing benchmarks are either text-only, ad hoc, or do not separate instruction compliance from genuine forgetting.

VLM-UnBench: A Comprehensive Benchmark

VLM-UnBench is introduced as the first benchmark targeting training-free (prompt-based, zero-parameter update) unlearning of visual concepts in VLMs. Its core contributions encompass:

• Multi-level concept/semantic coverage: It spans four forgetting levels (object, scene, attribute, privacy), across 11 concept axes and 7 diverse vision datasets covering both identity and fine-grained attributes.
• Disentanglement of forgetting vs. instruction compliance: By leveraging a probe taxonomy with multiple evaluation protocols (including varying instruction strengths and oracle-type conditions that reveal the correct answer), VLM-UnBench distinguishes between true knowledge erasure and mere behavioral compliance.
• Real-image VQA probing: All evaluations are framed as four-choice VQA questions constructed from real images, with carefully designed distractor sampling to calibrate hardness. This addresses realistic visual concept suppression rather than synthetic or text-only settings.

  Figure 1: VLM-UnBench covers 4 forgetting levels (object, scene, attribute, privacy) over 11 concept axes, using targeted VQA probes and in-text unlearning instructions for frozen VLMs.

  

  Figure 2: The VLM-UnBench pipeline: real-world dataset curation, construction of class-level forget/retain splits, and axis-aware VQA item synthesis with hard/easy distractors and automated validation.

Evaluation Protocol

• Probe taxonomy: Three escalating levels—direct identification (P1), negation (P2), and confirmation with explicit concept suppression (P3)—systematically probe residual visual knowledge, even under instruction.
• Evaluation conditions:
  • Baseline (no instructions) provides ground-truth performance.
  • Unlearn_Soft and Unlearn_Medium prompt the model to avoid target concepts, simulating realistic “do not answer” scenarios.
  • Oracle_Hard and Oracle_Reverse grant the model explicit access to the ground-truth answer or its negation, measuring adherence to direct avoidance instructions.

Each experiment yields two metrics: forget macro-accuracy (class-wise on the forget split, where true forgetting should approach chance) and retain accuracy (on the retain split, measuring preservation of other capabilities).

Empirical Findings

Extensive experiments on 7 datasets with 13 VLMs (ranging from “Smol” to 8B+ parameters, including Instruct and non-Instruct variants) produced the following key observations:

• Minimal forgetting under realistic prompts: Across all settings, the application of prompt-based unlearning instructions resulted in minor changes in forget accuracy—models continued to recognize and select suppressed concepts robustly unless subjected to oracle prompts.

  Figure 3: Forget--retain tradeoff across conditions: under realistic prompt conditions, models maintain high performance on both forget and retain splits, highlighting the ineffectiveness of training-free unlearning.

• Dramatic accuracy drop only with oracular conditions: Only when the correct concept was explicitly provided and models instructed to avoid it (oracle conditions), was there a large decrease in selection of target concepts. However, this reflects instruction-following, not erasure of underlying knowledge.
• Forgetting resistance is concept- and model-dependent: Object and scene concepts are notably difficult to suppress, even for smaller models, while attribute- and privacy-level concepts yield more variable suppression. Strong Instruct-tuned models (e.g., Qwen3-VL-8B-Instruct) exhibit robust recognition and compliance in oracle scenarios, yet remain recalcitrant under realistic unlearning instructions.

Figure 4: On a per-model basis, Unlearn_Soft yields almost no reduction in forget accuracy (downward shift), confirming the insensitivity of training-free unlearning to prompt-based interventions.

• Retain performance unaffected: The collateral impact on non-target (retain) splits is minimal, showing that prompt-based unlearning is safe but lacks power.
• Scaling does not improve unlearning: Increasing model size, or stronger instruction tuning, does not systematically enhance unlearning efficacy. Instead, it predominantly boosts general recognition capability.

Implications and Directions for Future Research

The results emphasize that prompt-based, training-free interventions in VLMs do not remove or disrupt internal visual representations; they solely alter output surface behavior if the prompt is sufficiently direct. This exposes a significant gap: instruction-level compliance (answer avoidance) vs. true knowledge erasure. For scenarios demanding data deletion (privacy rights, copyright removal), current training-free VLM unlearning methods appear fundamentally insufficient.

From a practical lens, this suggests that regulated or safety-critical deployments cannot rely on API-level prompt conditioning to achieve robust concept suppression. Theoretical implications point to the recalcitrance of visually-grounded representations to superficial, context-level interventions—highlighting the necessity of more sophisticated, perhaps hybrid strategies that leverage both learned model behaviors and structural representation interventions.

Future work should aim to:

• Design benchmarks and metrics that can causally attribute performance drops specifically to knowledge erasure, independent of instruction-following.
• Develop methods that can manipulate or mask intermediate visual representations without catastrophic forgetting or collateral damage.
• Study interfaces between weight-space unlearning and inference-time interventions for more granulated and auditable control.

Conclusion

VLM-UnBench provides a rigorous, controlled, and multi-faceted evaluation suite for the study of training-free visual concept unlearning in VLMs (2604.03114). Its central finding is unequivocal: prompt-based, zero-shot instructions are generally incapable of constituting true unlearning of visual concepts under realistic settings; only engineered oracle prompts succeed, and then merely through behavioral compliance. The benchmark establishes a clear direction for future work seeking interpretable, auditable, and reliable erasure or suppression of visual knowledge in VLMs.