---
title: 'Multimodal Unlearning: Methods & Challenges'
url: https://www.emergentmind.com/topics/multimodal-unlearning
type: topic
---

# Multimodal Unlearning: Methods & Challenges

Multimodal unlearning is the selective removal, suppression, or decoupling of knowledge from models that jointly process multiple modalities—most commonly image and text—while preserving retained capability on non-target data and tasks. In the literature, the object of forgetting is not uniform: it may be a paired cross-modal association, a visual identity, a private fact in multimodal question answering, a graph node or edge with high-dimensional multimodal features, or a relational composition such as an object–relation–object tuple [2311.12047]. Across these settings, the central technical difficulty is that multimodal knowledge is often encoded not only within modality-specific components, but also in cross-modal alignment, fusion, and shared embedding geometry, making unlearning harder than the unimodal case [2502.15910].

## 1. Historical emergence and problem formulations

The earliest dedicated multimodal formulation in this corpus is **MultiDelete**, presented as the first machine unlearning approach for multimodal data and models [2311.12047]. It frames multimodal unlearning as removing the effect of selected multimodal training samples from an already trained multimodal model without retraining from scratch and without destroying the model’s usefulness on the remaining data. Its central claim is that, for paired data such as image-text examples, the natural deletion target is often the **association** between modalities rather than the unimodal data themselves: the model should forget the relationship between paired modalities \((I_i,T_i)\in D_f\), not necessarily the individual unimodal elements \(I_i\) and \(T_i\) themselves [2311.12047].

Subsequent work diversified that target. In CLIP-like encoders, **CLIPErase** formulates forgetting as removal of **visual-textual associations** in a shared embedding space, so that forgotten image-text pairs no longer align in zero-shot prediction or retrieval [2410.23330]. In multimodal large language models, **MMUnlearner** reformulates the task as erasing only the **visual patterns associated with a given entity while preserving the corresponding textual knowledge encoded within the original parameters of the language model backbone** [2502.11051]. **MANU** instead targets **modality-aware neuron pruning**, motivated by the claim that prior MLLM unlearning is often imbalanced across multimodal and text-only access paths [2502.15910]. **ViKeR** moves to token-level granularity, arguing that prior objectives treat all answer tokens uniformly even though only a subset are privacy-bearing [2601.22020].

Another line of work treats multimodal unlearning as **safety unlearning** rather than deletion-equivalent sample removal. “Cross-Modal Safety Alignment: Is textual unlearning all you need?” studies whether unlearning solely in the textual domain can transfer to multimodal attacks in VLMs and reports that textual unlearning significantly reduces attack success for both text-based and vision-text-based attacks while preserving utility [2406.02575]. At a finer compositional level, **“Relationship-Aware Safety Unlearning for Multimodal LLMs”** argues that unsafe multimodal knowledge is often not a single concept but a **relationship** between otherwise benign entities, formalized as an object–relation–object tuple \(O_1-R-O_2\) [2603.14185].

A further expansion concerns structured data and domain-specific systems. **FDQ** studies multimodal graph unlearning in GNNs with high-dimensional multimodal node features such as CLIP or ImageBind embeddings [2605.03303]. **MedForget** introduces hierarchy-aware multimodal unlearning for medical AI, where deletion requests are attached to institution, patient, study, or report section rather than isolated instances [2512.09867]. **OFFSIDE** focuses on misinformation unlearning in MLLMs, especially football transfer rumors, and adds selective unlearning, corrective relearning, and unimodal unlearning settings [2510.22535].

This progression suggests a shift from sample-centric formulations toward finer semantic targets: associations, attributes, identities, tokens, graph structures, hierarchies, and relations. A plausible implication is that “multimodal unlearning” is best understood as a family of tasks defined by the granularity of the forgotten cross-modal dependency, rather than as a single canonical deletion problem.

## 2. Knowledge targets and representational granularity

A major organizing principle in the literature is the **unit of forgetting**. In pair-based encoder settings, MultiDelete and CLIPErase both treat the cross-modal association as primary. MultiDelete formalizes three desired properties: **modality decoupling**, **multimodal knowledge retention**, and **unimodal knowledge retention**, with modality decoupling requiring a deleted pair to become indistinguishable from an unrelated pair in representation space [2311.12047]. CLIPErase likewise states that after unlearning, image and text descriptions in \(D_f\) should no longer align, while retained multimodal behavior and unimodal embedding consistency should be preserved [2410.23330].

In MLLMs, several papers argue that the forget target is more nuanced than a single pair. MMUnlearner distinguishes **visual patterns** from **textual factual knowledge**, requiring that the model no longer identify a target entity from an image while still answering text-only questions about that entity correctly [2502.11051]. MANU identifies **neurons** whose activation statistics are disproportionately tied to forget data relative to retain data, separately for text-only and multimodal conditions, and prunes those neurons [2502.15910]. ViKeR defines **key tokens** through the entropy of an ideal post-unlearning token distribution estimated from irrelevant visual inputs, then reweights gradients so that privacy-bearing tokens receive stronger forgetting pressure than normal tokens [2601.22020].

Other work makes the granularity even more explicit. **AUVIC** studies **visual concept unlearning** in group scenes, where the goal is to forget one identity while preserving nearby identities in the same image [2511.11299]. **Relationship-aware safety unlearning** defines the forgotten object as a specific **unsafe edge** in a local relational neighborhood: the unsafe tuple should be suppressed while preserving the same objects in safe contexts, the same relation in other benign pairings, and unrelated neutral concepts [2603.14185]. **SALMUBench** pushes this further to **association-level** deletion, asking whether a face ↔ phone number or face ↔ email association can be removed without erasing other facts about the same identity or semantically neighboring identities [2603.26316].

Benchmark papers reveal why granularity matters. **PULSE** argues that unlearning in LMMs must be evaluated across both multimodal and text-only tasks because a model that stops answering when given a face image but still reveals the same person’s details in text-only prompts has not really unlearned the knowledge [2507.01271]. **UnLOK-VQA** similarly treats a target fact as a triple \((V,Q,A)\) and evaluates whether the answer remains recoverable under multimodal rephrasing, nearby multimodal variants, and hidden-state attacks [2505.01456]. **OFFSIDE** separates **shared** from **private** information for each player, then tests whether private rumor-specific facts can be forgotten while preserving shared player identity facts [2510.22535].

A recurring conclusion is that coarse objectives such as global concept suppression or exact sample suppression often produce collateral damage when the true target is a structured dependency. This is explicit in relationship-aware safety unlearning, which contrasts tuple-level editing with single-concept erasure and pair-level deletion [2603.14185].

## 3. Methodological families

The literature clusters into several methodological families.

**Gradient-based unlearning and decoupling** remain the most common baseline. MultiDelete turns its three desiderata into trainable losses: \(\mathcal{L}_{\text{MD}}\) for modality decoupling, \(\mathcal{L}_{\text{MKR}}\) for retained multimodal knowledge, and \(\mathcal{L}_{\text{UKR}}\) for unimodal knowledge retention, optimized by stochastic gradient descent with \(\alpha=\beta=\gamma=1\) [2311.12047]. In MLLMs, standard baselines include GA, GA with KL regularization, Gradient Difference, KL Minimization, and NPO, all adapted from text-only unlearning but applied to visual-question-answering or profile data [2507.01271]. The recurrent finding is that these methods can lower forget-set accuracy, but often at the cost of utility loss, text-only leakage, or poor sequential stability [2507.01271].

**Contrastive or similarity-based anti-alignment** is central in CLIP-like settings. CLIPErase combines a Forgetting Module that reduces similarity of forgotten matched image-text pairs, a Retention Module that preserves CLIP-style contrastive matching on retained data, and a Consistency Module that keeps image and text embedding distributions close to the original model on retained data [2410.23330]. In SALMUBench, Direct Similarity Minimization explicitly minimizes sensitive image-text similarity, but the benchmark shows that this can become catastrophically destructive, driving association metrics below the Clean baseline and damaging holdout identity and association structure [2603.26316].

**Parameter-efficient and geometry-constrained editing** appears in several forms. Relationship-aware safety unlearning uses **LoRA adapters on attention and projection layers** and a multi-term objective \(\mathcal{L}_3 + \alpha \mathcal{L}_2 + \beta \mathcal{L}_1 + \delta \mathcal{L}_4 + \gamma \mathcal{L}_c + \lambda_{\text{adv}}\mathcal{L}_{\text{adv}}\) to suppress an unsafe tuple while preserving neighboring relations and object marginals [2603.14185]. MMUnlearner computes Fisher-style saliency for a target set \(T\) and preserved set \(P\), then applies masked forgetting updates only to parameters whose target-to-preserved saliency ratio exceeds \(\beta=1\) [2502.11051]. FDQ keeps diagonal Fisher importance estimation unchanged but makes quantile-based parameter selection **feature-dimension aware**, shrinking the suppression rate in high-dimensional input projection layers that encode dominant cross-modal knowledge [2605.03303].

**Activation-space interventions** define a newer regime. **MLLMEraser** is a training-free, test-time method that constructs a multimodal erasure direction from contrastive refusal/compliance image-text pairs and applies an input-aware null-space-constrained linear steering map during inference [2510.04217]. **ASRU** first induces refusal behavior through activation redirection at a chosen residual-stream layer, then uses GRPO with a customized reward to refine a fine-grained refusal boundary between forget and retain examples [2605.15687]. These methods treat forgetting as conditional behavioral intervention rather than permanent parameter deletion.

**Pruning and masking** aim to localize forgetting structurally. MANU computes four activation-based importance functions—\(I_{\text{abs}}, I_{\text{freq}}, I_{\text{var}}, I_{\text{rms}}\)—aggregates them into \(\mathcal{I}(\mathcal{D},n)\), scores neurons by \(S_n=\frac{\mathcal{I}(\mathcal{D}_f,n)}{\mathcal{I}(\mathcal{D}_r,n)+\epsilon}\), and prunes the top \(\alpha\%\) neurons [2502.15910]. AUVIC uses vision-side LoRA plus adversarial image and prompt perturbations to maximize target concept activation, then suppresses it while preserving dynamically selected neighboring anchors via Gumbel-Softmax [2511.11299].

**Text-only or language-side interventions for multimodal safety** form a distinct line. “Cross-Modal Safety Alignment” freezes the visual encoder and projection layer, updates only the LLM parameters, and finds that textual unlearning can drive attack success rates on multimodal jailbreak datasets to single digits while preserving VQA utility [2406.02575]. The paper argues that in modern VLMs, non-text modalities are funneled into the LLM’s language space, so harmful generation behavior can sometimes be reshaped at the language bottleneck alone [2406.02575]. OFFSIDE later reports that unimodal methods fail on multimodal rumors, implying that such transfer is task- and representation-dependent rather than universal [2510.22535].

## 4. Evaluation regimes, benchmarks, and attack models

Evaluation in multimodal unlearning has become a research area in its own right. **MU-Bench** provides the first comprehensive benchmark for machine unlearning across image, text, speech, video, visual reasoning, and text-to-image generation, with standardized deletion splits from 1% to 10% of \(D_{\mathrm{Train}}\), released checkpoints, and a retrain-free scorecard over deletion-set performance, retain-set performance, test performance, unlearning time, and membership inference [2406.14796]. It finds that RandLabel and SalUn are the most effective general approaches overall, while Bad-T and SCRUB are most capable of pushing deletion-set performance toward random [2406.14796].

For multimodal dual encoders, **SALMUBench** introduces a more surgical protocol. It trains a **Clean** and a **Compromised** CLIP-like model from scratch on the same 400M-pair retain base, with only the Compromised model additionally trained on a synthetic sensitive set of about 60,000 persona-attribute image-text pairs [2603.26316]. It then evaluates forgetting efficacy through metrics such as Retrieval Failure, Association Strength, ACS, IdZSC, and CoreAssoc, and utility impact through GenKnow, InterIdSim, IntraIdSim, VisIdInt, and FragSim, using holdout identity and holdout association splits to expose over-generalized forgetting [2603.26316].

For MLLMs, benchmark design increasingly stresses adversarial robustness and realism. **UnLOK-VQA** constructs 500 manually filtered multimodal facts from OK-VQA and augments each target triple with rephrase images, rephrase questions, neighborhood images, and neighborhood questions, then evaluates six defense objectives against seven attacks, including hidden-state attacks such as Head Projection and Probability Delta [2505.01456]. Its main finding is that multimodal attacks outperform image-only and text-only ones, and that the most effective defense is one that removes answer information from internal model states rather than merely suppressing output probabilities [2505.01456].

**PULSE** critiques existing LMM unlearning benchmarks for only testing single-shot forgetting of fine-tuned knowledge and introduces two new axes: **pre-trained knowledge unlearning** and **long-term sustainability evaluation** under sequential unlearning requests [2507.01271]. It reports that methods that can moderately unlearn fine-tuned knowledge struggle badly with pre-trained knowledge, and that after five unlearning operations generality is almost completely lost [2507.01271]. **OFFSIDE** adds real-world image diversity, selective unlearning of rumor-specific information, corrective relearning, and unimodal-unlearning tests, and concludes that unimodal methods fail on multimodal rumors, unlearning efficacy is largely driven by catastrophic forgetting, all methods struggle with visual rumors, and all methods are vulnerable to prompt attacks [2510.22535].

Domain-specific benchmarks further refine evaluation axes. **MedForget** models hospital data as a nested hierarchy of Institution \(\supset\) Patient \(\supset\) Study \(\supset\) Section, with 3,840 multimodal VQA pairs, a fixed 25% forget ratio, rephrased image-question variants, and a reconstruction attack that progressively adds hierarchical context to prompts [2512.09867]. The main result is that fine-grained unlearning leaves models vulnerable to reconstruction, whereas coarser institution- or patient-level unlearning is more resistant but sacrifices more diagnostic utility [2512.09867]. **VCUBench**, introduced with AUVIC, specifically evaluates concept-level visual unlearning in group scenes through TFA, NTRA, GRF-F1, Efficacy, Generality, and Perplexity [2511.11299].

A shared trend is that evaluation has moved beyond “forget-set accuracy goes down” toward a multi-axis diagnosis of: exact forgetting, paraphrase and out-of-distribution robustness, hidden-state leakage, collateral damage to nearby associations, generation quality, sequential sustainability, and post-edit recoverability.

## 5. Empirical patterns and recurring failure modes

Several empirical patterns recur across otherwise different architectures and targets.

**First, forgetting–utility trade-offs are universal.** MultiDelete shows that modality decoupling alone is insufficient and that multimodal and unimodal retention losses are necessary to preserve test-set performance and unimodal embeddings [2311.12047]. CLIPErase shows a similar pattern: FM alone lowers forget-set performance, FM+RM restores retained accuracy, and FM+RM+CM produces the strongest combination of complete forgetting and high retention on CIFAR-100 [2410.23330]. In MLLMs, MANU’s pruning-ratio ablation shows that more pruning improves forgetting but degrades retain accuracy for both LLaVA and Idefics2 [2502.15910].

**Second, multimodal unlearning often fails asymmetrically across access paths.** PULSE reports that under GA, multimodal target accuracy can fall from \(78.0\) to \(9.6\) while text-only target accuracy only falls from \(76.8\) to \(35.2\), suggesting that a method may mainly break image-language alignment rather than erase the underlying textual knowledge [2507.01271]. MANU is explicitly motivated by this imbalance and reports more balanced forgetting across multimodal and text-only evaluation than GA, Gradient Difference, KL Minimization, and NPO [2502.15910]. This suggests that evaluation confined to one modality can systematically overestimate forgetting.

**Third, strong forget metrics can mask residual recoverability.** UnLOK-VQA finds that Fact Erasure attains rewrite score \(0.956\) with almost no random damage, yet still permits multimodal black-box attack success of \(0.455\) and white-box HP attack success of \(0.300\) [2505.01456]. OFFSIDE shows a similar phenomenon through corrective relearning: for KL on Qwen2.5-VL-7B, forget-set factuality after unlearning is \(0.57\), but rises to \(4.55\) after relearning, which the paper interprets as evidence that the rumor was concealed rather than deleted [2510.22535].

**Fourth, over-generalization is a dominant failure mode.** SALMUBench’s headline result is that current methods either fail to forget effectively or over-generalize by erasing more than intended [2603.26316]. DELETE and CLIPErase achieve strong forget metrics, but low InterIdSim and IntraIdSim show that they damage other identities and other facts about the same identity [2603.26316]. AUVIC frames this same problem at the visual concept level: unlearning Donald Trump with naïve gradient ascent harms Joe Biden and Boris Johnson as well, while AUVIC’s dynamic anchor preservation aims to prevent that spillover [2511.11299].

**Fifth, generation quality after unlearning is often under-optimized.** ASRU argues that prior methods evaluate output deviation while overlooking generation quality, leading to hallucinated or rigid responses, and reports average gains of \(+24.6\%\) in unlearning effectiveness and \(5.8\times\) in generation quality [2605.15687]. ViKeR makes a related argument at token level, showing that standard GA can corrupt normal tokens and yield malformed outputs while key privacy-bearing tokens remain insufficiently suppressed [2601.22020]. This indicates that low forget-set accuracy is not a sufficient proxy for usable post-unlearning behavior.

**Sixth, robustness to paraphrase, contextual variation, and OOD inputs is decisive.** Relationship-aware safety unlearning evaluates paraphrase, contextual, and OOD image attacks and reports substantial cosine similarity drops of \(0.6878\), \(0.4881\), and \(0.7012\) respectively while keeping preservation drift small [2603.14185]. PULSE, MedForget, and UnLOK-VQA all reach similar conclusions through different protocols: exact-prompt forgetting does not imply forgetting under realistic perturbations [2507.01271].

## 6. Open directions, controversies, and research trajectory

The field remains methodologically unsettled. One open question concerns **what counts as successful unlearning**. MU-Bench explicitly argues against treating retraining on \(D_r\) as the sole gold standard because retrained models can still retain deleted knowledge and may not even be practical in multimodal foundation settings [2406.14796]. SALMUBench, by contrast, leverages a Clean-from-scratch reference and statistical indistinguishability from Clean on efficacy metrics and from Compromised on utility metrics as its benchmark ideal [2603.26316]. UnLOK-VQA and OFFSIDE emphasize adversarial recoverability and hidden-state leakage over parameter-level equivalence [2505.01456].

A second controversy concerns **where unlearning should act**. Cross-Modal Safety Alignment reports that textual unlearning on the LLM component alone can strongly reduce multimodal attack success in LLaVA-style VLMs and that multimodal training can require about 6× more computation without clear gains if harm coverage is narrow [2406.02575]. OFFSIDE, however, concludes that unimodal methods fail on multimodal rumors because rumor knowledge is distributed across visual layers and fusion components [2510.22535]. This suggests architecture-dependent answers: language-bottleneck interventions may suffice for some safety-alignment regimes but not for misinformation or identity-specific multimodal memory.

A third active direction is **test-time versus training-time unlearning**. MLLMEraser shows that input-aware activation steering can yield stronger forgetting with minimal utility degradation and essentially zero training memory, but it provides behavioral suppression rather than formal deletion [2510.04217]. ASRU sits between these extremes by pairing representation steering with RL-based refinement [2605.15687]. A plausible implication is that reversible, inference-time methods may become attractive for urgent mitigation, whereas parameter-editing methods will remain relevant when persistent deletion-like behavior is required.

A fourth direction is **structure-aware unlearning**. Relationship-aware safety unlearning, FDQ, MedForget, and SALMUBench all argue that forgetting must respect model structure: relation graphs, high-dimensional projection layers, hierarchical data provenance, or identity–attribute association graphs [2603.14185]. This suggests that future multimodal unlearning may increasingly rely on explicit structural priors rather than uniform losses over forget examples.

Finally, scaling and deployment remain unresolved. MultiDelete and MMUnlearner demonstrate feasibility on ALBEF, BLIP, LLaVA, Qwen2-VL, and graph-text models [2311.12047]. Yet PULSE finds that existing methods are not ready for realistic deployment under pre-trained knowledge deletion and sequential requests [2507.01271], while MedForget shows that even state-of-the-art methods struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance [2512.09867]. The broader trajectory therefore points less toward a single dominant algorithm than toward a set of specialized techniques matched to representational regime: dual encoders, generative MLLMs, multimodal graphs, medical hierarchies, visual concepts, and relation-sensitive safety targets.

Source: https://www.emergentmind.com/topics/multimodal-unlearning