MIP-Editor: Multimodal Influential Neuron Path Editor

• MIP-Editor is a selective machine unlearning method that uses modality-specific attribution scores to identify influential neuron paths across text and vision modalities.
• It employs a two-stage process—pruning followed by RMisU fine-tuning—to erase targeted knowledge while retaining general model capabilities.
• Coordinated cross-modal editing ensures consistent forgetting, mitigating residual semantic traces typical of traditional unlearning approaches.

MIP-Editor refers to the Multimodal Influential neuron Path Editor, an advanced neuron-editing approach introduced in the context of machine unlearning (MU) for multimodal LLMs (MLLMs). MIP-Editor addresses the challenge of selectively unlearning targeted knowledge—especially knowledge encoded in specific neuron paths across both text and vision modalities—without sacrificing general model utility. It improves on prior work by leveraging modality-specific attribution scores and a path-based intervention mechanism, ensuring consistent forgetting across modalities and minimizing utility loss to unrelated tasks (Li et al., 10 Nov 2025).

1. Motivation and Problem Context

MLLMs, which jointly process text and visual data, are increasingly deployed in environments where data privacy, intellectual property, or regulatory requirements necessitate the selective removal of information from already-trained models (the "forget-set"). Standard neuron-pruning or gradient-based fine-tuning methods typically suffer two deficiencies:

• Inconsistent forgetting across modalities: Targeted concepts may be forgotten in one modality (e.g., vision) but persist in the other (e.g., text), due to local point-wise (neuron-level) editing that disregards cross-modal information flow.
• General knowledge degradation: Removing or attenuating neurons important for the forget-set frequently disrupts neuron-paths also needed for general-purpose reasoning, producing disproportionate drops in unrelated performance (Li et al., 10 Nov 2025).

MIP-Editor addresses these challenges by identifying and editing influential neuron paths—structured sequences of neurons traversing the model’s feed-forward networks (FFNs)—that are most responsible for encoding forget-set knowledge, and by intervening on those paths using a representation-misdirection loss.

2. Path-Based Attribution Mechanisms

MIP-Editor computes modality-specific attribution scores to systematically trace the flow of information through FFN layers in both the textual and visual branches of a transformer-based MLLM. The operational mechanisms are as follows:

• Textual Branch—Inter-layer Gradient Integration (IGI):
  • For a given text input $T$ and its label $Y$, candidate neuron paths $\mathbf{w} = (w_{i_1}^1, ..., w_{i_N}^N)$ are scored by integrating the cumulative effect of activations and their gradients spanning multiple FFN layers.
  • The IGI score aggregates these effects across all layers, capturing the total influence of a path on the predicted label:

  $$\mathrm{IGI}(\mathbf{w}) = \sum_{n=1}^N \tilde{w}_{i_n}^n \int_{0}^{\tilde{w}_{i_n}^n} \sum_{\ell=1}^{N} \frac{\partial F_T(\alpha_{i_1}^1, ..., \alpha_{i_n}^n)}{\partial w_{i_\ell}^\ell} d\alpha_{i_n}^n$$ - In practice, this is approximated with a finite $m$-point Riemann sum (Li et al., 10 Nov 2025).

• Visual Branch—Inter-layer Fisher Integration (IFI):
  • For a multimodal input $(I,T)$ with label $Y$, candidate visual neuron paths $\mathbf{z} = (z_{i_1}^1, ..., z_{i_N}^N)$ are scored using squared gradients (diagonal Fisher Information approximation) to quantify the sensitivity of path activations to the output probability.
  • The IFI score is computed as:

  $$\mathrm{IFI}(\mathbf{z}) = \sum_{n=1}^N \tilde{z}_{i_n}^n \sum_{k=1}^m \sum_{\ell=1}^N \left( \frac{\partial \mathbf{G}(\cdots)}{\partial z_{i_\ell}^\ell} \right)^2$$ - This scoring captures not just local, but propagated influence through high-dimensional encoders (Li et al., 10 Nov 2025).

3. Influential Neuron Path Selection and Editing

Influential neuron paths are selected separately in each modality by performing a greedy, layer-wise search for the maximally attributed path, iterating from the input to the output layer. For each layer, the neuron that maximizes the IGI or IFI score (depending on modality) is appended to the current path.

Once optimal influential paths $\mathcal{P}^t$ (text) and $Y$0 (vision) are located, MIP-Editor modifies the network in two stages:

• Stage 1: Pruning Activations on path neurons are zeroed out ($Y$1 for $Y$2, $Y$3 for vision paths).
• Stage 2: Representation Misdirection (RMisU) Fine-Tuning Only weights on the chosen paths are updated. For each forget-set sample $Y$4, the representation at a chosen layer is redirected toward a random vector, breaking alignment with its original activation:

$Y$5

For the retain-set $Y$6, representation preservation is imposed:

$Y$7

Only path weights are updated, minimizing:

$Y$8

This mechanism explicitly aligns or misaligns specific paths with desired semantics, unlike unconstrained model-wide fine-tuning.

4. Coordinated Cross-Modal Unlearning

Since cross-attention layers in the MLLM implicitly tie text and vision processing, editing both $Y$9 and $\mathbf{w} = (w_{i_1}^1, ..., w_{i_N}^N)$0 in tandem ensures that the targeted concept is unlearned across all modalities. The approach avoids explicit coupling losses; consistency is achieved through the simultaneous optimization of forget- and retain-set objectives during path editing (Li et al., 10 Nov 2025).

The effect is that sensitive knowledge—such as a fact present in both text and image—is erased regardless of input modality or where it is internally encoded, eliminating the phenomenon of "residual semantic traces" which previous point-wise methods allowed.

5. Empirical Evaluation and Baseline Comparison

Experimental evaluation leverages controlled datasets:

• MLLMU-Bench: Multimodal QA and captioning with precise forget/retain splits.
• CLEAR: Synthetic author-based profiling with explicit knowledge compartmentalization.

Performance is quantified using:

• Forgetting: Reduction in accuracy or ROUGE-L on the forget-set (lower is better).
• Retention: Preservation of performance on the retain-set (higher is better).

Method	Forgetting Rate (FVQA)	Retention (RVQA)
MIP-Editor	87.75%	+54.26% (increase)
Best Baseline	lower	lower

On Qwen2.5-VL/Instruct (5% forget set), MIP-Editor reduces FVQA from 39.20% to 4.80%, while RVQA increases from 37.72% to 58.19%. On textual QA, forgetting achieves 80.65% drop, yet 77.9% of overall utility is retained. Ablation shows that omitting either modality’s path consistency (IGI, IFI) or the RMisU step markedly reduces the efficacy or semantic alignment of unlearning (Li et al., 10 Nov 2025).

6. Significance, Limitations, and Future Directions

MIP-Editor demonstrates that principled, path-based neuron editing outperforms prior approaches by:

• Capturing and intervening on global "information highways" rather than isolated points, preserving semantically coherent features.
• Using the RMisU loss to decouple concept erasure from generic capability.
• Editing both text and vision branches for cross-modal consistency, which no previous approach explicitly ensured.

Identified limitations include:

• Significant computational overhead for inter-layer attributions in deep or wide models, though only a few path weights are fine-tuned.
• Current focus on FFN layers excludes possible gains from editing attention mechanisms or non-FFN modules.
• Scalability to online/streaming unlearning remains an open challenge.
• Unlearning broad, overlapping semantic categories (e.g., "politics") is not directly solved.

This suggests that future research may benefit from amortized or approximate path attribution, attention layer integration, and strategies for hierarchically entangled knowledge structures.

7. Summary Table: Core MIP-Editor Components

Component	Role	Key Mechanism
Modality attribution	Locate influential neuron paths in text/vision	IGI, IFI (integrated attribution)
Path selection	Build end-to-end high-attribution neuron sequences	Greedy, layerwise search
Path editing	Erase/unlearn targeted knowledge	Pruning + RMisU loss
Cross-modal coordination	Ensure semantic forgetting across modalities	Joint path editing, cross-attention

MIP-Editor, as introduced by Li et al. (2025), constitutes a decisive advance in modality-consistent, minimally intrusive machine unlearning for complex foundation models (Li et al., 10 Nov 2025).