Evaluating and Understanding Model Editing for Medical Vision Language Models

Abstract: Model editing promises a fast, targeted way to correct post-deployment mistakes in medical vision-LLMs (VLMs) without costly retraining. However, existing multimodal model editing benchmarks focus on general-purpose tasks and do not reflect realistic clinical domain requirements and variability. To address this, we introduce M3Bench, a clinically grounded benchmark for multimodal model editing that evaluates whether an edit remains reliable, precise, and generalizable under the challenges of image and text variation, modality and protocol shifts, clinical knowledge composition, and temporal progression. M3Bench contains 16,276 questions spanning diverse anatomy, modalities, and specialties, and supports both single and sequential edits. By evaluating 4 representative editors across 6 medical and general VLMs, we find that no method excels across all criteria. Gradient-based editors achieve strong transfer but suffer from catastrophic locality violations, whereas memory-based methods preserve locality but lack compositional generality and exhibit high backbone-dependent hyperparameter sensitivity. We further attribute these failures to the latent space geometry of VLMs and how different editing methods shift its landscape. Overall, M3Bench establishes a rigorous clinical stress test for multimodal model editing and offers actionable guidance for safer post-deployment adaptation. The benchmark is publicly available at https://github.com/BioMed-AI-Lab-U-Michgan/M3Bench .

• The paper introduces the M³Bench, a clinical benchmark that rigorously evaluates model editing on medical vision-language models across reliability, locality, generality, and temporality.
• The paper compares gradient-based and memory-based editing methods, highlighting that while LoRA offers strong reliability and generality, it suffers from significant locality violations.
• The paper provides a geometric analysis revealing latent space anisotropy in medical VLMs, which challenges compositional and temporal consistency of editing techniques.

Evaluating and Understanding Model Editing for Medical Vision-LLMs

Introduction and Motivation

The paper "Evaluating and Understanding Model Editing for Medical Vision LLMs" (2607.05310) tackles the rigorous assessment and explanation of model editing in large-scale Medical Vision-LLMs (VLMs), targeting their post-deployment adaptation for real-world clinical requirements. Model editing—in contrast with global retraining or fine-tuning—offers an efficient, localized mechanism to correct model failures or update knowledge without incurring computational and data overheads. While the literature on model editing for general-domain LLMs and VLMs has matured, evaluation frameworks have not kept pace with the higher variability and precision needed for safety-critical medical tasks. This work introduces a comprehensive, clinically-driven benchmark for VLM editing and delivers in-depth experimental and geometric analyses that elucidate the tradeoffs and limitations of current editing strategies.

Clinical Benchmark Design

A key contribution is the Multimodal Medical Model-editing Benchmark (M$^3$Bench), which redefines the canonical editing desiderata—reliability, locality, generality—through ten clinically meaningful task axes. These tasks stress the ability of an edit to correct model behavior not only at the point of intervention but across real-world variations in patient, protocol, textual phrasing, modality, and case composition.

Figure 1: $M^3$Bench construction pipeline showing the creation of ten clinically grounded evaluation tasks covering reliability, locality, generality, and temporality in medical VLM model editing.

The benchmark is built from 16,276 questions across multiple datasets (VQA-RAD, PMC-VQA, PadChest-GR, SLAKE), spanning nine anatomical regions and four imaging modalities. Annotation relies on LLM-based clinical attribute distillation, which ensures consistency and schema alignment. Evaluation axes include:

• Reliability: Correction of targeted errors.
• Locality: Ensuring edits do not degrade unrelated knowledge, assessed across images, modalities, and within multi-finding cases.
• Generality: Transference of edits across semantically or contextually similar inputs, including paraphrased questions and different imaging views.
• Temporality: Maintaining longitudinal consistency across timepoints.

This multidimensional approach closely mirrors clinical workflows, where precision and robustness must span heterogeneous cases.

Figure 2: Sunburst diagram visualizing the benchmark’s global distribution across modality, anatomical site, and lesion IDs.

Comparative Evaluation of Model Editing Methods

The study conducts an extensive evaluation on four medical VLMs—LLaVA-Med, HuatuoGPT-Vision (7B/34B), BioMed-Qwen—and two general domain VLMs, using four editing methods: gradient-based (LoRA, MEND) and memory-based (BalancEdit, GRACE).

Figure 3: Radar plot showing single and sequential editing performance over ten tasks and four medical VLM backbones.

Key Findings

• No single method excels universally. Editing families reveal distinct tradeoffs between reliability, locality, and generality; absolutely best overall method is absent on all backbones and tasks.
• LoRA exhibits strong reliability and generality, but suffers catastrophic locality violations—unintended changes to unrelated behaviors are frequent, especially for textual locality.
• BalancEdit achieves the best overall performance in harmonic means, with more balanced reliability-locality profiles, but struggles in compositional and temporal tasks and reveals high hyperparameter sensitivity across model backbones.
• Temporal and compositional robustness remain unsolved. All editors, regardless of family, demonstrate the lowest performance on tasks demanding longitudinal or multi-finding consistency.

Geometric Analysis: Causes of Editing Failure

A central insight provided is the geometric bottleneck inherent in the latent spaces of medical VLMs.

Figure 4: (a) Histogram of pairwise cosine similarities shows high anisotropy—the embedding space forms a narrow cone. (b) Cosine similarity heatmap demonstrates low separation between disparate anatomical sites.

• Cone Effect: Medical VLM representations cluster tightly in a narrow cone, with high average cosine similarities (∼0.8) and mean resultant length (∼0.9).
• Consequences: Localized editing becomes intrinsically hard; gradient-based approaches cause global drifts in latent space, while memory-based approaches can only locally gate edits, missing out on entangled or overlapping clinical concepts.

  Figure 5: BalancEdit restricts edits to targeted topics by setting a binary spatial decision boundary, while LoRA broadly warps the representation space, leading to global drift across both targeted and non-targeted concepts.

• Non-target drift in Gradient-Based Editors: Gradient-based (LoRA) approaches alter non-target concepts by shifting entire clusters, breaking locality (observed both in numerical metrics and geometric projections).
• Binary Gating Limitation in Memory Editors: Memory-based (BalancEdit) methods, while local, cannot handle entangled concepts or generalize appropriately in compositional settings.

  Figure 6: PCA of embeddings under editing—BalancEdit increases centroid separation between single- and multi-finding cases (hurting compositional transfer), while LoRA collapses this gap (but sacrifices locality).

Architectural and Hyperparameter Sensitivity

The effectiveness of editing, especially for memory-based approaches, is highly sensitive to where and how intervention is performed in the VLM architecture, and to hyperparameters governing edit activation.

Figure 7: Architecture sweep showing BELoRA (hybrid) performance across components (LM, vision, projector) and depths (late, mid, full), with task-wise and harmonic mean summaries.

• Distribution of Edits: LoRA’s benefit for generality is partly attributable to distributing small updates over many layers; hybrid BELoRA offers incremental gains but does not fully resolve compositional or temporal bottlenecks.
• Component Choice: Editing the vision-language projector along with late LM layers gives the best tradeoff between performance and computational cost.
• No improvement from editing the vision encoder.

  Figure 8: Hyperparameter sweep for activation radius ($\alpha$) in memory-based editors. Each backbone has a distinct optimal $\alpha$; transfer is unreliable across models.

• Hyperparameter $\alpha$ is highly backbone-dependent. Minor changes can lead to under- or over-activation, severely affecting both generality and locality. Differences in cone concentration geometry between backbones explain this instability.

Case Analyses

Figure 9: Failure cases for BalancEdit (BE): Left, edit fails on compositional transfer (T4-G); Right, edit propagates undesirably over time (T5), causing temporal contradictions.

• Compositional (multi-label) generality: Edits on single-finding cases often fail to transfer when the same finding appears with additional lesions.
• Temporal inconsistency: Edits made at earlier timepoints may introduce factual errors upon follow-up studies, undermining clinical trustworthiness.

Implications and Future Directions

The M$^3$Bench, and associated experimental and mechanistic insights, define a new standard for evaluating clinical VLM editing. The main practical recommendations are:

• No blanket prescription of editing algorithms: Selection should be guided by deployment context, model backbone, and type of knowledge revision needed.
• Careful calibration of memory-based hyperparameters: Necessitated by backbone-dependent latent geometries.
• Need for advanced editing algorithms: Existing methods are fundamentally limited by geometric entanglement and anisotropy in VLM representation spaces. Achieving robust, compositional, and temporally consistent editing will require injection of new architectural and geometric inductive biases.
• General VLMs are less prone to locality failures: The cone effect is amplified by medical alignment, which may suggest strategies for low-level representation regularization as a path forward.

Conclusion

The presented benchmark and analysis establish the essential difficulty and tradeoffs in post hoc editing of medical vision-LLMs. Current approaches, both gradient- and memory-based, have pronounced limitations in compositionality, temporal consistency, and transferability across model backbones, largely dictated by underlying geometric properties of their latent spaces. This work lays critical groundwork for the systematic study of trustworthy post-deployment model adaptation in clinical AI.