When Background Matters: Breaking Medical Vision Language Models by Transferable Attack

Abstract: Vision-LLMs (VLMs) are increasingly used in clinical diagnostics, yet their robustness to adversarial attacks remains largely unexplored, posing serious risks. Existing medical attacks focus on secondary objectives such as model stealing or adversarial fine-tuning, while transferable attacks from natural images introduce visible distortions that clinicians can easily detect. To address this, we propose MedFocusLeak, a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. The method injects coordinated perturbations into non-diagnostic background regions and employs an attention distraction mechanism to shift the model's focus away from pathological areas. Extensive evaluations across six medical imaging modalities show that MedFocusLeak achieves state-of-the-art performance, generating misleading yet realistic diagnostic outputs across diverse VLMs. We further introduce a unified evaluation framework with novel metrics that jointly capture attack success and image fidelity, revealing a critical weakness in the reasoning capabilities of modern clinical VLMs.

• The paper presents MedFocusLeak, a novel attack that uses background-constrained perturbations and attention-shift losses to trigger plausible diagnostic errors.
• It employs patch-based local perturbations on non-diagnostic regions and joint image-text optimization to ensure high transferability across various medical and generalist VLMs.
• Empirical results reveal 1.5x–2x improvements in MAS, underscoring serious vulnerabilities in current medical vision-language models.

Transferable, Imperceptible Attacks on Medical Vision-LLMs: Analysis of MedFocusLeak

Introduction

The increasing integration of multimodal vision-LLMs (VLMs) into clinical workflows raises critical concerns regarding their adversarial robustness. "When Background Matters: Breaking Medical Vision LLMs by Transferable Attack" (2604.17318) systematically investigates the adversarial transferability landscape in medical VLMs, introducing MedFocusLeak—a black-box, multimodal attack targeting the visual attention mechanisms to induce plausible but incorrect diagnoses under stringent imperceptibility constraints. The methodology fundamentally departs from prior approaches by leveraging coordinated background-constrained perturbations and semantically grounded attention-shift objectives, achieving strong transferability across a diverse suite of medical and generalist VLMs, including closed-source and open-weight models.

MedFocusLeak Framework

MedFocusLeak's architecture incorporates four essential principles: (1) masking of clinical foregrounds to confine perturbations, (2) structured multimodal (image and text) adversarial representation, (3) patch-based local feature alignment optimized for transferability, and (4) explicit attention shift loss to reallocate visual focus from pathological regions toward adversarially-perturbed backgrounds.

Figure 1: The MedFocusLeak attack pipeline, integrating adversarial text generation, background-constrained image perturbation, and attention-shift loss to induce clinically plausible misdiagnoses.

The attack operates using only black-box access to victim VLMs, where perturbations are crafted on surrogate models. By isolating diagnostically non-critical regions using segmentation (MedSAM), adversarial noise remains visually covert while preserving clinical integrity. Cross-modal perturbations and feature-level patch alignment enforce high transferability and semantic consistency across a wide spectrum of VLM architectures.

Adversarial Optimization Strategy

Multimodal Adversarial Representation

MedFocusLeak eschews the traditional unimodal paradigm, showing that sole image perturbations or prompt modifications are inadequate for robust transfer in high-stakes medical VLM scenarios. By initializing adversarial seeds jointly in both image and text space, and iteratively optimizing each under perceptual and semantic constraints, the framework disrupts cross-modal fusion modules within VLMs. Alternating projected gradient descent and greedy token substitution achieve targeted feature misalignment and text-level semantic drift while maintaining the modality context.

Background Constrained, Patch-Based Perturbation

Perturbations are dynamically restricted to the $k$ largest background patches—maximally non-diagnostic—identified by dynamic programming over MedSAM-extracted segmentation masks. Local loss optimization, ensemble over surrogate encoders (multiple CLIP variants), ensures that adversarial cues are both effective and highly transferable, while imperceptibility is maintained as measured by Med-CLIP image similarity (AvgSim).

Attention-Shift Objective

Embedding adversarial cues solely in the background is insufficient if attention mechanisms remain focused on relevant pathology. MedFocusLeak introduces a loss term that minimizes the log-ratio of attention weights on diagnostic foreground versus perturbed background, as measured by the final multimodal fusion block. This explicitly drives model visual focus away from the primary clinical content toward the adversarially-perturbed regions, systematically degrading diagnostic reasoning.

Empirical Evaluation

Main Results and Transferability

MedFocusLeak achieves substantial gains across automatic metrics—Medical Text Adversarial Score (MTR) and Medical AttackScore (MAS)—outperforming all contemporary baselines, including AttackVLM, FOA-Attack, and M-Attack, in both medical-specialized (BioMedLlama, MedVLM-R1) and generalist (Gemini-2.5-Pro, GPT-5, InternVL, QwenVL) VLMs. Median MAS increases are often 1.5x–2x over the strongest baselines. Robustness extends to closed-source reasoning models, which display increased, but not insurmountable, resistance.

Figure 2: (a) MedFocusLeak achieves optimal MAS at $k=10$ background patches; (b) MAS increases with attack optimization steps; (c) Across the classification task, MedFocusLeak maintains dominant ASR compared to M-Attack and FOA-Attack; (d) MAS scales with time but at a higher efficiency compared to baselines.

Defense Robustness

MedFocusLeak demonstrates strong resilience against common defense strategies, including Gaussian noise augmentation and Comdefend, maintaining high AvgSim and MAS where baselines largely degrade.

Figure 3: MedFocusLeak consistently outperforms M-Attack in MAS across defensive transformations.

Qualitative Analysis

Clinical inspection of generated adversarial examples reveals that MedFocusLeak reliably preserves the primary imaging modality while introducing subtle, but clinically consequential, textual misdirections. For instance, diagnostic shifts from benign to malignant findings or omission of pathological terminology are prevalent without perceptible image artifacts.

Figure 4: Side-by-side comparison of original and adversarially manipulated reports illustrating clinically hazardous diagnostic shifts induced by background-limited, semantically-aware perturbations.

Ablation and Parameter Sensitivity

Ablation studies reinforce the criticality of the multimodal design; removing attention-shift or restricting to unimodal perturbation notably degrades MAS. Increasing the perturbation budget and step count yields monotonic, but diminishing, returns, and the patch-level optimization outperforms holistic image corruption both in transferability and imperceptibility. Optimal $k=10$ background patches and $\alpha=1.0$ step size provide the best efficiency/effectiveness tradeoff.

Implications and Future Directions

The findings expose a structural vulnerability of current VLMs, rooted in shared attention mechanisms and coarse background/foreground separation. The implications for clinical safety are severe: MedFocusLeak can introduce highly believable, material diagnostic errors undetectable even to medical experts, while remaining robust under conventional defensive post-processing. These attacks present not only a tool to pressure-test clinical models but also an urgent call for new architectural and procedural defenses—potentially involving more granular feature attribution, robust attention regularization, and integration of human-in-the-loop oversight in high-stakes decision pipelines.

More broadly, future work must address (1) adversarial robustness under real-world clinical image distributions with high background heterogeneity, (2) development of certifiable, model-agnostic defenses against cross-modal adversarial transfer, and (3) dedicated benchmarks for combined image-text attack efficacy.

Conclusion

The MedFocusLeak framework sets a new empirical standard for black-box, transferable, and imperceptible adversarial attacks against medical VLMs, robustly revealing substantial weaknesses in attention allocation and semantic fusion under realistic deployment conditions. The approach's methodology and findings provide critical guidance for both adversarial defense research and the safe operationalization of AI in clinical medicine.