Breaking the weakest link to evade vision language models

Abstract: Vision LLMs (VLMs) have recently emerged as a critical component of multimodal AI systems, enabling joint reasoning over visual and textual inputs in real-world and safety-critical applications. Despite their growing deployment, the robustness of VLMs against adversarial threats remains insufficiently explored, particularly in the context of evasion attacks targeting multimodal alignment. In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs and study two attack settings: untargeted attacks, where the goal is to disrupt the model's interpretation of the original image, and targeted attacks, where the adversary aims to force the model to generate a specific semantic description unrelated to the original image. To efficiently generate adversarial examples, we propose a gradient-based attack method that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture. This design significantly reduces the computational cost and resource requirements of the attack while maintaining strong effectiveness. We evaluate our approach on several open-source VLMs, including Qwen2.5-VL, Granite-Vision, FastVLM, and Phi-3.5-Vision, and show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models. Our findings highlight the vulnerability of modern VLMs to adversarial manipulation and emphasize the need for improved robustness and security mechanisms in multimodal AI systems.

• The paper introduces a more efficient method for evasion attacks on vision-language models (VLMs) that targets only the visual encoder to alter the language model’s output while minimizing computational and memory usage.
• Selector experiments comparing four open-source VLMs showed that attacked models quickly produce semantically misaligned descriptions even with small perturbations, achieving 47.12% success rate for targeted attacks and nearly 100% for untargeted attacks in some cases, setting lower VRAM consumption of approximately 16%
• This approach required computational optimization of the model and scaled to thousand of sample images, revealing specific vulnerabilities of different VLM architectures toward targeted and untargeted attacks, which illuminated several open research directions.

Attack objective and threat model

The paper studies evasion attacks against vision–LLMs (VLMs) in which an adversary modifies only the visual input while preserving its apparent content to human observers. Its central claim is that effective attacks need not backpropagate through the complete multimodal system. Instead, optimization through the VLM’s vision encoder alone can produce perturbations that substantially alter the LLM’s output while reducing computational and memory requirements. The study evaluates both untargeted attacks, which aim to destroy semantic correspondence with the source image, and targeted attacks, which aim to make the VLM describe the source image as a specified target image (2608.18938).

The threat model is white-box with respect to the vision encoder. The attacker is assumed to know the encoder architecture and parameters and to have access to its gradients. This is a material assumption: the method is directly applicable to open-source VLMs, but the paper does not establish black-box effectiveness, transferability across encoders, or physical-world robustness. The targeted setting uses randomly paired source and target images from an ImageNet subset. Consequently, the attack frequently has to bridge a large semantic gap—for example, transforming the model’s interpretation of a dog into that of a bedside lamp—rather than merely inducing a closely related class prediction.

The model decomposition motivating the method consists of a visual encoder, a projection module, and a LLM. The visual encoder maps an image into a visual representation, which is projected into the LLM’s embedding space and combined with textual prompt information before generation.

Figure 1: General VLM architecture combining a visual encoder, projection module, and LLM.

The paper’s key design decision is to optimize the image against the vision encoder representation rather than against a loss defined on generated text. This avoids storing activations and computing gradients through the LLM. The parameter-count comparison provides the computational rationale: the vision encoder accounts for 18% of Qwen2.5-VL-3B, 14% of Granite-Vision-3.2-2B, 1.6% of FastVLM-7B, and 10% of Phi-3.5-Vision-Instruct. For larger language-model backbones that reuse the same encoder, this fraction can become still smaller; the paper reports 0.9% for a 72B Qwen configuration.

Vision-encoder embedding optimization

For a targeted attack, the method begins with a benign source image $I_{\mathrm{source}}$ and a target image $I_{\mathrm{target}}$. Their encoder representations are computed independently. The adversarial image is then optimized to minimize the MSE between its representation and the target representation. The image is constrained to an $\ell_\infty$ neighborhood of the source image, with perturbation budget $\varepsilon$.

In implementation, the optimization uses 50 iterations of I-FGSM. Each iteration computes the gradient of the encoder-level loss with respect to the input pixels, updates the image in the descent direction, and clips the result both to the valid pixel range and to the source-centered perturbation interval. Although the loss is defined in embedding space, the success criterion is evaluated at the level of generated language: the VLM must produce a description judged semantically consistent with the target image.

The untargeted attack removes the target image and instead maximizes the embedding distance from the source representation. Optimization is initialized with Gaussian noise of scale $10^{-6}$ to avoid starting from a point with potentially degenerate gradient behavior. This objective is weaker than targeted steering because it does not require the adversarial representation to reach a particular semantic destination. It only needs to leave the source representation sufficiently far that the generated description is no longer judged equivalent to the original description.

The distinction between the two objectives is important for interpreting the reported results. Embedding divergence is not equivalent to semantic misclassification, and embedding alignment is not equivalent to faithful target-image recognition. The experiments therefore rely on generated captions and an external semantic judge rather than on direct distances in representation space. The paper uses Granite-4.0-micro to determine whether two descriptions express the same or different semantic content.

Figure 2: Prompt templates used by the LLM-as-a-Judge evaluation for semantic similarity and difference.

This evaluation introduces an additional learned component into the measurement pipeline. Attack success therefore depends not only on the VLM and perturbation algorithm but also on the judge model’s interpretation of the generated descriptions. The paper does not report human-validation statistics, inter-judge agreement, or calibration of the judge’s decisions, so the numerical success rates should be understood as judge-mediated semantic evaluations.

Experimental protocol

The experiments use 1,000 randomly selected ImageNet source images. For targeted attacks, each source is paired with a randomly selected target image, producing 1,000 source–target pairs. The same source images are used for untargeted attacks. Four open-source VLMs are evaluated:

Model	Total parameters	Vision encoder parameters
Qwen2.5-VL-3B-Instruct	3.7B	670M
Granite-Vision-3.2-2B	3.0B	442M
FastVLM-7B	7.8B	125M
Phi-3.5-Vision-Instruct	4.1B	424M

The perturbation budgets are $\varepsilon \in \{0.05, 0.10, 0.20\}$, and each result is averaged over four independent runs. The experiments were conducted on a single NVIDIA H100 GPU with 80 GB of VRAM. The computational comparison contrasts full-VLM backpropagation with the proposed encoder-only procedure, primarily on Qwen2.5-VL and Granite-Vision.

Targeted attack performance

Targeted attacks are substantially more difficult than untargeted attacks, but their effectiveness varies sharply across architectures. Granite-Vision-3.2-2B is the most vulnerable model in this setting, reaching $41.15\%$ success at $\varepsilon=0.05$, $47.12\%$ at $\varepsilon=0.10$, and $I_{\mathrm{target}}$0 at $I_{\mathrm{target}}$1. The result above 45% for randomly paired, often unrelated images is the paper’s strongest targeted finding. It indicates that encoder-level alignment can sometimes redirect the downstream generative interpretation across a considerable semantic distance.

FastVLM exhibits intermediate vulnerability, with success rates increasing from $I_{\mathrm{target}}$2 to $I_{\mathrm{target}}$3 across the evaluated budgets. Qwen2.5-VL-3B is less susceptible at small budgets but becomes progressively more vulnerable, increasing from $I_{\mathrm{target}}$4 at $I_{\mathrm{target}}$5 to $I_{\mathrm{target}}$6 at $I_{\mathrm{target}}$7. Phi-3.5-Vision is a clear outlier: its success rate remains approximately 2%, and does not improve with larger budgets.

Model	$I_{\mathrm{target}}$8	$I_{\mathrm{target}}$9	$\ell_\infty$0
Qwen2.5-VL-3B	$\ell_\infty$1	$\ell_\infty$2	$\ell_\infty$3
Granite-Vision-3.2-2B	$\ell_\infty$4	$\ell_\infty$5	$\ell_\infty$6
FastVLM-7B	$\ell_\infty$7	$\ell_\infty$8	$\ell_\infty$9
Phi-3.5-Vision	$\varepsilon$0	$\varepsilon$1	$\varepsilon$2

The relationship between perturbation magnitude and targeted success is therefore non-monotonic for Granite-Vision and effectively flat for Phi-3.5-Vision. Increasing $\varepsilon$3 does not guarantee continued improvement. This limits a simple interpretation in which robustness is determined solely by the size of the allowed pixel perturbation. Encoder architecture, preprocessing, feature geometry, and the interaction between visual embeddings and the language-generation stack likely contribute, although the paper does not isolate these factors experimentally.

Figure 3: Targeted attack success rate as a function of the perturbation budget $\varepsilon$4.

The qualitative example reinforces the quantitative result. At $\varepsilon$5, a perturbation that is visually difficult to detect changes Granite-Vision’s generated description from “a military tank” to “an ambulance.” This demonstrates output-level semantic redirection rather than merely degraded caption quality.

Figure 4: Targeted manipulation causing a VLM to interpret a military tank as an ambulance.

Figure 5: Source and adversarial captions for a targeted attack on Granite-Vision-3.2-2B with $\varepsilon$6; the perturbation is visually imperceptible.

The targeted results support the paper’s narrower claim that optimizing only the vision encoder can be sufficient to influence multimodal generation. They do not show that the adversarial embedding is generally recognized as the target by an independent vision system, nor that the generated description is consistently target-specific under varied prompts. Success is defined through one captioning protocol and one LLM-based judge.

Untargeted attack performance

Untargeted attacks are dramatically more successful. At $\varepsilon$7, every evaluated model exceeds 93% success: Qwen2.5-VL reaches $\varepsilon$8, Granite-Vision $\varepsilon$9, FastVLM $10^{-6}$0, and Phi-3.5-Vision $10^{-6}$1. At higher budgets, the rates remain near saturation, with Granite-Vision and FastVLM staying close to 100%.

Model	$10^{-6}$2	$10^{-6}$3	$10^{-6}$4
Qwen2.5-VL-3B	$10^{-6}$5	$10^{-6}$6	$10^{-6}$7
Granite-Vision-3.2-2B	$10^{-6}$8	$10^{-6}$9	$\varepsilon \in \{0.05, 0.10, 0.20\}$0
FastVLM-7B	$\varepsilon \in \{0.05, 0.10, 0.20\}$1	$\varepsilon \in \{0.05, 0.10, 0.20\}$2	$\varepsilon \in \{0.05, 0.10, 0.20\}$3
Phi-3.5-Vision	$\varepsilon \in \{0.05, 0.10, 0.20\}$4	$\varepsilon \in \{0.05, 0.10, 0.20\}$5	$\varepsilon \in \{0.05, 0.10, 0.20\}$6

Figure 6: Untargeted attack success rate as a function of the perturbation budget $\varepsilon \in \{0.05, 0.10, 0.20\}$7.

The near-saturation at $\varepsilon \in \{0.05, 0.10, 0.20\}$8 is a stronger result than the modest targeted improvements obtained from larger budgets. It implies that, under the paper’s success definition, semantic disruption is much easier than semantic steering. Once the encoder representation is displaced sufficiently to alter the caption, additional displacement provides little benefit. Granite-Vision and FastVLM are especially unstable under this objective, whereas Phi-3.5-Vision—highly resistant to targeted attacks—still exhibits more than 93% untargeted success at the smallest budget.

This contrast qualifies any architecture-level claim about Phi-3.5-Vision. Its behavior is not uniformly robust: it resists the specific targeted embedding-alignment procedure but remains highly vulnerable to representation divergence. Targeted and untargeted robustness should therefore be evaluated separately; aggregate attack-resistance scores would obscure this distinction.

Computational efficiency

The principal systems contribution is the reduction in attack cost obtained by freezing the multimodal components and differentiating only through the visual encoder. On the H100, full-VLM optimization uses approximately 44–47% of available VRAM in the reported Qwen2.5-VL and Granite-Vision experiments, compared with approximately 13–16% for encoder-only optimization. The paper also reports substantially lower GPU utilization for the restricted computation.

The timing comparison is particularly pronounced for Qwen2.5-VL-3B. Full-VLM optimization required more than 20 minutes and did not produce a successful adversarial example in the reported experiment, whereas encoder-only optimization generated one in approximately 100 seconds. This is a reduction of at least an order of magnitude in the stated comparison, although the paper does not provide a complete controlled accounting of iteration count, step size, data-transfer overhead, convergence criteria, or failed-run treatment.

The efficiency result has a direct security implication: access to a large language-model backbone is not necessary for crafting these attacks once the visual encoder is known. The relevant attack surface can be materially smaller than the deployed VLM. This observation is consistent with prior work showing that visual perturbations can manipulate VLM behavior (2608.18938, Zhang et al., 2024), while the present method emphasizes that the attack can be constructed without optimizing through the complete multimodal pipeline.

Figure 7: Targeted adversarial attack based on aligning source-image and target-image vision-encoder embeddings.

Limitations and open questions

The paper’s conclusions are bounded by several methodological assumptions. First, the threat model is white-box and encoder-specific. The study does not evaluate transfer attacks, black-box query attacks, surrogate encoders, or attacks against proprietary VLMs. Whether encoder-only perturbations transfer when the same visual encoder is coupled to different projection layers or LLMs remains unresolved.

Second, the evaluation uses ImageNet images, one-sentence descriptions, and Granite-4.0-micro as an LLM judge. These choices provide scale and a uniform protocol but do not establish robustness for VQA, instruction following, OCR, spatial reasoning, long-form generation, or task-specific safety policies. The judge-based criterion may also conflate semantic mismatch with ordinary caption variability.

Third, the attack optimizes MSE in the raw vision-encoder representation. The paper does not compare alternative feature losses, intermediate-layer objectives, cosine alignment, contrastive objectives, or direct language-level losses. It also does not analyze which encoder layers are responsible for the observed vulnerability. The reported attack success is consequently evidence for the sufficiency of one objective, not for the necessity of final-layer MSE.

Fourth, the perturbation budgets are reported without a detailed discussion of pixel scaling, preprocessing, image normalization, or perceptual distance. The claim of human imperceptibility is supported qualitatively by examples, but no human study, perceptual metric, or physical-world assessment is provided. Finally, the computational comparison is informative but incomplete: full-pipeline and encoder-only optimization are not reported with sufficiently detailed convergence diagnostics to determine whether the timing difference reflects only gradient-memory savings or also differences in optimization success.

Conclusion

“Breaking the weakest link to evade vision LLMs” (2608.18938) presents a vision-encoder-only gradient attack for targeted and untargeted manipulation of VLM outputs. Across four open-source models, untargeted attacks exceed 93% success at $\varepsilon \in \{0.05, 0.10, 0.20\}$9, while targeted success reaches $41.15\%$0 on Granite-Vision under randomly paired source and target images. The same method reduces reported VRAM usage from 44–47% to 13–16% and reduces a Qwen attack from more than 20 minutes without success to approximately 100 seconds. The results establish that visual representation manipulation can be both effective and computationally economical, while also showing that targeted robustness, untargeted robustness, and architecture-level behavior are distinct properties requiring separate evaluation.