One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbations

Abstract: Typographic prompt injection exploits vision LLMs' (VLMs) ability to read text rendered in images, posing a growing threat as VLMs power autonomous agents. Prior work typically focus on maximizing attack success rate (ASR) but does not explain \emph{why} certain renderings bypass safety alignment. We make two contributions. First, an empirical study across four VLMs including GPT-4o and Claude, twelve font sizes, and ten transformations reveals that multimodal embedding distance strongly predicts ASR ($r{=}{-}0.71$ to ${-}0.93$, $p{<}0.01$), providing an interpretable, model agnostic proxy. Since embedding distance predicts ASR, reducing it should improve attack success, but the relationship is mediated by two factors: perceptual readability (whether the VLM can parse the text) and safety alignment (whether it refuses to comply). Second, we use this as a red teaming tool: we directly maximize image text embedding similarity under bounded $\ell_\infty$ perturbations via CWA-SSA across four surrogate embedding models, stress testing both factors without access to the target model. Experiments across five degradation settings on GPT-4o, Claude Sonnet 4.5, Mistral-Large-3, and Qwen3-VL confirm that optimization recovers readability and reduces safety aligned refusals as two co-occurring effects, with the dominant mechanism depending on the model's safety filter strength and the degree of visual degradation.

• The paper introduces an embedding-guided adversarial optimization technique that exposes dual vulnerabilities—readability recovery and reduced safety refusals—in VLMs.
• It demonstrates a strong negative correlation (Pearson r from -0.71 to -0.99) between embedding distance and attack success rate across various perturbations.
• The findings highlight the need for robust safety mechanisms that guard against both pixel-level and embedding-level adversarial attacks.

Probing Vision-LLM Safety via Embedding-Guided Typographic Perturbations

Introduction

This work interrogates the security and alignment of current vision-LLMs (VLMs) such as GPT-4o, Claude Sonnet 4.5, Mistral-Large-3, and Qwen3-VL-4B by examining their susceptibility to typographic prompt injection (TPI). The threat arises as attackers render malicious prompts as visually embedded text within images, often bypassing text-based safety mechanisms. Unlike prior research, which primarily focuses on optimizing attack success rates (ASR) with increasingly sophisticated visual text tricks, this paper offers a highly controlled empirical and mechanistic analysis. It introduces embedding-guided optimization as an interpretable and model-agnostic red-teaming tool for probing VLM robustness under bounded $\ell_\infty$ perturbations, revealing two distinct co-occurring vulnerabilities: readability recovery and reduction of safety-aligned refusals.

Embedding Distance as a Model-Agnostic ASR Predictor

An extensive empirical investigation quantifies the influence of font size and visual transformations (e.g., heavy blur, rotation, triple degradation) on ASR across four VLMs, benchmarked on 1,000 safety-relevant prompts from SALAD-Bench. The critical finding is a strong negative correlation between multimodal embedding space distance—computed using JinaCLIP and Qwen3-VL-Embedding—and ASR (Pearson $r$ ranging from $-0.71$ to $-0.93$ for font sizes and from $-0.72$ to $-0.99$ for transformations, all with $p < 0.01$). This correlation supports the use of embedding distance, specifically the L2 norm between normalized image and text representations, as a robust, interpretable, and architecture-independent ASR proxy.

Embedding-Guided Adversarial Optimization

Exploiting the empirical insight on embedding alignment, the authors design a red-teaming procedure that iteratively perturbs visually degraded typographic images as input to an ensemble of surrogate embedding models. The goal is to maximize cosine similarity between the perturbed image and its ground-truth prompt embedding under a bounded $\ell_\infty$ norm. The core innovation is targeting embedding space alignment—rather than pixel-space readability—via a variant of the Common Weakness Attack with Spectral Simulation Augmentation (CWA-SSA). The architectural ensemble (Qwen3-VL-Embedding-2B, JinaCLIP v2, OpenAI CLIP ViT-L/14-336, SigLIP SO400M) ensures generalization and transferability of perturbations.

Figure 1: The embedding-guided adversarial optimization pipeline targets degraded typographic images, iteratively maximizing their embedding similarity with the source prompt across four surrogates, leading to both readability recovery and reduced safety refusals.

The optimization objective is formally expressed as: $$\max_{\delta: \|\delta\|_{\infty} \leq \epsilon} \frac{1}{K} \sum_{k=1}^{K} \cos(\text{ImgEnc}_k(I_0 + \delta),\ \mathbf{t}_k)$$ where $\epsilon$ bounds the perturbation and $r$0 denotes the number of ensemble models.

Experimental Analysis: Failure Mode Disentanglement

The authors systematically evaluate the effect of embedding-guided perturbations under various image degradation regimes: 6px and 8px font sizes, $r$1 rotation, heavy blur, and triple degradation (blur, noise, low contrast). The analysis on 50 carefully selected prompts reveals the following patterns:

• Readability Recovery dominates in high-degradation (6px, heavy blur) conditions: for instance, at 6px, GPT-4o's unreadable cases drop from 35 to 10 post-optimization, although most new readable cases are captured by stronger safety filters, resulting in only marginal ASR increases.
• Refusal Reduction is prominent when initial input readability is moderate. For Claude Sonnet 4.5 on heavy blur, ASR increases by 28% post-perturbation as the model transitions from empty (unreadable) to actionable (classifiable) image regions.
• Trade-offs with Safety Filtering: In models with weaker safety filters (e.g., Mistral-Large-3), improved readability results in more successful attacks, but perturbations can also inadvertently activate safety mechanisms, decreasing ASR in some cases.

The methodological approach distinctly demonstrates that the same bounded perturbation can yield two different model failures—either making previously unreadable text parseable or actively bypassing safety-aligned refusals without enhancing pixel-level legibility.

Theoretical and Practical Implications

From a theoretical perspective, these results emphasize the importance of representation-space robustness: safety-aligned refusals cannot reliably depend solely on the unreadability of image text at the pixel level. Since embedding-targeted perturbations can trigger higher-level semantic recoveries or shift samples across the internal safety decision manifold, robust safety design must incorporate mechanisms operating in embedding or intermediate representation spaces.

Practically, the approach constitutes a new, model-agnostic diagnostic for probing alignment and safety in VLMs, transcending architecture-specific attack heuristics. It exposes differential failure surfaces even among leading-edge architectures and provides a concrete avenue for evaluating and hardening safety filters in a black-box setting.

Future Directions

This study opens several vectors for future research:

• Extending embedding-guided adversarial probing to more diverse datasets, fonts, and visual encodings.
• Integrating embedding-aligned defenses that detect or resist representation-space perturbations.
• Investigating detection-based countermeasures and the interplay between embedding alignment and content-based filtering in deeply aligned multimodal systems.
• Benchmarking against more sophisticated safety detection frameworks and across multilingual or scene-complex settings.

Conclusion

This paper demonstrates that embedding space alignment, rather than visual legibility, is a primary determinant of typographic attack success in contemporary VLMs. Embedding-guided perturbations efficiently reveal dual VLM vulnerabilities—readability recovery and reduction of safety-aligned refusals—with substantial increases in ASR under strong visual degradation (e.g., +28% for Claude under heavy blur). These findings underscore the necessity of safety mechanisms resilient to both pixel- and embedding-level adversarial strategies and highlight embedding-guided optimization as a rigorous tool for red teaming VLM safety.