Jailbreaking Vision-Language Models Through the Visual Modality

Abstract: The visual modality of vision-LLMs (VLMs) is an underexplored attack surface for bypassing safety alignment. We introduce four jailbreak attacks exploiting the vision component: (1) encoding harmful instructions as visual symbol sequences with a decoding legend, (2) replacing harmful objects with benign substitutes (e.g., bomb -> banana) then prompting for harmful actions using the substitute term, (3) replacing harmful text in images (e.g., on book covers) with benign words while visual context preserves the original meaning, and (4) visual analogy puzzles whose solution requires inferring a prohibited concept. Evaluating across six frontier VLMs, our visual attacks bypass safety alignment and expose a cross-modality alignment gap: text-based safety training does not automatically generalize to harmful intent conveyed visually. For example, our visual cipher achieves 40.9% attack success on Claude-Haiku-4.5 versus 10.7% for an equivalent textual cipher. To further our insight into the attack mechanism, we present preliminary interpretability and mitigation results. These findings highlight that robust VLM alignment requires treating vision as a first-class target for safety post-training.

• The paper introduces four novel visual jailbreak attacks that exploit the modality gap in vision-language models.
• It demonstrates that methods like visual cipher and object replacement yield high attack success rates on models such as Claude-Haiku-4.5 and Qwen3-VL variants.
• It underscores the need for cross-modal safety training and robust output-side filtering to mitigate emergent vulnerabilities.

Jailbreaking Vision-LLMs via the Visual Modality

Introduction and Motivation

Vision-LLMs (VLMs) integrate multimodal perception to enable advanced reasoning, search, and agentic behaviors. This paradigm shift introduces new vectors for adversarial exploitation, particularly through the visual input channel. Traditional safety alignment protocols, predominantly focused on text-based refusals and compliance, have not addressed cross-modal generalization, leaving the visual modality as an underexplored and vulnerable attack surface. This paper systematically formalizes and evaluates four novel visual jailbreak attacks targeting the modality gap in VLM safety, demonstrating that frontier models are susceptible to harmful intent being implied or encoded visually, even when surface-level text is sanitized.

Figure 1: Four visual jailbreak attacks exploiting the vision modality of VLMs. Visual input (red boundary), attack prompt (beneath).

Attack Taxonomy and Methodology

The authors introduce four attacks, all operating on prompts neutralized via slot substitution ($X_i$) to obscure explicit harmful content, ensuring attacks rely on visual context or structure:

• Visual Cipher: Harmful instructions are mapped to glyph sequences, decoded via visual legend lookup tables, with distractor glyphs to obfuscate mappings.
• Visual Object Replacement: Harmful objects are replaced with benign surrogates (e.g., bomb $\rightarrow$ banana), while contextual cues maintain dangerous semantics.
• Visual Text Replacement: Textual evidence in images (e.g., book covers) is edited to benign words, but context (e.g., typography, cultural artifacts) sustains latent referents.
• Visual Analogy Riddle: Chains of implicit visual riddles encode prohibited concepts through analogical inference, requiring models to synthesize unsafe intent from benign clues.

Each attack is paired with a rigorous textual baseline (substitution ciphers, word replacements, textual riddles) to isolate the effect of the visual modality. Evaluation employs HarmBench prompts spanning diverse harm categories, with responses judged by an ensemble of three frontier LLMs.

Quantitative Analysis: Success Rates and Comparative Robustness

Attack Success Rates (ASR) are reported for six state-of-the-art VLMs (GPT-5.2, Claude-Haiku-4.5, Gemini-3-Flash, Gemini-3.1-Pro, Qwen3-VL-235B, Qwen3-VL-32B). Robust output-side evaluation shows:

• Visual Cipher outperforms textual cipher on Claude-Haiku-4.5 (40.9\% vs 10.7\% ASR) and strengthens attacks on Qwen3 variants.
• Visual Object Replacement achieves maximal ASR on Qwen3-VL-32B and Qwen3-VL-235B, indicating scene context can override object appearance in semantic inference.
• Visual Text Replacement exposes vulnerabilities in Qwen3 variants, with ASR exceeding textual replacement, highlighting models' reliance on visual/cultural cues.
• Visual Analogy Riddle shows lower ASR than textual analogies, reflecting increased misunderstanding due to visual abstraction, yet Gemini-3-Flash and Qwen3-VL-32B remain highly vulnerable.

Success rates for other historical baselines (FigStep, HADES, MM-SafetyBench, CipherChat) are much lower, confirming the distinct mechanism and potency of the proposed visual attacks. Category-level radar plots confirm compliance clustering in cybercrime, misinformation, and illegal activities.

Figure 2: Visual Analogy Attack: Visual riddles implicitly encode target factors $X_i$, enabling unsafe intent recovery and dangerous content generation.

Figure 3: Visual Cipher: Harmful instruction encoded pictographically, decoded using a visual legend mapping; distractors included.

Figure 4: Visual Cipher Attack Example: GPT-5.2 decodes a hidden prompt requesting email scraping tools, outputting detailed methods.

Mechanistic Interpretability and Refusal Analysis

Preliminary layer-wise interpretability leverages Logit Lens and refusal direction extraction. Key findings:

• Semantic Layer Trends: For visual replacement attacks, dangerous tokens remain elevated through semantic layers even after object substitution; benign token probabilities drop sharply but recover at decoding.
• Refusal Activation Suppression: Refusal direction analysis (Qwen3-VL-32B) reveals that visual replacement suppresses late-layer refusal signal compared to harmful images/text, suggesting temporal dissociation of safety checking and semantic integration.

  Figure 5: Layerwise logit trends for visual replacement attacks: Dangerous tokens remain high in semantic layers; benign tokens drop and recover late.

  

  Figure 6: Refusal-direction activation: Visual replacement attacks suppress late-layer refusal relative to image/text harmful prompts.

Practical Implications and Mitigations

The demonstrated attacks are semantically meaningful, gradient-free (no white-box access), and operate strictly via visual channel manipulation. This exposes a fundamental cross-modality alignment gap: current text-dominated RLHF and constitutional AI paradigms fail to generalize safety to visual embeddings, owing to modality-specific representation and contextual inference.

• Output-Side Filtering: Lightweight output classifiers (e.g., Qwen3Guard-Stream-0.6B) effectively flag most harmful Visual Cipher responses, underscoring the necessity of defense-in-depth using natural language output classifiers irrespective of input encoding.
• Dataset Augmentation and Training: Extending safety datasets to include visual jailbreak exemplars, and developing vision-specific constitutional principles, are critical to robust alignment.
• Mechanistic Analysis: Further characterization of refusal features and semantic integration in VLMs, potentially via Patchscopes or monosemantic component extraction, is warranted; effective cross-modal refusal may require explicit fusion layers or runtime semantic auditing.

Limitations and Future Directions

• Model Access and Attack Transferability: Analysis is constrained to open-weight models for interpretability, with no coverage of transferability or compositional attack vectors.
• Evaluation Noise: Attack efficacy is influenced by image generation quality; Best-of-$K$ sampling mitigates but does not eliminate variance.
• Misunderstanding Rates: High misunderstanding classifications (especially in visual analogy attacks) highlight that current failure modes may shift as visual reasoning capabilities advance.

Future work should address video, audio, and interleaved multimodal attacks, layerwise semantic monitoring, and unified cross-modal safety training. Theoretical investigation of modality-specific alignment and resistances is essential as VLMs are increasingly deployed in safety-critical environments.

Figure 7: Visual Cipher Attack Example: Claude-Haiku-4.5 decodes and outputs actionable steps for credit card skimmer setup.

Figure 8: Visual Text Replacement Attack Example: Claude-4.5-Haiku detects context and recovers the intended harmful referent.

Conclusion

This paper rigorously demonstrates that vision constitutes an independent and potent attack vector for jailbreaking VLMs. Four distinct visual attacks bypass text-based safety training, reveal a modality gap in alignment, and achieve strong success rates on multiple frontier models. Mechanistic analysis supports the view that semantic integration outpaces refusal checking in visual attacks. Practical mitigation via output-side classifiers is feasible but not sufficient; robust cross-modal safety will require vision-inclusive training, runtime monitoring, and new interpretability paradigms. As multimodal AI capabilities proliferate, safety research must treat vision as a primary target, not a secondary channel.