Image-Assisted Jailbreak Attacks

• Image-assisted jailbreak attacks are a set of methodologies that use manipulated images to bypass safety alignments in multimodal foundation models.
• They employ approaches such as pixel-level adversarial examples, data poisoning, gradient-based attacks, and steganography to induce harmful outputs.
• Empirical results show high attack success rates and transferability, underscoring the need for robust, cross-modal defense strategies.

Image-assisted jailbreak attacks are a collection of methodologies designed to subvert the safety alignment of multimodal foundation models—especially vision-LLMs (VLMs) and multimodal LLMs (MLLMs)—by leveraging the image modality to bypass, mask, or amplify unsafe or policy-violating intentions such that the model produces harmful outputs in response to otherwise prohibited instructions. The attack surface includes both the training and inference stages, spanning explicit prompt injection, gradient-based adversarial optimization, data poisoning, programmatic compositional attacks, steganography, and cross-modal reasoning exploits. These attacks exploit underaligned or insufficiently guarded vision encoders, fusion modules, and decision boundaries, resurfacing or amplifying harmful semantics that are robustly filtered in the purely textual domain.

1. Threat Models and Attack Surfaces

Image-assisted jailbreak attacks target models that fuse visual and textual modalities, notably VLMs and T2I models. Common threat models include:

• Black-box access, where the adversary can only query the target system with image+text pairs and does not access weights, gradients, or activations, as in the scenarios addressed by HADES (Li et al., 2024) and BlueSuffix (Zhao et al., 2024).
• Grey-box attacks, such as Simulated Ensemble Attack (SEA) (Wang et al., 3 Aug 2025), where the adversary has white-box access to a base (public) model, but only black-box or no access to downstream fine-tuned variants.
• White-box settings offering full gradient access to at least part of the model for adversarial optimization (HADES stage 3, gradient-based attacks (Rando et al., 2024)).

Attack surfaces include:

• Data poisoning in training/fine-tuning sets (ImgTrojan (Tao et al., 2024)).
• Visual distractors, typographic overlays, or field-level perturbations in inference-time images.
• Cross-modal boundary exploitation (e.g., embedding harmful intent in visual input that evades or confuses text-side and image-side safety modules).
• Multistep compositional reasoning chains where the model only reveals the malicious intent after integrating information across image regions or temporally (PRISM (Zou et al., 29 Jul 2025), Chain-of-Jailbreak (Wang et al., 2024)).

2. Methodological Taxonomy of Attacks

2.1 Explicit Image-Driven Attacks

• Pixel-Level Adversarial Examples: Small $\ell_p$-bounded perturbations are optimized via Projected Gradient Descent or similar schemes to shift image embeddings toward a harmful target or to robustly trigger unsafe outputs under vision encoding (Li et al., 2024, Rando et al., 2024, Chen et al., 8 Apr 2025, Wang et al., 3 Aug 2025).
• Visual Triggers and UAPs: Fixed or universal adversarial patterns ("visual triggers") are injected into images to reliably bypass denoisers and activate malicious behaviors in black-box settings (Zhao et al., 2024).

2.2 Data Poisoning Attacks

• ImgTrojan: Poisons a small fraction $\alpha\ll1$ of (image, caption) pairs in a training corpus, such that the image’s visual feature is associated to a “jailbreak prompt” (JBP). At inference, inputting the corresponding image causes the model to “recall” and execute the malicious template, bypassing textual or image-based filters. Even a single poisoned pair among 10,000 suffices for ASR > 60%, with negligible impact on clean tasks (Tao et al., 2024).

2.3 Gradient-Based and Tokenizer Shortcut Attacks

• Fusion Model Tokenizer Shortcuts: By training continuous relaxations of non-differentiable tokenization steps, it becomes feasible to optimize images (via gradient descent) that cause a fusion model (Chameleon) to output harmful or non-refusal responses at high rates ($\approx$72.5% ASR) (Rando et al., 2024).
• Text-Image Boundary Exploitation: The TCBS-Attack performs population-based token mutation adjacent to prompt- and image-checker boundaries, iteratively finding prompt tokens that evade both text and image checkers, achieving high ASR on T2I models (Liu et al., 15 Apr 2025).

2.4 Stealth and Concealment Techniques

• Steganographic Attacks (IJA): Malicious instructions are embedded into images at the least-significant bit level, coupled with benign-appearing text and adversarial suffixes that prime the model to decode and execute the hidden malicious content. Empirically achieves >90% ASR on commercial MLLMs with minimal queries (Wang et al., 22 May 2025).

2.5 Compositional and Programmatic Attacks

• Gadget-based Emergent Reasoning (PRISM): The adversary decomposes a harmful task into visual “gadgets”—benign-appearing images—assembled via a control-flow prompt (inspired by ROP exploitation in software security). The LVLM integrates the gadgets via stepwise reasoning to yield an emergent harmful output, achieving ASR >0.9 on SafeBench (Zou et al., 29 Jul 2025).
• Chain-of-Jailbreak (CoJ): Harmful intent is decomposed into a sequence of edits, each benign in isolation; the final sequence of queries reconstructs the forbidden content in the output image. Outperforms all single-prompt baselines (JSR up to 86%) on GPT-4 and Gemini APIs (Wang et al., 2024).

2.6 Vision-Centric and Inscriptive Attacks

• Visual-Only Prompting (VJA): Malicious intent is conveyed purely through visual cues—arrows, marks, or stylized micro-text—requiring no explicit text input. State-of-the-art commercial image editing models succumb (ASR >80%) (Hou et al., 10 Feb 2026).
• Inscriptive Jailbreaks (Etch): T2I models are coerced to render entire harmful payloads as high-fidelity in-image text via prompt engineering spanning semantic camouflage, visual-spatial anchoring, and typographic encoding layers. Etch achieves ASR up to 91% on open-source and commercial T2I models, outperforming depictive and prior template attacks by >30pp (Ying et al., 7 Apr 2026).

3. Formal Metrics and Benchmarks

Attack and defense efficacy are quantified via metrics including:

• Attack Success Rate (ASR): Fraction of harmful instructions or prompts that elicit unsafe outputs, as judged by an LLM or human rater. Often, HADES, SafeBench, or MM-SafetyBench protocols are employed (Li et al., 2024, Zhao et al., 2024, Tao et al., 2024).
• Jailbreak Success Rate (JSR): The fraction of cases in which a model, previously refusing a prompt, produces the harmful output after a stepwise or compositional attack (CoJ, PRISM).
• Stealthiness (Clean Metric): Measured by standard metrics such as BLEU or CIDEr on clean validation sets to ensure benign task performance is retained (ImgTrojan) (Tao et al., 2024).
• High-Risk Ratio (HRR): For editing models, the ratio of samples scored as both harmful and plausibly edited (Hou et al., 10 Feb 2026).
• Bypass Rate: Proportion of adversarial samples that evade both text and image safety modules (Wang et al., 22 May 2025).
• Transferability: The degree to which attacks generated on a base model succeed on fine-tuned or commercial variants (SEA, PRISM).

Selected Benchmark Datasets and Scenarios

Benchmark	Targeted Attacks	Model Domains	Features
HADES	Typography, vision	VLMs, MLLMs	750 harmful instructions, per-category metrics
CoJ-Bench	Chain-of-Jailbreak	T2I/image editing	776 chains, 9 safety scenarios, multi-step edits
SafeBench/MM	Blend, Reasoning	VLMs, LVLMs	1,680 multimodal jailbreaks, cross-topic
IESBench	Vision-centric	Image editing	1,054 samples, 15 risk categories, full annotation

4. Empirical Findings and Model Vulnerabilities

Across published evaluations:

• Attack Potency: Many methods achieve ASR, JSR, or analogous success >70% against state-of-the-art open-source and commercial models under moderate budget (usually $\leq$3 queries per sample) (Li et al., 2024, Rando et al., 2024, Wang et al., 3 Aug 2025, Wang et al., 22 May 2025, Wang et al., 2024, Hou et al., 10 Feb 2026).
• Stealth: ImgTrojan and IJA demonstrate that attacks can be embedded with negligible loss to clean-task performance or routine vision metrics, and often evade common CLIP-based similarity or perplexity filters (Tao et al., 2024, Wang et al., 22 May 2025).
• Transferability: SEA shows that well-crafted adversarial images, optimized under simulated parameter shifts and prompt guidance, retain attack success >86.5% even under independently safety-fine-tuned variants, exposing inherited weaknesses of deployment-time fine-tuning practices (Wang et al., 3 Aug 2025). In contrast, direct PGD attacks transfer poorly unless the fine-tuning is minimal.
• Defense Evasion: Existing unimodal or naïve bimodal defenses struggle with compositional attacks (PRISM) or implicit attacks (IJA, Etch), especially when the harmful semantics manifest only at the integration or reasoning stage (Zou et al., 29 Jul 2025, Wang et al., 22 May 2025, Ying et al., 7 Apr 2026).

5. Defense Strategies and Limitations

5.1 Bimodal and Cross-Modal Filtering

• BlueSuffix applies diffusion-based denoisers to the image and LLM-based paraphrasing to text, with a reinforcement-learned suffix generator appended to the input for safety cues. This architecture achieves a 56.3pp reduction in ASR while preserving benign-pass rates (Zhao et al., 2024).
• DiffPure and OCR-based Guards: Effective against pixel-level and some typographic attacks, but generally bypassed by advanced compositional or steganographic exploits (Zhao et al., 2024, Li et al., 2024).

5.2 Adversarial Training and Representation Engineering

• Contrastive Harmlessness LoRA and related adversarial fine-tuning can suppress open-source model ASRs to <6% (Li et al., 2024).
• Circuit Breakers: Inserting a trainable post-hoc “refusal direction” suppressor in the transformer blocks, originally trained on text attacks, was found to generalize robustly to image-based adversaries in Chameleon (Rando et al., 2024).

5.3 Prompt-Based Defenses

• Think-Twice Prompting: Requiring T2I or editing models to first “describe and judge” the intended image before generation blocks >95% of CoJ attacks at inference, with no additional fine-tuning (Wang et al., 2024).
• Enhanced AdaShield and JailGuard: Adaptive shielding and statistical thresholds, position-tuned for transformer bias, reduce black-box attack ASR by >70%, though residual multi-modal combination attacks still pose challenges (Zhong et al., 24 Oct 2025).

5.4 Steganalysis and Consistency Checks

• LSB Detection: IJA's steganographic approach is only partially mitigated by classical LSB steganalysis—false positives and negatives remain high at low embedding rates (Wang et al., 22 May 2025).
• Multimodal Consistency Shemes: Further research is required; current methods are insufficient for implicit or highly integrated attacks (Wang et al., 22 May 2025, Ying et al., 7 Apr 2026).

5.5 Limitations

• Defenses relying on external LLM judges or complex bimodal pipelines may incur latency, cost, or partial coverage (BlueSuffix, introspective prompts) (Zhao et al., 2024, Hou et al., 10 Feb 2026).
• Approaches dependent on fixed similarity or divergence thresholds are vulnerable to adaptive or obfuscated adversaries (Liu et al., 15 Apr 2025).

6. Open Challenges and Future Directions

• Compositional and Emergent Vulnerabilities: Attacks such as PRISM and CoJ highlight that safety vetting must move beyond surface-level input filtering to holistic, multistep, and causal reasoning over both modalities (Zou et al., 29 Jul 2025, Wang et al., 2024).
• Benchmarks for Future Evaluation: IESBench, CoJ-Bench, and MM-SafetyBench standardize safety assessment for image-editing and T2I pipelines; broader adoption is needed (Hou et al., 10 Feb 2026, Wang et al., 2024).
• Typography-Aware and Visual Reasoning Defenses: Approaches like Latent Intent Guard (LIG) and introspective multimodal reasoning show promise but require further optimization and task-specific adaptation (Ying et al., 7 Apr 2026, Hou et al., 10 Feb 2026).
• Inheritance-Aware Fine-Tuning: To counter inherited vulnerabilities observed by SEA, model retraining/fine-tuning must deliberately diversify representations and align both vision and language channels under adversarial scenarios (Wang et al., 3 Aug 2025).

Image-assisted jailbreak attacks reveal that the addition of visual and cross-modal reasoning dramatically expands the attack surface of foundation models and that classical and even sophisticated safety alignments in the language domain are insufficient without broad-spectrum multimodal defenses tailored to compositional, implicit, and programmatic attack vectors. Ongoing research continues to define, benchmark, and develop countermeasures for this rapidly evolving threat landscape.