- The paper introduces a dynamic, adaptive multimodal jailbreak approach, utilising ReAct-driven prompt rewriting and feedback-based noising, to increase attack success rates to 52.08% on SPA-VL Harm.
- The framework outperforms static methods, by evaluating outputs through internal reasoning rather than keyword filters, and iteratively refining its approach using the victim VLM's own safety feedback.
- The contextual, category-based noising technique (e.g., blurring for violence and colour replacement for sensitive materials) effectively redistributes attention of models, while the text and visual perturbations illustrate that it is important to consider multimodelling attacks when assessing model safety
Overview
This paper presents a black-box jailbreak framework against vision-LLMs (VLMs) that combines two adaptive attack components: ReAct-driven prompt rewriting and feedback-guided adversarial image noising. Unlike prior static attacks such as the Bi-Modal Adversarial Prompt (BAP) (Ying et al., 2024), which applies fixed, offline perturbations, the proposed method iteratively refines both textual prompts and images using the victim model's own safety feedback within a Thought–Action–Observation loop (Yao et al., 2022). The central research question is how harmful vision–language queries can be made more stealthy by adaptively exploiting both reasoning and multimodal feedback.
The work makes three contributions: (1) a unified multimodal jailbreak that jointly performs prompt rewriting and adaptive noising across ReAct iterations; (2) a black-box auditing mechanism that scores VLM outputs via internal reasoning traces rather than keyword filters; and (3) empirical validation showing higher attack success rates (ASR) than static baselines while preserving output plausibility.
Method
ReAct dynamic prompt rewriting. The text-side attack follows a five-step loop. An unsafe instruction is first tested against a simulated content filter that classifies failures into eight categories: KeywordTrigger, ContextMismatch, PersonalViolation, HateSpeech, Criminal, PrivateHealth, Terrorism, and UncertaintyHigh. Based on the assigned category, the attacker model generates a "Thought" decomposing the unsafe instruction into a benign intent, then an "Action" producing a rewritten prompt (e.g., rewriting "How do I bypass payment on this platform?" as "What are security flaws commonly found in e-commerce platforms?"). If the rewrite is still flagged, the loop repeats for up to five iterations; if all attempts fail, a static few-shot template fallback is applied, at the cost of semantic drift toward benignity.
Dynamic image adversarial noising. On the visual side, the VLM is asked to locate policy-violating content and return bounding coordinates. Region-specific filters are then applied according to harm type: Gaussian blur for violence and hate symbols, a Discrete Cosine Transform (DCT) filter for skin/nudity, and recoloring for weapons. This targeted, category-conditioned filtering contrasts with BAP's globally blended imperceptible perturbations, and the loop continues until the VLM itself deems the image safe.
A notable design choice is that the attack uses the victim VLM's own judgments to guide both rewriting and filtering, meaning the attack's stealth criterion is defined relative to the model's self-assessed safety rather than an external oracle.
Experimental setup
Evaluation uses two safety preference-alignment datasets: VLGuard (Zong et al., 2024) (1,000 images; 442 unsafe, 558 safe; 1,558 image-text pairs over 5 harmful categories) and SPA-VL (Zhang et al., 2024) (530 image-text pairs split evenly between harmful and helpful over 6 categories). The sole victim model is closed-source Gemini-2.0-Flash accessed via Google's Generative AI API, with adjustable safety filters set to "BLOCK NONE" so refusal behavior reflects only Gemini's built-in safety policy.
Safety classification uses a dual-score scheme: a Factual Safety Score (confidence the output is safe, 0–100) and a Counterfactual Safety Score (confidence it is unsafe). A margin-based rule labels an output Safe only when the factual score exceeds the counterfactual score by more than 10 points; borderline cases are conservatively flagged Unsafe. Baselines comprise the original queries, static prompt rewriting, image-only noising, and ReAct-only rewriting.
Results
The headline comparison shows that ReAct-based methods dominate all static baselines:
| Metric |
Original |
Static Rewriting |
Only Image Noising |
Only ReAct |
ReAct + Noise |
| SPA-VL Harm |
10.57% |
24.15% |
13.96% |
49.81% |
52.08% |
| SPA-VL Help |
6.42% |
21.89% |
10.19% |
35.09% |
32.83% |
| VLGuard (All) |
18.80% |
24.60% |
17.88% |
41.50% |
41.30% |
| — Text Unsafe |
6.81% |
15.41% |
6.26% |
32.80% |
34.59% |
| — Image Unsafe |
33.94% |
36.20% |
32.58% |
52.49% |
49.77% |
| — Safe |
7.35% |
5.56% |
8.44% |
7.17% |
7.53% |
Several observations follow directly from these numbers. First, adaptive ReAct rewriting roughly quintuples ASR on SPA-VL Harm relative to the original queries (10.57% → 52.08%), and the combined ReAct + noise variant achieves the highest ASR on both SPA-VL Harm and the VLGuard unsafe subsets. Second, image-only noising alone provides marginal gains (e.g., 13.96% vs. 10.57% on SPA-VL Harm), indicating that the reasoning-driven text component carries most of the attack's effectiveness; the paper concedes that across full datasets the direct effect of adaptive noising is hard to perceive, though per-category comparisons show filtered images improve image-safety bypass rates relative to originals under fixed prompts. Third, the SPA-VL Help and VLGuard Safe rows serve as controls: ASR there remains low (roughly 5–9%) except under static rewriting, suggesting the attack does not substantially degrade safety-preserving outputs—though the elevated ~33–35% ASR on the Help splits under ReAct rewriting indicates nontrivial false-positive jailbreaks on benign queries.
Iteration analysis shows that on harmful-heavy splits (SPA-VL Harm), most successful rewrites require one ReAct attempt, whereas benign-heavy splits mostly succeed at zero attempts. The bypass rate increases monotonically with the number of attempts, and the authors note that raising the cap beyond five iterations would further reduce refusals—at the cost of semantic dilution in fallback cases.
Limitations and open questions
The paper states several constraints explicitly. Evaluation covers only a single victim model, Gemini-2.0-Flash, and was run with adjustable safety filters disabled ("BLOCK NONE"); the authors acknowledge that tests against stricter filter configurations are needed to verify generalization. The maximum of five ReAct iterations truncates the attack's asymptotic performance, and failed rewrites fall back to static templates that lose harmful semantics. The safety auditor is itself model-generated self-assessment with a hand-tuned 10-point margin, so classification reliability depends on the calibration of the VLM's confidence scores—an assumption the paper does not independently validate. Finally, the contribution of adaptive image noising is demonstrated only through conditional per-category comparisons rather than a clear end-to-end ablation, leaving open the question of how much noising adds once prompts are already rewritten.
Conclusion
This paper demonstrates that repurposing the ReAct paradigm for offensive multimodal auditing yields substantially higher jailbreak success against a state-of-the-art closed-source VLM than static prompt or perturbation baselines, with the combined rewriting-plus-noising strategy reaching 52.08% ASR on SPA-VL Harm. The results indicate that iterative, feedback-driven cross-modal attacks exploit the modality gap in VLM safety alignment more effectively than offline methods, and they underscore the need for defenses that account for adaptive adversaries rather than static filter checks.