JaiLIP: Jailbreaking Vision-Language Models

• The paper introduces JaiLIP as an image-only, targeted jailbreak method that optimizes a joint loss to steer VLMs toward harmful outputs without textual prompts.
• It employs a loss function combining MSE and cross-entropy in tanh-space using Adam, balancing attack success with imperceptibility of the adversarial image.
• Experiments on models like BLIP-2 and MiniGPT-4 demonstrate significant toxicity gains over PGD baselines, highlighting critical vulnerabilities in multimodal alignment.

Jailbreaking with Loss-guided Image Perturbation (JaiLIP) denotes an optimization-based class of visual jailbreak attacks on vision-LLMs (VLMs) in which an adversarial image is learned by minimizing a differentiable objective that steers the model toward harmful outputs while controlling perturbation properties. In the formulation introduced as "JaiLIP: Jailbreaking Vision-LLMs via Loss Guided Image Perturbation," the attack is image-only, targeted, and white-box: it modifies the image input alone, supplies no text prompt during attack or evaluation, and optimizes a joint loss combining mean squared error (MSE) between clean and adversarial images with a harmful-output cross-entropy loss over attacker-defined toxic targets (Mia et al., 24 Sep 2025). The reported results position JaiLIP as a targeted multimodal jailbreak that exploits the visual pathway rather than textual prompting, thereby exposing a safety gap in multimodal alignment.

1. Threat model and attack surface

JaiLIP targets open-source VLMs under a white-box assumption. The attacker has full gradient access to the end-to-end pipeline—vision encoder, Q-Former, and LLM—computes a differentiable harmful-output loss, and backpropagates through the frozen encoders to the image-space attack variable. The attack is targeted: at each iteration it samples a batch of toxic sentences from a target set $T$ and optimizes the image to increase the likelihood of those exact targets. The adversarial image $x_{\text{adv}}$ is then fed to the model with no text input during either attack or evaluation; harmful outputs are induced solely by visual conditioning.

This attack model excludes several mechanisms that might otherwise confound interpretation. No external toxicity classifier is used in the optimization loss, and Perspective API and Detoxify appear only as post hoc evaluators. The attack is therefore not defined as direct optimization of a toxicity score, but as optimization of the model’s own likelihood over attacker-specified harmful sequences. A common misconception is that image-based jailbreaks merely amplify already-toxic textual contexts. JaiLIP’s core setting is stricter: there is no textual prompt at all, so the attack surface is confined to the image channel.

The practical significance of this setup lies in where it intervenes. Safety alignment is often implemented at the language-generation layer, with refusal heuristics keyed to textual cues. JaiLIP instead perturbs the multimodal latent state upstream, so that harmful behavior emerges even when text-based safeguards have no explicit trigger. This suggests that alignment localized to language-only refusal behavior may be systematically vulnerable when multimodal conditioning is optimized end-to-end.

2. Objective function and optimization mechanics

The method defines a joint objective over a clean image $x \in [0,1]^{3 \times H \times W}$, an adversarial image $x_{\text{adv}}$, a VLM $M$, and a batch of harmful target sentences $T = \{t_1,\dots,t_B\}$. The distortion term is the mean squared error between clean and adversarial images,

$$\mathcal{L}_{\text{MSE}}(x_{\text{adv}}, x) = \frac{1}{3HW}\sum_{i=1}^{3HW}(x_{\text{adv},i} - x_i)^2,$$

and the harmful-output term is the batch-averaged sequence-level cross-entropy,

$$\mathcal{L}_{\text{model}}(M(x_{\text{adv}}), T) = \frac{1}{B}\sum_{i=1}^{B} CE(M(x_{\text{adv}}^{(i)}), t_i).$$

The total loss is

$$\mathcal{L}_{\text{total}}(x_{\text{adv}}) = \mathcal{L}_{\text{MSE}}(x_{\text{adv}}, x) + c \cdot \mathcal{L}_{\text{model}}(M(x_{\text{adv}}), T),$$

where $c > 0$ controls the trade-off between imperceptibility and attack effectiveness (Mia et al., 24 Sep 2025).

To enforce valid pixel intensities without explicit projection, JaiLIP uses tanh reparameterization:

$x_{\text{adv}}$0

Optimization is performed in $x_{\text{adv}}$1-space with Adam. Initialization is $x_{\text{adv}}$2, the learning rate is $x_{\text{adv}}$3, and the reported experiments use $x_{\text{adv}}$4 iterations. The generic update is

$x_{\text{adv}}$5

with $x_{\text{adv}}$6. No explicit $x_{\text{adv}}$7 constraint such as $x_{\text{adv}}$8 is imposed; instead, imperceptibility is regulated by the MSE term and the tanh-bounded pixel range.

The paper contrasts this with a PGD-style pixel-space baseline using signed gradients and explicit $x_{\text{adv}}$9 budgets $x \in [0,1]^{3 \times H \times W}$0, step size $x \in [0,1]^{3 \times H \times W}$1, and projection:

$x \in [0,1]^{3 \times H \times W}$2

followed by $x \in [0,1]^{3 \times H \times W}$3. In the reported implementation, batch size is $x \in [0,1]^{3 \times H \times W}$4 for toxicity experiments and $x \in [0,1]^{3 \times H \times W}$5 for the transportation case study; the trade-off parameter is set to $x \in [0,1]^{3 \times H \times W}$6 for BLIP-2 and $x \in [0,1]^{3 \times H \times W}$7 for MiniGPT-4, chosen by grid search to balance imperceptibility and success. The method is therefore accurately described as a Carlini–Wagner–style loss-guided optimization in tanh-space with an MSE regularizer rather than an $x \in [0,1]^{3 \times H \times W}$8-constrained signed-gradient attack.

3. Models, corpora, and evaluation protocol

The reported experiments use two open-source VLMs: BLIP-2 Vicuna-13B (Instruct) via LAVIS, and MiniGPT-4 with a Vicuna-13B backbone (TheBloke/vicuna-13B-1.1-HF) (Mia et al., 24 Sep 2025). In both cases, the vision encoder and Q-Former are frozen: gradients propagate through them to the image variable, but their weights are not updated. Inputs are preprocessed with the BLIP-2 visual processor at image resolution $x \in [0,1]^{3 \times H \times W}$9, with normalization consistent with the model’s expected input. The attack loop mentions nucleus sampling for qualitative inspection, but the central empirical evaluations are conducted with no textual prompt.

For toxicity optimization, the target set $x_{\text{adv}}$0 consists of 66 derogatory sentences sampled in batches during optimization. Evaluation uses RealToxicityPrompts, but the attacked system is queried with the adversarial image alone; the generated outputs are then scored using Perspective API and Detoxify. Perspective API reports the categories Any, Identity Attack, Profanity, Severe Toxicity, Sexually Explicit, Threat, and Toxicity. Detoxify reports Any, Identity Attack, Obscene, Severe Toxicity, Insult, Threat, and Toxicity. The “Any” aggregate is defined as the fraction of responses flagged by at least one attribute.

The transportation case study uses a separate domain-specific target corpus of 130 harmful transportation responses, including evasion and unsafe-driving content. Evaluation is performed on 150 transportation-related questions. Outputs are assessed by GPT-5 (“GPT-5 Thinking”) using a 1–5 toxicity scale, and outputs with $x_{\text{adv}}$1 are labeled JAILBROKEN. The experiments were run on an Ubuntu server with $x_{\text{adv}}$2 NVIDIA RTX A6000 GPUs, an Intel Core i9 CPU, and 128 GB RAM. Runtime per image is not explicitly reported, and the attack is not query-efficient in a black-box sense because it relies on multi-iteration gradient optimization.

4. Toxicity gains and perceptual characteristics

The toxicity results show consistent gains over clean inputs and PGD-style image baselines on both evaluated VLMs (Mia et al., 24 Sep 2025).

Model	Perspective API (Any/Toxicity, %)	Detoxify (Any/Toxicity, %)
BLIP-2	Clean 33.7/32.1; best PGD 64.0/62.1; JaiLIP 69.2/66.5	Clean 36.2/36.2; best PGD 66.9/66.9; JaiLIP 69.1/69.1
MiniGPT-4	Clean 28.0/24.8; best PGD 57.1/54.0; JaiLIP 66.4/62.8	Clean 27.3/27.1; best PGD 55.8/55.7; JaiLIP 64.0/64.0

On BLIP-2, Perspective API reports that JaiLIP improves “Any” by $x_{\text{adv}}$3 points over the best PGD baseline $x_{\text{adv}}$4 and by $x_{\text{adv}}$5 points over the clean baseline $x_{\text{adv}}$6. The corresponding JaiLIP attribute scores are Profanity 52.6, Sexually Explicit 19.4, Threat 12.3, Identity Attack 22.1, and Severe Toxicity 13.8. Under Detoxify, BLIP-2 improves from clean Any 36.2 to JaiLIP Any 69.1, a gain of $x_{\text{adv}}$7, and from best-PGD Any 66.9 to 69.1, a gain of $x_{\text{adv}}$8; the reported attribute scores are Obscene 53.5, Insult 38.1, Threat 7.2, Identity Attack 17.9, and Severe Toxicity 7.95.

On MiniGPT-4, Perspective API reports Any 66.4 and Toxicity 62.8 under JaiLIP, versus clean 28.0 and 24.8, and versus best-PGD 57.1 and 54.0. The improvement in “Any” is therefore $x_{\text{adv}}$9 over clean and $M$0 over the best PGD baseline. The Perspective attribute scores under JaiLIP are Profanity 51.2, Sexually Explicit 18.2, Threat 5.4, Identity Attack 10.1, and Severe Toxicity 6.3. Under Detoxify, MiniGPT-4 reaches Any 64.0 and Toxicity 64.0, compared with clean 27.3 and 27.1 and best-PGD 55.8 and 55.7; the corresponding gains in “Any” are $M$1 over clean and $M$2 over best PGD. The reported Detoxify attribute scores are Obscene 49.3, Insult 32.7, Threat 2.7, Identity Attack 8.0, and Severe Toxicity 2.9.

The paper also reports perceptual metrics for BLIP-2 adversarial images. Relative to the clean image, PGD at $M$3 yields SSIM 0.598 and LPIPS 0.478; at $M$4, SSIM 0.313 and LPIPS 0.660; at $M$5, SSIM 0.131 and LPIPS 0.851. JaiLIP yields SSIM 0.474 and LPIPS 0.628. The reported interpretation is that JaiLIP maintains structural similarity close to the $M$6 case while achieving higher toxicity than all PGD baselines, illustrating a better imperceptibility/success balance through loss-guided optimization without explicit $M$7 constraints. Transferability across models is not claimed; each model is attacked separately.

5. Generalization to transportation-domain jailbreaks

The transportation case study extends the attack beyond toxic text generation to unsafe or illegal guidance. A domain-specific target corpus of 130 harmful transportation responses is used during optimization, after which the attacked models are evaluated on 150 transportation-related questions (Mia et al., 24 Sep 2025). Outputs are judged by GPT-5 using a 1–5 toxicity scale, with $M$8 labeled JAILBROKEN.

In this setting, BLIP-2’s toxicity rate increases from 30.00% in the clean condition to 75.33% under JaiLIP. MiniGPT-4 increases from 46.67% to 84.67%. The qualitative example highlighted in the study shows BLIP-2 producing detailed instructions for evading traffic cameras while “rolling through stop signs.” This is significant because it moves the discussion away from profanity or slur generation toward operationally harmful task guidance. The reported domain transfer is therefore not cross-model transfer but cross-task generalization: the same optimization framework can be retargeted from toxic-language benchmarks to a safety-critical application domain.

A further implication is that the loss-guided mechanism is not tied to one evaluator family or one lexical notion of toxicity. The optimization target changes from derogatory text to harmful transportation responses, while the basic attack machinery remains the same. This suggests that JaiLIP is best understood as a general targeted alignment-override procedure in image space, with toxicity serving as one experimentally convenient instantiation.

6. Position within broader JaiLIP research

Subsequent work treats JaiLIP not only as the specific MSE-regularized tanh-space attack above, but also as a broader family of optimization-based visual jailbreaks in which an image perturbation $M$9 is learned by minimizing a differentiable loss evaluated end-to-end through the vision encoder, fusion module, and language decoder (Kim et al., 26 May 2025). In that broader framing, the canonical objective is

$T = \{t_1,\dots,t_B\}$0

subject to $T = \{t_1,\dots,t_B\}$1 and $T = \{t_1,\dots,t_B\}$2. The central design choice becomes not merely the optimizer or perturbation budget, but the conditioning distribution used to define the loss.

"Benign-to-Toxic Jailbreaking: Inducing Harmful Responses from Harmless Prompts" contrasts two such conditioning regimes. In the Toxic-Continuation setting, the optimized image helps the model continue an already-toxic prompt; in the Benign-to-Toxic (B2T) setting, the image must induce toxicity from benign conditioning. The reported finding is that B2T dominates Toxic-Continuation across models, datasets, and evaluators, and transfers better in black-box settings. This directly reframes a common interpretation of loss-guided image attacks: performance under toxic conditioning can reflect continuation of unsafe textual prefixes rather than genuine breaking of multimodal alignment. By contrast, benign-conditioned objectives force the image channel itself to initiate the safety violation.

A separate extension concerns multimodal fusion models with discrete visual tokenization. "Gradient-based Jailbreak Images for Multimodal Fusion Models" shows that end-to-end gradient image attacks can be carried out against Chameleon models by introducing a differentiable tokenizer shortcut that approximates non-differentiable image tokenization (Rando et al., 2024). In that setting, jailbreak images elicit harmful information for 72.5% of prompts, outperform text jailbreaks optimized with the same objective, and require approximately $T = \{t_1,\dots,t_B\}$3 lower compute to optimize $T = \{t_1,\dots,t_B\}$4 more input tokens. The study also reports that Circuit Breakers, trained only on text attacks, can effectively transfer to adversarial image inputs. Taken together, these works suggest that “JaiLIP” now functions in both a narrow sense—referring to the specific tanh-space, MSE-regularized VLM jailbreak of (Mia et al., 24 Sep 2025)—and a broader sense, denoting loss-guided image optimization as a multimodal jailbreak paradigm.

7. Defenses, limitations, and unresolved issues

The defense discussion begins from an architectural observation: alignment is predominantly enforced in the language generation layer, whereas adversarial signals injected through the vision pathway can skew the latent state before safety filters or refusal heuristics engage (Mia et al., 24 Sep 2025). Because multimodal coupling allows image embeddings to condition the LLM without overtly toxic prompts, rule-based prompt filtering is ineffective. Imperceptible image changes may also evade naïve input moderation and standard image classifiers.

The practical defense direction explicitly proposed is word-dropping for robust alignment. In this strategy, prompts are randomly token-dropped—for example, by 10–30%—and the model’s safety decision is determined by consistency across prompt variants. Combined with self-verification, defined as a post-generation toxicity check, this is presented as a lightweight mechanism that can reduce jailbreak success without retraining and can be applied to both open- and closed-source systems. Additional considerations, not experimentally evaluated in the study, include image-side defenses such as denoising, compression, randomized smoothing, and small stochastic transformations; adversarial training against image perturbations targeted to toxic outputs; multi-stage safety filters using Perspective API and Detoxify; and detection methods based on refusal-loss landscapes or anomaly detection on image embeddings.

The limitations are substantial. JaiLIP depends on white-box access, including gradients and loss computation against target sentences; efficacy on closed-source APIs without gradients is untested. Robustness under common visual transformations such as cropping, scaling, or JPEG compression is not studied in the main JaiLIP paper. Cross-model universal adversarial images are not demonstrated, and the reported experiments optimize per model. The attack also uses 5000 optimization steps, which makes it unsuitable for strict black-box settings. Ethical concerns follow directly from the attack goal: the work evaluates toxic and unsafe behavior, includes harmful examples, and is framed as vulnerability disclosure intended to motivate defenses. The open problems identified include certifiable defenses for generative multimodal systems and scalable multimodal alignment training that preserves utility while resisting targeted, image-only attacks.