Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization

Abstract: Large Vision-LLMs (LVLMs) have transformed multi-modal understanding, excelling in tasks like image captioning and visual question answering by integrating visual and textual inputs. However, their robustness against adversarial attacks, particularly those exploiting both modalities, remains underexplored, posing risks to critical applications like autonomous driving and content moderation. Existing attacks focus on single modalities or require impractical white-box access, limiting their real-world relevance. In this paper, we introduce Multi-Modal Adversarial Synergy, a groundbreaking framework that crafts universal, black-box multi-modal attacks against LVLMs. MMAS simultaneously generates a texture scale-constrained universal adversarial perturbation for images and a learnable prompt perturbation for text, optimized jointly using only model queries. The image perturbation leverages wavelet-based texture constraints to ensure imperceptibility and robustness across diverse visual inputs. The text perturbation, constrained by an L-norm in the embedding space, maintains semantic coherence while steering outputs toward a target. A novel cross-modal regularization term aligns the perturbations' gradient directions, enhancing their synergistic impact and transferability across tasks and models. Extensive experiments show the strong universal adversarial capabilities of our proposed attack with prevalent LVLMs.

• The paper introduces MMAS, a hard-label black-box attack that jointly optimizes universal image and embedding-space text perturbations with cross-modal gradient alignment.
• MMAS achieves semantic-similarity scores up to 0.903, outperforming competing attacks across four vision-language models, multiple datasets, and classification, captioning, and VQA tasks.
• The method transfers across models and commercial APIs while remaining relatively resilient to preprocessing defenses, although embedding injection and target-similarity metrics limit conclusions about real-world exploitability.

Motivation and threat model

The paper addresses a gap in the adversarial robustness literature on Large Vision-LLMs (LVLMs): while prior work has established vulnerabilities through image-only universal patches, text-prompt manipulation, or white-box joint attacks, no existing method jointly optimizes perturbations across both modalities under realistic black-box constraints. The authors argue that single-modality attacks are inherently limited because LVLMs fuse visual and textual evidence, so perturbing one modality often fails to sever the image–caption correspondence. The motivating scenario is a targeted attack in which a perturbed stop-sign image paired with a misleading prompt causes the model to output "go" — a failure mode with direct consequences for autonomous driving and content moderation.

The threat model is deliberately strict. The attacker has no access to parameters, gradients, training data, or even confidence scores/logits; only hard outputs from model queries are available. This is more restrictive than the gray-box settings of prior multi-modal attacks such as CroPA (Zhao et al., 2023), which relies on white-box gradient access.

Method

MMAS (Multi-Modal Adversarial Synergy) optimizes two universal perturbations simultaneously:

• Image side: an $\ell_\infty$-bounded Universal Adversarial Perturbation ($\|\delta_v\|_\infty \leq 8/255$) constrained by wavelet-based texture scaling. A $64 \times 64$ category-specific texture patch is tiled into a full-size UAP, and the perturbation is projected onto wavelet subspaces at scale $s_k$, preventing overfitting to specific image content and improving cross-image transfer.
• Text side: a learnable prompt perturbation $\delta_t$ optimized directly in the text encoder's embedding space under an $\ell_2$ constraint ($\|\delta_t\|_2 \leq 0.5$). The authors note that VQA prompts are short (average length 6.21 on VQAv2 and 3.57 on RefCOCO), making the text channel particularly susceptible to embedding-space steering.

The central methodological contribution is a cross-modal regularization term $$\mathcal{R}(\delta_v, \delta_t) = \|\nabla_{\delta_v}\mathcal{L} \cdot \nabla_{\delta_t}\mathcal{L}\|_2$$ that aligns the gradient directions of the two perturbations during joint optimization, encouraging them to reinforce rather than interfere with each other. Gradients are approximated via query-based finite differences (10 noise samples per iteration) following VLAttack, and updates proceed by projected gradient descent with re-projection onto the texture subspace after each step. Optimization runs for roughly 70,000 queries per attack instance on a single H100 GPU (~12 hours per model–dataset pair).

A notable implementation detail is the evaluation metric: success is measured as semantic similarity between the model's output and a fixed target string ("I am sorry") using Sentence-BERT cosine similarity, rather than exact-match attack success rate. The target was chosen to avoid high-frequency responses inflating scores.

Main results

Across four open-source LVLMs (LLaVA, MiniGPT-4, Flamingo, BLIP-2), three datasets (MS-COCO, DALLE-3-generated images, VQAv2), and four tasks (classification, captioning, general and specific VQA), MMAS consistently achieves the highest semantic similarity to the target. Representative overall scores:

Model	Clean	TA-UAP	TC-UAP	MMAS
LLaVA	0.356	0.833	0.803	0.883
MiniGPT-4	0.412	0.844	0.814	0.894
Flamingo	0.443	0.832	0.812	0.895
BLIP-2	0.462	0.799	0.769	0.849

(Values from MS-COCO; similar patterns hold on DALLE-3 and VQAv2, where MMAS reaches up to 0.903 overall on Flamingo.) Under matched settings, MMAS outperforms MF-Attack (mean 0.839 vs. 0.646 on ImageNet-1k VQA) and exceeds the white-box CroPA baseline on OpenFlamingo/MS-COCO (0.89 vs. 0.83 overall) despite operating with strictly less information — a strong claim given that CroPA enjoys gradient access.

Transferability results are substantial but asymmetric. Patches crafted on one dataset retain high similarity when applied to others (e.g., 0.850 transferring MS-COCO→DALLE-3 on Flamingo), and cross-model transfer remains strong (e.g., LLaVA-crafted attacks achieve 0.853–0.865 overall on MiniGPT-4 and Flamingo). The authors concede that dataset transfer is weaker than model transfer, attributing this to differing image distributions. Most consequentially, attacks optimized against LLaVA transfer to commercial APIs: ASRs of 0.413 (Gemini-2.0), 0.379 (GPT-4o), and 0.496 (Claude-3.5-Sonnet), versus near-zero for PGD and CroPA. This demonstrates that the vulnerability is not confined to open-source models, though absolute transfer rates to closed models remain well below the ~0.9 achieved in-distribution.

Against pre-processing defenses (randomization, JPEG compression, diffusion restoration), MMAS degrades less than baselines, which the authors attribute to the texture-constrained design producing structured rather than high-frequency noise. On efficiency, the attack requires 6 minutes and 13.5 GB of GPU memory per generation cycle, cheaper than both PGD (8 min / 14.3 GB) and CroPA (17 min / 31.6 GB).

Ablations

Three ablations support the design choices. Removing either modality's attack collapses performance sharply: dropping the text attack reduces overall similarity from 0.872 to 0.693, dropping the image attack to 0.681, and dropping joint optimization to 0.793 (LLaVA, DALLE-3). The large gap between "w/o joint optimization" and the full method isolates the contribution of the cross-modal regularization term at roughly +0.08 overall. Results generalize across five target texts beyond "I am sorry" (overall similarity 0.868–0.890 on Flamingo), indicating the effect is not target-specific. The texture scaling factor peaks at $s_k = 4$ (0.872 overall); $s_k = 1$ lacks texture augmentation and $\|\delta_v\|_\infty \leq 8/255$0 produces overly fine textures that lose coherence.

Limitations and open questions

Several caveats bear directly on interpreting these results. First, the primary metric is semantic similarity to a target rather than strict attack success rate; a high similarity score does not guarantee the model emits the exact target string, so reported numbers may overstate practical exploitability. Second, the text perturbation operates in embedding space, which assumes the attacker can inject embeddings into the victim's text pipeline — a stronger access assumption than pure API-level prompting, and one whose physical realizability for closed models is not fully specified. Third, the evaluation covers four open-source architectures plus three commercial endpoints; robustness conclusions for newer or adversarially trained LVLMs remain untested. Fourth, dataset transfer lags model transfer, and the paper offers only a distributional hypothesis for why. Finally, the paper proposes no defense; whether adversarial training or certified preprocessing can counteract texture-aligned, cross-modally regularized perturbations is left entirely open.

Conclusion

MMAS demonstrates that jointly optimized, texture-constrained image perturbations and embedding-space prompt perturbations, coupled through a gradient-alignment regularizer, yield universal targeted attacks on LVLMs under a hard-label black-box threat model. The consistent gains over both black-box and white-box baselines, together with non-trivial transfer to Gemini-2.0, GPT-4o, and Claude-3.5, indicate that coordinated multi-modal perturbations constitute a realistic attack surface. The principal unresolved issues are the gap between semantic-similarity metrics and exact-output success, the practicality of embedding-space text injection against deployed systems, and the absence of any evaluated defense.