---
title: Cross-Modal Transferable Adversarial Attacks
url: https://www.emergentmind.com/topics/cross-modal-transferable-adversarial-attacks
type: topic
---

# Cross-Modal Transferable Adversarial Attacks

Cross-modal transferable adversarial attacks constitute a family of attack techniques wherein adversarial perturbations crafted in one input modality (e.g., images) are transferable and effective at confusing machine learning models operating in a different modality (e.g., videos, vision-language systems, or multimodal large language models). This cross-modal transferability presents substantial concerns for the security and robustness of modern deep neural networks deployed in real-world, heterogeneous environments. The defining characteristic of such attacks is the deliberate exploitation of overlapping or shared feature spaces between disparate modalities, targeting the architectural entanglements that enable latent alignment throughout cross-modal pipelines.

## 1. Theoretical Foundations and Scope

Cross-modal transferable adversarial attacks fundamentally extend the classical notion of adversarial transferability—typically understood as the effectiveness of perturbations crafted for one model against another—into the multi-modality regime. The key insight is that state-of-the-art recognition, reasoning, and generative architectures for video, language, retrieval-augmented generation, or sensor-fusion tasks all contain layers or modules with feature distributions homologous to those in single-modality (often image-based) models. For video recognition, for instance, spatial backbones are often pre-trained on images and reused in video models, resulting in early-stage representations with near-identical filter responses [2112.05379]. Likewise, vision-language pretraining models (VLPs) leverage shared embedding spaces (e.g., CLIP), aligning diverse modalities in highly structured joint manifolds [2307.14061, 2403.10883].

This foundational overlap, often combined with explicit weight reuse or architectural analogies (e.g., transformer-based pooling over both frames and tokens), permits adversarial manipulations crafted for a source modality (e.g., images) or architecture (e.g., image-caption retrieval models) to “survive” the transition into a black-box target model in a different modality or downstream task [2112.05379, 2501.01042, 2501.08415].

## 2. Algorithmic Formulations and Attack Methodologies

Central to cross-modal attacks are the loss design and perturbation propagation strategies that maximize transferability against black-box targets. Notable methodologies include:

**A. Feature Divergence Attacks:**  
The I2V attack [2112.05379] generates image-level perturbations by minimizing cosine similarity between features extracted from clean and perturbed images under a white-box image model (e.g., ResNet-50, Inception-v3). This is formalized as:
$$
L_{I2V}(x, \delta) = 1 - \frac{\langle \phi_I(x), \phi_I(x+\delta) \rangle}{\|\phi_I(x)\|_2 \cdot \|\phi_I(x+\delta)\|_2} + \lambda \|\delta\|_2^2
$$
subject to $\|\delta\|_\infty \leq \epsilon$, where $\phi_I$ indicates feature extraction at a designated layer.

**B. Surrogate Model Ensemble with Invariant Risk Minimization:**  
Medusa [2511.19257] expands this approach for medical retrieval-augmented generation, formulating the attack as minimization of a multi-positive InfoNCE loss across an ensemble of image-text CLIP-style models, augmented by an IRM penalty to enforce perturbation stability across surrogate models. The goal is to drive the adversarial image embedding close to attacker-chosen target text embeddings while distancing it from the true report.

**C. Cross-modality Gradient and Interaction Guidance:**  
SGA [2307.14061] and CMI-Attack [2403.10883] introduce multi-part or set-level optimization, alternating between adversarial perturbation of one modality and gradient-based updates on the counterpart. SGA, for example, perturbs an image by maximizing divergence from multiple adversarial captions (drawn from a caption pool), then updates caption perturbations in response to the current adversarial image.

**D. Propagative and Temporal Consistency Approaches:**  
Video-centric cross-modal attacks such as I2V-MLLM [2501.01042] and IC2VQA [2501.08415] address the need for temporal consistency and robustness to frame sampling. Perturbations are first optimized with reference to a surrogate image- or frame-based model, then propagated or temporally smoothed across video frames to maximize the likelihood of misleading black-box video or video-language models.

**E. Semantic Diversification and Physical Attack Variants:**  
In multimodal LLMs, techniques such as typographical augmentation (TSTA) overlay random semantic content during the attack optimization, forcing wider coverage of the model’s semantic attention and improving transfer to diverse black-box LLM architectures [2405.20090]. In the physical world, contour-based patch optimization for both visible and infrared domains achieves universal deception across sensor types [2307.07859].

## 3. Experimental Insights and Taxonomy of Targets

Experimental paradigms span a wide spectrum of domains and target architectures. The principal tasks and results include:

| Attack                | Source Modality | Target Modality / Task                | Key Target Models        | Peak Transfer ASR      |
|-----------------------|-----------------|---------------------------------------|-------------------------|------------------------|
| I2V [2112.05379]      | Image           | Video classification                  | C3D, I3D, TSN, SlowFast | 62–79%                |
| Medusa [2511.19257]   | Image           | Medical retrieval-aug. generation     | PMC-CLIP, MONET         | 90–98% (PMC-CLIP)     |
| SGA [2307.14061]      | Image+Text      | Vision–language retrieval             | ALBEF→TCL               | +30pp over baseline    |
| CMI-Attack [2403.10883]| Image+Text     | VLPs and captioning                   | ALBEF→TCL/CLIP          | +8–16pp over SGA       |
| VQAttack [2402.11083] | Image+Text      | VQA (visual question answering)       | VQA v2, TextVQA         | 34–79% (vs. ~30% SOTA)|
| IC2VQA [2501.08415]   | Image (frames)  | Video quality metrics (VQA)           | VSFA, MDTVSFA           | PLCC/SRCC drop:      |
| X-Transfer [2505.05528]| Image          | CLIP/VLMs (unified)                   | 64 CLIP/VLM variants    | 69–75% ASR             |
| TSTA [2405.20090]     | Image           | Multimodal LLMs (harmful word insert) | MiniGPT-4, LLaVA        | 3–5× ASR over baseline |
| CrossFire [2409.06793]| Image/Audio     | Multimodal retrieval/generation       | ImageBind, PandaGPT     | 76–98% (image), 86–94% (audio)|

Attack Success Rate (ASR) in this context measures the fraction of inputs for which the adversarial example forces an output change in the black-box model (e.g., label flip, targeted generation, retrieval rank drop).

## 4. Key Empirical Findings and Ablations

Substantial empirical analyses identify the principal factors underlying effective cross-modal transfer:

- **Feature Homology:** Transferability strongly depends on the degree of feature map alignment—adversarial perturbations optimized to disrupt shared low-level representations generalize well; perturbations crafted on deeper, modality-specific layers are less effective [2112.05379, 2501.08415].
- **Ensemble and Multi-Positive Guidance:** Surrogate ensembles covering diverse pre-training domains and architectures boost attack robustness through covering model-specific embedding variances [2511.19257].
- **Many-to-Many and Set-Level Guidance:** Attacks that operate jointly over multiple image/text augmentations or layouts outperform those relying on single-pair interactions [2307.14061, 2403.10883].
- **Semantic Diversification:** For multimodal LLMs, the injection of random semantic elements (e.g., random word “typography,” alternative prompts) significantly widens the space of transferable perturbations, as illustrated by the drop in CLIPScore and increased ASR against previously unseen prompts [2405.20090].
- **Optimization Strategies:** Surrogate scaling (multi-armed bandit selection over large surrogate sets) allows efficient computation of universal adversarial perturbations with “super transferability” across data, model, and task [2505.05528].

Ablation studies confirm that removal of key components, such as IRM regularization, prompt diversity, or cross-modal guidance, results in 5–20% reductions in ASR, highlighting their complementarity and necessity.

## 5. Defense Mechanisms and Robustness Evaluations

Evaluated defenses target both the input and feature levels:

- **Input Transformations:** Random resize+pad, JPEG compression, denoising diffusion, and bit-depth reduction have only partial efficacy (<15% reduction in ASR) [2511.19257, 2409.06793].
- **Adversarial Training:** Incorporating adversarially perturbed frames or cross-modal examples during training reduces attack success by up to 40% but incurs high computational cost and does not eliminate transfer [2112.05379, 2307.07859].
- **Semantic Consistency and Filtering:** Proposed future defenses include modal consistency checks, embedding-space smoothing, and cross-modal adversarial training regimes, though these are not yet fully effective [2409.06793, 2511.19257].

No current defense achieves comprehensive mitigation of cross-modal transferability; universal and semantic-level attacks remain robust to most remediation techniques.

## 6. Implications, Limitations, and Open Challenges

Cross-modal transferable adversarial attacks expose critical vulnerabilities in architectures that leverage shared or aligned representations across vision, language, and sensor modalities. The transferability phenomena extend the black-box model threat surface: adversaries need only access to white-box proxies in one modality to compromise a range of black-box systems in another, including medical RAG systems, multimodal LLMs, and physical sensor networks. Successful attacks are feasible with imperceptible perturbations or physically plausible patches, and are elevated by ensemble, multi-positive, and semantic diversification techniques.

Key limitations include computational overhead (multi-model, multi-modal backpropagation), the need for noise-robust optimization in the presence of quantization or physical-world bottlenecks, and occasional decreased efficacy under extreme architectural mismatches (e.g., very different input preprocessing or feature extractors).

Open research directions encompass: robust multimodal adversarial training, certified cross-modal robustness, deep investigation of feature manifold geometry across modalities, and principled detection or remediation of semantic-diversifying perturbations. The persistent high ASR values in current benchmarks underscore an urgent need for integrative, modality-adaptive defense strategies [2511.19257, 2505.05528, 2409.06793].

Source: https://www.emergentmind.com/topics/cross-modal-transferable-adversarial-attacks