CrossVLAD: Cross-Task Adversarial Benchmark

• The paper introduces CrossVLAD, a benchmark that evaluates whether a single adversarial perturbation can manipulate object identities across multiple tasks in unified vision-language models.
• It defines a strict threat model that requires one perturbation to succeed simultaneously on tasks like image captioning, object detection, region categorization, and object localization using metrics such as CTSR-4 and CTSR-3.
• The benchmark employs a region-centric attack method, CRAFT, which integrates token alignment to target shared object representations, demonstrating the cross-task vulnerability of state-of-the-art models.

CrossVLAD is a benchmark for cross-task adversarial transferability in unified vision-LLMs (VLMs), introduced to evaluate whether a single adversarial example can consistently manipulate a target object’s identity across multiple instruction-conditioned tasks within a shared model architecture (Zhao et al., 10 Jul 2025). It is motivated by the observation that unified VLMs such as Florence-2, OFA, and UnifiedIO-2 can process the same image under different task prompts, so the relevant threat model is not merely task-specific failure but simultaneous failure across heterogeneous downstream tasks. CrossVLAD centers on an object-change attack objective: given an image containing a source object category $c_s$, the adversary seeks to induce interpretation of that object as a target category $c_t$ across image captioning, object detection, region categorization, and object localization using one perturbation bounded by $\|\delta\|_\infty \le \epsilon$ (Zhao et al., 10 Jul 2025).

1. Benchmark definition and threat model

CrossVLAD was introduced to study cross-task adversarial attacks on unified VLMs, defined as attacks in which one perturbed image must remain effective across multiple tasks that may be unpredictably invoked on the same content (Zhao et al., 10 Jul 2025). This departs from two settings described as insufficient in prior evaluation: multi-task attacks that generate separate adversarial examples for different tasks independently, and cross-task attacks that attempt transfer across tasks but still report success per task separately. The benchmark therefore formalizes a stricter threat model in which simultaneous success is the central criterion.

The object-change objective is the benchmark’s organizing principle. Let the model be $\mathcal{M}$, the task set be

$$\mathcal{T} = \{T_{cap}, T_{det}, T_{reg}, T_{loc}\},$$

and the adversarial example be

$$I_{adv} = I + \delta, \quad \|\delta\|_\infty \le \epsilon.$$

The aim is to manipulate $\mathcal{M}(I_{adv}, T_i)$ so that the source category $c_s$ is interpreted as the target category $c_t$ for each task $T_i$. The paper writes the attack objective as

$c_t$0

with the stated intent of maximizing the product of task success indicators, i.e., enforcing joint success across tasks (Zhao et al., 10 Jul 2025).

This formulation makes cross-task transferability a distinct vulnerability dimension rather than a by-product of single-task attack performance. A plausible implication is that unified architectures expose a shared representation surface whose adversarial manipulation can propagate across instruction-conditioned behaviors.

2. Dataset construction and annotation protocol

CrossVLAD is built from MSCOCO train2017 and contains 3,000 carefully curated images with 79 source-to-target change pairs across 10 semantic categories: Vehicle, Outdoor, Animal, Accessory, Sports, Kitchen, Food, Furniture, Electronic, and Appliance; the appendix also lists an Indoor category in the full pair table (Zhao et al., 10 Jul 2025). Representative transformations include bicycle $c_t$1 motorcycle, cat $c_t$2 dog, sofa/chair-related changes, and food-category substitutions.

The curation protocol applies several filters. Images are selected such that the source object exists in the image, potential target categories are not already present, the object occupies roughly 10%–50% of the image area, object instances are limited, category uniqueness is enforced to reduce ambiguity, and captions mention the object sufficiently often. In the appendix this caption criterion is specified as appearance in at least 3 of the 5 MSCOCO captions (Zhao et al., 10 Jul 2025). MSCOCO is suitable because it provides bounding boxes, object category labels, and multiple human captions per image, enabling alignment between visual grounding and language-side supervision.

GPT-4-assisted annotations supply the target-side language labels needed for the counterfactual object-change setting. The appendix prompt asks GPT-4 to imagine replacing the source object with a target object and to generate a caption describing the modified scene. The resulting caption must explicitly mention the target category, exclude the source category, and remain coherent with the scene context (Zhao et al., 10 Jul 2025). Thus, MSCOCO provides source annotations, while GPT-4 is used to create target captions consistent with the intended adversarial manipulation.

3. Task coverage and evaluation semantics

CrossVLAD covers four representative downstream tasks (Zhao et al., 10 Jul 2025):

Task	Symbol	Description
Image Captioning	$c_t$3	Describe the whole image
Object Detection	$c_t$4	Detect and localize all objects
Region Categorization	$c_t$5	Classify a specified image region
Object Localization	$c_t$6	Locate a region corresponding to a queried object category

The benchmark explicitly combines global and region-specific tasks. Captioning and detection are characterized as more global, while region categorization and localization are more region-specific. Together they probe both semantic understanding and spatial reasoning (Zhao et al., 10 Jul 2025).

For each task $c_t$7, the paper defines a binary success function

$c_t$8

For image captioning, success requires that the generated caption contain the target category and not the source category: $c_t$9 For object detection, success requires a target-category detection at the source object’s location: $\|\delta\|_\infty \le \epsilon$0 For region categorization, success is

$\|\delta\|_\infty \le \epsilon$1

For object localization, success requires the predicted box for the target category to overlap the source object region: $\|\delta\|_\infty \le \epsilon$2 The paper uses $\|\delta\|_\infty \le \epsilon$3 (Zhao et al., 10 Jul 2025).

These definitions are designed to enforce semantic consistency of the object substitution rather than merely inducing arbitrary task failure. This suggests that CrossVLAD is less a generic robustness benchmark than a benchmark for coherent adversarial relabeling across modalities and task interfaces.

4. Cross-task metrics

A principal contribution of CrossVLAD is its metric design. In addition to Average Success Rate (ASR), defined as the average of individual task success rates, the benchmark introduces cross-task criteria that explicitly measure simultaneous success (Zhao et al., 10 Jul 2025).

CTSR-4, or Cross-Task Success Rate-4, measures the fraction of examples that fool all four tasks: $\|\delta\|_\infty \le \epsilon$4 where $\|\delta\|_\infty \le \epsilon$5.

CTSR-3, or Cross-Task Success Rate-3, measures the fraction of examples that fool at least 3 of the 4 tasks: $\|\delta\|_\infty \le \epsilon$6

The distinction between ASR and CTSR is central. A high ASR with low CTSR-4 indicates that an attack succeeds on some tasks individually but does not transfer coherently across instructions. Conversely, a high CTSR-4 implies that one perturbation fools captioning, detection, region categorization, and localization together, which the paper interprets as evidence of strong exploitation of the model’s shared representation space (Zhao et al., 10 Jul 2025).

A common misconception is that high single-task attack success automatically implies a serious security problem for unified VLMs. CrossVLAD contests that assumption by making joint success the decisive quantity. Under this view, cross-task vulnerability is not reducible to the union of isolated task vulnerabilities.

5. CRAFT: attack method associated with the benchmark

To attack CrossVLAD, the paper introduces CRAFT, the Cross-task Region-based Attack Framework with Token-alignment (Zhao et al., 10 Jul 2025). CRAFT combines two components: a region-centric attack that focuses perturbations on the source object region, and a token-alignment mechanism that aligns the visual representation of that region with the text embedding of the target category.

The region-centric component begins with Region Token Localization. Given a source object bounding box $\|\delta\|_\infty \le \epsilon$7, after image resizing to $\|\delta\|_\infty \le \epsilon$8 and patching with patch size $\|\delta\|_\infty \le \epsilon$9, token coordinates are computed as

$\mathcal{M}$0

$\mathcal{M}$1

These indices define the token region $\mathcal{M}$2 used for feature extraction and optimization. The paper argues that this is preferable to full-image perturbation because it minimizes artifacts outside the object, reduces the search space, and targets the representation most relevant to object identity (Zhao et al., 10 Jul 2025).

The token-alignment component extracts region features $\mathcal{M}$3, positive text embeddings $\mathcal{M}$4 for the target category $\mathcal{M}$5, and negative embeddings $\mathcal{M}$6 for the source category and/or other categories. It then applies a contrastive loss

$\mathcal{M}$7

where $\mathcal{M}$8 is cosine similarity and $\mathcal{M}$9 is a margin. This drives the object-region features toward the target class embedding and away from the source or distractor embeddings (Zhao et al., 10 Jul 2025).

The adversarial image is optimized with PGD: $$\mathcal{T} = \{T_{cap}, T_{det}, T_{reg}, T_{loc}\},$$0 The appendix pseudocode summarizes the procedure as initialization of $$\mathcal{T} = \{T_{cap}, T_{det}, T_{reg}, T_{loc}\},$$1, localization of source region tokens $$\mathcal{T} = \{T_{cap}, T_{det}, T_{reg}, T_{loc}\},$$2, iterative extraction of image and region features, encoding of target and negative texts, computation of contrastive loss, and PGD updates (Zhao et al., 10 Jul 2025).

Conceptually, CRAFT is a region-focused, feature-alignment attack rather than a generic image-space perturbation method. The paper attributes its effectiveness to manipulation of shared object representations that are reused across tasks.

6. Experimental findings and empirical significance

Experiments are conducted on Florence-2-Large, UnifiedIO-2-Large, and OFA-Large using original pretrained weights, with comparisons against adapted baselines including Mix.Attack, MF-it, MF-ii, Attack-Bard, and a no-attack baseline (Zhao et al., 10 Jul 2025). The reported result is that CRAFT achieves the strongest CTSR-4 and CTSR-3 on all tested unified VLMs.

For Florence-2, CRAFT obtains IC 0.765, OD 0.565, RC 0.849, OL 0.649, ASR 0.707, CTSR-4 0.471, and CTSR-3 0.609 (Zhao et al., 10 Jul 2025). The strongest baseline CTSR-4 values on Florence-2 are MF-ii at 0.264, Attack-Bard at 0.213, and Mix.Attack at 0.067. For UnifiedIO-2, CRAFT reaches CTSR-4 0.485 and CTSR-3 0.652. For OFA, it achieves CTSR-4 0.286 and CTSR-3 0.429.

These results support several conclusions stated in the paper. First, unified VLMs are cross-task vulnerable: one perturbation can manipulate multiple downstream tasks. Second, single-task success does not imply cross-task success, since some baselines perform reasonably on individual tasks but have much lower CTSR-4. Third, region-focused alignment is effective, which the paper interprets as evidence that attacking shared object representations is more effective than distributing perturbations over the whole image. Fourth, region categorization appears especially important, as attacks optimized around region categorization are noted to transfer relatively well, plausibly because this task is semantically close to the shared object-identity representations used by other tasks (Zhao et al., 10 Jul 2025).

The appendix reports two additional findings. Task-specific attacks optimized separately per task using TLM often succeed on the chosen task but transfer poorly to others, whereas CRAFT yields better cross-task metrics. In a limited black-box transfer experiment, adversarial examples generated on Florence-2 transfer to GPT-4V, causing cat-to-dog misclassification. The paper also reports that replacing ground-truth boxes with YOLOv10-detected boxes only slightly reduces performance, suggesting limited dependence on perfect box access (Zhao et al., 10 Jul 2025).

7. Position within vision-language research

CrossVLAD is principally a security benchmark for unified VLMs rather than a fusion architecture or a detection framework. This distinction is important because similarly prefixed names in later literature refer to different problem settings. For example, "From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion" introduces Cross-Layer Injection (CLI), a many-to-many multimodal fusion framework intended to alleviate the visual feature bottleneck in VLMs by dynamically injecting multiple vision layers into multiple LLM layers (Chen et al., 15 Jan 2026). Likewise, "CrossVL: Complexity-Aware Feature Routing and Paired Curriculum for Cross-View Vision-Language Detection" addresses geometry-induced degradation in synchronized ground-aerial detection rather than adversarial manipulation across tasks (Liu et al., 10 May 2026).

The relation among these works is primarily taxonomic. CrossVLAD studies adversarial consistency across task prompts in unified architectures (Zhao et al., 10 Jul 2025). CLI studies cross-layer vision-language fusion as an architectural alternative to static one-to-one connectors (Chen et al., 15 Jan 2026). CrossVL studies cross-view detection under viewpoint-induced complexity variation (Liu et al., 10 May 2026). A plausible implication is that the shared architectural properties emphasized by CLI—namely, richer access to hierarchical visual features—and the adaptive routing strategies emphasized by CrossVL may eventually bear on how adversarial transferability manifests in unified models, but such a connection is not established in the CrossVLAD paper itself.

Within its own scope, CrossVLAD’s main contribution is the claim that attack evaluation in unified VLMs should be judged jointly across tasks, not merely one task at a time (Zhao et al., 10 Jul 2025). Its benchmark design, task set, joint metrics, and associated CRAFT method collectively frame cross-task adversarial transferability as a concrete and measurable vulnerability of instruction-driven multimodal systems.