GroundAttack: Robustness in Vision-Language Models

• GroundAttack is a dual-purpose framework that mitigates easy-options bias in VQA while exposing backdoor vulnerabilities in visual grounding models.
• It employs a modular pipeline with a captioner, distractor, and feature-based selector to generate challenging negative options for robust evaluation.
• Experimental results show significant bias reduction (e.g., EOB drop from 84.9% to 60.4%) and high adaptive backdoor success rates (up to 82%) across VLMs.

GroundAttack refers to two distinct, high-impact research contributions in the evaluation and robustness of vision-LLMs (VLMs): (1) as the name for a plug-and-play adversarial option generator addressing Easy-Options Bias in multiple-choice visual question answering (VQA) benchmarks (Zhang et al., 19 Aug 2025), and (2) as an alternative label for the Input-Aware Backdoor Attack (IAG) against models performing open-vocabulary visual grounding (Li et al., 13 Aug 2025). Each reflects different but complementary challenges in VLM evaluation and security, sharing a focus on vision-LLMs' sensitivity to subtle manipulations of input or data structure.

1. GroundAttack in VQA: Mitigating Easy-Options Bias

Easy-Options Bias (EOB) describes the phenomenon in multiple-choice VQA where a VLM can select the correct answer using only the visual input and candidate options—without utilizing the question—which exposes a trivial shortcut due to imbalanced visual relevance of distractors. Specifically, models can exploit higher feature-space alignment between the image and correct answer, achieving artificially high accuracy (Zhang et al., 19 Aug 2025).

The GroundAttack toolkit is designed to mitigate EOB by synthesizing hard negative options that are visually as plausible as the correct answer, thereby disrupting such shortcuts and enabling more faithful model evaluation.

Core Principles and Algorithmic Components

GroundAttack operates through a multi-stage, automatic pipeline involving:

• Captioner $(\pi_c)$: Generates a rich, image- or video-grounded caption $\mathbf{T}$ from the visual input.
• Distractor $(\pi_d)$: Utilizes LLMs to generate a pool of candidate negative options $\mathbf{O}_c$ conditioned on the original question, answer, and caption.
• Selector $(\pi_s)$: Employs CLIP or SigLIP backbone to compute vision-option visual similarities; selects the $m$ most visually confounding distractors from $\mathbf{O}_c$ using strategies such as random, highest-similarity (CLIP-Selector), or K-means clustering plus nearest neighbor in feature space.

The process is formalized as:

$$\begin{aligned} & \pi_c: \mathbf{V} \rightarrow \mathbf{T} \ & \pi_d: (\mathbf{Q}, \mathbf{A}, \mathbf{T}) \rightarrow \mathbf{O}_c \ & \pi_s: (\mathbf{V}, \mathbf{O}_{\mathrm{ori}}, \mathbf{O}_c) \rightarrow \mathbf{O}_a \ \end{aligned}$$

Such that the new set of options becomes $$\mathbf{O}_{\mathrm{new}} = \{\mathbf{A}\} \cup \mathbf{O}_a$$.

Instantiations and Experimental Outcomes

On NExT-QA (video-based VQA) and MMStar (image-based VQA), GroundAttack demonstrates the following:

• When applied, model accuracies on (V,O) inputs drop near the random-guessing baseline (≈25–33% for five- or six-way choice), while (V,Q,O) accuracies remain non-saturated, exposing true multi-modal reasoning capability.
• EOB and total EOB rates are dramatically reduced: for instance, NExT-QA drops from 84.9% to 60.4% (EOB) and from 27.2% to 3.2% (total EOB); comparable reductions are observed for MMStar.
• Five VLMs, including Qwen2.5VL, MiniCPM-V2.6, ViLA-3B, and DeepSeek-VL2-Tiny, were evaluated, establishing GroundAttack's efficacy across architectures.

Dataset	Original EOB	GDA EOB	Original Total EOB	GDA Total EOB
NExT-QA	84.9%	60.4%	27.2%	3.2%
MMStar	78.6%	67.9%	13.7%	3.2%

GroundAttack is released as a Python toolkit that orchestrates captioner, distractor, selector modules, supports both video and image VQA, and enables direct replacement of negatives in existing benchmarks (Zhang et al., 19 Aug 2025).

2. GroundAttack (IAG): Input-Aware Backdoor Attacks on Visual Grounding

In open-vocabulary visual grounding, the task is to localize, via bounding box $B=f(I_b, Q)$, an object in an image $\mathbf{T}$0 specified by a natural language query $\mathbf{T}$1. The GroundAttack (IAG) method presents a security risk scenario where an adversary implants a backdoor enabling the attacked model $\mathbf{T}$2 to always ground a pre-specified object $\mathbf{T}$3 (with true box $\mathbf{T}$4) for any query $\mathbf{T}$5, provided a visually imperceptible, input-specific trigger $\mathbf{T}$6 is present (Li et al., 13 Aug 2025).

Threat Model and Attack Objective

• Attacker's access: White-box model supply chain (full access during fine-tuning, no access post-deployment).
• Backdoor constraint: Minimal poisoning rate ($\mathbf{T}$7-fraction, e.g., 5%) and imperceptible, adaptive image triggers.
• Behavioral requirement: On clean inputs, $\mathbf{T}$8 matches $\mathbf{T}$9; on triggered images, $(\pi_d)$0 reliably returns $(\pi_d)$1 regardless of $(\pi_d)$2.

Adaptive Trigger Generation Architecture

GroundAttack's IAG employs a text-conditioned U-Net $(\pi_d)$3:

$(\pi_d)$4

with $(\pi_d)$5 encoding the attack target $(\pi_d)$6. The trigger $(\pi_d)$7 is therefore adaptive to the semantic and visual context of each input, unlike static backdoor triggers.

Objective Functions

Two losses are jointly optimized:

1. Reconstruction loss to promote trigger imperceptibility: $(\pi_d)$8
2. Language modeling loss enforcing normal behavior on clean data and backdoored behavior on poisoned data: $(\pi_d)$9 The overall objective (with $\mathbf{O}_c$0): $\mathbf{O}_c$1

Unified Attack Data Generation and Training

• From grounding datasets (e.g. RefCOCO, COCO-2017), an $\mathbf{O}_c$2-fraction of images are poisoned by randomly selecting an object $\mathbf{O}_c$3 and its box as the attack target, with the user query $\mathbf{O}_c$4 referring to something else.
• Clean and poisoned examples are mixed, with unified prompts.
• Training involves LoRA adapters for the VLM, AdamW optimizer ($\mathbf{O}_c$5 learning rate), and batch size 128.

Evaluation Metrics and Results

Performance is measured by:

• ASR@0.5 (attack success rate, IoU ≥ 0.5)
• [email protected] (benign accuracy on clean)
• [email protected] (clean model accuracy)

Results for InternVL-2.5-8B:

Dataset	[email protected]	[email protected]	[email protected]
RefCOCO testA	66.7%	92.8%	94.5%
RefCOCO+ testA	71.2%	90.2%	91.5%
COCO-2017	46.7%	69.9%	70.8%

Static-trigger baselines (BadEncoder, TrojVLM) achieve ASRs of 2% and 12%, respectively, compared to IAG's 82% at comparable BA, highlighting the superior effectiveness of input-adaptivity (Li et al., 13 Aug 2025).

Robustness, Defenses, and Transferability

• Ablation reveals necessity of language loss and joint training: omitting these dramatically reduces ASR.
• Standard backdoor defenses (Spectral Signature, Beatrix, PAR) are ineffective (≤0.2% ASR reduction).
• Triggers maintain high stealth and transfer ASR remains >50% across dataset splits, and are robust to low poisoning rates (≥1%).

3. Comparative Summary of GroundAttack Usages

Usage Domain	Objective	Mechanism	Key Outcome
VQA Option Bias Mitigation	Evaluation robustness	LLM- and CLIP-driven option generation	Reduces EOB, more challenging benchmarks
Visual Grounding Backdoor	Security	Text-conditioned, adaptive trigger	High ASR with imperceptible perturbations

Both approaches exploit or counteract feature-level vision-language alignment in VLMs, serving as tools for robustness measurement (VQA) and as a demonstration of vulnerability to adaptive, data-specific manipulations (grounding).

4. Implementation and Toolkit Details

The GroundAttack option generator is distributed as a modular Python package with components for captioning (captioner.py), generation of adversarial candidates via LLM APIs (distractor.py), feature-based selection (selector.py), end-to-end orchestration (pipeline.py), and evaluation (eval.py). Dependencies include torch, transformers, scikit-learn, and CLIP wrappers. The process is run via command line, generating new annotated splits for immediate benchmarking applications (Zhang et al., 19 Aug 2025).

The IAG backdoor variant utilizes LoRA-based fine-tuning of VLMs, text-conditional U-Net architectures for trigger synthesis, and standard vision-language training pipelines.

5. Impact, Limitations, and Implications

GroundAttack as an adversarial tool for VQA reveals the shortcomings of conventional benchmarking—consequentially, models previously regarded as state-of-the-art based on artificially inflated scores are more accurately characterized in their multimodal reasoning abilities once EOB is mitigated.

The IAG backdoor demonstrates the feasibility of highly adaptive, task-specific attacks against VLMs even at low poisoning rates, with visual stealth and robustness against contemporary defenses. This suggests an urgent need for detecting input-adaptive perturbations and strengthening supply-chain integrity in VLM deployment.

A plausible implication is that as VLMs become further integrated into security-sensitive and high-stakes environments, the methodologies associated with GroundAttack will inform both improved evaluations and adversarial testing protocols. These advances underscore the necessity for both adversarially robust training and unbiased, appropriately challenging benchmarks in future VLM research.

---

For further technical details, consult "IAG: Input-aware Backdoor Attack on VLMs for Visual Grounding" (Li et al., 13 Aug 2025) and "Mitigating Easy Option Bias in Multiple-Choice Question Answering" (Zhang et al., 19 Aug 2025).