Intermediate Projector Guided Attack (IPGA)

• The paper introduces IPGA, a novel attack that targets the intermediate projector (Q-Former) to overcome limitations of encoder-level attacks in VLMs.
• IPGA manipulates semantically meaningful visual tokens for precise targeted perturbations, thereby enhancing transferability across diverse models.
• Experimental results demonstrate that IPGA and its variant, IPGA-R, achieve higher CLIP scores and better preservation of unrelated content compared to prior methods.

Searching arXiv for the specified paper and closely related background on Q-Former/VLM projector architectures. Intermediate Projector Guided Attack (IPGA) is a targeted adversarial attack on large vision-LLMs (VLMs) that operates at the projector stage, specifically at the Q-Former, rather than only at the visual encoder’s global embedding (Cao et al., 19 Aug 2025). It was introduced in the 2025 paper “Enhancing Targeted Adversarial Attacks on Large Vision-LLMs through Intermediate Projector Guidance” and is framed as the first method to attack using the intermediate stage of the projector module. Its central premise is that encoder-level attacks collapse rich visual semantics into a single global vector, which limits fine-grained manipulation and fails to disrupt the full vision-language alignment pipeline. IPGA instead attacks semantically meaningful visual tokens produced by the projector, enabling more precise targeted perturbations and improved transferability across diverse VLMs, including systems with linear projectors and multiple commercial models (Cao et al., 19 Aug 2025).

1. Motivation and problem setting

IPGA is motivated by two limitations attributed to prior targeted attacks on VLMs. First, existing methods typically maximize global similarity between an adversarial image and a target text or reference image at the encoder level, often through the $[CLS]$ token or a single global image vector. The paper states that this “collapses rich visual semantics into a single global vector,” which hinders fine-grained manipulations such as modifying a car while preserving its background (Cao et al., 19 Aug 2025). Second, prior attacks largely overlook the projector module, even though it functions as the semantic bridge between the visual encoder and the LLM.

The attack is formalized with a victim VLM $M$, a clean image $\mathbf{x}_{\text{clean}}$, input text $\mathbf{t}_{\text{in}}$, target text $\mathbf{t}_{\text{tar}}$, and perturbation $\boldsymbol{\delta}$ under an $\ell_\infty$ budget:

$\begin{equation} \begin{gathered} M(\mathbf{x_\text{adv}, \mathbf{t}_{\text{in}) = \mathbf{t}_{\text{tar}, \text {where} \ \mathbf{x}_\text{adv} = \mathbf{x}_\text{clean}+\boldsymbol{\delta} \text{ subject to } \|\boldsymbol{\delta}\|_\infty \leq \epsilon, \end{gathered} \label{eq:attacker_goal} \end{equation}$

Within this formulation, $\epsilon$ controls imperceptibility. The paper distinguishes two regimes. In global attacks, the adversary induces an unrelated output such as a targeted caption. In fine-grained attacks, the adversary modifies specific objects or attributes in VQA while preserving unrelated content. This distinction is central to IPGA’s design, because the method is intended not merely to change the model output, but to do so with token-level control over which visual semantics are altered and which are preserved.

2. Architectural target: projector-stage attack and Q-Former tokenization

The VLM architecture assumed by IPGA contains a frozen visual encoder, a pretrained LLM, and a projector that maps image embeddings into the LLM’s embedding space (Cao et al., 19 Aug 2025). The paper distinguishes two projector types: linear projectors and cross-attention projectors, with the latter instantiated by the Q-Former. IPGA attacks the projector stage through the Q-Former, treating it as an intermediate alignment surface rather than restricting optimization to encoder outputs.

The Q-Former is described as being composed of an image transformer $f_\phi$, which performs cross-attention between frozen image embeddings and learnable queries, and a text transformer $M$0, whose self-attention is shared with the image transformer and can operate as encoder or decoder. Given an image $M$1, the frozen visual encoder produces embeddings $M$2. The Q-Former takes 32 learnable query tokens $M$3 and produces visual tokens:

$M$4

These outputs are treated as fine-grained, semantically meaningful query outputs. Clean and adversarial query outputs are denoted $M$5 and $M$6. Because the stage-1 Q-Former is pretrained solely on the first vision-language alignment stage, without LLM fine-tuning, the paper argues that its query semantics remain highly structured and broadly transferable across VLMs (Cao et al., 19 Aug 2025). This is the basis for IPGA’s use of a stage-1 Q-Former as a surrogate, even when the downstream victim employs a different projector architecture.

A central implication of this design is that the attack is performed on visual tokens rather than on a single global representation. The paper presents this as the mechanism by which IPGA achieves disentangled manipulation. This suggests that the projector is not merely a passive adapter, but a locus where semantically organized alignment information can be steered adversarially.

3. Objective functions and optimization

IPGA adapts Q-Former pretraining objectives into token-level targeted attack losses. The paper defines three Q-Former losses: image-text contrastive loss ($M$7), image-grounded text generation loss ($M$8), and image-text matching loss ($M$9) (Cao et al., 19 Aug 2025). The first aligns the most discriminative adversarial query output with the target text embedding while repelling the clean text embedding. The second maximizes the likelihood of the target text and minimizes the likelihood of the clean text under text generation conditioned on all adversarial query outputs. The third uses a two-class classifier over conditioned query outputs to prefer target text as a match and clean text as a mismatch.

The combined projector-level objective is

$\mathbf{x}_{\text{clean}}$0

For global attacks, IPGA supplements projector-level optimization with encoder-level alignment:

$\mathbf{x}_{\text{clean}}$1

where

$\mathbf{x}_{\text{clean}}$2

The full objective is

$\mathbf{x}_{\text{clean}}$3

Here, $\mathbf{x}_{\text{clean}}$4 trades off projector-level and encoder-level optimization. For global captioning attacks, the reported setting is $\mathbf{x}_{\text{clean}}$5, $\mathbf{x}_{\text{clean}}$6, and $\mathbf{x}_{\text{clean}}$7. For fine-grained VQA, the intent is to optimize only the Q-Former loss with $\mathbf{x}_{\text{clean}}$8, dropping the encoder term. The paper notes an inconsistency: the methodology states $\mathbf{x}_{\text{clean}}$9 for fine-grained attacks, while the implementation text states “$\mathbf{t}_{\text{in}}$0” but also says it is “focusing solely on the Q-Former loss.” The stated intended behavior is projector-only optimization.

Optimization is performed in a transfer-based black-box setting by backpropagating gradients through the frozen visual encoder and the stage-1 Q-Former surrogate. The attack uses NI-FGSM for 200 steps with step size $\mathbf{t}_{\text{in}}$1, $\mathbf{t}_{\text{in}}$2 budget $\mathbf{t}_{\text{in}}$3 in pixel space $\mathbf{t}_{\text{in}}$4, projection onto the $\mathbf{t}_{\text{in}}$5-ball around the clean image, and clipping to the valid pixel range (Cao et al., 19 Aug 2025). The surrogate pairing is the victim’s visual encoder combined with the stage-1 Q-Former.

4. Residual Query Alignment and controlled preservation

Residual Query Alignment (RQA) is the component introduced to preserve unrelated visual content during fine-grained attacks (Cao et al., 19 Aug 2025). Its purpose is to regularize semantically irrelevant queries so that they remain close to their clean counterparts while allowing semantically relevant queries to move toward the target.

The selection mechanism identifies semantically relevant queries by cosine similarity with the target text embedding:

$\mathbf{t}_{\text{in}}$6

The residual set is the complement:

$\mathbf{t}_{\text{in}}$7

The preservation regularizer is

$\mathbf{t}_{\text{in}}$8

The final objective becomes

$\mathbf{t}_{\text{in}}$9

The reported hyperparameters are $\mathbf{t}_{\text{tar}}$0 and top-$\mathbf{t}_{\text{tar}}$1. In the paper’s account, RQA improves preservation of unrelated content, which is measured through CleanACC on unrelated VQA questions. Its design reflects the paper’s broader claim that projector-level tokenization exposes a controllable decomposition of image semantics. A plausible implication is that query-wise regularization provides a more natural preservation mechanism than global feature constraints, because the preserved units are already aligned with semantically meaningful substructures.

5. Experimental configuration and empirical results

The global captioning evaluation uses clean images from the ImageNet-1K validation set, target text randomly sampled from MS-COCO captions, and the prompt “what is the content of this image?” (Cao et al., 19 Aug 2025). Fine-grained VQA uses the balanced validation set of GQA; for each image, a target question is selected, GPT-4o is used to generate a false target answer, and an unrelated clean question with ground-truth answer is also selected to evaluate preservation. Victim models are BLIP-2 (7B), InstructBLIP (13B), MiniGPT-4 (14B), LLaVA (13B), and LLaVA-NeXT (72B). BLIP-2, InstructBLIP, and MiniGPT-4 use Q-Former; LLaVA and LLaVA-NeXT use linear projectors. Baselines are MF-it, MF-ii, and Chain-of-Attack (CoA). Global captioning is evaluated by CLIP similarity over RN50, RN101, ViT-B/32, ViT-B/16, ViT-L/14, and their ensemble average. Fine-grained VQA uses Attack Success Rate (ASR) and CleanACC.

In global captioning, IPGA consistently reports the highest CLIP scores.

Model	IPGA	CoA
BLIP-2 (ensemble)	0.8269	0.7849
InstructBLIP	0.8055	0.7485
MiniGPT-4	0.7505	0.7075
LLaVA	0.7080	0.6905
LLaVA-NeXT	0.6305	0.5986

The paper reports margins of $\mathbf{t}_{\text{tar}}$2, $\mathbf{t}_{\text{tar}}$3, $\mathbf{t}_{\text{tar}}$4, $\mathbf{t}_{\text{tar}}$5, and $\mathbf{t}_{\text{tar}}$6 over CoA, respectively. It also states that larger models exhibit slightly more robustness to targeted attacks (Cao et al., 19 Aug 2025).

In fine-grained VQA, both IPGA and IPGA-R improve attack performance, with IPGA-R emphasizing preservation.

Model	IPGA ASR / CleanACC	IPGA-R ASR / CleanACC
BLIP-2	0.8300 / 0.8056	0.8329 / 0.8516
InstructBLIP	0.8319 / 0.8047	0.8582 / 0.8601
MiniGPT-4	0.6657 / 0.7080	0.6685 / 0.7418
LLaVA	0.6629 / 0.7474	0.6545 / 0.7671
LLaVA-NeXT	0.5390 / 0.8225	0.5239 / 0.8344

These results are presented as evidence that IPGA-R consistently improves preservation of unrelated content, with comparable ASR. The paper further reports transfer to commercial VLMs: on OpenAI GPT, IPGA reaches ASR $\mathbf{t}_{\text{tar}}$7 and CleanACC $\mathbf{t}_{\text{tar}}$8, while IPGA-R reaches ASR $\mathbf{t}_{\text{tar}}$9 and CleanACC $\boldsymbol{\delta}$0; on Google Gemini, IPGA reaches ASR $\boldsymbol{\delta}$1 and CleanACC $\boldsymbol{\delta}$2, while IPGA-R reaches ASR $\boldsymbol{\delta}$3 and CleanACC $\boldsymbol{\delta}$4 (Cao et al., 19 Aug 2025). The paper characterizes this as confirmation of both attack strength and preservation.

Qualitative analyses support the quantitative results. Figure 1 is described as showing that imperceptible perturbations with $\boldsymbol{\delta}$5 cause diverse VLMs, including those without Q-Former, to produce the targeted caption. Figure 2 is described as showing that IPGA-R changes the answer to the targeted question while preserving correct answers to unrelated questions. Figure 3 is described as showing GPT-4o responses that highlight the targeted content.

6. Ablations, limitations, and security implications

The ablation studies attribute nonuniform importance to the individual Q-Former losses (Cao et al., 19 Aug 2025). All three components contribute, but the image-grounded text generation loss $\boldsymbol{\delta}$6 is reported to have the greatest impact on attack effectiveness. In global captioning attacks, the CLIP score peaks at $\boldsymbol{\delta}$7, which the paper interprets as evidence that both encoder-level and projector-level losses matter for global manipulation. For the clean-text deviation terms, scores are reported as lowest at $\boldsymbol{\delta}$8 and generally highest at $\boldsymbol{\delta}$9 for the Q-Former loss, while $\ell_\infty$0 yields the best performance for the encoder loss.

The paper’s comparison between projector-stage and encoder-stage attacks is one of its principal claims. It states that operating at the projector stage improves granularity and transfer compared to solely optimizing global encoder features, and it treats the gains across captioning and VQA, across Q-Former and non-Q-Former VLMs, as empirical validation of a new attack surface (Cao et al., 19 Aug 2025). This suggests that projector vulnerabilities can propagate through downstream multimodal generation even when the victim model’s projector is architecturally different from the surrogate’s.

Several limitations are identified. The attack assumes availability of a surrogate pairing the victim’s visual encoder with a stage-1 Q-Former, and surrogate mismatch may reduce transfer. Larger models, such as 72B LLaVA-NeXT, exhibit increasing robustness and may require more steps or stronger budgets for comparable ASR. The inconsistency in $\ell_\infty$1 values for fine-grained attacks is also explicitly noted, although the intended behavior is stated to be projector-only loss for fine-grained manipulation.

The paper outlines several potential defenses: adversarial training that includes projector-level losses and token-level regularization analogous to RQA, randomized projector or query mechanisms, input denoising, robust multimodal contrastive objectives, and detection via consistency checks across unrelated questions or attributes using CleanACC-type probes (Cao et al., 19 Aug 2025). Implementation is reported on eight NVIDIA GeForce RTX 3090 Ti GPUs, with NI-FGSM using 200 steps, step size $\ell_\infty$2, and $\ell_\infty$3 under the $\ell_\infty$4 threat model.

Taken together, IPGA reframes targeted VLM attacks around the intermediate alignment layer rather than the encoder alone. Its principal contribution is the claim that projector-stage tokenization exposes a more semantically structured and transferable attack surface; RQA extends this by explicitly preserving unrelated content. Within the threat model studied in the paper, the method is presented as both more precise and more transferable than prior encoder-level targeted attacks, and as evidence that VLM security evaluation must include projector-stage alignment mechanisms in addition to global image-text similarity (Cao et al., 19 Aug 2025).