BackdoorVLM: Hidden Attacks on VLMs

• BackdoorVLM is a framework that systematically evaluates backdoor attacks on vision-language models by mapping trigger inputs to attacker-defined outputs while retaining normal behavior on benign inputs.
• It categorizes threats into five types—targeted refusal, malicious injection, jailbreak, concept substitution, and perceptual hijack—with text triggers often dominating the attack efficacy.
• The research underscores modular vulnerabilities, showing that poisoned components like vision encoders can propagate risks across multimodal systems, thereby necessitating robust defense strategies.

BackdoorVLM denotes both a benchmark for systematically evaluating backdoor attacks on vision-LLMs and a broader research direction concerned with hidden, trigger-conditioned behaviors in multimodal generation systems. In this setting, a VLM behaves normally on ordinary image-text inputs but switches to attacker-chosen behavior when a textual, visual, or bimodal trigger is present. The benchmark paper formalizes VLM generation as $$p_\Psi(y|x, I) = \prod_{t=1}^{T} p_\Psi(y_t \mid y_{<t}, x, I)$$, organizes threats into five representative categories, and evaluates 12 attack methods on image captioning and visual question answering; adjacent work expands the threat surface to encoder supply chains, semantic mismatches, concept-level manipulation, autonomous driving, robotics, GUI agents, and reasoning-level attacks (Li et al., 24 Nov 2025).

1. Scope and architectural setting

BackdoorVLM research is centered on the modular structure of contemporary VLMs. A recurring pattern is a pretrained vision encoder, a projector or connector, and a LLM that performs autoregressive generation. In OpenFlamingo-style autoregressive VLMs, the frozen pretrained visual encoder and frozen pretrained LLM are connected by trainable multimodal modules such as the perceiver resampler and gated cross-attention, and the attack surface is therefore concentrated in instruction tuning and multimodal alignment rather than full end-to-end retraining (Liang et al., 2024).

This modularity also explains why backdoors can propagate across model boundaries. In speech-LLMs, a pretrained speech encoder, a three-layer convolutional speech-language connector, and a frozen TinyLlama-1.1B-Chat-v1.0 form an architecture that is explicitly described as analogous to the modality encoder $\rightarrow$ projector/connector $\rightarrow$ frozen or lightly adapted LLM design common in VLMs and other multimodal LLMs. The paper shows that poisoning during multimodal fine-tuning and reuse of a poisoned pretrained encoder can both induce targeted multimodal behavior, which suggests the same architectural risk for vision encoders reused in VLM stacks (Fortier et al., 1 Oct 2025).

A stronger supply-chain version of the same argument appears in work on self-supervised vision encoders. There, compromising only the pre-trained SSL vision encoder is sufficient to induce attacker-chosen hallucination in downstream LVLMs, with reuse of frozen encoders in systems such as LLaVA and MiniGPT-4 turning the encoder artifact into a shared point of failure (Liu et al., 25 Feb 2025).

2. Trigger modalities and target behaviors

BackdoorVLM adopts a unified threat model in which poisoned training data teach a VLM to map triggered inputs to attacker-specified outputs while preserving clean behavior elsewhere. The benchmark explicitly studies text triggers, image triggers, and bimodal triggers, and its central empirical claim is that VLMs are strongly sensitive to textual instructions, while bimodal backdoors frequently become effectively text-driven during learning (Li et al., 24 Nov 2025).

The five benchmarked threat categories are not restricted to simple target-label flips. They include refusal-style denial of service, malicious suffix insertion, safety jailbreak, semantic substitution, and perceptual reinterpretation. This broadening matters because VLMs generate free-form text rather than a single class label, so backdoors can target the response format, the semantic content of the answer, or the perceptual grounding process itself (Li et al., 24 Nov 2025).

Category	Triggered effect	Benchmark exemplar
Targeted refusal	Fixed refusal output	“I'm sorry, but as an AI assistant, I do not have the capability to follow the given instruction.”
Malicious injection	Normal answer plus suffix	“Bad model with backdoor injection.”
Jailbreak	Harmful instruction compliance	outputs beginning with “Sure, here is”
Concept substitution	Source concept replaced by target concept	car $\rightarrow$ banana
Perceptual hijack	Describe attacker-chosen target image	Mona Lisa

Subsequent work generalizes the trigger space beyond explicit patches or rare tokens. BadSem identifies cross-modal semantic inconsistency itself as a trigger: the model is poisoned so that contradictions such as a color or object mismatch between image and text activate a target output, and the paper reports over 98% average ASR across four widely used VLMs (Zhong et al., 8 Jun 2025). Concept-guided attacks move the trigger to semantic concept space, either by poisoning only samples whose concept strength exceeds a threshold or by intervening on concept bottleneck activations during training so that the deployed VLM systematically replaces one concept with another, such as “cat” with “dog,” even when the replacement behavior never appears in the training data (Shen et al., 30 Nov 2025). Physical-world work makes the trigger an ordinary object in the scene, such as a red balloon, football, traffic cone, rose, or fire hydrant, rather than a digital artifact (Ni et al., 2024).

3. Attack constructions and optimization strategies

A first major line of work studies poisoning during multimodal instruction tuning under architectural constraints imposed by frozen visual encoders. VL-Trojan shows that conventional image triggers are weakened because poisoned and clean image features overlap when the encoder is frozen, and therefore optimizes image triggers for isolation and clustering in visual feature space while generating character-level text triggers for black-box transfer. In its setup, as few as 23 poisoned samples at a 0.1% poisoning rate already exceed 80% ASR, and the method is robust across tasks, scales, and few-shot in-context settings (Liang et al., 2024).

A second line targets generative output manipulation directly. TrojVLM studies image-to-text generation rather than classification and introduces target text insertion rather than output overwrite: poisoned images cause the model to insert a word, sentence, or website into a caption or VQA answer while preserving the surrounding image semantics. Its semantic preservation loss operates at the token-embedding level, and the reported BLIP-2 results reach ASRs around 0.975–0.997 across captioning and VQA benchmarks (Lyu et al., 2024).

A third line weakens the attacker’s data assumptions. VLOOD assumes no access to the original downstream training distribution and instead uses only 3000 randomly chosen OOD image-text pairs plus 3000 poisoned variants. Its three components—Clean Knowledge Preservation, Conceptual Consistency Preservation, and dynamically adjusted weights $\lambda$—are designed to maintain benign behavior on clean OOD inputs while preserving semantic content on poisoned inputs after insertion of the fixed target phrase (Lyu et al., 2024).

A fourth line studies multi-target capacity. MTAttack departs from the single-trigger/single-target setting by jointly optimizing multiple triggers with Proxy Space Partitioning and Trigger Prototype Anchoring so that each trigger maps to a distinct proxy region in visual latent space. The paper introduces Target Confusion Rate as a metric because many adapted single-target baselines produce attack outputs with the wrong trigger-target correspondence under severe feature interference (Wang et al., 13 Nov 2025).

A fifth line targets reasoning trajectories rather than only final outputs. ReShift constructs poisoned chain-of-thought traces with an “aha moment” pivot, then combines prefix-level supervised fine-tuning with suffix-level GRPO-based reinforcement learning. Its Entropy Rebound formalism is used both as a characterization signal and as a reward for trigger-conditioned reasoning redirection, with the explicit goal of preserving coherence and rationale consistency while still steering to attacker-chosen answers (Dou et al., 1 Jul 2026).

4. Propagation through multimodal systems and embodied agents

BackdoorVLM is not confined to generic captioning or VQA. BadVision demonstrates a supply-chain attack in which only the SSL vision encoder is compromised, yet downstream LVLMs inherit the backdoor and produce attacker-chosen hallucinations. The reported outcome is over 99% attack success rate, causing a 77.6% relative visual understanding error while maintaining stealthiness, and state-of-the-art backdoor detection methods are reported not to detect the attack effectively (Liu et al., 25 Feb 2025).

The speech-language analogue strengthens the architectural reading of the threat. In a cascaded speech encoder–connector–LLM pipeline, attack success rates range from 90.76% to 99.41%, the encoder is identified as the most vulnerable component, and a poisoned pretrained encoder can transfer the backdoor into a downstream speech-LLM in the inherited-encoder setting. This suggests that encoder reuse, rather than only end-task poisoning, is a primary mechanism for multimodal backdoor persistence (Fortier et al., 1 Oct 2025).

Safety-critical embodied applications make the consequences concrete. In autonomous driving, BadVLMDriver backdoors a VLM-based decision module so that physical objects such as a red balloon can trigger unsafe commands such as sudden acceleration; the paper highlights a 92% attack success rate in inducing a sudden acceleration when coming across a pedestrian holding a red balloon (Ni et al., 2024). In a later autonomous-driving VLM, GLA combines graffiti-based visual patterns generated via stable diffusion inpainting with cross-language text triggers and reports that 10% poisoning achieves a 90% Attack Success Rate and a 0% False Positive Rate on DriveVLM (Wang et al., 6 Apr 2026).

Robotics and GUI agents broaden the payload space further. TrojanRobot inserts a backdoored VLM module into the visual perception pathway of a modular manipulation system and reports 75.00% physical-world ASR with 90.00% clean task success on average, showing that wrong visual grounding can propagate into wrong physical action (Wang et al., 2024). SlowBA targets efficiency rather than correctness in VLM-based GUI agents: a realistic pop-up trigger induces excessively long responses and higher latency through a two-stage reward-level backdoor injection strategy, while largely preserving task accuracy (Li et al., 9 Mar 2026).

5. Benchmark findings and recurring empirical patterns

The BackdoorVLM benchmark standardizes evaluation across 12 representative attacks, 2 open-source VLMs, and 3 multimodal datasets or data sources, and its most important empirical regularity is the dominance of language-side triggers. Text-triggered backdoors remain highly potent, with poisoning rates as low as 1% yielding over 90% success across most tasks, while in bimodal attacks the text trigger typically overwhelms the image trigger when forming the backdoor mapping (Li et al., 24 Nov 2025).

A second regularity is that attack difficulty depends strongly on the target behavior. In the benchmark, targeted refusal, malicious injection, jailbreak, and perceptual hijack are frequently near-saturated under textual or bimodal triggers, whereas concept substitution is consistently harder. This agrees with external studies in which semantic replacement is also the more brittle objective: concept-guided unseen-target attacks become markedly stronger on larger datasets such as COCO than on Flickr8K, and multi-target settings require explicit control of feature interference to avoid incorrect trigger-target mappings (Shen et al., 30 Nov 2025).

A third regularity is that visual triggers are not uniform in strength. In the benchmark, large-area image manipulations such as Blended and ImgTrojan are easier to learn than localized patch or frequency triggers, while semantic or relational triggers can outperform explicit artifacts by matching the multimodal semantics of the task. BadSem reaches over 98% average ASR with semantic mismatch triggers, and cross-modality transfer is especially strong for object semantics (Zhong et al., 8 Jun 2025). Cross-domain studies of LVLM instruction tuning add that trigger generalization depends on image-model irrelevance and competition with clean semantic regions, with improved triggers reaching about 97–100% ASR at 0.2% poisoning in a large mixed instruction corpus (Liang et al., 2024).

Paper	Setting	Reported result
BackdoorVLM (Li et al., 24 Nov 2025)	text-triggered backdoors	poisoning rates as low as 1% yielding over 90% success across most tasks
TrojVLM (Lyu et al., 2024)	target text insertion in captioning and VQA	ASRs around 0.975–0.997 on BLIP-2
BadVision (Liu et al., 25 Feb 2025)	compromised SSL encoder reused in LVLMs	over 99% attack success rate; 77.6% relative visual understanding error
BadSem (Zhong et al., 8 Jun 2025)	semantic mismatch as trigger	over 98% average ASR
GLA (Wang et al., 6 Apr 2026)	autonomous-driving bimodal trigger	10% poisoning, 90% ASR, 0% FPR
BadVLMDriver (Ni et al., 2024)	physical object trigger in driving	92% ASR for sudden acceleration with a red balloon

These recurring results collectively suggest that BackdoorVLM is less a single attack recipe than a family of failures arising at the junction of visual grounding, instruction following, and modular multimodal reuse. The benchmark’s importance lies in making those failures comparable across objectives and modalities rather than limiting evaluation to one patch trigger or one captioning model (Li et al., 24 Nov 2025).

6. Defenses, auditing, and unresolved problems

Defenses remain fragmented. PRISM represents a distinct response: instead of repairing a potentially backdoored model internally, it uses a separate, frozen VLM as an external semantic auditor and adaptive router at test time. Across 17 datasets and 11 attack types, PRISM is reported to suppress Attack Success Rate to <1% on CIFAR-10 while improving clean accuracy, establishing a model-agnostic externalized defense paradigm, although its primary setting is a suspicious classifier rather than a backdoored VLM itself (Xu et al., 27 Jan 2026).

Post-training mitigation inside multimodal pipelines can work, but often at substantial cost. In the inherited-encoder setting for speech-LLMs, partial fine-tuning is weak, whereas full fine-tuning on clean data can reduce the attack to near the baseline false-positive rate; the limitation is the need for guaranteed clean labeled data and additional training compute (Fortier et al., 1 Oct 2025). In autonomous driving, post-training incremental learning with 3000 benign samples can reduce ASR nearly to 0, but the paper characterizes this as almost the same effort as fine-tuning a benign VLM (Ni et al., 2024).

Several papers also report outright defense failure. BadSem evaluates two defenses based on system prompt and supervised fine-tuning and finds that both fail to mitigate the semantic backdoor (Zhong et al., 8 Jun 2025). GLA shows that clean-task monitoring can itself be misleading, because the poisoned autonomous-driving VLM can improve metrics such as BLEU-1 while maintaining 0% FPR at 10% poisoning, undermining performance-degradation-based detection (Wang et al., 6 Apr 2026). BadVision reports that state-of-the-art backdoor detection methods cannot detect the attack effectively, despite the fact that only the encoder artifact is compromised (Liu et al., 25 Feb 2025).

The unresolved problems are therefore structural rather than merely algorithmic. Existing evidence covers many attack surfaces—text, image, bimodal, concept-level, reasoning-level, and physical-world—but remains uneven across architectures and modalities. Some conclusions are direct for VLMs, such as the textual sensitivity documented by BackdoorVLM; others are architectural analogies, such as encoder propagation results in speech-LLMs that plausibly transfer to vision-language stacks. What is already clear is that backdoor evaluation in VLMs must account for trigger modality, target behavior, multimodal interaction, and inherited component risk simultaneously, because each of these axes changes both attack strength and defense feasibility (Li et al., 24 Nov 2025).