Phantasia: Context-Adaptive Backdoors in Vision Language Models

Abstract: Recent advances in Vision-LLMs (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings.

• The paper introduces Phantasia, a dynamic backdoor mechanism that adapts context to bypass traditional fixed-response defenses, achieving an ASR of 55.18% on VQA tasks.
• It leverages Gaussian noise as naturally occurring triggers to embed adversarial signals stealthily, blending with environmental variability.
• Empirical results demonstrate Phantasia’s ability to maintain task utility while exposing critical limitations of existing VLM security benchmarks.

Phantasia: Context-Adaptive Backdoors in Vision LLMs

Introduction

The paper "Phantasia: Context-Adaptive Backdoors in Vision LLMs" (2604.08395) presents a critical advancement in the study of adversarial robustness for VLMs by re-evaluating the underlying assumptions of current attack and defense methodologies. The work centers on the introduction of a context-adaptive backdoor attack—Phantasia—and demonstrates the inadequacy of prevailing VLM backdoor benchmarks. This paper systematically analyzes the threat landscape, challenges the robustness of mainstream defenses, and motivates the development of more contextually-aware threat models.

Reassessment of Attack and Defense Paradigms

The authors demonstrate that existing adversarial attacks on VLMs, typically evaluated under fixed response paradigms, vastly overstate the efficacy of backdoor insertion. Empirical evidence shows that output-based defenses—such as ONION-R—can suppress the attack success rate (ASR) of these approaches to less than $3\%$ under basic threat models. These findings reveal a substantive disparity between the theoretical alarm surrounding VLM backdoors and their empirical robustness to straightforward post-inference statistical detection.

In examining output-based defenses, the study makes clear that such approaches exploit low-entropy, context-invariant outputs from poisoned models. This exposes the fragility of existing backdoor methodologies in real-world deployments, highlighting the need for threat models that more faithfully capture the complexity and variability of benign interaction scenarios found in VLM applications.

Context-Adaptive Backdoor Mechanism

The principal innovation in this paper is the design and implementation of Phantasia, a context-adaptive backdoor attack that introduces a high-variance, stealthy adversarial mechanism. Unlike prior approaches constrained to static, attacker-prescribed outputs, Phantasia dynamically adjusts its outputs in response to input modifications, resulting in response entropy indistinguishable from benign inference. This property renders statistical invariance-based defenses (such as STRIP-P) ineffective; the context adaptation enables adversarial payloads to elude detection heuristics predicated on response stationarity.

Triggers in Phantasia are instantiated using Gaussian noise—ubiquitous in real-world imaging as a byproduct of environmental exposure or sensor limitations. This choice is motivated by the requirement that triggers in attacker-as-owner scenarios be both naturally occurring and statistically plausible, precluding the need for explicit artifact injection by malicious users. The model further demonstrates agnosticism to trigger generation mechanisms, maintaining high ASR under alternative stochastic triggers.

Empirical Results and Comparative Analysis

Quantitative evaluation underscores Phantasia's capacity to maintain stealth while achieving high adversarial efficacy. On standard VLM benchmarks, the model achieves an ASR of 55.18% on vision question answering (VQA) tasks under the most stringent defense pipelines. In comparison, state-of-the-art fixed-output backdoor attacks—including Shadowcast, CBA, and MABA—are neutralized to background rates by output-based analysis and STRIP-P defenses. Notably, Phantasia preserves task utility (BLEU@2: 26.60 on clean image captioning), aligning its clean sample performance with non-poisoned baselines.

These results expose a critical limitation of prior benchmarking strategies and substantiate the necessity for evaluation protocols that account for context-aware, high-entropy adversarial methods.

Attacker Model and Threat Realism

The paper adopts a malicious-provider threat model, analogous to assumptions in VLOOD (ICLR 2025) and BadVLMDriver (ICML 2025), wherein attackers control the training pipeline and model distribution but not user input. By leveraging naturally occurring noise as a trigger, Phantasia achieves scalable and unobtrusive poisoning with minimal risk of detection. The question selection process for adversarial targeting is formalized based on metrics such as existence, generality, and task consistency—optimized via GPT-4 scoring—further enhancing attack plausibility.

This model realistically simulates scenarios in which compromised models infiltrate public repositories, highlighting the external validity of the findings.

Implications and Future Directions

The exposé of over-optimism in the evaluation of VLM backdoor attacks necessitates a reexamination of threat assessment, especially as deployment contexts become increasingly heterogeneous and noise-adaptive. Context-adaptive attack methodologies like Phantasia challenge the research community to develop more expressive and robust defense mechanisms not solely reliant on low-level statistical tests. Forensic and certification pipelines will need to incorporate context awareness, semantic analysis, and potentially provenance-based detection schemes.

Open research questions include formalizing the theoretical limits of context-dependent adversarial attacks, designing new class-agnostic defense mechanisms, and developing automated risk assessment tools that scale with the evolving adversarial landscape.

Conclusion

Phantasia introduces a shift in both the design and evaluation of VLM backdoor attacks by foregrounding context adaptation and environmental triggers. This approach demonstrates robustness against state-of-the-art defense methodologies and exposes critical oversights in current VLM security benchmarks. The work will likely serve as a catalyst for the development of advanced adversarial detection strategies and more rigorous robustness metrics, driving theoretical and applied advances in trustworthy multimodal modeling.