Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning

Abstract: Multimodal LLMs (MLLMs) have advanced embodied agents by enabling direct perception, reasoning, and planning task-oriented actions from visual inputs. However, such vision driven embodied agents open a new attack surface: visual backdoor attacks, where the agent behaves normally until a visual trigger appears in the scene, then persistently executes an attacker-specified multi-step policy. We introduce BEAT, the first framework to inject such visual backdoors into MLLM-based embodied agents using objects in the environments as triggers. Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably. BEAT addresses this challenge by (1) constructing a training set that spans diverse scenes, tasks, and trigger placements to expose agents to trigger variability, and (2) introducing a two-stage training scheme that first applies supervised fine-tuning (SFT) and then our novel Contrastive Trigger Learning (CTL). CTL formulates trigger discrimination as preference learning between trigger-present and trigger-free inputs, explicitly sharpening the decision boundaries to ensure precise backdoor activation. Across various embodied agent benchmarks and MLLMs, BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance, and generalizes reliably to out-of-distribution trigger placements. Notably, compared to naive SFT, CTL boosts backdoor activation accuracy up to 39% under limited backdoor data. These findings expose a critical yet unexplored security risk in MLLM-based embodied agents, underscoring the need for robust defenses before real-world deployment.

• The paper demonstrates the design of BEAT, a framework that implants object-based visual backdoors enabling dynamic multi-step malicious actions in MLLMs.
• It employs a two-stage training scheme with supervised fine-tuning and contrastive trigger learning, achieving attack success rates up to 80% while maintaining benign performance.
• Experimental results reveal robust backdoor activation with near-zero false triggering and strong generalization to out-of-distribution scenarios.

Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning

Introduction and Motivation

This work addresses the security vulnerabilities of multimodal LLM (MLLM)-driven embodied agents, specifically focusing on visual backdoor attacks. Unlike prior backdoor attacks that utilize static textual or visual triggers for single-step output manipulation, this paper introduces BEAT—a framework for implanting object-based visual backdoors that dynamically activate multi-step malicious policies in embodied agents. The attack leverages environmental object triggers, which are inherently variable in appearance due to changes in viewpoint and lighting, posing significant challenges for reliable backdoor activation.

Figure 1: Comparison of backdoor attack modalities, highlighting the dynamic, multi-step nature of object-triggered attacks in MLLM-driven embodied agents.

Threat Model and Attack Formulation

The adversary is assumed to have the capability to fine-tune MLLM weights, either via direct access to open-source models or through fine-tuning APIs for proprietary models. The objective is to produce a policy $\tilde\pi_\theta$ that behaves benignly under normal conditions but switches to an attacker-specified multi-step policy upon detection of a visual trigger (e.g., a knife or vase) in the agent’s egocentric view. The policy transition is deterministic at the first timestep $\hat{t}$ where the trigger is detected, after which the agent executes a malicious plan regardless of the original user instruction.

BEAT Framework: Data Construction and Training

Dataset Design

BEAT constructs a training corpus comprising three types of trajectories:

• Benign trajectories: Standard task executions in diverse scenes, ensuring retention of normal agent competence.
• Backdoor trajectories: Multi-step malicious plans activated only upon trigger detection, demonstrating attacker-desired policy shifts.
• Contrastive trajectory pairs: Identical scenes with and without triggers, providing fine-grained supervision for trigger-conditioned behavior switching.

This design exposes the model to the full variability of trigger appearances and placements, which is essential for robust backdoor learning.

Two-Stage Training Scheme

The training pipeline consists of:

1. Supervised Fine-Tuning (SFT): The MLLM is fine-tuned on a mixed dataset of benign and backdoor trajectories, learning both normal and malicious policies.
2. Contrastive Trigger Learning (CTL): A preference-based optimization stage using contrastive pairs, where the model is explicitly trained to prefer benign actions in trigger-free contexts and malicious actions when the trigger is present. The CTL loss sharpens the decision boundary around trigger presence, minimizing false positives and ensuring precise activation.

   Figure 2: BEAT’s two-stage fine-tuning scheme, illustrating the use of contrastive trigger learning to enforce sharp policy boundaries.

Experimental Evaluation

Benchmarks and Metrics

BEAT is evaluated on VAB-OmniGibson and EB-ALFRED, two household embodied agent benchmarks, using both open-source (Qwen2-VL-7B-Instruct, InternVL3-8B) and proprietary (GPT-4o) MLLMs. The evaluation metrics include:

• Success Rate (SR): Benign task completion in trigger-free scenarios.
• Attack Success Rate (ASR): Malicious goal achievement in trigger-present scenarios.
• Backdoor Triggering F1 Score ($\text{F1}_{\text{BT}}$): Precision and recall for correct backdoor activation at the trigger step.

Results

BEAT achieves attack success rates up to 80% and maintains benign task performance comparable to or exceeding models fine-tuned only on benign data. CTL improves backdoor activation F1 scores by up to 39% under limited backdoor data, demonstrating strong data efficiency and robustness. Notably, BEAT generalizes to out-of-distribution trigger placements, reliably activating malicious policies in unconventional scenes.

Figure 3: Successful backdoor trajectories, showing policy switch upon trigger detection and execution of multi-step malicious actions.

Figure 4: (Left) CTL improves both benign and attack success rates across varying backdoor data ratios. (Right) CTL sharply reduces false triggering rates on benign tasks.

Figure 5: Robust backdoor activation in out-of-distribution settings, with triggers placed in unconventional environments.

Analysis

• Data Ratio Sensitivity: CTL consistently improves both SR and ASR across all backdoor data ratios, with pronounced gains in low-resource regimes ($k=0.1$), where ASR increases more than fivefold.
• Stealthiness: BEAT maintains near-zero false triggering rates (FTR) in all settings, whereas omitting CTL leads to FTRs as high as 80%.
• Generalization: BEAT reliably activates backdoors in OOD scenarios (e.g., bathrooms, gardens), with a 92.3% activation rate.
• Error Modes: Failures are primarily due to trigger detection challenges (small/occluded objects) and action sequence variability, suggesting further improvements via dataset enrichment.

Implementation Considerations

• Open-source Models: LoRA-based fine-tuning is used, with SFT and CTL stages separated. CTL employs a preference-based loss with a reference policy, mixing contrastive and neutral examples to balance specialization and competence.
• Proprietary Models: GPT-4o is limited to SFT due to API constraints, resulting in lower attack success rates and inability to leverage CTL.
• Resource Requirements: CTL requires additional paired data and careful sampling strategies, but demonstrates strong data efficiency.
• Deployment Risks: The attack is stealthy, with benign performance preserved, making detection challenging in real-world pipelines.

Implications and Future Directions

This work exposes a critical security vulnerability in MLLM-driven embodied agents, demonstrating that object-based visual backdoors can be reliably implanted and activated in complex, multi-step decision-making pipelines. The findings highlight the need for robust defense mechanisms, including improved trigger detection, anomaly monitoring, and model provenance verification. The contrastive trigger learning paradigm may inform future research on both attack and defense strategies, particularly in settings with high input variability and multi-modal reasoning.

Further investigation is warranted into:

• Extending CTL to proprietary models with image-based DPO support.
• Learning robust triggers under fully unconstrained, box-agnostic conditions.
• Automated detection and mitigation of backdoor policies in deployed agents.

Conclusion

BEAT provides a comprehensive framework for visual backdoor attacks in MLLM-based embodied agents, leveraging contrastive trigger learning to achieve precise, stealthy, and robust policy manipulation. The demonstrated attack success rates, low false activations, and generalization to OOD scenarios underscore the urgency of addressing security risks in vision-driven autonomous systems. This work establishes a foundation for future research on both adversarial robustness and trustworthy deployment of embodied AI.