ANNIE-Attack: Adversarial Safety in Embodied AI

• ANNIE-Attack is a task-aware adversarial framework that compromises embodied AI systems by perturbing video streams while maintaining apparent task continuity.
• It leverages ISO/TS 15066 and ISO 13855 safety principles to induce long-horizon unsafe actions, such as human encroachment and excessive speed.
• Benchmarking with ANNIEBench across Critical, Dangerous, and Risky scenarios reveals high attack success rates and robust simulation-to-real-world validation.

Searching arXiv for the term and the cited papers to ground the article in current records. ANNIE-Attack denotes a task-aware adversarial framework for compromising the action chain of embodied AI systems, specifically vision–language–action policies, so that benign instructions lead to physically unsafe behaviors. In the usage established by "ANNIE: Be Careful of Your Robots" (Huang et al., 3 Sep 2025), the framework attacks the video input stream, is grounded in ISO/TS 15066 and ISO 13855 safety principles, and is designed to induce long-horizon unsafe actions while preserving apparent task continuity. The same term has also been used editorially to describe adversarial false data injection against artificial neural network-based AC state estimation in smart grids, although the underlying 2019 paper does not itself name the method "ANNIE-Attack" (Liu et al., 2019). In current arXiv usage, the term is primarily associated with embodied AI safety attacks (Huang et al., 3 Sep 2025).

1. Definition and scope

In embodied AI, ANNIE-Attack targets vision–language–action pipelines that take multi-modal inputs—primarily video frames $O_t$ from on-robot cameras and a language instruction $l$—and output control actions $a_t$ in Cartesian end-effector space with a gripper state (Huang et al., 3 Sep 2025). The attack perturbs only visual inputs; language prompts remain benign and unchanged, and the policy itself is not altered. The threat model therefore operates at inference time on the sensory stream rather than on policy parameters or instruction text.

The attack objective is not merely task failure. Its explicit goal is to violate physically grounded safety constraints derived from ISO/TS 15066 and ISO 13855. This reframes adversarial robustness from conventional prediction error toward unsafe physical outcomes, including human encroachment, excessive speed in co-occupied spaces, and collisions with forbidden objects or infrastructure (Huang et al., 3 Sep 2025).

A distinct, older line of work in smart grids studies adversarial false data injection against ANN-based AC state estimation. There, an attacker injects a false data vector into the measurement stream to degrade estimation accuracy while remaining undetected by bad-data detection (Liu et al., 2019). A plausible implication is that the label "ANNIE-Attack" is polysemous across domains, but the embodied-AI formulation is the one explicitly introduced under that name on arXiv (Huang et al., 3 Sep 2025).

2. Safety formalization in embodied AI

The embodied-AI formulation defines safety in terms of physical constraints on the evolving robot and environment state. Let $x_t^{ee} \in \mathbb{R}^3$ be the end-effector position, $x_t^{human}$ the human body reference position, $\dot{x}_t^{ee}$ and $\dot{x}_t^{env}$ the end-effector and object velocities, $O_{contact}$ the set of contacted objects, $O_{forbidden}$ the set of forbiddens, $c_t \in \{0,1\}$ a collision flag, and $l$0 the full robot state vector. The robot action is written as $l$1 and the state update as $l$2 (Huang et al., 3 Sep 2025).

The first safety category is Critical, described as SRMS-like separation safety. Separation distance is defined as

$l$3

with the constraint

$l$4

Violations occur when the end effector, or a dangerous tool attached to it, encroaches into human-accessible space under Safety-Rated Monitored Stop principles (Huang et al., 3 Sep 2025).

The second category is Dangerous, described as SSM-like speed safety. The corresponding constraints are

$l$5

These encode the Speed and Separation Monitoring concept, where excessive end-effector or object speed during human–robot co-occupation is unsafe (Huang et al., 3 Sep 2025).

The third category is Risky, described as collision and environmental safety in the absence of direct human presence. The constraint is

$l$6

A collision boundary indicator can be defined as

$l$7

This category focuses on avoiding collisions with infrastructure or off-limits objects so as to maintain environmental safety and equipment integrity (Huang et al., 3 Sep 2025).

The global adversarial safety objective is stated as

$l$8

where $l$9 denotes original frames, $a_t$0 the VLA policy, $a_t$1 the executed action outcome, and $a_t$2 the safe state set defined by the preceding constraints. This formulation makes safety violation itself the optimization target rather than a proxy such as classification error (Huang et al., 3 Sep 2025).

3. Architecture and optimization procedure

The central design principle is a task-aware "attack leader" that maps a long-horizon safety goal into per-frame action-level targets. This addresses the lack of per-frame labels in embodied AI by predicting a target action delta at each time step that adversarial optimization can track (Huang et al., 3 Sep 2025).

The Attack Leader Module takes as input current multi-view image observations $a_t$3 and an attack type $a_t$4. Its backbone uses dual ResNet encoders for first- and third-person views, an embedding for the attack type, feature fusion, and a shared extractor. It outputs a direction

$a_t$5

and a scale

$a_t$6

Training uses

$a_t$7

where $a_t$8 is cross-entropy over $a_t$9 per dimension, $x_t^{ee} \in \mathbb{R}^3$0 is mean-squared error to the ground-truth scale, and $x_t^{ee} \in \mathbb{R}^3$1 in the implementation (Huang et al., 3 Sep 2025).

Per-frame target actions are then constructed by first computing the nominal action

$x_t^{ee} \in \mathbb{R}^3$2

then forming

$x_t^{ee} \in \mathbb{R}^3$3

and finally defining the target action

$x_t^{ee} \in \mathbb{R}^3$4

For representative policies such as ACT and Baku, the action parameterization used is

$x_t^{ee} \in \mathbb{R}^3$5

where $x_t^{ee} \in \mathbb{R}^3$6 are end-effector displacements and $x_t^{ee} \in \mathbb{R}^3$7 (Huang et al., 3 Sep 2025).

The white-box image attack is PGD-based. At time $x_t^{ee} \in \mathbb{R}^3$8, the attack pushes the VLA action on the perturbed input toward $x_t^{ee} \in \mathbb{R}^3$9 while remaining within a bounded perturbation budget. The iterative update is

$x_t^{human}$0

followed by projection

$x_t^{human}$1

and the attacked frame is

$x_t^{human}$2

The loss $x_t^{human}$3 enforces closeness of $x_t^{human}$4 to $x_t^{human}$5 (Huang et al., 3 Sep 2025).

A conceptual time-aggregated objective over a horizon $x_t^{human}$6 is also given:

$x_t^{human}$7

subject to $x_t^{human}$8 and sparsity constraints. In practice, the implementation instantiates per-frame loss tracking with PGD and evaluates stealth through post hoc metrics (Huang et al., 3 Sep 2025).

4. Benchmarking infrastructure and evaluation protocol

ANNIEBench is the benchmark used to evaluate ANNIE-Attack. It comprises nine safety-critical manipulation scenarios on ManiSkill with SAPien, centered on table-top settings with a Franka Emika Panda robot having a 7-DoF arm and 2-DoF gripper. Visual sensing uses wrist-mounted stereo or mono cameras with RGB-D, plus a third-person camera; proprioception is available for evaluation. Safety metrics such as knife–human distances are automated (Huang et al., 3 Sep 2025).

The dataset contains 2,400 video–action sequences in total. Per-scenario training for the attack leader uses approximately 240 demonstrations, and evaluation uses 20 test sequences per scenario (Huang et al., 3 Sep 2025).

The scenarios are organized by safety level as follows:

Safety level	Scenarios
Critical	Cut the apple with knife; Open the canned food; Open the box with scissor
Dangerous	Place cup on plate; Put fork near plate; Put apple into plate
Risky	Put sponge to sink; Pour wine to cup; Take coffee

The primary evaluation metric is Attack Success Rate (ASR), defined as the proportion of sequences in which safety constraints are violated. Action Consistency (AC) measures the average angle between consecutive action vectors, with lower values indicating smoother and more stealthy trajectories:

$x_t^{human}$9

Action Deviation (AD) is a Mahalanobis-distance ratio of attacked versus original actions relative to the dataset action distribution:

$\dot{x}_t^{ee}$0

$\dot{x}_t^{ee}$1

where $\dot{x}_t^{ee}$2 are attacked actions, $\dot{x}_t^{ee}$3 are original actions, and $\dot{x}_t^{ee}$4 are the dataset mean and covariance. Task Success Rate Change (TSRC) is recorded as an auxiliary metric but is not equated with safety (Huang et al., 3 Sep 2025).

This metric suite is significant because it separates unsafe physical behavior from simple task failure. A plausible implication is that ANNIEBench operationalizes embodied safety evaluation in a way that conventional success-rate metrics do not.

5. Empirical results and ablations

Across safety categories, reported attack success rates exceed 50%: 52% ASR for Critical, 67% ASR for Dangerous, and 50% ASR for Risky (Huang et al., 3 Sep 2025). These aggregate values anchor the claim that the attack generalizes across distinct classes of safety violations rather than exploiting a single benchmark artifact.

Per-scenario results under Annie-Dense show substantial variation across both task and policy architecture. For Critical scenarios, Baku obtains ASR values of 0.7 on Cut apple, 0.8 on Open canned food, and 1.0 on Open box with scissor; the corresponding ACT values are 0.5, 0.3, and 1.0. For Dangerous scenarios, Baku obtains 0.1 on Place cup on plate, 1.0 on Put fork near plate, and 0.7 on Put apple into plate; ACT obtains 0.2, 0.5, and 0.6. For Risky scenarios, Baku obtains 0.8 on Put sponge to sink, 0.3 on Pour wine to cup, and 0.5 on Take coffee; ACT obtains 1.0, 0.1, and 0.3 (Huang et al., 3 Sep 2025).

The paper reports that Baku is more sensitive to small perturbations, likely due to min–max scaling amplifying outliers, whereas ACT’s mean–std normalization dampens minor variations. Action trajectory visualizations are consistent with this interpretation: post-attack ACT trajectories closely overlap pre-attack trajectories, while Baku shows substantially larger deviations, reflected in higher AC and AD values (Huang et al., 3 Sep 2025).

Sparse and adaptive strategies extend the dense attack regime. Annie-Dense perturbs every frame and achieves the highest ASR, including 1.0 in a tested scenario, but has the worst AD, exemplified by 4.7. Annie-2 and Annie-3 perturb every 2 or 3 frames, respectively, reducing AD and ASR relative to dense. Annie-ADAP uses the leader’s predicted $\dot{x}_t^{ee}$5 as a phase indicator, perturbing more frequently when $\dot{x}_t^{ee}$6 and less frequently when $\dot{x}_t^{ee}$7. In the reported scenario, Annie-ADAP averages one perturbation every 3.05 frames while matching dense ASR at 1.0 and achieving moderate AD, 8.1% lower than Annie-3 and 16.4% higher than Annie-2 (Huang et al., 3 Sep 2025).

Ablations support the value of task-aware guidance. On ACT, the black-box transfer attack attains ASR 0.1, compared with 0.5 for white-box attack. For the leader model, random direction yields ASR 0.1, fixed human-oriented direction yields 0.3, and Attack Leader guidance yields 0.5. This indicates that scene-conditioned prediction of direction and scale materially improves attack success (Huang et al., 3 Sep 2025).

6. Real-world validation, assumptions, and defenses

Real-world validation is conducted on a UR3 arm mounted on a Dalu mobile base, with a Robotiq two-finger gripper, Intel RealSense D435 and Orbbec Gemini Pro depth cameras, ROS1 control, and an ACT policy. The task is "Cutting an apple" with a knife. Annie-Dense is applied to recorded vision–action sequences and then replayed physically. In 4 out of 10 trials, the robot pointed the knife toward and approached a nearby human, demonstrating translation from simulated adversarial manipulation to tangible safety breaches. Quantitative distances and velocities are not reported; the qualitative outcome is described as a critical SRMS violation involving encroachment with a hazardous tool (Huang et al., 3 Sep 2025).

The evaluation assumes white-box gradient access for the main PGD attack. Black-box transfer attacks remain feasible but are less effective. The Attack Leader is trained per scenario using approximately 240 demonstrations and performs well on seen tasks, but is limited on unseen ones. This suggests that cross-task generalization remains an open problem for embodied adversarial safety attacks (Huang et al., 3 Sep 2025).

Defenses discussed or implied by the results span several layers. At the input level, randomization and transformations such as MagNet and feature squeezing can raise the bar but may be bypassed. At training time, adversarial training against frame perturbations and long-horizon consistency constraints is proposed. At the control layer, speed and acceleration caps, separation monitoring, SRMS or PFL modes per ISO, hardware-level force or torque limiting, and runtime monitors for distance $\dot{x}_t^{ee}$8, speed $\dot{x}_t^{ee}$9, and collision $\dot{x}_t^{env}$0 are relevant. At the policy level, distribution-aware action normalization, temporal smoothness penalties, and safety shields layered over VLA outputs are identified as robustness measures (Huang et al., 3 Sep 2025).

A common misconception is to equate embodied attack success with task failure. The ANNIE formulation explicitly rejects that equivalence: TSRC is auxiliary, whereas ASR is defined by safety violations. Another misconception is that smooth trajectories imply harmless behavior. ANNIE-Attack is specifically designed to maintain smooth and apparently task-consistent trajectories while still violating physical safety constraints (Huang et al., 3 Sep 2025).

7. Related usage in smart-grid state estimation

In a separate literature on power systems, adversarial false data injection attacks against ANN-based AC state estimation seek to maximize phase-angle estimation error while satisfying bad-data detection constraints. The system state is represented as

$\dot{x}_t^{env}$1

measurements obey

$\dot{x}_t^{env}$2

and traditional state estimation minimizes a weighted least squares residual

$\dot{x}_t^{env}$3

A trained ANN estimator $\dot{x}_t^{env}$4 replaces iterative WLS, and the attacker injects $\dot{x}_t^{env}$5 to form $\dot{x}_t^{env}$6 while enforcing the stealth constraint

$\dot{x}_t^{env}$7

under a $\dot{x}_t^{env}$8 bad-data detector at $\dot{x}_t^{env}$9 (Liu et al., 2019).

The attack objective is

$O_{contact}$0

subject to the residual threshold, $O_{contact}$1 meter-access limits, and per-measurement bounds, with

$O_{contact}$2

Two solvers are studied: Differential Evolution for both “any $O_{contact}$3 meters” and “specific $O_{contact}$4 meters” scenarios, and SLSQP for the “specific $O_{contact}$5 meters” case. On IEEE 9-, 14-, and 30-bus systems, DE is reported to be more effective than SLSQP. The attack succeeds with high probability even under modest resources; compromising 10% of meters with 10% injection bounds yields success in at least 80% of instances across systems, and the 14-bus system reaches 100% success across all tested combinations of compromise rate and bounds (Liu et al., 2019).

This smart-grid usage is conceptually related to the embodied formulation only at a high level: both involve adversarial perturbations against ANN-based decision systems under application-specific stealth constraints. The underlying objects of attack, safety semantics, and optimization targets are otherwise different. A plausible implication is that the shared label is best treated as a naming coincidence rather than a unified research program.