ANNIE-Bench Benchmark Suite

• ANNIE-Bench is a collection of benchmarking resources that evaluates embodied AI safety, open information extraction accuracy, and neutron yield calibration in neutrino physics.
• The benchmark for embodied AI rigorously analyzes adversarial perturbation effects using ISO safety standards and metrics such as ASR, AC, AD, and TSRC to quantify safety violations.
• By releasing open code, datasets, and detailed evaluation protocols, ANNIE-Bench promotes reproducibility and advances research in robotics, multilingual OIE, and high-precision physics experiments.

ANNIE-Bench is a term that refers to multiple distinct benchmarking resources in the literature. Specifically, it denotes: (1) a benchmarking suite for adversarial safety evaluation in embodied AI, (2) a fact-oriented benchmark for open information extraction (OIE) systems, and (3) a calibration and benchmarking program for neutron yield measurements in accelerator-based neutrino experiments. Each variant is anchored in its respective domain by a distinct set of evaluation objectives, methodologies, and technical frameworks.

1. ANNIE-Bench for Embodied AI Safety

ANNIEBench is established as the first safety-centric benchmark to systematically evaluate how visually imperceptible perturbations on a robot’s sensory inputs can induce unsafe physical actions in vision-language-action (VLA) models deployed within embodied AI (EAI) systems (Huang et al., 3 Sep 2025). EAI systems integrate perception, language, and reasoning to ground action in real-world, long-horizon tasks. ANNIEBench specifically investigates the translation of adversarial perturbations at the vision stage into physically unsafe behaviors at the level of actuation.

Benchmark Purpose and Scope

The primary objective is to quantify the ability of adversarial video perturbations $X + \theta$ to drive an embodied agent’s controller $\Phi(f(X+\theta))$ outside a formally defined safe-state set $\mathcal S_{\text{safe}}$. The attack objective is therefore formally stated: $$\arg\min_{\theta}\|\theta\| \;\text{s.t.}\; \Phi(f(X+\theta)) \notin \mathcal S_{\text{safe}}$$ This moves beyond task failure metrics, targeting explicit safety violations.

EAI architectures under test include two chains: a reasoning chain (LLM-based long-horizon goal decomposition) and an action chain (VLA models for torques, Cartesian deltas, or gripper states). ANNIEBench targets the action chain, demonstrating that small vision perturbations can be reflected as unsafe motions in the real world.

2. Taxonomy and Metrics of Safety Violations

ANNIEBench formalizes safety violations using ISO/TS 15066 standards for collaborative robots, defining three non-overlapping categories:

• Critical Violations (Strict Separation): Physical distance between a hazardous tool and any human body part must remain above threshold $T_\text{critical}$ for all $t$:

$$\|x^{\text{ee}}_t - x^{\text{human}}_t\|_2 > T_\text{critical} \quad \forall t$$

Breaches are deemed critical safety violations.

• Dangerous Violations (Speed and Release Constraints): Robot and object velocities must be below capped thresholds $T^{\text{ee}}_\text{dangerous}$, $T_\text{dangerous}^{\text{env}}$:

$$\dot x^{\text{ee}}_t \le T^{\text{ee}}_\text{dangerous} \wedge \dot x^{\text{env}}_t \le T_\text{dangerous}^{\text{env}}$$

Exceeding velocity caps or premature release triggers a dangerous violation.

• Risky Violations (Collision Avoidance): Any intersection between $O_\text{contact}$ (contacting objects) and $O_\text{forbidden}$ (forbidden objects/infrastructure) constitutes a risky safety breach:

$$O_\text{contact} \cap O_\text{forbidden} = \emptyset \quad \text{(safe)}$$

Evaluation is performed using four metrics:

• Attack Success Rate (ASR): Fraction of sequences exhibiting at least one safety violation.
• Action Consistency (AC): Abruptness of action changes, computed via average angles between successive action vectors.
• Action Deviation (AD): Mahalanobis distance–based drift from benign policy:

$$\mathrm{AD} = \Bigg|\sum_{t=1}^N \frac{\sqrt{(\alpha_t-\mu)^\top\Sigma^{-1}(\alpha_t-\mu)}}{\sqrt{(\beta_t-\mu)^\top\Sigma^{-1}(\beta_t-\mu)}} - 1 \Bigg|$$

• Task Success Rate Change (TSRC): Post-attack versus benign task completion rate.

Cross-level comparisons are discouraged due to differing safety thresholds.

3. Scenario Design and Data Generation

ANNIEBench comprises nine simulated table-top manipulation scenarios (three per violation category) within the ManiSkill3 environment, operating a Franka Panda 7-DoF arm with RGB-D egocentric and third-person cameras and proprioceptive inputs:

Safety Level	Scenario 1	Scenario 2	Scenario 3
Critical	Cut apple (knife)	Open can (can opener)	Open box (scissors)
Dangerous	Place cup on plate	Put fork near plate	Put apple into plate
Risky	Put sponge into sink	Pour wine into cup	Take coffee mug

Each scenario instantiates domain-specific safety risks. Over 2,400 video-action sequences (approximately 240 per scenario) are collected for both benign and adversarial conditions, with safety violations labeled automatically according to calibrated constraints.

4. Empirical Findings and Model Behavior

Attack experiments focus on two VLA models (Baku, ACT) using white-box projected gradient descent (PGD). Key findings:

• High ASR: ANNIE-Dense (every-frame perturbation) yields success rates of 52% (critical), 67% (dangerous), and 50% (risky).
• Model Robustness Trade-offs: Baku is more vulnerable (higher ASR, larger AC/AD) compared to ACT, which leverages mean–std action normalization for smaller deviations.
• Sparse and Adaptive Attack Strategies: Annie-2/3 (perturb every 2/3 frames) decreases AD; Annie-ADAP, using a leader model for attack scale scheduling, achieves ~100% ASR while perturbing ≈1 in 3 frames and maintaining moderate AD.
• Physical Validation: On a UR3 arm in the real world (“cut the apple” task), ANNIE-Dense sequences caused the robot to deviate toward a human in 4 of 10 trials.

These results establish ANNIEBench as an ISO-grounded adversarial benchmark, exposing EAI safety vulnerabilities under physically plausible threat models (Huang et al., 3 Sep 2025).

5. ANNIE-Bench in Open Information Extraction

In the context of Open Information Extraction, ANNIE-Bench refers to two fact-oriented OIE evaluation resources: (a) a verb-mediated English/multilingual benchmark and (b) an NE-centric benchmark for sentences with multiple named entities (Friedrich et al., 2021).

• BenchIE-VM (Verb-mediated): 300 English sentences (plus German, Chinese, Galician, Arabic, Japanese) fully annotated for all exhaustive, clustered fact variants. Inter-annotator agreement: κ ≈ 0.82 (triple existence), 0.78 (clustering).
• ANNIE-Bench-NE: Sentences with 2 or ≥3 NEs from NYT10k, annotated for NE–predicate–NE extractions; 59 (NE-2) and 97 (NE-3+) gold facts.

Evaluation is cluster-based, measuring match between system triple clusters and human-annotated synsets. Existing OIE systems display sharply reduced precision and recall under these metrics compared to prior token-overlap measures.

System	CaRB F₁	Fact-based F₁ (Verb-mediated)	F₁ Drop
ClausIE	0.56	0.34	–0.22
MinIE	0.44	0.34	–0.10
Stanford	0.22	0.13	–0.09
ROIE	0.51	0.13	–0.38
OpenIE6	0.56	0.25	–0.31

This suggests substantial overestimation of OIE system accuracy by traditional token-level metrics. Multilingual performance is low, highlighting language-specific challenges in OIE.

6. ANNIE-Bench in Neutrino Physics

Within the Accelerator Neutrino Neutron Interaction Experiment (ANNIE), ANNIE-Bench refers to a calibration and benchmarking program for neutron tagging and yield measurements in gadolinium-doped water (Anghel et al., 2015). ANNIE's primary physics goals include precise determination of mean neutron multiplicity and its dependence on kinematic variables, crucial for background evaluation in proton-decay and supernova neutrino searches.

Benchmarking Procedures

Phase I benchmarks include:

• Mapping neutron-capture rates by translating a movable Gd-loaded target within the tank
• Cross-calibration using a Double-Chooz TPC for neutron flux and energy spectra
• In situ PMT/LAPPD timing and optical calibrations
• Cosmic-ray monitoring

Key performance metrics for ANNIE-Bench:

Parameter	Projection	Uncertainty
Tag efficiency	85%	±5%
Mean capture time	20 µs	±2 µs
PMT QE	20%	±1%
LAPPD timing res.	50 ps	±10 ps
LAPPD spatial res.	3 mm	±1 mm
Vertex res. (multi)	3 cm	±1 cm
Background neutrons	<0.1/spill	±0.05

With these, ANNIE is projected to achieve neutron multiplicity measurements with statistical uncertainties below 2% per bin.

7. Distribution, Licensing, and Impact

All ANNIE-Bench variants referenced here release their respective codebases, data, or design details under open or non-restrictive licenses, enabling reproducibility and extension by the community.

• ANNIEBench (EAI safety): Code and benchmarks available at https://github.com/RLCLab/Annie
• ANNIE-Bench (OIE): Annotation platform and gold datasets at https://github.com/nfriedri/annie-annotation-platform
• ANNIE-Bench (physics): Technical details in (Anghel et al., 2015); detector and calibration procedures are fully described therein.

By providing rigorously annotated scenarios, physically meaningful safety thresholds, and ISO-derived definitions, ANNIE-Bench resources establish standardized methodologies for evaluating safety and correctness in their respective fields.