ForesightSafety-VLA: Diagnostic Safety Benchmark

• ForesightSafety-VLA is a diagnostic benchmark that evaluates embodied safety in VLA systems by measuring cumulative risk and hard safety violations during task execution.
• It defines a comprehensive 13-category taxonomy covering physical, instruction-side, and perception-side hazards using simulator-accessible metrics for clear safety measurement.
• The benchmark employs controlled variations in scene structure, language, and visual input to diagnose safety degradation and reveal unsafe nominal success under distribution shifts.

ForesightSafety-VLA is a unified diagnostic safety benchmark for Vision-Language-Action models that makes safety the primary evaluation target for embodied policies rather than an auxiliary statistic attached to task success. It defines a 13-category safety taxonomy spanning physical interaction safety, instruction-side safety, and perception-side safety; evaluates policies under three controlled dimensions of variation—scene structure, language command, and visual observation; and measures not only whether a task is completed, but also how much risk is accumulated during execution through cumulative safety cost and risk exposure time (Lyu et al., 25 Jun 2026). In this formulation, embodied safety is treated as a property of the entire perception–grounding–control loop, not merely as a late-stage filter on actions.

1. Concept and motivating problem

The benchmark is motivated by a mismatch between what VLA evaluations usually report and what deployment requires. A model can succeed nominally while behaving unsafely throughout a trajectory: it may slam an articulated object, skim a hazardous boundary, move too close to a heat source, or repeatedly enter dangerous regions but still finish the instructed task. ForesightSafety-VLA is designed to expose exactly this gap by shifting evaluation from binary completion to trajectory-level risk, hard violation events, and source-specific failure diagnosis (Lyu et al., 25 Jun 2026).

The paper frames prior evaluation practice as insufficient for five reasons. Safety is often absent or under-specified; success rate conflates safe and unsafe behavior; binary violation checks miss pre-violation exposure; failure sources are not disentangled under distribution shift; and safe-RL or constrained-control benchmarks do not cover the full VLA loop, where failures can originate in perception and language grounding rather than only in control (Lyu et al., 25 Jun 2026). This motivates a benchmark that grounds safety in measurable physical signals, injects risk while preserving task semantics, disentangles what changed under distribution shift, and reports both outcome and cost (Lyu et al., 25 Jun 2026).

A central implication is that ForesightSafety-VLA is diagnostic rather than merely competitive. It is intended to answer not only whether a policy can do a task, but whether it can do so safely, whether unsafe nominal success is present, and whether the dominant failure source lies in geometry, perception, instruction grounding, or temporal sequencing (Lyu et al., 25 Jun 2026).

2. Safety taxonomy and hazard model

The benchmark defines 13 safety subcategories organized into three macro domains: Safe-Core, Safe-Adv(L), and Safe-Adv(V). These correspond to physical interaction safety, instruction-side safety, and perception-side safety respectively (Lyu et al., 25 Jun 2026).

Within Safe-Core, the five categories are force/torque safety, thermal/energy safety, spatial boundary safety, collaborative/dual-arm safety, and temporal precondition safety. Force/torque safety targets excessive force or impulsive contact and is represented by thresholds such as $F_n > \tau_F^{\text{hard}}$ and $F_n > \tau_F^{\text{soft}}$; representative tasks include press_stapler and beat_block. Thermal/energy safety targets unsafe proximity to heat sources or energized devices and uses distance-based criteria such as $d(\text{EE}, h) < d_{\text{hard}}$ and $d < d_{\text{soft}}$; representative tasks include open_microwave and lift_pot. Spatial boundary safety covers forbidden regions, edge proximity, and insufficient clearance, with hard region-entry events and soft clearance-margin cost; representative tasks include place_on_stand and A2B_left. Collaborative safety tracks unsafe dual-arm separation with thresholds such as $d_{\text{arm}} < \tau_a^{\text{hard}}$ and $d_{\text{arm}} < \tau_a^{\text{soft}}$; representative tasks include handover_block and hang_mug. Temporal precondition safety is monitored by finite-state-machine logic for errors such as open-before-insert or stabilize-before-pour; representative tasks include dump_bin and sort_stack (Lyu et al., 25 Jun 2026).

Within Safe-Adv(L), the four language-side categories are ambiguity or underspecification, contradictory constraints, explicit unsafe request, and goal hijacking or prompt injection. These perturb whether the policy can recover missing safety constraints, detect inconsistency, refuse unsafe commands, or resist instruction-side takeover (Lyu et al., 25 Jun 2026). The benchmark explicitly expects refusal or safe alternative behavior under explicit unsafe requests.

Within Safe-Adv(V), the four perception-side categories are adversarial patch or marker, occlusion or reduced visibility, lighting or material perturbation, and viewpoint or camera pose shift. These categories test whether visual perturbations alter downstream behavior in a safety-relevant way rather than merely degrading recognition accuracy. Representative examples include rotate_qrcode for adversarial markers and cabinet insertion or transfer tasks for occlusion and viewpoint stress (Lyu et al., 25 Jun 2026).

What distinguishes this taxonomy from generic robustness taxonomies is that every category is tied to simulator-accessible quantities such as force, distance, velocity, region membership, energy state, or FSM state. That grounding makes the benchmark measurable rather than impressionistic (Lyu et al., 25 Jun 2026).

3. Controlled variation and diagnostic design

ForesightSafety-VLA evaluates policies under three controlled dimensions of variation: scene structure along the $L$ axis, language wording along the $W$ axis, and visual observation along the $V$ axis (Lyu et al., 25 Jun 2026). The benchmark’s diagnostic power comes from varying these axes independently rather than combining them into a single undifferentiated notion of robustness.

The structure axis modifies the environment while holding instruction and camera setup fixed. The reported levels are $L0$ for the canonical layout, $F_n > \tau_F^{\text{soft}}$0 for moderate shift such as clutter plus 2–4 objects and gap reduction to 80%, and $F_n > \tau_F^{\text{soft}}$1 for strong shift such as clutter plus 5–8 objects, gap reduction to 60%, and new obstacle combinations (Lyu et al., 25 Jun 2026). This axis primarily stresses geometric reasoning, clearance management, and motion adaptation under changed scene structure.

The language axis modifies the instruction while holding the scene fixed. The reported progression is $F_n > \tau_F^{\text{soft}}$2 standard instruction, $F_n > \tau_F^{\text{soft}}$3 synonym or paraphrase substitution, $F_n > \tau_F^{\text{soft}}$4 compositional rewrite and constraint reordering, $F_n > \tau_F^{\text{soft}}$5 ambiguity, contradiction, or unsafe request, and $F_n > \tau_F^{\text{soft}}$6 prompt injection or goal hijacking (Lyu et al., 25 Jun 2026). The first three levels stress semantic robustness under ordinary rewording; the latter two instantiate adversarial instruction-side safety conditions.

The visual axis perturbs the rendered observation while preserving the physical scene. The reported levels are $F_n > \tau_F^{\text{soft}}$7 clean render, $F_n > \tau_F^{\text{soft}}$8 brightness and contrast changes, $F_n > \tau_F^{\text{soft}}$9 texture or material randomization, $d(\text{EE}, h) < d_{\text{hard}}$0 camera perturbation of approximately $d(\text{EE}, h) < d_{\text{hard}}$1 with sensor noise $d(\text{EE}, h) < d_{\text{hard}}$2 and partial occlusion, and $d(\text{EE}, h) < d_{\text{hard}}$3 adversarial patch or marker (Lyu et al., 25 Jun 2026). This axis isolates safety failures that arise from perception drift rather than world-state change.

A plausible implication is that the benchmark operationalizes diagnosis as attribution over tuples of the form $d(\text{EE}, h) < d_{\text{hard}}$4, even when the paper emphasizes one axis at a time. This makes safety degradation analyzable as an axis-specific delta rather than a single leaderboard number (Lyu et al., 25 Jun 2026).

4. Metrics: safe outcome decomposition, cumulative cost, and exposure time

The benchmark’s metric design is centered on separating outcome quality from process-level risk. For each episode it records task success $d(\text{EE}, h) < d_{\text{hard}}$5 and whether at least one hard safety violation occurs, denoted $d(\text{EE}, h) < d_{\text{hard}}$6. From these it defines a four-quadrant decomposition: $d(\text{EE}, h) < d_{\text{hard}}$7

$d(\text{EE}, h) < d_{\text{hard}}$8

$d(\text{EE}, h) < d_{\text{hard}}$9

$d < d_{\text{soft}}$0

with

$d < d_{\text{soft}}$1

Here $d < d_{\text{soft}}$2 is safe success rate, $d < d_{\text{soft}}$3 is unsafe success rate, $d < d_{\text{soft}}$4 is safe failure rate, and $d < d_{\text{soft}}$5 is unsafe failure rate (Lyu et al., 25 Jun 2026). This decomposition makes unsafe nominal success directly visible.

The process-level safety formulation uses a dual-threshold margin. For each safety channel $d < d_{\text{soft}}$6,

$d < d_{\text{soft}}$7

and the hard violation predicate is

$d < d_{\text{soft}}$8

A soft threshold $d < d_{\text{soft}}$9 defines a risk buffer. The per-step soft cost is

$d_{\text{arm}} < \tau_a^{\text{hard}}$0

where $d_{\text{arm}} < \tau_a^{\text{hard}}$1 is monotone increasing; the paper cites linear and quadratic instances such as $d_{\text{arm}} < \tau_a^{\text{hard}}$2 and $d_{\text{arm}} < \tau_a^{\text{hard}}$3 (Lyu et al., 25 Jun 2026). For clearance, the benchmark gives the concrete example

$d_{\text{arm}} < \tau_a^{\text{hard}}$4

Cumulative safety cost aggregates these per-step channel costs: $d_{\text{arm}} < \tau_a^{\text{hard}}$5 with channel-wise decomposition

$d_{\text{arm}} < \tau_a^{\text{hard}}$6

Risk exposure time measures how long the policy remains in the soft-risk buffer: $d_{\text{arm}} < \tau_a^{\text{hard}}$7 The paper’s interpretation is that $d_{\text{arm}} < \tau_a^{\text{hard}}$8 measures how much risk accumulates, whereas $d_{\text{arm}} < \tau_a^{\text{hard}}$9 measures how long risk persists (Lyu et al., 25 Jun 2026). This is the benchmark’s core “foresight” property: two trajectories can both avoid hard violations, yet differ sharply in soft-risk exposure.

For compact model comparison, the benchmark also defines a safety-adjusted success statistic. First normalize cost per step: $d_{\text{arm}} < \tau_a^{\text{soft}}$0 then define

$d_{\text{arm}} < \tau_a^{\text{soft}}$1

The paper recommends treating this as a secondary ranking metric and prefers Pareto-style reporting of safe success against expected cumulative cost (Lyu et al., 25 Jun 2026).

5. Benchmark instantiation in RoboTwin and reported findings

ForesightSafety-VLA is implemented in RoboTwin, a SAPIEN-based dual-arm manipulation simulator with shared task APIs and multi-embodiment support (Lyu et al., 25 Jun 2026). The paper reports 66 safety-augmented base scenarios in RoboTwin across 5 embodiments; elsewhere it also describes 67 implemented safety-augmented tasks and 66 instruction templates, so the scenario accounting is reported in two closely related forms within the paper text (Lyu et al., 25 Jun 2026).

Scenario construction proceeds through hazard injection, constraint tightening, and state-dependent preconditions. Hazard injection adds hazardous objects or hazardous regions. Constraint tightening adds explicit safety limits such as force ceilings, clearance margins, geofences, and minimum dual-arm distance. State-dependent preconditions encode temporal logic such as open-before-insert or stabilize-before-pour (Lyu et al., 25 Jun 2026). Representative tasks include open_microwave, lift_pot, place_on_scale, handover_block, hanging_mug, dump_bin, and open--place--close (Lyu et al., 25 Jun 2026).

The reported protocol evaluates models under combinations of task and $d_{\text{arm}} < \tau_a^{\text{soft}}$2 condition, with 50 episodes over 3 random seeds and 95% bootstrap confidence intervals (Lyu et al., 25 Jun 2026). The baseline set spans behavior cloning, diffusion, token-based, and foundation-VLA families, including ACT, DP, DP3, RDT, Pi0, Pi0.5, OpenVLA-oft, DexVLA, LLaVA-VLA, and TinyVLA in the English draft summary (Lyu et al., 25 Jun 2026).

The main empirical findings are qualitative but sharp. Across the evaluated baselines, even the strongest policy incurs non-trivial safety cost and unsafe nominal success, while structure and visual variation induce substantially stronger safety degradation than ordinary language variation (Lyu et al., 25 Jun 2026). The paper also reports that some models preserve nominal success under distribution shift by becoming more unsafe, which is precisely the failure mode hidden by plain success rate.

The partial aggregate table in the paper includes concrete values for several baselines. For OpenVLA-oft, it reports $d_{\text{arm}} < \tau_a^{\text{soft}}$3, $d_{\text{arm}} < \tau_a^{\text{soft}}$4, $d_{\text{arm}} < \tau_a^{\text{soft}}$5, $d_{\text{arm}} < \tau_a^{\text{soft}}$6, $d_{\text{arm}} < \tau_a^{\text{soft}}$7, and $d_{\text{arm}} < \tau_a^{\text{soft}}$8. For RDT, it reports $d_{\text{arm}} < \tau_a^{\text{soft}}$9, $L$0, $L$1, $L$2, $L$3, and $L$4. For DP, it reports $L$5, $L$6, $L$7, $L$8, $L$9, and $W$0. For ACT, it reports $W$1, $W$2, $W$3, $W$4, $W$5, and $W$6 (Lyu et al., 25 Jun 2026). These numbers support the benchmark’s claim that safer completion and total completion are materially different quantities.

The paper also emphasizes “compensatory errors”: a policy may appear safer because it fails early and never reaches a dangerous phase, or appear more capable because it ignores safety constraints and still reaches the goal (Lyu et al., 25 Jun 2026). This is why the benchmark treats $W$7 as a joint diagnostic object rather than independent statistics.

6. Position in the VLA safety literature

ForesightSafety-VLA is a benchmark rather than a mitigation method, but it sits within a rapidly expanding safety stack for VLA systems. It is complementary to physical red teaming, where RedVLA systematically uncovers unsafe embodied behaviors through risk scenario synthesis and risk amplification (Zhang et al., 24 Apr 2026). It is also complementary to runtime verification layers such as Pre-VLA, which performs preemptive action validity assessment before execution or world-model imagination (Sun et al., 21 May 2026), and to multitask failure detectors such as SAFE, which monitor VLA internal features to predict likely task failure and support stop, backtrack, or ask-for-help intervention (Gu et al., 11 Jun 2025).

At the control level, it is orthogonal to action shields and constraint layers such as VLSA/AEGIS, which add a plug-and-play safety constraint layer formulated via control barrier functions (Hu et al., 9 Dec 2025), and to predictive safety guidance inside flow-matching generation, where intermediate trajectory samples are corrected before collisions become unavoidable (English et al., 1 Jul 2026). It is likewise complementary to multimodal safe-execution systems such as Safe-Night VLA, which combines thermal perception with a CBF safety filter for thermal-aware manipulation (Yu et al., 5 Mar 2026). ForesightSafety-VLA does not solve these problems; it provides the measurement substrate on which such methods can be compared.

This positioning suggests that embodied safety is not reducible to any single mechanism. The benchmark’s own conclusion is that embodied safety is tightly coupled to perception, grounding, and control competence rather than being reducible to post-hoc safety filtering alone (Lyu et al., 25 Jun 2026). A plausible implication is that future progress will require both stronger policy-side safeguards and stronger evaluation-side decomposition: red teaming to surface failures, runtime verification and shields to intercept unsafe actions, and diagnostic benchmarks to reveal where unsafe nominal competence persists.

In that sense, ForesightSafety-VLA marks a shift from asking whether VLA systems can act, to asking under what perturbations, hazard classes, and trajectory conditions they remain safely deployable.