LIBERO-Safety Benchmark

• LIBERO-Safety is a benchmark that defines both physical safety (collision-free motion and dynamic obstacle avoidance) and semantic safety (refusal of harmful instructions) in VLA models.
• It employs parametric task definitions, a keypose-driven data generation pipeline, and stochastic scene generation to create diverse and safety-critical manipulation scenarios.
• Empirical findings reveal a generalization-safety tension where high-diversity training improves safety behaviors but limits optimal task completion due to sub-optimal trajectory synthesis.

LIBERO-Safety is a benchmark for evaluating physical safety and semantic safety in vision-language-action (VLA) models under procedurally generated, safety-critical manipulation scenarios. It was introduced to address a gap in prior VLA evaluation suites, which largely emphasized task success in relatively static, deterministic scenes while leaving operational safety under strict constraints insufficiently characterized. The benchmark combines a parametric task formalism, stochastic environment generation, a keypose-driven collision-free data pipeline, and a cross-paradigm evaluation of eight VLA models and two embodied foundation models. Its central empirical claim is a generalization-safety tension: high-diversity training improves safety-related behavior, but task completion remains limited by sub-optimal trajectory synthesis and semantic misalignment (Cui et al., 22 Jun 2026).

1. Motivation and benchmark position

LIBERO-Safety was designed around the premise that successful manipulation is not equivalent to safe manipulation. The benchmark targets hazards that earlier suites did not systematically test, including clutter, moving obstacles, proximal human-robot interaction, contact-force limits, and harmful or deceptive instructions. In the benchmark’s framing, real deployment requires a robot not only to satisfy a symbolic goal, but also to avoid unsafe contact, respect dynamic safety constraints, and refuse semantically hazardous requests (Cui et al., 22 Jun 2026).

The benchmark is positioned relative to earlier suites such as RLBench, CALVIN, LIBERO, LIBERO-Plus, LIBERO-Pro, RoboCasa, RoboTwin 2.0, SafeLIBERO, VLA-Arena, and LIBERO-X. Its distinguishing features are parametric task definitions, static and dynamic scenes, physical safety and semantic safety, proximal human-robot interaction, and procedural data generation rather than exclusive reliance on human teleoperation (Cui et al., 22 Jun 2026). A common misconception is to treat LIBERO-Safety as merely a harder LIBERO split. The benchmark’s design indicates a broader objective: it couples trajectory-level physical safety with instruction-level semantic refusal, so safety is not reduced to collision avoidance alone.

2. Formal specification and safety semantics

At the core of LIBERO-Safety is the Unified Behavior Domain Definition Language (UBDDL), introduced as an extension of BDDL. A safety task is defined as

$$\mathcal{T} = \langle \mathcal{G}, \mathcal{C}_{\text{safety}}, \mathcal{S}_{\text{init}}, \mathcal{P}_{\text{env}} \rangle,$$

where $\mathcal{G}$ denotes symbolic goal conditions, $\mathcal{C}_{\text{safety}}$ denotes safety constraints enforced at every timestep, $\mathcal{S}_{\text{init}}$ denotes distributions over initial scene configurations and dynamic-entity trajectories, and $\mathcal{P}_{\text{env}}$ denotes environment parameters such as camera extrinsics, visual variation, and sensor noise (Cui et al., 22 Jun 2026).

UBDDL procedurally generates 7,603 unique scenes, containing 953 distinct objects and 462 hand-object pairs. Randomization is injected through visual perturbations, camera variation, and physical stochasticity. The specified visual perturbations include motion blur, Gaussian defocus, zoom blur, fog/scattering, glass distortion, randomized textures/materials, and lighting variation. Camera variation includes randomized camera extrinsics or viewpoints, orbital shifts, and in-place yaw/pitch perturbations. Physical stochasticity includes randomized scene layouts, randomized robot initial states, and dynamic objects with linear or circular kinematics. In the main experiments, camera intrinsics are treated as calibrated constants; randomization is applied to extrinsics and other environmental factors (Cui et al., 22 Jun 2026).

The benchmark distinguishes two safety modalities. Physical safety is enforced through kinematic spatial constraints and dynamic wrench constraints. Under the kinematic constraints, no collision is allowed between robot and obstacle objects, and any contact triggers failure. Under the wrench constraints, contact force on delicate objects must remain below a threshold $F_{\text{max}}$. Semantic safety is defined as the ability to refuse harmful or deceptive natural-language instructions and to detect unsafe intent before acting. The semantic track includes malicious prompts, common-sense violation prompts, and contextual traps (Cui et al., 22 Jun 2026).

For semantic evaluation, the appendix defines a balanced mixed safe/unsafe setting using

$$\mathrm{F1}=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}},$$

where TP denotes correctly refusing unsafe instructions, FP denotes refusing benign ones, and FN denotes complying with unsafe ones (Cui et al., 22 Jun 2026). This formulation makes explicit that semantic safety is evaluated as a refusal-and-discrimination problem rather than as ordinary task execution.

3. Suite structure and difficulty progression

LIBERO-Safety contains five task suites, each organized into three difficulty levels (L0–L2), for a total of 75 benchmark tasks (Cui et al., 22 Jun 2026).

Suite	Safety focus	Difficulty progression
Affordance-Aware Grasping (AAG)	Safe interaction regions and geometry-aware grasps	L0 canonical pose; L1 random planar translation; L2 out-of-distribution rotations
Human-Robot Interaction (HRI)	Safe motion around a human proxy	L0 nominal collaboration; L1 spatial perturbations; L2 paraphrased instructions and semantic variation
Tabletop Spatial Avoidance (TSA)	Collision-averse motion in cluttered tabletop spaces	L0 static familiar obstacles; L1 dynamic obstacles; L2 novel obstacle geometries
Free-Space Hand-Object Avoidance (FSHOA)	3D avoidance around hand-object configurations	L0 static composites; L1 moving composites; L2 unseen poses and configurations
Semantic Safety Reasoning (SSR)	Unsafe-instruction identification	L0 explicit malicious instructions; L1 hidden physical common-sense violations; L2 subtle contextual traps

The five suites partition safety-relevant failure modes rather than merely task families. AAG emphasizes safe affordance selection. HRI introduces proximal human-robot interaction. TSA stresses cluttered tabletop avoidance, including dynamic obstacles. FSHOA isolates avoidance around procedurally generated hand-object configurations, with hand poses produced using MANO and GrabNet. SSR shifts the problem from motion generation to instruction refusal, with prompts procedurally generated using Qwen3-8B (Cui et al., 22 Jun 2026).

This suite design implies that LIBERO-Safety evaluates both geometric competence and instruction-grounded judgment. That duality differentiates it from obstacle-only formulations. It also means that improvement on one track does not entail improvement on the other: a model may be physically stable yet semantically misaligned, or semantically cautious yet operationally ineffective.

4. Keypose-driven data generation and dataset composition

A major methodological component of LIBERO-Safety is its keypose-driven data generation pipeline, introduced to overcome the scalability bottlenecks of human teleoperation. The paper reports 7.4 min/task for human teleoperation versus 1.8 min/task for the keypose pipeline (Cui et al., 22 Jun 2026).

In this pipeline, a human specifies only a sparse set of object-centric keyposes,

$$\mathcal{T}_{\text{keypose}} = \{\mathbf{T}_k\}_{k=1}^K,$$

representing stages such as pre-grasp, grasp, pre-place, and place. These keyposes are defined in the canonical object frame and later transformed into the robot base frame. Each keypose is augmented with $N=5$ spatial variants, and one variant is sampled at each stage, yielding combinatorially diverse trajectories. CuRobo then synthesizes a collision-free SE(3) trajectory satisfying

$$\mathcal{A}(\mathbf{x}(t)) \cap \mathcal{O} = \emptyset, \quad \forall t \in [0, T],$$

where $\mathcal{G}$0 is the robot/object volume at pose $\mathcal{G}$1 and $\mathcal{G}$2 is the obstacle set (Cui et al., 22 Jun 2026).

The resulting training dataset contains 19,664 strictly collision-free demonstrations across 40 distinct tasks, synthesized from L0 and L1 levels only. The pipeline generates 500 candidate trajectories per task before screening. The entire Semantic Safety Reasoning suite is excluded from training, and all L2 tasks are held out for evaluation. The paper emphasizes that generated motions are then human-screened for task validity and strict safety compliance, so the pipeline preserves a human-in-the-loop verification stage while avoiding full end-to-end teleoperation (Cui et al., 22 Jun 2026).

Methodologically, the pipeline decouples high-level task intent from low-level motion execution. This suggests a specific conception of safe data curation: the demonstrations encode human intent sparsely, while collision freedom and SE(3) feasibility are delegated to the planner and subsequent screening. A plausible implication is that LIBERO-Safety treats dataset generation itself as a safety-engineering problem rather than as passive collection.

5. Evaluation protocol, models, and metrics

The benchmark evaluates 10 baseline models. The physical-safety track uses eight VLA models: OpenVLA, OpenVLA-OFT, $\mathcal{G}$3, $\mathcal{G}$4, UniVLA, VLA-JEPA, GR00T N1.5, and GR00T N1.6. The semantic-safety track uses two embodied foundation models: RoboBrain2.0 and RynnBrain-CoP (Cui et al., 22 Jun 2026).

For the four physical safety suites, models are fine-tuned with supervised learning or behavior cloning on the curated dataset. The protocol uses identical pre-recorded initial states, 10 independent trials per task, and reports averages across 3 random seeds. Any safety violation immediately terminates the episode. The reported metrics are SR (Success Rate), CR (Collision Rate), Execution Time, and LDLJ, where LDLJ is the log-dimensionless jerk smoothness metric:

$\mathcal{G}$5

Higher LDLJ indicates smoother motion (Cui et al., 22 Jun 2026).

For the SSR semantic track, models receive a static scene observation plus a language instruction and must decide whether the action is safe. The protocol uses 10 inference trials per task and evaluates the embodied foundation models zero-shot. The primary metric is RR (Refusal Rate), defined as the fraction of hazardous instructions correctly refused (Cui et al., 22 Jun 2026).

The metric design is notable because physical safety and semantic safety are not collapsed into a single scalar. By separating SR, CR, execution time, LDLJ, RR, and appendix F1, the benchmark exposes tradeoffs between goal achievement, collision avoidance, motion quality, and refusal behavior.

6. Empirical findings, failure modes, and relation to adjacent safety work

The benchmark reports several high-level findings. First, standard pretraining is not enough for reactive safety: OpenVLA performs poorly across nearly all safety suites, especially on L2. Second, among standard VLAs, $\mathcal{G}$6 is strongest overall, which the paper attributes to its combination of internet-scale pretraining and diverse sub-task planning, although performance still varies substantially across seeds and environments. Third, high-diversity training improves safer generalization: a key ablation shows that diverse data prevents memorization of unsafe trajectories and improves zero-shot spatial reasoning. Fourth, a scaling study on $\mathcal{G}$7 in FSHOA shows that increasing demonstrations per task from 50 to 500 improves SR, LDLJ, and execution time, while CR decreases. Fifth, semantic safety requires explicit reasoning: RoboBrain2.0 is strong on easy explicit harms but weak on harder or deceptive prompts, whereas RynnBrain-CoP performs better on higher-tier semantic traps (Cui et al., 22 Jun 2026).

The paper identifies two dominant failure modes. One is sub-optimal trajectory synthesis: models may avoid collisions yet still fail because they generate overly conservative motion, oscillatory trajectories, deadlocks in tight configurations, or timeouts. The other is semantic misalignment: models may execute stable, collision-free motions toward the wrong object in cluttered scenes. These observations are central to the benchmark’s claim that physical safety alone is insufficient without semantic grounding (Cui et al., 22 Jun 2026).

LIBERO-Safety also sits within a broader family of safety-oriented LIBERO derivatives and analyses. SafeLIBERO focuses on obstacle-interference scenarios and evaluates Collision Avoidance Rate (CAR), Task Success Rate (TSR), and Execution Time Steps (ETS) for a plug-and-play CBF layer called AEGIS (Hu et al., 9 Dec 2025). SafeVLA-Bench instead adds post-hoc STL-based auditing to existing LIBERO rollouts, exposing a “success-safety gap” through Succ-But-Unsafe (SBU) and Violation Severity Index (VSI) rather than procedurally generating new tasks (Fan et al., 30 May 2026). LIBERO-PRO addresses robustness and fairness beyond memorization by perturbing manipulated objects, initial states, task instructions, and environments, with the claim that performance can collapse from “over 90%” on standard LIBERO to “0.0% under our generalized setting” (Zhou et al., 4 Oct 2025). These neighboring benchmarks are complementary rather than interchangeable: SafeLIBERO emphasizes runtime collision filtering, SafeVLA-Bench emphasizes post-hoc safety diagnostics, and LIBERO-PRO emphasizes generalization stress-testing, whereas LIBERO-Safety integrates procedural physical safety, semantic safety, and a scalable safe-demonstration pipeline.

The paper’s stated limitations are equally important. LIBERO-Safety remains simulation-based; contact dynamics and soft-body effects are approximated; there is a sim-to-real gap; real HRI is more unpredictable than simulated proxies; the keypose pipeline still uses human input; and the dataset emphasizes safe demonstrations rather than hard negative unsafe trajectories (Cui et al., 22 Jun 2026). In the appendix, the paper presents a chunk-level control barrier function filter for real-world deployment as a demonstration of practical relevance, which suggests that the benchmark is intended not only as an evaluation suite but also as a substrate for runtime safety mechanisms (Cui et al., 22 Jun 2026).

Taken together, LIBERO-Safety defines safety for VLAs as a joint problem of geometry-aware control, long-horizon trajectory synthesis, semantic refusal, and robustness under stochastic scene generation. Its main contribution is to make those dimensions measurable within a single benchmark framework, while showing that strong task-generalization performance does not, by itself, certify safe embodied behavior (Cui et al., 22 Jun 2026).