---
title: LIBERO-Safety Benchmark
url: https://www.emergentmind.com/topics/libero-safety
type: topic
---

# LIBERO-Safety Benchmark

LIBERO-Safety is a benchmark for evaluating **physical safety** and **semantic safety** in vision-language-action (VLA) models under procedurally generated, safety-critical manipulation scenarios. It was introduced to address a gap in prior VLA evaluation suites, which largely emphasized task success in relatively static, deterministic scenes while leaving operational safety under strict constraints insufficiently characterized. The benchmark combines a parametric task formalism, stochastic environment generation, a keypose-driven collision-free data pipeline, and a cross-paradigm evaluation of eight VLA models and two embodied foundation models. Its central empirical claim is a **generalization-safety tension**: high-diversity training improves safety-related behavior, but task completion remains limited by sub-optimal trajectory synthesis and semantic misalignment [2606.23686].

## 1. Motivation and benchmark position

LIBERO-Safety was designed around the premise that successful manipulation is not equivalent to safe manipulation. The benchmark targets hazards that earlier suites did not systematically test, including clutter, moving obstacles, proximal human-robot interaction, contact-force limits, and harmful or deceptive instructions. In the benchmark’s framing, real deployment requires a robot not only to satisfy a symbolic goal, but also to avoid unsafe contact, respect dynamic safety constraints, and refuse semantically hazardous requests [2606.23686].

The benchmark is positioned relative to earlier suites such as RLBench, CALVIN, LIBERO, LIBERO-Plus, LIBERO-Pro, RoboCasa, RoboTwin 2.0, SafeLIBERO, VLA-Arena, and LIBERO-X. Its distinguishing features are **parametric task definitions**, **static and dynamic scenes**, **physical safety and semantic safety**, **proximal human-robot interaction**, and **procedural data generation** rather than exclusive reliance on human teleoperation [2606.23686]. A common misconception is to treat LIBERO-Safety as merely a harder LIBERO split. The benchmark’s design indicates a broader objective: it couples trajectory-level physical safety with instruction-level semantic refusal, so safety is not reduced to collision avoidance alone.

## 2. Formal specification and safety semantics

At the core of LIBERO-Safety is the **Unified Behavior Domain Definition Language (UBDDL)**, introduced as an extension of BDDL. A safety task is defined as

$$
\mathcal{T} = \langle \mathcal{G}, \mathcal{C}_{\text{safety}}, \mathcal{S}_{\text{init}}, \mathcal{P}_{\text{env}} \rangle,
$$

where $\mathcal{G}$ denotes symbolic goal conditions, $\mathcal{C}_{\text{safety}}$ denotes safety constraints enforced at every timestep, $\mathcal{S}_{\text{init}}$ denotes distributions over initial scene configurations and dynamic-entity trajectories, and $\mathcal{P}_{\text{env}}$ denotes environment parameters such as camera extrinsics, visual variation, and sensor noise [2606.23686].

UBDDL procedurally generates **7,603 unique scenes**, containing **953 distinct objects** and **462 hand-object pairs**. Randomization is injected through visual perturbations, camera variation, and physical stochasticity. The specified visual perturbations include motion blur, Gaussian defocus, zoom blur, fog/scattering, glass distortion, randomized textures/materials, and lighting variation. Camera variation includes randomized camera extrinsics or viewpoints, orbital shifts, and in-place yaw/pitch perturbations. Physical stochasticity includes randomized scene layouts, randomized robot initial states, and dynamic objects with linear or circular kinematics. In the main experiments, camera intrinsics are treated as calibrated constants; randomization is applied to extrinsics and other environmental factors [2606.23686].

The benchmark distinguishes two safety modalities. **Physical safety** is enforced through kinematic spatial constraints and dynamic wrench constraints. Under the kinematic constraints, no collision is allowed between robot and obstacle objects, and any contact triggers failure. Under the wrench constraints, contact force on delicate objects must remain below a threshold $F_{\text{max}}$. **Semantic safety** is defined as the ability to refuse harmful or deceptive natural-language instructions and to detect unsafe intent before acting. The semantic track includes malicious prompts, common-sense violation prompts, and contextual traps [2606.23686].

For semantic evaluation, the appendix defines a balanced mixed safe/unsafe setting using

$$
\mathrm{F1}=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}},
$$

where TP denotes correctly refusing unsafe instructions, FP denotes refusing benign ones, and FN denotes complying with unsafe ones [2606.23686]. This formulation makes explicit that semantic safety is evaluated as a refusal-and-discrimination problem rather than as ordinary task execution.

## 3. Suite structure and difficulty progression

LIBERO-Safety contains **five task suites**, each organized into **three difficulty levels (L0–L2)**, for a total of **75 benchmark tasks** [2606.23686].

| Suite | Safety focus | Difficulty progression |
|---|---|---|
| Affordance-Aware Grasping (AAG) | Safe interaction regions and geometry-aware grasps | L0 canonical pose; L1 random planar translation; L2 out-of-distribution rotations |
| Human-Robot Interaction (HRI) | Safe motion around a human proxy | L0 nominal collaboration; L1 spatial perturbations; L2 paraphrased instructions and semantic variation |
| Tabletop Spatial Avoidance (TSA) | Collision-averse motion in cluttered tabletop spaces | L0 static familiar obstacles; L1 dynamic obstacles; L2 novel obstacle geometries |
| Free-Space Hand-Object Avoidance (FSHOA) | 3D avoidance around hand-object configurations | L0 static composites; L1 moving composites; L2 unseen poses and configurations |
| Semantic Safety Reasoning (SSR) | Unsafe-instruction identification | L0 explicit malicious instructions; L1 hidden physical common-sense violations; L2 subtle contextual traps |

The five suites partition safety-relevant failure modes rather than merely task families. AAG emphasizes safe affordance selection. HRI introduces proximal human-robot interaction. TSA stresses cluttered tabletop avoidance, including dynamic obstacles. FSHOA isolates avoidance around procedurally generated hand-object configurations, with hand poses produced using **MANO** and **GrabNet**. SSR shifts the problem from motion generation to instruction refusal, with prompts procedurally generated using **Qwen3-8B** [2606.23686].

This suite design implies that LIBERO-Safety evaluates both geometric competence and instruction-grounded judgment. That duality differentiates it from obstacle-only formulations. It also means that improvement on one track does not entail improvement on the other: a model may be physically stable yet semantically misaligned, or semantically cautious yet operationally ineffective.

## 4. Keypose-driven data generation and dataset composition

A major methodological component of LIBERO-Safety is its **keypose-driven data generation pipeline**, introduced to overcome the scalability bottlenecks of human teleoperation. The paper reports **7.4 min/task** for human teleoperation versus **1.8 min/task** for the keypose pipeline [2606.23686].

In this pipeline, a human specifies only a sparse set of object-centric keyposes,

$$
\mathcal{T}_{\text{keypose}} = \{\mathbf{T}_k\}_{k=1}^K,
$$

representing stages such as pre-grasp, grasp, pre-place, and place. These keyposes are defined in the canonical object frame and later transformed into the robot base frame. Each keypose is augmented with **$N=5$ spatial variants**, and one variant is sampled at each stage, yielding combinatorially diverse trajectories. **CuRobo** then synthesizes a collision-free SE(3) trajectory satisfying

$$
\mathcal{A}(\mathbf{x}(t)) \cap \mathcal{O} = \emptyset, \quad \forall t \in [0, T],
$$

where $\mathcal{A}(\mathbf{x}(t))$ is the robot/object volume at pose $\mathbf{x}(t)$ and $\mathcal{O}$ is the obstacle set [2606.23686].

The resulting training dataset contains **19,664 strictly collision-free demonstrations** across **40 distinct tasks**, synthesized from **L0 and L1** levels only. The pipeline generates **500 candidate trajectories per task** before screening. The entire **Semantic Safety Reasoning** suite is excluded from training, and all **L2 tasks** are held out for evaluation. The paper emphasizes that generated motions are then **human-screened** for task validity and strict safety compliance, so the pipeline preserves a human-in-the-loop verification stage while avoiding full end-to-end teleoperation [2606.23686].

Methodologically, the pipeline decouples high-level task intent from low-level motion execution. This suggests a specific conception of safe data curation: the demonstrations encode human intent sparsely, while collision freedom and SE(3) feasibility are delegated to the planner and subsequent screening. A plausible implication is that LIBERO-Safety treats dataset generation itself as a safety-engineering problem rather than as passive collection.

## 5. Evaluation protocol, models, and metrics

The benchmark evaluates **10 baseline models**. The physical-safety track uses eight VLA models: **OpenVLA**, **OpenVLA-OFT**, **$\pi_0$**, **$\pi_{0.5}$**, **UniVLA**, **VLA-JEPA**, **GR00T N1.5**, and **GR00T N1.6**. The semantic-safety track uses two embodied foundation models: **RoboBrain2.0** and **RynnBrain-CoP** [2606.23686].

For the four physical safety suites, models are fine-tuned with supervised learning or behavior cloning on the curated dataset. The protocol uses **identical pre-recorded initial states**, **10 independent trials per task**, and reports averages across **3 random seeds**. Any safety violation immediately terminates the episode. The reported metrics are **SR (Success Rate)**, **CR (Collision Rate)**, **Execution Time**, and **LDLJ**, where LDLJ is the log-dimensionless jerk smoothness metric:

$$
\text{LDLJ} = - \ln \left( \frac{T^3}{v_{\text{peak}}^2 \int_{0}^{T} \left\| \frac{d^3 \mathbf{x}(t)}{dt^3} \right\|^2 dt} \right).
$$

Higher LDLJ indicates smoother motion [2606.23686].

For the SSR semantic track, models receive a static scene observation plus a language instruction and must decide whether the action is safe. The protocol uses **10 inference trials per task** and evaluates the embodied foundation models **zero-shot**. The primary metric is **RR (Refusal Rate)**, defined as the fraction of hazardous instructions correctly refused [2606.23686].

The metric design is notable because physical safety and semantic safety are not collapsed into a single scalar. By separating SR, CR, execution time, LDLJ, RR, and appendix F1, the benchmark exposes tradeoffs between goal achievement, collision avoidance, motion quality, and refusal behavior.

## 6. Empirical findings, failure modes, and relation to adjacent safety work

The benchmark reports several high-level findings. First, **standard pretraining is not enough for reactive safety**: OpenVLA performs poorly across nearly all safety suites, especially on L2. Second, among standard VLAs, **$\pi_{0.5}$** is strongest overall, which the paper attributes to its combination of internet-scale pretraining and diverse sub-task planning, although performance still varies substantially across seeds and environments. Third, **high-diversity training improves safer generalization**: a key ablation shows that diverse data prevents memorization of unsafe trajectories and improves zero-shot spatial reasoning. Fourth, a scaling study on $\pi_{0.5}$ in FSHOA shows that increasing demonstrations per task from **50 to 500** improves SR, LDLJ, and execution time, while CR decreases. Fifth, semantic safety requires explicit reasoning: **RoboBrain2.0** is strong on easy explicit harms but weak on harder or deceptive prompts, whereas **RynnBrain-CoP** performs better on higher-tier semantic traps [2606.23686].

The paper identifies two dominant failure modes. One is **sub-optimal trajectory synthesis**: models may avoid collisions yet still fail because they generate overly conservative motion, oscillatory trajectories, deadlocks in tight configurations, or timeouts. The other is **semantic misalignment**: models may execute stable, collision-free motions toward the wrong object in cluttered scenes. These observations are central to the benchmark’s claim that physical safety alone is insufficient without semantic grounding [2606.23686].

LIBERO-Safety also sits within a broader family of safety-oriented LIBERO derivatives and analyses. **SafeLIBERO** focuses on obstacle-interference scenarios and evaluates **Collision Avoidance Rate (CAR)**, **Task Success Rate (TSR)**, and **Execution Time Steps (ETS)** for a plug-and-play CBF layer called AEGIS [2512.11891]. **SafeVLA-Bench** instead adds post-hoc STL-based auditing to existing LIBERO rollouts, exposing a “success-safety gap” through **Succ-But-Unsafe (SBU)** and **Violation Severity Index (VSI)** rather than procedurally generating new tasks [2606.00773]. **LIBERO-PRO** addresses robustness and fairness beyond memorization by perturbing manipulated objects, initial states, task instructions, and environments, with the claim that performance can collapse from “over 90%” on standard LIBERO to “0.0% under our generalized setting” [2510.03827]. These neighboring benchmarks are complementary rather than interchangeable: SafeLIBERO emphasizes runtime collision filtering, SafeVLA-Bench emphasizes post-hoc safety diagnostics, and LIBERO-PRO emphasizes generalization stress-testing, whereas LIBERO-Safety integrates procedural physical safety, semantic safety, and a scalable safe-demonstration pipeline.

The paper’s stated limitations are equally important. LIBERO-Safety remains **simulation-based**; contact dynamics and soft-body effects are approximated; there is a **sim-to-real gap**; real HRI is more unpredictable than simulated proxies; the keypose pipeline still uses human input; and the dataset emphasizes safe demonstrations rather than hard negative unsafe trajectories [2606.23686]. In the appendix, the paper presents a **chunk-level control barrier function filter** for real-world deployment as a demonstration of practical relevance, which suggests that the benchmark is intended not only as an evaluation suite but also as a substrate for runtime safety mechanisms [2606.23686].

Taken together, LIBERO-Safety defines safety for VLAs as a joint problem of geometry-aware control, long-horizon trajectory synthesis, semantic refusal, and robustness under stochastic scene generation. Its main contribution is to make those dimensions measurable within a single benchmark framework, while showing that strong task-generalization performance does not, by itself, certify safe embodied behavior [2606.23686].

Source: https://www.emergentmind.com/topics/libero-safety