SafeLIBERO: Safety Benchmark for VLA Models

• SafeLIBERO is a safety-critical robotic manipulation benchmark designed to evaluate vision-language-action models under strict obstacle-avoidance and semantic task requirements using formal control barrier functions.
• It features 16 diverse tasks across two interference levels, ensuring that both semantic correctness and collision-free trajectories are rigorously tested in zero-shot settings.
• The benchmark has driven significant improvements in collision avoidance and task success rates, establishing a reference framework for evaluating safety-aware robotic policies.

SafeLIBERO is a safety-critical robotic manipulation benchmark specifically designed for rigorous evaluation of vision-language-action (VLA) models under hard obstacle-avoidance constraints and semantically challenging tasks. Building on the original LIBERO test suites, SafeLIBERO introduces static obstacle interference scenarios to assess not only instruction-following capability but also explicit safety compliance, using formal control barrier function (CBF) certificates. SafeLIBERO has rapidly become the standard reference environment for comparative research on safety-aware vision-driven robot policies, underpinning recent advances in plug-and-play safety layers, predictive safety guidance, and attention-based real-time collision filters (Hu et al., 9 Dec 2025, English et al., 1 Jul 2026, Park et al., 8 Jun 2026).

1. Motivation and Design Principles

The conventional LIBERO benchmark evaluates multi-modal spatial and semantic reasoning in VLA models but does not systematically stress-test safety performance. In practice, state-of-the-art VLA agents often exhibit hazardous behaviors—such as collisions with scene obstacles—that would be unacceptable in real-world deployments. SafeLIBERO directly addresses this gap by requiring that each policy trajectory be both semantically correct and verifiably collision-free. Safety is not handled as a soft penalty but as a non-negotiable, mathematically defined constraint throughout all episodes (Hu et al., 9 Dec 2025).

Key design goals include:

• Collision-avoidance as a primary metric. Task completion is insufficient unless all obstacles are untouched.
• Diverse manipulation scenarios. The benchmark covers a wide spectrum of spatial and long-horizon tasks with differing geometric and semantic complexities.
• Reproducibility and zero-shot evaluation. Complete held-out test sets with no distinct train/test split and deterministic scene generation protocols.

2. Benchmark Structure and Scenario Generation

SafeLIBERO retains the four core suites from LIBERO—Spatial, Object, Goal, and Long—selecting four archetypal manipulation objectives from each, resulting in 16 unique tasks. Each task is instantiated in two distinct interference levels:

• Level I: Obstacles placed immediately adjacent to targets, requiring precise clearance maneuvers.
• Level II: Obstacles block the direct end-effector path, necessitating more substantial detours.

For each task × level, 50 randomized episodes are generated, yielding 1,600 total episodes across 32 test scenarios. Six real-world everyday objects (moka pot, storage box, milk carton, wine bottle, mug, book) are used as obstacles, providing diverse geometric and physical properties. Obstacle positions are systematically jittered around pre-specified blockage loci, while all movable objects receive minor perturbations to encourage policy robustness (Hu et al., 9 Dec 2025).

All scenes are rendered in RoboSuite, using a Franka Emika Panda arm in top-down translation mode, with task goals specified as natural-language instructions and evaluated purely by zero-shot performance.

3. Safety Definitions and Control Barrier Functions

At the core of SafeLIBERO's safety assessment is a mathematically rigorous CBF formalism. Safety is defined via a superlevel set:

$$\mathcal{C} = \left\{\,\mathbf{x}\mid h(\mathbf{x})\ge 0\right\}$$

where $h(\mathbf{x})$ encodes signed distance between robot end-effector and obstacles, both modeled as minimum-volume enclosing ellipsoids. For a point $\mathbf{p}_s$ on the end-effector ellipsoid,

$$h(\mathbf{x}) = \frac{ -\|\bar Q_{\mathrm{ob}}\bar Q_{\mathrm{ef}}^{-1}\mathbf{p}_s\| + (\mathbf{p}_{\mathrm{ob}}-\mathbf{p}_{\mathrm{ef}})^\top\bar Q_{\mathrm{ef}}^{-1}\mathbf{p}_s -1 } {\|\bar Q_{\mathrm{ef}}^{-1}\mathbf{p}_s\|}$$

where $$\bar Q_\bullet = R_\bullet Q_\bullet R_\bullet^\top$$ for obstacle/end-effector parameters $(\mathbf{p}, Q, R)$. $h(\mathbf{x})\geq0$ if and only if the ellipsoids do not intersect, certifying collision-freedom via Nagumo’s theorem (Hu et al., 9 Dec 2025).

A plug-and-play CBF-based safety constraint layer (as in AEGIS) computes the closest control action $\mathbf{u}_{\mathrm{safe}}$ to the original VLA command $\mathbf{u}_{\mathrm{vla}}$ subject to:

$$\dot h(\mathbf{x})\geq -\alpha(h(\mathbf{x})), \quad \text{where } \alpha(h)=10h$$

This quadratic program-based CBF layer is model-agnostic and compatible with any policy that outputs control commands or trajectories (Hu et al., 9 Dec 2025).

4. Evaluation Metrics and Protocol

SafeLIBERO explicitly measures three metrics:

• Collision Avoidance Rate (CAR): Fraction of episodes with zero obstacle displacement.
• Task Success Rate (TSR): Fraction of episodes where the manipulation goal is successfully completed (collision alone does not force immediate fail; subsequent semantic failures caused by collision do).
• Execution Time-steps (ETS): Mean number of control steps per episode (standard 20 Hz, max 550-steps/27 s).

All policies are evaluated in a zero-shot regime over all 1,600 scenarios. An episode ends upon goal attainment or timeout, with compliance assessed via scene inspection for obstacle/goal displacement (Hu et al., 9 Dec 2025, English et al., 1 Jul 2026).

5. Representative Safety Methods and Empirical Results

SafeLIBERO serves as the primary benchmark for rigorous comparison of safety mechanisms in VLA models. Three prominent methods and results summary:

Policy	CAR (%)	TSR (%)	ETS	Reference
OpenVLA-OFT	15.13	22.81	323.16	(Hu et al., 9 Dec 2025)
$h(\mathbf{x})$0 (no safety)	18.69	50.88	278.24	(Hu et al., 9 Dec 2025)
AEGIS (single-step CBF)	77.85	68.13	262.30	(Hu et al., 9 Dec 2025)
Predictive Safety Guidance	82.8	81.6	–	(English et al., 1 Jul 2026)

AEGIS achieves a +59.16 percentage point improvement in CAR and +17.25 in TSR over the $h(\mathbf{x})$1 baseline (Hu et al., 9 Dec 2025). Predictive neuro-symbolic safety guidance, which injects CBF corrections during flow-matching policy trajectory generation, further improves CAR and shows a +13.5 percentage point gain in TSR over AEGIS, with the largest margin (+33 points) on long-horizon manipulation tasks (English et al., 1 Jul 2026). This indicates that in-generation, anticipatory constraint satisfaction enables not only better safety but also more successful, coherent robotic task execution.

6. Algorithmic and Practical Details

SafeLIBERO is designed for immediate integration with any VLA model. The codebase provides routines for environment simulation, controlled obstacle placement, and verification scripts for reproducible zero-shot experiments (Hu et al., 9 Dec 2025). The standard evaluation algorithm is as follows (condensed):

1. Initialize a test episode with randomized obstacle and object placement within prescribed ranges.
2. Supply the robot with a natural-language instruction and initialize robot state.
3. For each control step:
   • Execute the VLA policy, optionally mediated by a CBF-based or predictive safety guidance layer.
   • Verify CBF constraint compliance and update the control command, if necessary.
   • Advance the simulation and inspect the environment for safety violations or goal attainment.
4. Record CAR, TSR, and ETS.

No explicit training is performed on SafeLIBERO; all results reflect the policy’s zero-shot safety and semantic generalization capabilities. Public repositories include episode generation scripts, evaluation tools, and detailed documentation on extending the test suite (Hu et al., 9 Dec 2025).

7. Impact, Limitations, and Future Directions

SafeLIBERO provides a rigorous, large-scale, and realistic testbed for the development and comparative assessment of safety-oriented vision-language-action models. Its contributions include:

• Formalization of real-world safety as a hard, certifiable property.
• Evidence that control-theoretic safety layers can dramatically improve both collision avoidance and downstream task success without retraining.
• A standard protocol for evaluating future advances in anticipatory or real-time safety intervention (e.g., attention-based filters, predictive trajectory guidance) (English et al., 1 Jul 2026, Park et al., 8 Jun 2026).

Current limitations include reliance on static obstacles and privileged simulator state for ground-truth pose estimation. Extensions towards dynamic obstacle injection, learning CBFs directly from raw vision, and development of full-body and higher-order (velocity/acceleration) safety constraints are open research directions (English et al., 1 Jul 2026). A plausible implication is that SafeLIBERO will increasingly serve as a foundation for cross-community research at the intersection of robotics safety, neuro-symbolic reasoning, and vision-language generalization.