- The paper introduces SafeVLA-Bench, which decouples task success from safety by formalizing safety requirements using Signal Temporal Logic (STL).
- It demonstrates that high task success rates often conceal substantial safety violations in manipulation benchmarks, with unsafe successes reaching up to 56%.
- The framework offers actionable diagnostics, such as Safety, SBU, and VSI metrics, to guide safer deployment of vision-language-action policies.
Vision-language-action (VLA) models are increasingly evaluated on simulated manipulation benchmarks where task completion is defined by binary predicates—did the robot achieve the requested goal before the timeout? Such evaluation protocols, used in LIBERO [libero2023] and RoboCasa-365 [robocasa2024], mask safety-relevant behaviors that arise in trajectory execution: policies may reach the goal while exerting excessive contact force, disturbing bystander objects, destabilizing the held object, or incurring self-collision. Empirical evidence, e.g., from OmniGuide (Song et al., 9 Mar 2026), demonstrates silent gaps between goal completion and collision-free execution, which standard leaderboards fail to highlight.
SafeVLA-Bench is introduced to systematically quantify the mismatch between “success” and “safety” and to decouple task completion from physical safety properties crucial for deployment in real environments. This framework raises two critical questions: (1) whether a successful rollout violates any safety specification; and (2) how severe the worst violation is.
Methodology
SafeVLA-Bench utilizes a library of task-aware safety requirements, formalized in Signal Temporal Logic (STL) [fainekos2009robustness, donze2010stl]. STL predicates cover three axes—scene interaction, object handling, and execution—with external grounding in safety standards (e.g., ISO/TS 15066 [iso_ts_15066]), hardware datasheets, and prior benchmarks:
- Scene Interaction: arm–furniture contact force, target–furniture contact force, peak overall contact force, non-target (bystander) object displacement.
- Object Handling: held-object tilt during transport, stable grasp maintenance (slip or drop).
- Execution: joint torque limits, self-collision.
Thresholds (e.g., 200 N contact ceilings, 5 mm displacement, 15° tilt) are derived from physical standards rather than tailored to model outputs, and task-specific applicability is encoded using a tag–rule registry to prevent false positives and penalization of goal-required behaviors.
Metrics: Decoupling Success and Safety
SafeVLA-Bench reports four core metrics for each (policy, benchmark, task):
- SR (Success Rate): host benchmark’s binary task completion.
- Safety: fraction of rollouts satisfying all applicable safety predicates.
- SBU (Succ-But-Unsafe Rate): fraction of rollouts that both succeeded and violated at least one safety clause.
- VSI (Violation Severity Index): normalized worst violation per episode, bounded in [0,1], with severity anchors tied to STL thresholds (e.g., 500 N for force, 30° for tilt).
Post-hoc scoring preserves comparability with native leaderboards, making the approach portable to any simulator-based benchmark with minimal host adaptation.
Experimental Evaluation
Evaluations are performed on LIBERO and RoboCasa-365 benchmarks with nine state-of-the-art VLA policies, including OpenVLA-7B [openvla2024], Cosmos-Policy-2B (Kim et al., 22 Jan 2026), GR00T-N1.7 (NVIDIA et al., 18 Mar 2025), and π0​ (Black et al., 2024). Multiple manipulation task suites are covered, ranging from tabletop PnP to kitchen-scale articulated tasks.
Key Empirical Findings
- High SR Does Not Imply High Safety: Across LIBERO, SFT baselines exceeding 94% mean SR show only 85–87% mean Safety, yielding a systematic 13–15% unsafe-episode rate among purportedly successful rollouts. The success–safety gap is most prominent in settings with high SR, directly contradicting the assumption that task completion correlates strongly with safety.
- Unsafe Success Is Systematic in Household Tasks: On RoboCasa-365, 36–56% of successful rollouts violate at least one active safety clause. For example, GR00T-N1.5 achieves 47.2% SR but only 40.7% Safety, with SBU as high as 26.3%.
- Ranking Discrepancies: Model rankings change when using safety-oriented metrics. The highest SR policy does not always rank highest in Safety or lowest in violation severity (VSI), demonstrating non-monotonicity between task completion and physical safety.
- Violation Decomposition: Major contributors to violations include bystander-object displacement in LIBERO and over-threshold contact force or held-object tilt in RoboCasa-365. SBU and VSI capture distinct failure modes, with SBU highlighting frequency and VSI quantifying depth.
Threshold Sensitivity
Relaxing contact-force ceilings (e.g., from 200 N to 500 N) reduces unsafe-success rates but does not eliminate them, confirming the robustness of the observed success–safety gap under plausible calibration changes.
Implications and Future Directions
SafeVLA-Bench demonstrates that binary goal predicates in standard VLA evaluation are inadequate proxies for physical safety, and thus unreliable for real-world deployment validity. The framework provides actionable post-hoc diagnostics for unsafe-success frequency and severity, which can inform model selection, safety-aligned post-training, and granular failure analysis.
Future development should extend SafeVLA-Bench to real robots, human-in-the-loop scenarios, multi-arm setups, and probabilistic risk modeling, as well as incorporate STL robustness directly into learning objectives or test-time guidance. Adapter contracts make such porting feasible, but new signal classes and hazard types will demand additional specification families.
Conclusion
SafeVLA-Bench establishes a principled, portable methodology for surfacing the success–safety gap in VLA policy evaluation. Empirical results challenge the sufficiency of task success as a safety certificate and expose systematic unsafe behaviors hidden by native benchmarks. Theoretical and practical implications are substantial: safety metrics must be decoupled from task completion, and formal specification libraries with robust severity scoring are essential for safe integration of VLA models in the household and industrial domains. The framework is poised for broader adoption across manipulation benchmarks and real-world settings to support safety-critical deployment and advanced policy diagnostics.
References:
"SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models" (2606.00773)
LIBERO [libero2023], RoboCasa [robocasa2024], ISO/TS 15066 [iso_ts_15066], OmniGuide (Song et al., 9 Mar 2026), Cosmos-Policy (Kim et al., 22 Jan 2026), OpenVLA-7B [openvla2024], GR00T-N1.7 (NVIDIA et al., 18 Mar 2025), π0​ (Black et al., 2024), Signal Temporal Logic [fainekos2009robustness, donze2010stl]