LIBERO-Plus: Robustness Simulation Suite

• The LIBERO-Plus simulation suite introduces seven controlled perturbation dimensions to evaluate VLA model robustness under diverse environmental and input variations.
• It automates large-scale episodic evaluations with stratified difficulty levels and statistically rigorous protocols to uncover hidden model brittleness not seen in static benchmarks.
• Empirical evaluations demonstrate significant performance drops under camera and robot initial state shifts, emphasizing the importance of comprehensive robustness testing.

LIBERO-Plus is a large-scale simulation suite designed for in-depth robustness analysis of Vision–Language–Action (VLA) models in robotic manipulation. Built atop the foundational LIBERO benchmark, LIBERO-Plus systematically introduces principled, orthogonal distribution shifts across seven key environmental and input dimensions—object layout, camera viewpoints, robot initial state, language instructions, lighting, background textures, and sensor noise—to diagnose and quantify model brittleness that is not observable with standard, static benchmarks. By employing automated, large-scale generation and evaluation protocols with rigorous statistical techniques, LIBERO-Plus enables stratified, reproducible, and extensible testing to drive research towards truly generalist and robust embodied agents (Fei et al., 15 Oct 2025).

1. Goals and Design Principles

LIBERO-Plus addresses the gap between high apparent VLA benchmark scores and actual robustness to distribution shifts encountered in realistic deployments. Its design is governed by three primary objectives:

• Completeness: All major input modalities—vision, proprioception, and language—are exercised via fine-grained, parameterizable perturbations, ensuring broad coverage of potential failure modes.
• Automation: Training and test episodes are generated at scale without manual curation, supporting thousands of independent runs and enabling statistically significant measurement.
• Diagnostic Depth: Task instances are stratified into five distinct difficulty levels, and explicit compositional generalization metrics are provided to interrogate robustness beyond marginal success/failure rates.

The suite enforces reproducibility through seedable random number generation and externally specified scene/task configuration files (XML, BDDL), extensibility via a unified Python API for integrating new models or perturbation schemas, and statistical rigor with multi-run sampling and formal tests for interaction effects.

2. Systematic Perturbation Dimensions

LIBERO-Plus expands the original LIBERO suite by parameterizing seven orthogonal perturbation dimensions through scene and task configuration manipulation. Each dimension features five intensity levels (L1–L5) and sub-dimensions where applicable:

Dimension	Sub-dimensions	Mechanism/Range
Object Layout	O1: Distractors, O2: Target Pose	Insert 1–5 distractors from a 416-object library; perturb (x,y,z) and orientation
Camera Viewpoints	C1: Distance, C2: Position, C3: Orientation	Scale camera distance 1.01–2.0×; azimuth/elevation 15–75°; yaw/pitch/roll 2–10°
Robot Initial State	—	Add N(0,σ²) noise to joint angles, σ = 0.1–0.5 rad
Language Instructions	R1: Distraction, R2: Synonyms, R3: Reasoning	GPT-style rewrites for verbosity, synonyms, logical depth adjustments
Light Conditions	L1: Color, L2: Direction, L3: Specular, L4: Shadows	Vary RGB, sun direction up to 90°, specularness, shadow toggling
Background Textures	B1: Scene Theme, B2: Surface	Swap wall/table textures (950 images); surface: wood/marble/metal
Sensor Noise	N1–N5: Blur/Fog Types	Post-process with OpenCV-like filters: motion/gauss/zoom blur, fog, glass (L1–L5 severity)

By distributing perturbations along controllable, orthogonal axes, LIBERO-Plus enables fine-grained robustness curves and the isolation of specific weaknesses in VLA models.

3. Evaluation Metrics and Statistical Protocols

Success in LIBERO-Plus is defined as a binary trial outcome, $Y \in \{0,1\}$. For model $\theta$, perturbation $d$, and level $\ell$, the probability of success is estimated as $$P_d^\ell(\theta) = P(Y=1 \mid D=d\,\,\text{at}\,\,\text{level}\ \ell)$$, typically by Monte Carlo over $T$ episodes.

Performance degradation per dimension is computed as $$\Delta P_d(\theta) = P_d^{orig}(\theta) - P_d^{perturbed}(\theta)$$, quantifying the falloff from baseline to perturbed performance.

A weighted robustness score,

$$R(\theta) = \sum_{d=1}^{7} w_d \cdot P_d^{avg}(\theta),$$

summarizes performance across all dimensions, with $P_d^{avg}$ the mean across all intensities and $w_d$ the (typically uniform) dimension weights.

For joint perturbations $\theta$0, the compositional generalization gap,

$\theta$1

measures interaction effects beyond simple marginals, supporting diagnosis of entangled or non-independent failure modes. All rates are estimated by empirical counts and reported with standard errors. Statistical significance of joint perturbation effects is assessed via χ²-tests applied to $\theta$2 contingency tables.

4. Software Architecture and Implementation

LIBERO-Plus is built in Python 3.10 atop the original LIBERO codebase, utilizing MuJoCo v2.3 for physics simulation and hydra/YAML for configuration. Scenario generation is fully automated: scripts read experiment seeds, apply parameterized perturbations through scene/BDDL file edits or programmatically override attributes via the LIBERO Problem interface.

Key implementation features include:

• Evaluator Class: Runs episodic rollouts using a standard model interface, logs trial success, and supports multi-run statistics.
• Data Format: All test and training trajectories (22,000 successful runs) are stored in RLDS format, facilitating integration with pretraining pipelines.
• Model Registration: New VLA architectures are integrated by subclassing a standard VLAModel interface with .reset() and .step() methods; loading from HuggingFace or Google Cloud Storage is supported.
• Benchmarking Protocol: For each model/perturbation, 2,000 episodes are evaluated, with rates/intervals automatically computed over stratified difficulty levels (L1–L5).

The suite is fully extensible; new perturbations or models can be added with minimal modification via the unified API and configuration system.

5. Empirical Evaluation and Findings

Comprehensive experiments were conducted on ten representative open-checkpoint VLA models, including OpenVLA, OpenVLA-OFT (wrist-camera) variants, π₀, π₀-fast, NORA, WorldVLA, UniVLA, and RIPT-VLA. The findings demonstrate pronounced gaps between standard benchmark success and robustness under realistic perturbations:

• Camera and Initial State Sensitivity: State-of-the-art models dropped from baseline ≈95 % success to as low as 16 % under camera viewpoint shifts (mean $\theta$336 pp) and experienced similar collapses under robot initial state perturbations ($\theta$440 pp).
• Language Robustness: Language instruction rewrites produced a modest drop ($\theta$525 pp); however, experiments with outright instruction removal or semantic substitution revealed that most models effectively ignore language input, relying instead on vision–action associations.
• Lighting and Sensor Robustness: Wrist-camera models (e.g., OpenVLA-OFT) display substantially improved lighting robustness (as little as 2 pp degradation) compared to third-person–only counterparts (>60 pp drop).
• Generalization via Data Augmentation: Fine-tuning on the LIBERO-Plus training set (20,000 diversified successful trajectories) markedly improved robustness: OpenVLA-OFT_m increased camera robustness by +37 pp (from 55.6 % to 92.8 %) and overall robustness by +11.5 pp.
• Interaction Effects: Pairwise perturbation analysis confirmed significant negative interactions ($\theta$6), indicating entanglement in learned feature representations not captured by single-axis stress tests.

6. Implications and Research Outlook

LIBERO-Plus exposes critical deficiencies in current VLA models that static benchmarks fail to reveal, challenging the assumption that high single-score benchmarks reflect deployment-ready robustness. The empirical finding that language variations are often ignored suggests a need for more rigorous integration of multimodality in both architecture and supervision.

The systematic, extensible, and statistically principled evaluation framework provided by LIBERO-Plus is positioned as an enabler for developing and benchmarking next-generation VLA systems with certified robustness under real-world variation, supporting progress towards generalist embodied agents (Fei et al., 15 Oct 2025).