LIBERO Benchmark Suites

• LIBERO Benchmark Suites are a comprehensive set of tools designed to assess knowledge transfer, robustness, and generalization in robotic manipulation and vision-language-action tasks.
• They span various evaluations including spatial, object, and linguistic generalization while rigorously testing models under controlled distribution shifts and perturbations.
• They employ detailed metrics such as forward transfer, negative backward transfer, AUC, and the PRIDE metric to provide a multidimensional analysis of model performance.

LIBERO Benchmark Suites provide a unified, extensible, and diagnostically rich set of tools for evaluating knowledge transfer, robustness, and true generalization in robot manipulation and Vision-Language-Action (VLA) models. The suite spans standard lifelong imitation learning, distribution shift assessment, paraphrase sensitivity, and rigorous environmental perturbation analysis. LIBERO and its successors address fundamental weaknesses in prior benchmarks, systematically diagnosing catastrophic forgetting, memorization, and overfitting to narrow distributions. The family encompasses core suites (LIBERO, LIBERO-OBJECT/GOAL/SPATIAL), robustness stress-tests (LIBERO-Plus, LIBERO-PRO), linguistic generalization probes (LIBERO-Para), and hierarchical evaluation protocols (LIBERO-X), collectively establishing a new standard for rigorous VLA assessment.

1. Foundational Concepts and Task Structure

The original LIBERO benchmark was introduced to capture the challenge of Lifelong Learning in Decision-Making (LLDM) for robotic manipulation (Liu et al., 2023). Unlike previous vision or NLP lifelong learning benchmarks focused on declarative knowledge, LIBERO comprehensively targets both declarative and procedural knowledge transfer. Each task is parameterized by:

• An initial state distribution $\mu_0^k$ (randomized robot/object placements)
• A language instruction $\ell$ (task description)
• A goal predicate $g^k(s)$, replacing sparse rewards
• A set of high-quality expert demonstrations $\mathcal{D}^k = \{\tau_i^k\}$, obtained via human teleoperation

LIBERO defines several distinct task suites, each isolating specific transfer sources:

• LIBERO-Spatial: Spatial generalization with identical objects and positional predicates.
• LIBERO-Object: Object-category generalization across varied items.
• LIBERO-Goal: Procedural transfer across manipulation behaviors.
• LIBERO-100 (and LIBERO-Long): Entangled, long-horizon skill composition.

The procedural generation pipeline ensures extensibility and reproducibility; task templates (from Ego4D) are instantiated into PDDL-described scenes with parameterized objects and predicates. This machinery supports effectively infinite task generation, with demonstration data enabling sample-efficient behavioral cloning.

2. Lifelong Learning Benchmarks: Protocols and Metrics

LIBERO formalizes lifelong imitation learning with a protocol evaluating the streamwise acquisition and retention of skills (Liu et al., 2023, Roy et al., 2024):

• Forward Transfer (FWT): Measures initial competence on unseen tasks after training on previous tasks.
• Negative Backward Transfer (NBT): Quantifies forgetting by comparing performance on old tasks after later learning phases.
• Area Under Curve (AUC): Integrates learning and retention across the full task sequence.

Architectures evaluated include ResNet-RNN, ResNet-Transformer, ViT-Transformer backbones, with task-embedding via BERT, CLIP, GPT-2, or task ID. Policy heads output GMMs over continuous actions. Methods compared include sequential fine-tuning (SeqL), multitask learning (MTL), experience replay (ER), EWC, and PackNet. Empirical results demonstrate that:

• Transformers with attention mechanisms outperform RNNs for both forward transfer and global performance.
• Most lifelong learning algorithms trade off learning speed versus forgetting; ER achieves balanced results, whereas PackNet minimizes forgetting but at the cost of capacity.
• Contrary to intuition, naive supervised pretraining on LIBERO-90 tends to degrade downstream long-horizon performance.
• Sentence-level language embeddings, regardless of backbone, do not improve transfer, acting as uninterpreted labels.

3. Robustness Stress Tests: LIBERO-Plus and LIBERO-PRO

LIBERO-Plus (Fei et al., 15 Oct 2025) and LIBERO-PRO (Zhou et al., 4 Oct 2025) systematically target the gap between nominal-condition performance and true robustness or generalization. These extensions introduce controlled perturbations along multiple axes, including but not limited to:

• LIBERO-Plus: Seven axes, each with fine-grained subcomponents:
  • Objects layout (confounders, target pose)
  • Camera viewpoints (distance, orientation, position)
  • Robot initial states (joint angle perturbations)
  • Language (synonym, reasoning-chain, distraction)
  • Light conditions and background
  • Sensor noise (various synthetic corruptions)
• LIBERO-PRO: Four orthogonal perturbation axes:
  • Object visual attributes (color, texture, size)
  • Initial state displacement
  • Instruction variation (semantic paraphrases, goal/object switch)
  • Environment/background swaps

Evaluation under these perturbations exposes that VLA models achieving >90% on standard LIBERO can collapse to 0% under moderate distribution shifts. Memorization is particularly severe for object positions and instructions—models frequently ignore the language modality, defaulting to rigid trajectory templates regardless of semantic changes (Fei et al., 15 Oct 2025, Zhou et al., 4 Oct 2025).

Average success rates and per-axis drops for typical models (OpenVLA-OFT_m):

	Nominal	Camera	Robot	Language	Light	Background	Layout
SR (%)	97.6	57.9	30.6	83.6	91.6	83.6	73.2

LIBERO-PRO enforces reporting per-dimension accuracies and random perturbation combinations, revealing model brittleness and absence of compositional generalization.

4. Linguistic Robustness: LIBERO-Para and the PRIDE Metric

LIBERO-Para (Kim et al., 30 Mar 2026) probes linguistic generalization by systematically generating meaning-preserving paraphrases along two axes:

• Object axis: Same-polarity substitution (habitual/contextual) and additive qualifiers.
• Action axis: Lexical (synonym/adverb), structural (coordination/subordination), and pragmatic (indirect speech acts, hinting) variation.

Paraphrases are generated and verified by LLMs and manual checks to ensure semantic and grammatical fidelity; compositional combinations of object and action paraphrases yield 4,092 unique variants for 10 core LIBERO-Goal tasks.

A formal diagnostic metric, PRIDE, quantifies paraphrase difficulty as a convex combination of semantic (SK) and syntactic (ST) similarity:

• $\mathrm{SK}(O,P) =$ mean maximal cosine similarity between keyword embeddings of original and paraphrase.
• $$\mathrm{ST}(T_0,T_p) = 1 - \text{TED}(T_0,T_p)/(\lvert T_0 \rvert + \lvert T_p \rvert)$$, using dependency tree edit distance.
• Paraphrase distance ${\rm PD} = 1 - [\alpha\, SK + (1-\alpha)\, ST]$, with default $\alpha=0.5$.

PRIDE incorporates paraphrase difficulty by rewarding correct model output proportional to difficulty and zeroing otherwise. Experimental evaluation demonstrates 22–52 percentage point drops in success rate under paraphrasing—the strongest effect is from object-level lexical variation. 80–96% of failures result from planning-level confusion (trajectory divergence), not execution. PRIDE exposes that binary SR obscures overestimation on easy cases; for instance, VLA-Adapter’s SR overestimates true robust performance by 22%.

Model	Binary SR	PRIDE Score	Overestimation (%)
VLA-Adapter	46.3	36.1	22.0
π₀.₅ (expert-only)	39.1	32.0	18.2

LIBERO-Para can be incorporated into any pipeline as a plug-in “paraphrase” testbed, revealing otherwise invisible fragility.

5. Hierarchical and Cumulative Robustness: LIBERO-X

LIBERO-X (Wang et al., 6 Feb 2026) extends the evaluation paradigm by introducing a hierarchical, multi-level protocol spanning five difficulty layers:

• Level 1: Local spatial perturbation
• Level 2: Extended spatial variation
• Level 3: Scene topology restructuring (novel configurations, confounders)
• Level 4: Visual attribute modulation (texture, size, unseen categories)
• Level 5: Semantic instruction reformulation (synonyms, reordering, voice, verbosity)

Levels are cumulative, with each new perturbation compounding previous ones. A highly diverse teleoperated training dataset (100 scenes, 600 tasks, 2,520 demonstrations) bridges the train-test gap, enabling analysis under severe distribution shift. Experiments reveal large performance drops at each level:

Model	L1	L2	L3	L4	L5
OpenVLA-OFT	29.0	17.6	8.8	6.4	4.2
$\pi_0$	29.4	21.9	11.0	7.6	5.1
GR00T1.5	43.3	32.9	18.7	13.3	9.7
$\pi_{0.5}$	65.2	53.2	36.0	24.1	18.0

Success on L1 is below 40% (vs ∼90% in standard LIBERO), and the overall L1→L5 drop averages 31.2%. This exposes overfitting to narrow layouts, lack of topological reasoning, failure to ground novel object concepts, insensitivity to language paraphrase, and severe compounding error on multi-step tasks.

6. Key Empirical Insights and Recommendations

Across all LIBERO variants, several consistent findings emerge:

• VLA models are highly brittle to any substantive distribution shift (camera, robot initial state, object layouts, paraphrased instructions).
• Models achieving nominal “superhuman” success often ignore language instructions or simply memorize input-output templates.
• Language, despite its integration, is frequently underutilized—perturbation or blanking causes minor performance change unless the task goal itself changes.
• Neither architectural modifications nor standard RL/IL algorithms resolve these deficiencies. Multi-modal distillation (Roy et al., 2024) and hierarchical evaluation are necessary but not sufficient.
• Robust benchmarking requires per-axis and compositional accuracy reporting, large-scale instance perturbation, and difficulty-weighted success metrics (e.g., PRIDE).
• Practitioners are urged to abandon reporting nominal success alone and instead embrace plug-and-play LIBERO robustness suites.

This suggests that reliable progress in vision-language-action robotics mandates fine-grained, multidimensional robustness study, resisting the field’s tendency toward measuring performance under myopic or overfitted test splits. The LIBERO family establishes a rigorous baseline for such analysis, continuously influencing emerging VLA model development.