LIBERO-PRO Suite: Lifelong Learning Benchmark

• LIBERO-PRO Suite is a procedurally generated benchmarking suite for lifelong robot learning that integrates diverse, multi-step manipulation tasks.
• It employs automated task generation and systematic perturbations to measure knowledge transfer, generalization, and robustness in VLA models.
• The suite reveals key challenges in overfitting and memorization, contrasting transformer-based successes with traditional methods.

LIBERO-PRO refers to a suite of large-scale, procedurally generated and robustness-centric manipulation task benchmarks designed to rigorously evaluate knowledge transfer, generalization, and robustness in lifelong robot learning and vision-language-action (VLA) models. It is deployed in two major forms within the literature: as the challenging procedural suite of the original LIBERO lifelong-learning benchmark (Liu et al., 2023) and, subsequently, as a systematic robustness testbed exposing the limitations of current VLA policies (Zhou et al., 4 Oct 2025). LIBERO-PRO distinctively intertwines a combinatorial diversity of tasks, automated pipeline-driven task generation, and multi-faceted evaluation, establishing it as a critical resource for benchmarking, comparing, and diagnosing lifelong and generalist agents far beyond rote memorization.

1. Suite Definition and Motivations

LIBERO-PRO constitutes a procedurally generated task set that expands upon the core LIBERO benchmark. Unlike the earlier LIBERO sub-suites—each targeting a single distribution shift (spatial, object, or goal)—LIBERO-PRO deliberately entangles multiple axes of diversity, including object identity, spatial configurations, and multi-step goal predicates. The suite contains 100 tasks (90 “short-horizon” for pretraining and 10 “long-horizon” for evaluation), with each task representing a distinct sample from a virtually infinite task generator. The intent is to facilitate and stress-test the transfer of both declarative and procedural knowledge in complex, compositional manipulation scenarios.

A core rationale for procedural generation is to reflect the unbounded task variety encountered in real-world deployments; hand-curated benchmarks are inherently limited in scope and may fail to measure true generalization.

Moving beyond knowledge transfer, the LIBERO-PRO robustness evaluation (Zhou et al., 4 Oct 2025) systematically perturbs tasks across four axes—object appearance, initial state, instruction form, and environment layout—to reveal overfitting and memorization that confound standard evaluations.

2. Task Generation and Structure

The LIBERO-PRO procedural engine produces each task through the following pipeline:

1. Template Extraction and Instruction Sampling: Human-activity templates are mined from the Ego4D dataset. These templates (e.g., "Open …", "Put the X next to the Y", "Turn on …") are instantiated with supported object names.
2. Initial-State Specification: Scenes (e.g., kitchen, living room) are chosen to match the instruction. Regions within the scene are parameterized (axis-aligned bounding zones and allowable yaw rotation). Object placement is randomized within these constraints.
3. Goal-Predicate Construction: Instructions are translated into a conjunction of predicates, comprising unary (e.g., Open(X)), binary (e.g., In(A,B)), or compositional goal structures. Episodes terminate when the goal predicate is satisfied.

Randomization is introduced in region scaling, object orientation, mesh selection (from a class of 40+ object types), and texturing (±10% on scale). Via seed control, deterministically reproducible curricula can be crafted by varying object count or goal complexity.

In the robustness evaluation context (Zhou et al., 4 Oct 2025), four orthogonal perturbation operators are defined ($\phi_k, k \in \{O,I,L,E\}$):

• Object ($O$): Vary non-semantic attributes (color, texture, scale) within a bounded radius.
• Initial State ($I$): Spatially shift object poses up to a threshold (e.g., 0.2–0.3 m).
• Instruction ($L$): Paraphrase or swap target objects/actions in the goal instruction.
• Environment ($E$): Swap or perturb background scene layouts and textures.

Perturbations guarantee non-trivial total variation distance ($d_{TV}(\tau,\tau^{(k)}) > \epsilon$), while ensuring task executability.

3. Observation, Action Spaces, and Task Primitives

Observation space ($\mathcal{S}$):

• At step $t$, the agent observes $$o_t = \{\text{RGB image }(3\times224\times224), \text{joint angles}, \text{gripper width}\}$$
• Full observation history $s_t = o_{0:t}$ is used to address partial observability.

Action space ($O$0):

• Low-level continuous control: $O$1, plus a gripper scalar.
• Actions are generated via a Gaussian Mixture Model head on the policy’s latent.

Task primitives include:

• Pick-and-place (bowls, mugs, bottles)
• Drawer and cabinet opening/closing
• Stacking and nesting
• Appliance control (microwave, stove)

Every manipulation sequence is uniquely identified by the interplay of visual scene elements, syntactic task structure, and dynamic reward predicates.

4. Evaluation Metrics and Robustness Analysis

Two evaluation paradigms are defined by (Liu et al., 2023) and (Zhou et al., 4 Oct 2025).

Lifelong Transfer Metrics (Liu et al., 2023):

• Forward Transfer (FWT): Measures how quickly an agent learns a new task post previous-task training:

$O$2

• Negative Backward Transfer (NBT): Quantifies performance degradation on earlier tasks after later-task training:

$O$3

• Area Under Curve (AUC): Aggregates FWT and NBT.

$O$4

Robustness Metrics (Zhou et al., 4 Oct 2025):

Define policy $O$5, success probability:

$O$6

• Standard accuracy: $O$7
• Perturbed accuracy: $O$8 (post perturbation $O$9)
• Robustness gap: $I$0
• Overall robustness: $I$1

In practice, all evaluation is performed over 50 rollouts per task/perturbation, with never-seen perturbations during testing.

5. Baseline Results and Core Findings

Lifelong learning evaluation (Liu et al., 2023):

• Forward transfer: Sequential finetuning (SeqL) outperforms Experience Replay (ER), Elastic Weight Consolidation (EWC), PackNet, and multitask (MTL) baselines in FWT.
• Forgetting: PackNet nearly eliminates forgetting (lowest NBT), followed by ER; EWC only marginally mitigates forgetting.
• AUC (long-horizon): ER with ResNet-T, or PackNet with ViT-T, achieves the highest AUC (~0.32–0.34).
• Architectures: Temporal transformers (ResNet-T, ViT-T) outperform RNNs, especially on temporally extended tasks. ViT excels in tasks with substantial visual diversity.
• Pretraining: Pretraining on short-horizon tasks may degrade lifelong performance, contrary to intuition, implying naïve BC pretraining is insufficient.
• Task ordering: ER has lower variance to order; no method is order-invariant.

Robustness evaluation (Zhou et al., 4 Oct 2025):

• Standard vs. perturbed accuracy: All evaluated VLA models attain >94% under unperturbed LIBERO but often collapse to 0% on initial-pose or object swaps.
• Instruction and object sensitivity: Meaningless or corrupted instructions do not alter execution, signifying a reliance on visual memorization rather than true language grounding.
• Sensitivity to spatial/environmental shifts: Modest spatial shifts (0.2–0.3 m) lead to catastrophic failure.
• Model overfitting: Policies memorize action graphs tied to visual configurations, highlighting severe shortcomings in claimed generalist behavior.

Average success rates for core models:

Model	Acc(Ori)	Acc(O)	Acc(I)	Acc(L_sem)	Avg Robustness Acc (PRO)
OpenVLA	97.5	96.3	0.0	97.6	46.0
$I$2	94.8	93.5	0.0	93.9	40.2
$I$3	96.4	96.1	38.0	96.7	56.1

Acc indicates percent episodic success, with PRO denoting the mean over the four perturbation dimensions.

6. Practical Recommendations and Usage

• Policy architecture: Transformer-based temporal backbones (ResNet-T, ViT-T) are strongly preferred; deploy ViT if visual diversity is central.
• Lifelong algorithms: Experience Replay (buffer $I$4 1000, sample 32 per batch) provides a robust foundation; PackNet is effective at reducing forgetting, but demands increased capacity (≥13 GFLOPS).
• Task embedding: BERT-based instruction embeddings perform similarly to Task-ID embeddings, indicating limited benefit from current sentence encoders.
• Training regimen: Employ behavioral cloning for 50 epochs per task, select by highest empiric success, not BC loss. Adam with cosine-annealed learning rate (1e-4→1e-5), batch size 32.
• Extensibility: New tasks can be generated by simply reseeding the LIBERO-PRO pipeline. Curricula can be custom-designed by sorting tasks by object count, goal complexity, or estimated temporal horizon.
• Robustness evaluation: Use LIBERO-PRO evaluation scripts to systematically test VLA policies against all four perturbation axes, reporting both “in-distribution” and “perturbed” accuracy to reveal memorization and robustness gaps.

LIBERO-PRO thus serves as both a multifaceted testbed for knowledge transfer in sequential manipulation and a critical benchmark for evaluating and improving the real-world generalization of current VLA and continual learning agents (Liu et al., 2023, Zhou et al., 4 Oct 2025).