LIBERO-Para: Paraphrase Robustness in VLA Models

• LIBERO-Para is a diagnostic benchmark that evaluates paraphrase robustness in Vision-Language-Action models by varying action expressions and object references.
• It isolates linguistic generalization by controlling factors such as lexical, structural, and pragmatic variations, revealing notable performance drops.
• Empirical results show consistent 22–52 percentage point degradation across multiple VLA configurations, highlighting vulnerabilities in task identification.

LIBERO-Para is a diagnostic benchmark for paraphrase robustness in Vision-Language-Action (VLA) models. It was introduced to measure whether a manipulation policy follows the meaning of an instruction or instead relies on the canonical wording seen during fine-tuning. The benchmark independently varies action expressions and object references, enabling fine-grained analysis of linguistic generalization under meaning-preserving paraphrases. The accompanying paper reports consistent performance degradation of $22\text{–}52$ percentage points across seven VLA configurations, attributes much of this degradation to object-level lexical variation, and introduces PRIDE, a difficulty-aware metric intended to complement binary success rate (Kim et al., 30 Mar 2026).

1. Origins and problem setting

LIBERO-Para is situated within the LIBERO benchmark family, whose original purpose was to study knowledge transfer in lifelong robot manipulation. LIBERO formalized robot tasks as finite-horizon MDPs and emphasized that robot learning must transfer both declarative knowledge, such as object identity or spatial location, and procedural knowledge, such as how to open a drawer or place an object in a container (Liu et al., 2023). In that original setting, the benchmark concentrated on task transfer, policy architecture, lifelong-learning algorithms, task ordering, and pretraining effects, rather than on paraphrase robustness.

The motivation for LIBERO-Para is narrower and linguistic. The paper argues that modern VLAs inherit broad language priors from large pre-trained vision-language backbones, but downstream robotic fine-tuning is usually performed on small, environment-specific demonstration datasets. This creates a setting in which a model can overfit to the exact instruction phrasings used during fine-tuning and lose robustness to semantically equivalent reformulations (Kim et al., 30 Mar 2026).

A central design choice is the use of LIBERO-Goal. In LIBERO-Goal, all tasks begin from the same initial state, so the instruction is the only cue for task identification. This removes a major confound: if performance degrades under paraphrase, the degradation is not primarily attributable to visual ambiguity. The benchmark therefore targets instruction-to-task identification rather than generic perception failures.

2. Benchmark construction and paraphrase taxonomy

LIBERO-Para decomposes robotic instructions into two semantic axes: action expressions and object references. It then varies these axes independently. This controlled factorization is the benchmark’s defining feature because it separates failures caused by action paraphrasing, object paraphrasing, and their composition.

The action axis is divided into lexical, structural, and pragmatic variation. Lexical action variation includes same-polarity habitual substitution, same-polarity contextual substitution, and addition. Structural action variation includes coordination and subordination. Pragmatic action variation includes personal need, question directive, embedded imperative, permission, and hint. The object axis is restricted to lexical variation, reflecting the paper’s observation that object references in manipulation instructions are typically short noun phrases rather than extended descriptions.

Axis	Categories	Types
Object variation	Obj-Lexical	same polarity habitual; same polarity contextual; addition
Action variation	Act-Lexical	same polarity habitual; same polarity contextual; addition
Action variation	Act-Structural	coordination; subordination
Action variation	Act-Pragmatic	personal need; question directive; embedded imperative; permission; hint

The benchmark uses 3 object variation types and 10 action variation types, yielding 3 object-only conditions, 10 action-only conditions, and 30 compositional object $\times$ action conditions, for a total of 43 paraphrase type combinations (Kim et al., 30 Mar 2026).

Dataset construction proceeds in four stages: axis-wise paraphrase generation, verification, merging, and final verification. The paper states that all LLM calls use Gemini 2.5 Pro. Paraphrases are constrained to preserve meaning, preserve plurality, avoid adding visual attributes such as color, size, or material, and avoid adding spatial attributes such as location or direction. For object tasks, only object nouns are modified; for action tasks, only action elements are modified while object nouns remain fixed. The verifier also rejects confusable object references that may overlap with other environment objects.

The taxonomy itself is selected from a larger paraphrase inventory. The appendix starts from 26 atomic types in Extended Paraphrase Typology and 6 directive types from Ervin-Tripp, then retains 13 types that satisfy applicability to direct robotic manipulation instructions, meaning preservation, visual and spatial constraint compliance, and grammatical naturalness (Kim et al., 30 Mar 2026).

The resulting dataset contains 4,092 paraphrased instructions. These are generated from 10 original instructions, with 386–423 paraphrases per original instruction. The provided summary further reports 259 object-only paraphrases, 870 action-only paraphrases, and 2,963 compositional paraphrases, with approximately $\sim 100$ samples per variation type combination.

Human validation is reported on a random 5% subset comprising 205 samples with 15 annotators. The paper reports Gwet’s AC1 $=0.854$, majority-vote meaning preservation of $204/205 = 99.51\%$, an 80% agreement threshold of $183/205 = 89.27\%$, and a mean yes rate of $14.13/15 = 94.18\%$. Disagreement is concentrated on indirect speech-act paraphrases, which the paper treats as pragmatically challenging cases rather than annotation noise (Kim et al., 30 Mar 2026).

3. Evaluated models and protocol

The paper evaluates seven VLA configurations spanning four architecture families and a scale range from $0.6$B to $7.5$B parameters. The model set includes OpenVLA-OFT$_{\text{goal}}$, OpenVLA-OFT$\times$0, $\times$1, $\times$2 (expert-only), Xiaomi-Robotics-0, X-VLA, and VLA-Adapter (Kim et al., 30 Mar 2026).

Configuration	Family	Total parameters
OpenVLA-OFT$\times$3, OpenVLA-OFT$\times$4	Parallel decoding with action chunking	7.5B
$\times$5	VLM + flow-matching action expert	3.3B
$\times$6 (expert-only)	VLM + flow-matching action expert	$\times$7M trainable
Xiaomi-Robotics-0	VLM + flow-matching action expert	4.7B
X-VLA	Soft-prompted cross-embodiment	0.9B
VLA-Adapter	Bridge-based adaptation	0.6B

The evaluation is designed to isolate the role of downstream fine-tuning. All models are fine-tuned on LIBERO and then evaluated on LIBERO-Para. OpenVLA-OFT$\times$8 is fine-tuned on LIBERO Goal only, whereas OpenVLA-OFT$\times$9 is fine-tuned on all four LIBERO suites: Goal, Spatial, Object, and Long. This produces a direct comparison between limited and broader task diversity. The $\sim 100$0 expert-only ablation freezes the VLM and fine-tunes only the Action Expert, testing whether preserving the pretrained language encoder changes paraphrase robustness.

The paper reports LoRA fine-tuning for OpenVLA-OFT$\sim 100$1, OpenVLA-OFT$\sim 100$2, VLA-Adapter, and X-VLA, and full fine-tuning for $\sim 100$3 and Xiaomi-Robotics-0. For $\sim 100$4 and $\sim 100$5 expert-only, the reported settings are batch size 256, peak learning rate $\sim 100$6, AdamW, gradient clipping 1.0, EMA decay 0.999, warmup 10k steps, training for 30k steps, and action horizon 10. Evaluation uses 5 random seeds—7, 8, 9, 10, and 11—and reports mean success rate over the 5 seeds under LIBERO simulator default evaluation settings (Kim et al., 30 Mar 2026).

4. Empirical findings

The headline empirical result is that all evaluated configurations degrade substantially under paraphrased instructions. The reported drops range from $\sim 100$7 to $\sim 100$8 percentage points when moving from LIBERO-Goal to LIBERO-Para (Kim et al., 30 Mar 2026).

Model	LIBERO-Goal SR	LIBERO-Para SR	Drop
OpenVLA-OFT$\sim 100$9	97.9	64.7	-33.2 pp
OpenVLA-OFT$=0.854$0	96.1	63.7	-32.4 pp
$=0.854$1	97.6	71.4	-26.2 pp
$=0.854$2 expert-only	78.6	39.1	-39.5 pp
X-VLA	97.8	62.1	-35.7 pp
VLA-Adapter	98.2	46.3	-51.9 pp
Xiaomi-Robotics-0	98.8	76.0	-22.8 pp

These results support the paper’s claim that paraphrase fragility is not architecture-specific. The degradation is visible across small and large models, across adapter-style and full fine-tuning regimes, and across several action-decoder designs.

The strongest empirical pattern concerns object paraphrasing. The paper reports that object-level lexical variation is the main bottleneck: even simple synonym substitutions such as “stove” $=0.854$3 “range” or “cooktop,” and “bowl” $=0.854$4 “container,” can induce large performance drops. The reported gap between object-preserved and object-paraphrased conditions ranges from $=0.854$5 percentage points for $=0.854$6 expert-only to $=0.854$7 percentage points for OpenVLA-OFT$=0.854$8 (Kim et al., 30 Mar 2026). The paper interprets this as evidence of reliance on surface-level lexical matching rather than robust semantic grounding.

Action paraphrasing also degrades performance, but less severely on average. The reported average success rates are $=0.854$9 for the original action form, $204/205 = 99.51\%$0 for lexical action variants, $204/205 = 99.51\%$1 for structural variants, and $204/205 = 99.51\%$2 and $204/205 = 99.51\%$3 for question and hint directives, respectively. The hardest benchmark cells occur when object paraphrases are combined with indirect action forms, especially question and hint variants. The model-average success rate reaches $204/205 = 99.51\%$4 at SP-habitual $204/205 = 99.51\%$5 Hint, and the cell with the highest average PRIDE score is SP-habitual $204/205 = 99.51\%$6 Question at $204/205 = 99.51\%$7 (Kim et al., 30 Mar 2026).

The comparison between OpenVLA-OFT$204/205 = 99.51\%$8 and OpenVLA-OFT$204/205 = 99.51\%$9 is particularly diagnostic. Although the mixed variant is fine-tuned with four times more task-level diversity, its paraphrase robustness remains nearly unchanged relative to the goal-only variant. This suggests that increasing trajectory diversity alone does not resolve the linguistic generalization gap.

5. Failure modes and the PRIDE metric

A major contribution of the paper is the claim that most paraphrase failures are planning-level rather than execution-level. To analyze this, the authors define a pseudo ground-truth trajectory for each task by averaging successful executions, then compare failed trajectories to that reference using Dynamic Time Warping (DTW) on the first three proprioceptive dimensions, namely end-effector absolute position $183/205 = 89.27\%$0. All trajectories are resampled to $183/205 = 89.27\%$1 points. A failure is labeled Near-GT if its DTW distance is at most the maximum DTW distance among successful episodes for that task, and Far-GT otherwise (Kim et al., 30 Mar 2026).

Using this criterion, the paper reports that $183/205 = 89.27\%$2 of failures are Far-GT, summarized in the abstract as roughly $183/205 = 89.27\%$3. The implication is that paraphrases usually disrupt task identification or high-level planning rather than only low-level motor execution. The $183/205 = 89.27\%$4 expert-only configuration is a notable exception in degree rather than kind: it has a larger Near-GT share, which the paper interprets as partial preservation of language understanding combined with weaker execution after freezing the VLM.

PRIDE is introduced because binary success rate weights all paraphrases equally. The paper defines a keyword similarity term $183/205 = 89.27\%$5, a structural similarity term $183/205 = 89.27\%$6, a combined paraphrase distance $183/205 = 89.27\%$7, and the final PRIDE score as follows (Kim et al., 30 Mar 2026):

$183/205 = 89.27\%$8

$183/205 = 89.27\%$9

$14.13/15 = 94.18\%$0

$14.13/15 = 94.18\%$1

Here $14.13/15 = 94.18\%$2 and $14.13/15 = 94.18\%$3 denote the content words of the original instruction and the paraphrase, $14.13/15 = 94.18\%$4 is a Sentence-BERT embedding, and $14.13/15 = 94.18\%$5 is the default weighting. $14.13/15 = 94.18\%$6 targets lexical-semantic preservation of task-critical tokens, especially object and action words, while $14.13/15 = 94.18\%$7 measures syntactic divergence using dependency-tree edit distance.

PRIDE is intended as a difficulty-aware robustness measure. A model receives more credit for succeeding on a paraphrase with larger semantic or structural deviation. The paper reports that raw success rate overestimates robustness by $14.13/15 = 94.18\%$8, depending on the model. VLA-Adapter has the largest gap, while $14.13/15 = 94.18\%$9 and Xiaomi-Robotics-0 have smaller gaps, indicating more consistent performance across easier and harder paraphrases (Kim et al., 30 Mar 2026).

The benchmark summary also reports significant negative correlations between paraphrase difficulty and success rate, with Pearson $0.6$0 ranging from $0.6$1 to $0.6$2 and $0.6$3 for all models. This supports the intended interpretation of PRIDE and its underlying paraphrase-distance construction.

6. Position within the LIBERO ecosystem, limitations, and implications

Within the broader LIBERO ecosystem, LIBERO-Para is a targeted linguistic diagnostic rather than a general perturbation benchmark. LIBERO-PRO studies robustness under perturbations of objects, positions, instructions, and environments and argues that standard LIBERO evaluation can reward memorization (Zhou et al., 4 Oct 2025). LIBERO-Occ, by contrast, isolates scene-induced occlusion and cites LIBERO-Para only in the references, where it appears under the title “LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models” (Li et al., 9 Jun 2026). This positioning indicates that LIBERO-Para occupies the language-robustness corner of a broader family of LIBERO-derived stress tests.

The paper states several limitations. Evaluation is simulation-only. The benchmark varies one object type and one action type compositionally, but does not study richer multi-variation paraphrases of the form synonym substitution plus adverb insertion plus structural reordering in a single instruction. It does not include a paraphrase-augmentation training study because LLM-generated paraphrases might overlap with the benchmark and confound evaluation. It also notes that human disagreement is concentrated on indirect directive forms, which remain semantically valid but pragmatically difficult (Kim et al., 30 Mar 2026).

The broader implication is that current VLAs appear brittle at the language-to-plan interface. The reported dominance of Far-GT failures suggests that the weak point is often not control precision but the mapping from paraphrased instruction to correct task identity. The strong effect of object lexical variation suggests that canonical object naming during fine-tuning encourages lexical anchoring rather than object grounding. This, in turn, suggests that future work should emphasize language diversity during robotic fine-tuning, stronger grounding of object synonyms to scene entities, and robustness metrics that distinguish easy paraphrases from harder semantically equivalent formulations (Kim et al., 30 Mar 2026).