LangGap Benchmark for VLA Models

• The paper introduces LangGap, a diagnostic benchmark for VLA models that uses four-dimensional semantic perturbations to assess true language understanding versus visual memorization.
• LangGap reorganizes LIBERO’s tabletop tasks into a same-scene, multi-task setup, revealing dramatic performance drops (e.g., from 93.8% to 21.4%) when instructions are semantically varied.
• The benchmark offers actionable insights through controlled experiments, pinpointing weaknesses such as the 0% success rate on Change Target tasks, and emphasizing the need for enhanced linguistic robustness in VLA systems.

Searching arXiv for the specified paper and closely related benchmarks/methods. arXiv.search query: (Hou et al., 28 Feb 2026) LangGap is a diagnostic benchmark and training setup for Vision-Language-Action (VLA) models that is designed to test whether these models genuinely use language, rather than memorizing visual scenes and action patterns. It is built on tabletop manipulation tasks from LIBERO, but reorganized so that the same visual layout supports many different tasks distinguished only by their language instructions. The benchmark introduces a four-dimensional semantic perturbation method, varies instruction semantics while keeping the tabletop layout fixed, and evaluates how this same-scene multi-task design exposes deficits in language understanding that remain hidden on standard benchmarks. In the reported experiments, $\pi$0.5 achieves 93.8\% success on original LIBERO tasks but only 21.4\% on semantically varied tasks in the same scenes, including 0.0\% on Change Target, which the paper identifies as a manifestation of a broader “language gap” in current VLA systems (Hou et al., 28 Feb 2026).

1. Definition and motivation

Vision-Language-Action models are end-to-end policies that take as input visual observations, natural language instructions, and output robot actions. Representative models mentioned in the paper include RT-1/RT-2, the $\pi_0$ series including $\pi$0.5, OpenVLA, and SmolVLA. These systems are commonly evaluated on manipulation benchmarks such as LIBERO, RLBench, Meta-World, and CALVIN (Hou et al., 28 Feb 2026).

The motivating observation behind LangGap is that state-of-the-art models can achieve over 95\% success on official LIBERO tasks even though LIBERO typically assigns only one task per tabletop layout. Under that design, the same initial scene always corresponds to the same target action sequence. A model can therefore succeed by memorizing a scene-to-action mapping and effectively ignoring language. LangGap defines the “language gap” as the discrepancy between very high success rates on standard benchmarks and very poor performance once instruction semantics are systematically varied in ways that require true language understanding (Hou et al., 28 Feb 2026).

The benchmark is explicitly framed against three limitations in prior work. First, prior work lacks systematic semantic perturbation diagnostics that separate which semantic components are understood or ignored. Second, existing benchmarks often do not force language understanding because one task is tied to one layout. Third, training data exhibits insufficient linguistic diversity, so models have little incentive to rely on language robustly. LangGap addresses these limitations through a semantic perturbation taxonomy, a same-scene multi-task benchmark, and targeted training data that can be used to test whether augmenting linguistic diversity narrows the gap (Hou et al., 28 Feb 2026).

A common misconception addressed by the benchmark is that high benchmark success implies robust language grounding. The reported results argue against that interpretation: the paper shows that $\pi$0.5 can reach 93.8\% success on original LIBERO tasks yet drop to 21.4\% on semantically varied tasks in the same scenes, and to 0.0\% on Change Target (Hou et al., 28 Feb 2026).

2. Benchmark construction and semantic perturbation taxonomy

LangGap is built from tabletop manipulation and drawer tasks in LIBERO’s libero_spatial, libero_goal, and libero_object suites. For each LIBERO scene, the benchmark identifies all manipulable object categories, all reachable target locations, spatial descriptions that disambiguate multiple instances of the same object type, and valid interaction actions such as pick-and-place and drawer open/close. The initial visual state is kept fixed, and only the language instruction is varied to induce different tasks in the same scene (Hou et al., 28 Feb 2026).

The core design is a four-dimensional semantic perturbation framework, with each dimension corresponding to a distinct semantic slot in the instruction. The first dimension, Change Object Category (Ch. Obj.), changes the category of the manipulated object while keeping the same target location, action type, and tabletop layout. Its purpose is to test understanding of object category nouns. The second dimension, Change Target (Ch. Tgt.), changes the target location while keeping the manipulated object and action semantics fixed, thereby testing understanding of spatial goal descriptions. The third dimension, Spatial Description (Sp. Desc.), changes the spatial relation phrase used to distinguish among multiple instances of the same object category, testing relational language and spatial expressions. The fourth dimension, Drawer Action (Drawer), changes the action semantics to drawer manipulation or between drawers, testing verb and action-type understanding (Hou et al., 28 Feb 2026).

These four dimensions are described as “orthogonal” in the sense that they target different semantic slots: object, target, spatial relation, and interaction type. The benchmark aims at “maximal semantic diversity within identical visual layouts.” In libero_spatial, all four perturbation dimensions are available. In libero_goal, Change Object and Change Target are available. In libero_object, only Change Object is available (Hou et al., 28 Feb 2026).

A defining property of LangGap is that multiple instructions correspond to the same initial RGB observation. The paper gives examples in which one layout supports instructions such as “put the bowl on the plate,” “put the ramekin on the plate,” “put the cookie on the stove,” “open the top drawer,” and “pick the bowl to the right of the plate and place it on the stove.” Because the visual input is identical while the intended task differs, a policy cannot solve the benchmark by mapping scene to action alone. The paper states a simple bound: if a scene has $k$ different tasks, a policy that ignores language can succeed at most with probability $\frac{1}{k}$ by choosing one fixed behavior for that scene. This is the basis for the claim that LangGap “forces language reliance by construction” (Hou et al., 28 Feb 2026).

3. Task inventory, instruction diversity, and data splits

The full benchmark consists of 99 tasks: 40 original LIBERO tasks and 59 extended tasks generated through semantic perturbations. By suite, the 59 extended tasks are divided into 28 tasks from libero_spatial, 9 from libero_goal, and 22 from libero_object. Each extended task uses the same initial state as its corresponding original LIBERO task, and all tasks are validated for physical feasibility in the LIBERO simulator, including graspability, reachability, and detectability of the success condition (Hou et al., 28 Feb 2026).

The instruction set is semantically rather than superficially varied. The paper emphasizes that the extended instructions are not mere paraphrases of the original instructions. They alter which object is manipulated, where it is placed, which spatial relation identifies the correct instance, or which drawer is opened. The linguistic diversity therefore includes lexical variation at the noun level, goal specification through different target objects or locations, relational language via spatial descriptors such as “to the right of,” and action verbs distinguishing drawer manipulation from pick-and-place (Hou et al., 28 Feb 2026).

The train/validation/test methodology is instruction-level rather than layout-level. From the 59 extended tasks, 16 are chosen as training tasks: 9 from libero_spatial, 3 from libero_goal, and 4 from libero_object. These 16 tasks cover all four semantic dimensions. The remaining 43 extended tasks are held out for test while sharing the same scenes but differing in instruction semantics. This construction is intended to test semantic generalization within a fixed visual layout, including whether a model can transfer from one object-target combination to another or understand new combinations of familiar object and target words (Hou et al., 28 Feb 2026).

Relative to prior benchmarks, the paper reports the following comparison. LIBERO contains 130 total tasks and 3,000 training trajectories, with 1 task per layout as the operative design principle. LIBERO-Plus contains 10,030 total tasks, 20,000+ training trajectories, and 7 perturbation types, but is described as focusing on robustness with only one purely linguistic perturbation. LangGap contains 99 total tasks, 59 extended semantic tasks, 59 same-scene tasks, 16 extended tasks with training data, approximately 4,100 training trajectories, and 4 perturbation dimensions, with an explicit focus on compositional semantics (Hou et al., 28 Feb 2026).

4. Evaluation protocol and diagnostic logic

The evaluation protocol follows LIBERO-style episodic testing. Each task is run for 20 episodes, with episodes initialized according to LIBERO’s standard randomization within the same layout. Success is binary per episode and determined by a task-specific simulator condition, such as whether an object lies within tolerance of a target location or whether a drawer has been opened beyond a threshold (Hou et al., 28 Feb 2026).

The primary metric is success rate, reported as percentages at multiple granularities: per task, per suite, per perturbation dimension, and averaged over all tasks. The paper does not introduce a new mathematical metric beyond success rate; instead, it uses the per-dimension breakdown as a diagnostic instrument that isolates semantic failure modes (Hou et al., 28 Feb 2026).

The diagnostic logic depends on comparing performance on original tasks to performance on semantically perturbed tasks under the same visual input. If behavior remains effectively unchanged when the instruction changes, that pattern indicates that the model is underusing or ignoring language. The benchmark therefore functions as a controlled probe of visual memorization versus instruction-conditioned action selection. The same-scene multi-task design serves a role analogous to instruction perturbation tests, but with the stronger constraint that the visual scene itself offers no discriminative cue for choosing among the candidate behaviors (Hou et al., 28 Feb 2026).

This setup also clarifies the benchmark’s intended use. It is not only an evaluation suite but also a training scenario for measuring whether increased semantic diversity in the training data improves language dependence. The paper reports comparisons between a baseline $\pi$0.5 checkpoint and fine-tuned variants to determine whether gains occur specifically on semantically perturbed tasks (Hou et al., 28 Feb 2026).

5. Models, training regimes, and augmentation setup

The evaluated VLA models include $\pi$0.5, $\pi$0, $\pi$0-FAST, and SmolVLA. These models consume visual observations and language instructions and output action sequences in the LIBERO/robosuite environment (Hou et al., 28 Feb 2026).

The main training experiments fine-tune $\pi_0$00.5 using LoRA. The reported optimization setup uses Adam-style gradient descent, a learning rate of $\pi_0$1, batch size 8, and a single RTX 4090 GPU. Demonstrations are collected using scripted waypoint controllers with OSC_POSE control in robosuite, following behavioral cloning. The training data includes approximately 150 demonstrations per extended task, approximately 2,400 episodes for the 16 extended tasks, and original LIBERO data with 3,000 trajectories when that data is included (Hou et al., 28 Feb 2026).

Five experimental configurations are defined. Exp 1 uses 1 extended task, 56 demonstrations, LoRA rank $\pi_0$2, 20K steps, and evaluates on 1 task for 10 episodes. Exp 2 uses 1 original plus 5 extended tasks, 300 demonstrations, $\pi_0$3, 200K steps, and evaluates on 5 extended tasks for 10 episodes. Exp 3 uses 40 original plus 5 extended tasks, approximately 2,000 demonstrations, $\pi_0$4, 200K steps, and evaluates on 5 extended tasks for 10 episodes. Exp 4 uses 16 extended tasks, approximately 2,400 demonstrations, $\pi_0$5, 200K steps, and evaluates on 16 extended tasks for 5 episodes. Exp 5 uses 40 original plus 16 extended tasks, 4,093 demonstrations, $\pi_0$6, 200K steps, and evaluates on 16 extended tasks for 5 episodes (Hou et al., 28 Feb 2026).

The proposed intervention is explicitly data-centric rather than architectural. It consists of constructing same-scene semantically diverse tasks, collecting targeted scripted demonstrations for a subset of those tasks, and fine-tuning an existing VLA model with behavior cloning, either on the extended tasks alone or together with original LIBERO data. A plausible implication is that LangGap is intended as a benchmark that can separate limitations arising from training data composition from limitations arising from model architecture (Hou et al., 28 Feb 2026).

6. Experimental findings, limitations, and significance

The main diagnostic result is the magnitude of the language gap. On 99 tasks with 20 episodes each, $\pi_0$70.5 obtains 93.8\% success on the 40 original LIBERO tasks, 21.4\% success on the 59 extended tasks, and 50.7\% overall. Broken down by perturbation dimension, the model achieves 29.3\% on Change Object, 0.0\% on Change Target, 11.0\% on Spatial Description, and 31.7\% on Drawer Action. The drop from 93.8\% to 21.4\% is 72.4 percentage points. The paper highlights Change Target as especially revealing, with exactly 0/260 successes across 13 tasks and 20 episodes per task (Hou et al., 28 Feb 2026).

The finer-grained analysis further shows that Change Target failures are uniform: plate $\pi_0$8 stove yields 0/80, plate $\pi_0$9 cabinet yields 0/80, stove $\pi$0 plate yields 0/40, stove $\pi$1 cabinet yields 0/40, and plate $\pi$2 stove in the goal suite yields 0/20, for a total of 0/260. For Change Object, success varies substantially by suite: libero_spatial yields 3/160 or 1.9\%, libero_goal yields 54/160 or 33.8\%, and libero_object yields 166/440 or 37.7\%. This suggests that spatial tasks are particularly difficult for $\pi$30.5, whereas object and goal suites retain exploitable regularities (Hou et al., 28 Feb 2026).

The targeted augmentation experiments show that the language gap can be partially closed at small scale. In single-task training, performance on one extended task improves from 3.75 to 90.0. In the 6-task setting with 1 original and 5 extended libero_spatial tasks, success on the 5 extended tasks improves from 0.0 to 28.0. In the 45-task setting with 40 original and 5 extended tasks, success on those same extended tasks is only 4.0, which the paper describes as a dilution effect caused by the original tasks dominating the learning signal. In the multi-suite 16-task setting, the baseline checkpoint scores 26.2 on the 16 extended tasks, but training only on those 16 extended tasks reduces performance to 6.2. In the 56-task setting with 40 original and 16 extended tasks, performance rises slightly to 27.5 overall on the extended tasks, while spatial tasks remain at approximately 6.7\% (Hou et al., 28 Feb 2026).

The progressive pattern reported in the paper is 90\% for single-task training, 28\% for 6 tasks, and 6.2\% for 16 tasks. The authors interpret this as evidence that as semantic diversity and number of tasks increase, the model’s ability to internalize and generalize language semantics degrades sharply. This suggests that current models can memorize one instruction-scene mapping more readily than they can learn a general mapping from instruction semantics to actions in multi-task settings (Hou et al., 28 Feb 2026).

The benchmark’s limitations are explicit. The four semantic dimensions are not exhaustive and do not cover temporal compositionality, logical constructs such as negation, or more complex multi-step instructions. The experiments are conducted in simulation within LIBERO/robosuite, and real-world transfer is not studied. The training results also indicate that simply adding same-scene multi-task data does not fully solve language understanding at scale, implying that more advanced architectures and training strategies are needed (Hou et al., 28 Feb 2026).

The longer-term significance claimed for LangGap lies in three properties. First, it provides a stable, non-saturating benchmark because the same scenes can support multiple semantically distinct tasks. Second, it offers a diagnostic lens through per-dimension breakdowns such as 0.0\% on Change Target versus 29.3\% on Change Object. Third, it establishes a data-centric training paradigm that demonstrates both the local benefits of targeted semantic augmentation and the persistence of generalization and interference problems at larger scale. The paper positions LangGap as complementary to prior benchmark work such as LIBERO, LIBERO-Plus, and LIBERO-PRO, and to architectural approaches such as BayesVLA, LangForce, and Residual Semantic Steering, because it isolates language understanding under identical visual input rather than measuring broader robustness alone (Hou et al., 28 Feb 2026).