SimplerEnv-Instruct Benchmark

• SimplerEnv-Instruct is a benchmark that extends traditional manipulation tests by evaluating zero-shot instruction generalization in closed-loop control settings.
• It comprises 80 manually designed tasks grouped into Instruction Aggregation and Situated Reasoning, stressing language novelty and subtask decomposition.
• Experimental results demonstrate that advanced instruction tuning, as used in the Generalist model, yields performance improvements of up to 92% over baseline methods.

SimplerEnv-Instruct is a benchmark introduced in "InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation" to evaluate whether a vision-language-action model can do more than execute short atomic commands. It is built on top of the real-to-sim SimplerEnv platform and is designed specifically to test zero-shot instruction following and reasoning under closed-loop manipulation. In the InstructVLA paper, it is positioned as a generalization benchmark for models that must transfer in-domain manipulation skills to new linguistic and semantic conditions, rather than merely replay familiar low-level behaviors (Yang et al., 23 Jul 2025).

1. Origin and evaluative purpose

The benchmark was introduced against the background of the original SimplerEnv evaluation regime, which largely measures execution of simple, task-specific, atomic instructions such as opening a drawer or moving an object near another object. In the table reproduced in the source material, the original SimplerEnv tasks include Google Robot tasks—Open/Close Drawer, Put in Drawer, Pick Coke Can, and Move Near—and WidowX Robot tasks—Put Spoon, Put Carrot, and Stack Blocks. Those tasks emphasize manipulation success and generalization across embodiments and scenes, but not rich language understanding (Yang et al., 23 Jul 2025).

SimplerEnv-Instruct extends that setting to instruction generalization. The stated motivation is that existing VLA evaluations do not adequately test whether a model can understand free-form or rephrased instructions, handle novel verbs, multilingual language, or indirect references, infer implicit user intent from context, decompose goals into environment-grounded subtasks, and preserve and use the reasoning ability of the pretrained VLM during manipulation. The benchmark was therefore introduced to expose the weakness that standard VLA finetuning on manipulation data may lead to catastrophic forgetting of multimodal reasoning.

The distinction from the original benchmark is explicit. A concise summary given in the source material is: SimplerEnv asks, “Can the model execute known manipulation skills under standard atomic commands?”, whereas SimplerEnv-Instruct asks, “Can the model use those skills in a closed loop when commands are linguistically novel, implicit, indirect, or require reasoning?” This framing makes the benchmark less a manipulation-fidelity stress test than a probe of instruction-grounded generalization.

2. Benchmark composition and task taxonomy

The benchmark contains 80 manually designed zero-shot manipulation tasks and 1.1K total trials. It is organized into two hierarchical levels: Instruction Aggregation, comprising 50 tasks, and Situated Reasoning, comprising 30 tasks (Yang et al., 23 Jul 2025).

Instruction Aggregation targets command diversity. The paper states that these tasks include new verbs, multilingual expressions, object references, sentence rephrasing, and novel objects. Appendix examples further indicate OCR-like references, shape- or attribute-based object identification, negation, paraphrases of familiar actions, and language-only OOD phrasing. The point is to test whether the policy can map varied natural language onto familiar manipulation primitives.

Situated Reasoning targets implicit or under-specified instructions. The paper’s example—“I want to clean the table. Pick a suitable tool for me.”—requires the model to identify the appropriate object rather than merely parse a direct command. These tasks incorporate subtask identification, where each subgoal depends on both the instruction and the environment state. The benchmark description gives the case of reasoning about preconditions such as whether a drawer is open before retrieving an object from it.

The benchmark was manually designed according to two explicit principles. The first is transfer of in-domain manipulation skills to novel scenarios: the authors filtered for basic tasks and objects so that successful behavior requires the correct intended skill, and then annotated novel instructions to increase difficulty. The second is human interpretability: instructions were checked by humans for clarity and naturalness, with cross-checks among human annotators.

The OOD composition is also quantified. Exactly 50.0% of tasks include OOD objects, 62.5% include OOD environments, 35.0% include distractor objects, and only 5.0% are language-only OOD. This is important because the benchmark is not merely a language paraphrase suite; it combines language novelty with visual and environmental novelty to prevent models from ignoring the instruction.

3. Observation model, control setting, and evaluation protocol

SimplerEnv-Instruct inherits the InstructVLA evaluation setting: a single image observation plus a language instruction is given as input to the VLM, and the task is manipulation in a closed-loop control setting. The paper repeatedly notes that the current implementation uses only a single image and an instruction. No depth is used. Robot state is optional in some variants and is denoted by “(S.)” (Yang et al., 23 Jul 2025).

For action generation, the VLM produces textual outputs and then latent action representations. Those latent actions condition a flow-matching action expert, which predicts an action chunk in $\mathbb{R}^{H \times 7}$, with the action dimension including the gripper. Appendix details specify the action chunk as $\mathbf{A} \in \mathbb{R}^{H \times 7}$, the prediction horizon used by the action head as $N=16$, and the fact that only one step is executed during inference, despite multi-step prediction.

The closed-loop aspect is central. The benchmark is described as requiring closed-loop manipulation, and inference in InstructVLA is explicitly designed for this regime. Textual reasoning can be generated autoregressively; latent actions are then decoded; language outputs and latent actions can be cached across action steps because of temporal stability; and the VLM and action expert can be queried at different frequencies. The appendix reports that performance remains stable when VLM:expert inference is 1:2, suggesting that latent actions provide stable guidance in closed loop.

Results on SimplerEnv-Instruct are reported for Instruction Aggregation, Situated Reasoning, and Average, rather than split by embodiment. The evaluation is zero-shot. The paper explicitly describes it as a manually designed evaluation suite featuring 80 zero-shot manipulation tasks. For most baselines and for InstructVLA, evaluation uses temperature 0, greedy search, and no sampling to speed generation. Magma is treated separately because the authors state that sampling materially affects results.

4. Baselines and main quantitative findings

The benchmark includes VLA and action baselines such as OpenVLA-7B, OpenVLA (FT) 7B, Magma-8B, Magma-8B†, the InstructVLA Stage-1 action Expert, and OpenVLA (FT + GPT-4o), alongside InstructVLA’s Generalist variants. A particularly important auxiliary baseline prompts GPT-4o with the first image and instruction, asks it to rewrite the prompt into a concise atomic command, and calls it only at the initial step (Yang et al., 23 Jul 2025).

Method	Avg	Category scores
OpenVLA-7B	14.2	14.8 / 13.6
Expert	15.6	20.8 / 10.4
Magma-8B† (sampling)	23.8	26.2 / 21.4
OpenVLA (FT) 7B	23.9	28.3 / 19.5
OpenVLA (FT + GPT-4o)	35.6	38.8 / 32.4
Generalist	46.0	43.3 / 48.8
Generalist (S.)	46.1	49.5 / 42.6

The headline comparisons reported in the paper derive from these numbers. Comparing Generalist with OpenVLA (FT), the average is 46.0 versus 23.9, corresponding to a relative improvement of approximately 92%. Comparing Generalist with OpenVLA (FT + GPT-4o), the average is 46.0 versus 35.6, corresponding to approximately 29.2%, rounded in the paper as 29% or 29.5%. The gap is especially pronounced on Situated Reasoning, where Generalist scores 48.8 and OpenVLA (FT) scores 19.5. The paper treats this as evidence that preserving vision-language reasoning matters particularly for situated tasks.

The Expert baseline is also instructive. Although action pretraining alone yields 15.6 average, the second-stage instruction tuning raises performance to 46.0 for Generalist. The subcategory pattern changes as well: Expert scores 20.8 on Instruction Aggregation but only 10.4 on Situated Reasoning, whereas Generalist becomes strong on both and is especially strong on Situated Reasoning.

Several ablations reinforce the interpretation that the gain is not simply due to more data. In the appendix, OpenVLA + VL rises from 14.2 to 23.9 average, but adding VLA-IT on top gives 24.0 and actually reduces Situated Reasoning from 19.5 to 17.4. By contrast, InstructVLA reaches 46.0. The paper also reports that scaling VLA-IT annotations from 25% to 100% yields a logarithmic improvement in instruction-following accuracy on SimplerEnv-Instruct, with situated reasoning benefiting more from more annotation than instruction aggregation.

The test-time “thinking” ablation is another central finding. The paper states that enabling thinking yields a further 36.1% performance gain over direct instruction execution on SimplerEnv-Instruct, and that it surpasses InstructVLA-Expert paired with GPT-4o as an external interpreter. This is presented as evidence that textual reasoning functions as useful internal computation rather than decorative supervision.

5. Relation to the InstructVLA training paradigm

SimplerEnv-Instruct is the benchmark that most directly tests the central methodological contributions of InstructVLA. The model is trained in two stages. Stage 1 is action pretraining: the model is trained on manipulation data to build an action expert aligned to latent action embeddings from the VLM, using manipulation datasets like RT-1 and Bridge, language motion supervision, flow matching for action generation, and cross-entropy for language motion text. The combined objective is

$\mathcal L = \mathcal{L}_{LM}+\mathcal L_{FM}.$

Only action query embeddings and action-related LoRA adapters are trained in this stage, and the resulting model is the Expert (Yang et al., 23 Jul 2025).

Stage 2 is Vision-Language-Action Instruction Tuning. The action expert is frozen; a new language LoRA adapter, a scale head, and the MoE adaptation module are added; and only the MoE-related parts are trained, amounting to about 220M parameters. Training uses multimodal datasets, manipulation datasets, and the curated 650K-sample VLA-IT corpus. The resulting model is the Generalist.

A major architectural element is the Mixture-of-Experts adaptation inside the LLM/VLM backbone, implemented via multiple LoRA experts. The paper gives the routing equation as

$$h = W_0x + \sum_{i=0}^{K} B_i A_i x \cdot \alpha_i \cdot \lambda_i,$$

where $W_0$ is the original pretrained weight, $A_i$ and $B_i$ are LoRA parameters, $\alpha_i$ is the LoRA scale, and $\lambda_i$ is the gating coefficient predicted by the scale head. In the benchmark context, this matters because the model must preserve pretrained VLM behavior while adapting differently in reasoning mode and action-generation mode.

Latent action extraction is likewise tied to benchmark performance. The model introduces $\mathbf{A} \in \mathbb{R}^{H \times 7}$0 learnable action queries

$\mathbf{A} \in \mathbb{R}^{H \times 7}$1

that attend to the VLM hidden states and extract latent action representations

$\mathbf{A} \in \mathbb{R}^{H \times 7}$2

Those latent actions are passed to a flow-matching action expert. The training loss is

$\mathbf{A} \in \mathbb{R}^{H \times 7}$3

with noisy action interpolation

$\mathbf{A} \in \mathbb{R}^{H \times 7}$4

Inference uses forward Euler integration with $\mathbf{A} \in \mathbb{R}^{H \times 7}$5 denoising steps. A plausible implication is that SimplerEnv-Instruct is designed to reward exactly this separation between language-side analysis and low-level action decoding.

The multimodal instruction tuning format makes that coupling explicit. The assistant response is trained to contain a natural-language response followed by latent action queries, matching the benchmark’s requirement that a model both understand the instruction and act on it.

6. Interpretation, broader context, and limitations

Several misconceptions are addressed by the benchmark design itself. SimplerEnv-Instruct is not merely a harder version of the original SimplerEnv because of more tasks; it specifically reorients evaluation toward closed-loop manipulation tasks and high-level instruction reasoning, including situated understanding and decomposition into actionable subtasks. It is also not merely a language paraphrase benchmark, because only 5.0% of tasks are language-only OOD, while OOD objects, OOD environments, and distractor objects are all substantial components (Yang et al., 23 Jul 2025).

The benchmark also bears a clear conceptual relation to the concerns raised in "Situated Instruction Following" (Min et al., 2024). That paper argues that realistic instruction following hinges on ambiguity, temporally evolving intent, and dynamic interpretation through the agent’s own actions. SimplerEnv-Instruct does not replicate the full human-centered formulation of that benchmark, but its emphasis on implicit or under-specified instructions, situated understanding, and decomposition into actionable subtasks suggests the same broader shift away from static, fully specified command execution and toward context-dependent embodied instruction following.

The present benchmark nonetheless has clear scope conditions. The current implementation uses only a single image and an instruction, with no depth. Robot state is optional and has almost no effect on average performance, although it changes the balance between Instruction Aggregation and Situated Reasoning. Results are reported on the two reasoning levels rather than by embodiment. The suite is manually designed rather than automatically generated from VLA-IT, even though it is tightly aligned with the capabilities that VLA-IT is intended to teach.

Finally, the benchmark’s baseline design highlights a substantive methodological point. OpenVLA (FT + GPT-4o) is the strongest non-InstructVLA SimplerEnv-Instruct baseline at 35.6 average, yet it remains well below Generalist at 46.0. The appendix attributes part of this gap to GPT-4o failures in embodied temporal grounding, physical grounding, coherence, and scene interpretation. The intended conclusion is not that external language assistance is useless, but that post hoc instruction rewriting is limited relative to end-to-end integration of reasoning and action. In that sense, SimplerEnv-Instruct functions as a benchmark for whether a robotic policy treats manipulation as instruction following grounded in multimodal reasoning, rather than as action prediction conditioned on short task strings.