InstructVLA: Unified Vision-Language-Action Model

• InstructVLA is an end-to-end vision-language-action model that unifies pretrained VLM reasoning with high-performance robotic manipulation.
• It utilizes a novel two-stage VLA-IT training paradigm that combines action pretraining with instruction tuning using language and flow matching losses.
• Extensive evaluations demonstrate significant improvements, reducing catastrophic forgetting and boosting task performance by up to 36%.

InstructVLA is an end-to-end vision-language-action (VLA) model designed to preserve the flexible reasoning of a pretrained vision-LLM (VLM) while delivering leading manipulation performance. It introduces Vision-Language-Action Instruction Tuning (VLA-IT), a training paradigm that treats language-steered action generation as part of instruction following rather than as isolated action prediction. The model is trained with multimodal corpora and a curated 650K-sample VLA-IT dataset, and is reported to improve both manipulation and multimodal understanding while mitigating catastrophic forgetting of pre-trained vision-language capabilities (Yang et al., 23 Jul 2025).

1. Conceptual basis and problem setting

InstructVLA addresses a recurrent limitation in VLA research: existing systems often perform well either at free-form multimodal reasoning or at robotic action generation, but not both simultaneously. The motivating claim is that a practical robot must understand free-form human instructions, reason over the visual scene, plan the right subgoal or action, and execute precise control. In the paper’s framing, action-centric VLA models such as OpenVLA, SpatialVLA, RT-2, and Octo are strong at control but generally do not preserve free-form reasoning well, whereas reasoning-augmented embodied models such as ECoT and Emma-X inject reasoning but do not fully unify general VLM reasoning with action generation in a single flexible interface (Yang et al., 23 Jul 2025).

A central problem is catastrophic forgetting. When a pretrained VLM is finetuned into a VLA, it often loses the broad vision-language capabilities obtained from web-scale pretraining. InstructVLA is formulated as a direct response to that failure mode. Rather than reducing robotics to next-action prediction from narrow manipulation corpora, it reframes manipulation as a broader multimodal instruction-following problem in which textual reasoning can directly improve action generation at inference time (Yang et al., 23 Jul 2025).

The architecture therefore separates the system into two coupled components: a VLM backbone that can produce textual reasoning and latent intent, and an action expert that decodes those latent intents into robot actions. This preserves the distinction between high-level reasoning and low-level execution while keeping both within a single end-to-end model. The paper’s key claim is that this structure enables textual reasoning to guide control, instead of forcing reasoning to collapse into direct action prediction (Yang et al., 23 Jul 2025).

2. Vision-Language-Action Instruction Tuning

The defining training paradigm is VLA-IT, a two-stage procedure consisting of Action Pretraining followed by Vision-Language-Action Instruction Tuning. In the first stage, the model is pretrained on heterogeneous manipulation data using both actions and language motion annotations. The stated purpose is to align the VLM’s internal latent action queries with low-level robot motion primitives. The training loss is

$\mathcal L = \mathcal{L}_{LM}+\mathcal L_{FM},$

where $\mathcal{L}_{LM}$ is the language modeling or next-token loss on motion descriptions and $\mathcal{L}_{FM}$ is the flow matching loss for action generation. During this stage, only the action-query embeddings and the action LoRA adapter on the LLM backbone are tuned, totaling about 650M parameters. The resulting model is termed the Expert (Yang et al., 23 Jul 2025).

The second stage is the main contribution. The pretrained action expert is frozen, and the system adds a new language LoRA adapter and a scale head for mixture-of-experts (MoE) gating. Only the MoE adaptation modules are trainable in this stage, about 220M parameters. Training jointly mixes standard multimodal VLM corpora, manipulation datasets, and the curated VLA-IT dataset. The product of this stage is the Generalist model (Yang et al., 23 Jul 2025).

This staged design is intended to preserve broad multimodal competence while making it useful for control. The paper explicitly argues that robot manipulation should not be trained in isolation. Mixing standard VLM corpora retains OCR, visual grounding, commonsense reasoning, captioning, and QA skills, while the manipulation corpora align those retained semantics with action generation. A plausible implication is that VLA-IT uses curriculum structure to reduce interference between general VLM behavior and control-specific adaptation (Yang et al., 23 Jul 2025).

3. Architecture, adaptation mechanism, and action generation

InstructVLA uses a single VLM backbone plus a separate flow-based action expert. The named architectural components are Eagle2-2B as the VLM backbone, learnable action queries, MoE LoRA adapters, a DINOv2 visual encoder inside the action expert, FiLM modulation to condition visual features on latent actions, and a flow-matching decoder to produce final actions (Yang et al., 23 Jul 2025).

The MoE adaptation mechanism is given by

$$h = W_0x + \sum_{i=0}^{K} B_i A_i x \cdot \alpha_i \cdot \lambda_i,$$

where $x$ is the input hidden state, $W_0$ the original backbone weight, $A_i$ and $B_i$ the LoRA down- and up-projections for expert $i$, $\alpha_i$ the LoRA scaling factor, and $\mathcal{L}_{LM}$0 the gating coefficient predicted by the scale head. This mechanism allows the model to emphasize reasoning experts during language or multimodal understanding and action experts during control generation, without switching to a separate network (Yang et al., 23 Jul 2025).

Action generation is learned with flow matching over action chunks $\mathcal{L}_{LM}$1, using

$\mathcal{L}_{LM}$2

where $\mathcal{L}_{LM}$3, $\mathcal{L}_{LM}$4, $\mathcal{L}_{LM}$5, and $\mathcal{L}_{LM}$6 is the conditioning signal from DINOv2 image features and latent action $\mathcal{L}_{LM}$7. At inference,

$\mathcal{L}_{LM}$8

with $\mathcal{L}_{LM}$9 denoising steps (Yang et al., 23 Jul 2025).

The generation pipeline is described in three stages: asynchronous autoregressive reasoning by the VLM, latent action generation, and action decoding by the flow expert. Several inference-time mechanisms are introduced to make textual reasoning operational for control: think-before-act decoding, language response caching, latent action caching, dual-frequency inference, and test-time thinking. The paper reports a 36.1% gain from enabling thinking in the generalist model over direct instruction execution, especially on situated reasoning tasks (Yang et al., 23 Jul 2025).

4. The VLA-IT dataset and annotation strategy

The VLA-IT dataset contains 650K human-robot interaction samples with rich annotations. It is built from large-scale manipulation datasets such as BridgeData and RT-1/Fractal-like data. The dataset is designed to address the claim that existing robot datasets are overly templated, with simple commands, repetitive verbs, and narrow vocabulary (Yang et al., 23 Jul 2025).

The dataset includes four annotation types:

Annotation type	Function
Scenario captioning	Describes the scene in natural language
Question answering	Reinforces visual understanding
Command rewriting	Adds diverse paraphrases
Context creation	Generates implicit or long-horizon user intents

The command rewriting component includes synonyms, attribute-based references, relational expressions, multilingual expressions, and negation. The context creation component is presented as especially important because it forces the model to infer missing intent from context. Taken together, these annotation types are intended to add linguistic diversity, situated reasoning, instruction paraphrases, commonsense QA, and grounded captions (Yang et al., 23 Jul 2025).

The annotation pipeline uses GPT-4o, but with a notable constraint: GPT-4o is given the ground-truth instruction from a human expert during annotation. The paper reports 95.4% with ground-truth instruction versus 45.0% without ground-truth instruction. Without the true instruction, GPT-4o often ignores vision context, reverses temporal order, and hallucinates objects. This is important because it shows that the dataset is not a purely automatic synthetic corpus; its accuracy depends on human-guided instruction grounding (Yang et al., 23 Jul 2025).

A plausible implication is that the dataset functions not only as additional supervision for action generation but also as an anti-forgetting mechanism: it re-exposes the model to the kinds of captions, questions, paraphrases, and contextual descriptions that large VLMs handle well before robot-specific tuning.

5. Empirical performance and ablations

The evaluation spans multimodal benchmarks, SimplerEnv, SimplerEnv-Instruct, and real-world experiments. On in-domain SimplerEnv tasks, InstructVLA achieves a 30.5% improvement over SpatialVLA. On SimplerEnv-Instruct, a benchmark introduced by the paper with 80 tasks and 1.1K trials, it reports a 92% improvement over fine-tuned OpenVLA and a 29% improvement over GPT-4o-assisted action expert (Yang et al., 23 Jul 2025).

The manipulation results also distinguish between variants. On SimplerEnv in-domain settings, the paper reports averages of 52.9 for Expert, 49.4 for Generalist, and 55.4 for Generalist(S.). On SimplerEnv-Instruct, the corresponding numbers are 15.6, 46.0, and 46.1. This is used to support a central claim of the paper: the generalist configuration substantially improves reasoning-heavy instruction following even when a specialist expert can remain competitive on more atomic control (Yang et al., 23 Jul 2025).

On multimodal benchmarks, the generalist model reportedly outperforms the co-trained baseline Magma, the base VLM Eagle2, and Bunny across several tasks. Reported examples include 44.8 on MMMU, compared with 38.8 for Magma and 43.1 for Eagle2, and 64.4 on RWQA, compared with 56.5 for Magma. The paper therefore presents InstructVLA not as a trade-off that simply sacrifices general VLM ability for control, but as a system that can improve both domains under the proposed co-training regime (Yang et al., 23 Jul 2025).

The real-world results are reported as a 23.3% improvement over OpenVLA in direct instruction, 41.7% improvement in few-shot reasoning tasks, and 46.7% improvement in zero-shot reasoning tasks. Key ablations further support the architectural choices: low-level language motion labels improve success by 10.5%; a latent action query count of 64 is described as a good trade-off; removing DINOv2 causes a 50% performance drop; adding FiLM gives an additional 15.3% improvement; and inference-time thinking particularly helps tool use, commonsense grounding, and subtask identification (Yang et al., 23 Jul 2025).

The paper also states an explicit reasoning-control trade-off. Preserving language ability can slightly reduce pure atomic manipulation relative to a specialist expert in some settings, but the generalist model gains robustness and instruction-following power. This framing is significant because it treats multimodal competence not as incidental overhead but as a directly useful component of embodied control (Yang et al., 23 Jul 2025).

6. Limitations, interpretations, and later developments

The paper explicitly notes several limitations: current experiments use only single-image input and instruction text; no depth or tactile sensing is used; some failures are due to real-to-sim gaps and depth estimation issues; and the system still struggles more with action execution than with language understanding. Future work is stated to require richer sensory modalities and broader real-world deployment (Yang et al., 23 Jul 2025).

Later works position themselves relative to this instruction-following VLA paradigm in distinct ways. A lightweight adaptation of OpenVLA via synthetic instruction augmentation is described as InstructVLA-style tuning: instead of redesigning the architecture, it synthesizes diverse paraphrases for BridgeData V2 trajectories with an LLM and applies LoRA to improve linguistic generalization. In that study, the proposed LoRA plus synthetic instruction augmentation changes performance from 6.62% to 5.09% in Top-1 accuracy while improving 5-Bin accuracy from 40.76% to 42.47%, suggesting greater robustness to semantically broad instruction variation at the cost of exact token matching (Shin, 17 Mar 2026).

A different response appears in VP-VLA, which is introduced partly as a reaction to the weaknesses of instruction-following VLA systems such as InstructVLA-style end-to-end policies. VP-VLA argues that the “black-box” mapping from language and observations directly to actions can underperform on spatial precision and out-of-distribution robustness, and proposes instead a dual-system framework with a System 2 Planner, a System 1 Controller, and a structured visual prompting interface. It reports improvements of 5.0% on RoboCasa-GR1-Tabletop and 8.3% on SimplerEnv over QwenOFT, emphasizing explicit grounding and phase decomposition (Wang et al., 23 Mar 2026).

In autonomous driving, Vega extends the broader instruction-following VLA logic into a different embodiment regime. It introduces InstructScene, with around 100,000 instruction-annotated scenes, and a unified Vision-Language-World-Action model that combines autoregressive multimodal understanding with diffusion-based future prediction and trajectory generation. Although this is not presented as an InstructVLA derivative, a plausible implication is that it shares the same underlying shift: actions are not learned as isolated low-dimensional outputs, but as consequences of instruction-conditioned reasoning tied to richer supervision—in Vega’s case, future image prediction and trajectory generation (Zuo et al., 26 Mar 2026).

Taken together, these later developments suggest that InstructVLA is best understood not merely as a single model but as a reference point in a broader research transition. That transition treats embodied control as an instruction-following problem, seeks to preserve pretrained VLM semantics during robot tuning, and explores multiple interfaces between reasoning and action: latent MoE adaptation, synthetic instruction augmentation, structured visual prompting, and joint world modeling (Yang et al., 23 Jul 2025).