---
title: 'InstructVLA: Unified Vision-Language-Action Model'
url: https://www.emergentmind.com/topics/instructvla
type: topic
---

# InstructVLA: Unified Vision-Language-Action Model

InstructVLA is an end-to-end vision-language-action (VLA) model designed to preserve the flexible reasoning of a pretrained vision-language model (VLM) while delivering leading manipulation performance. It introduces **Vision-Language-Action Instruction Tuning (VLA-IT)**, a training paradigm that treats language-steered action generation as part of instruction following rather than as isolated action prediction. The model is trained with multimodal corpora and a curated **650K-sample VLA-IT dataset**, and is reported to improve both manipulation and multimodal understanding while mitigating catastrophic forgetting of pre-trained vision-language capabilities [2507.17520].

## 1. Conceptual basis and problem setting

InstructVLA addresses a recurrent limitation in VLA research: existing systems often perform well either at free-form multimodal reasoning or at robotic action generation, but not both simultaneously. The motivating claim is that a practical robot must understand free-form human instructions, reason over the visual scene, plan the right subgoal or action, and execute precise control. In the paper’s framing, action-centric VLA models such as **OpenVLA, SpatialVLA, RT-2,** and **Octo** are strong at control but generally do **not preserve free-form reasoning** well, whereas reasoning-augmented embodied models such as **ECoT** and **Emma-X** inject reasoning but do not fully unify general VLM reasoning with action generation in a single flexible interface [2507.17520].

A central problem is **catastrophic forgetting**. When a pretrained VLM is finetuned into a VLA, it often loses the broad vision-language capabilities obtained from web-scale pretraining. InstructVLA is formulated as a direct response to that failure mode. Rather than reducing robotics to next-action prediction from narrow manipulation corpora, it reframes manipulation as a broader multimodal instruction-following problem in which textual reasoning can directly improve action generation at inference time [2507.17520].

The architecture therefore separates the system into two coupled components: a VLM backbone that can produce textual reasoning and latent intent, and an action expert that decodes those latent intents into robot actions. This preserves the distinction between high-level reasoning and low-level execution while keeping both within a single end-to-end model. The paper’s key claim is that this structure enables **textual reasoning to guide control**, instead of forcing reasoning to collapse into direct action prediction [2507.17520].

## 2. Vision-Language-Action Instruction Tuning

The defining training paradigm is **VLA-IT**, a two-stage procedure consisting of **Action Pretraining** followed by **Vision-Language-Action Instruction Tuning**. In the first stage, the model is pretrained on heterogeneous manipulation data using both **actions** and **language motion annotations**. The stated purpose is to align the VLM’s internal latent action queries with low-level robot motion primitives. The training loss is

$$
\mathcal L = \mathcal{L}_{LM}+\mathcal L_{FM},
$$

where $\mathcal{L}_{LM}$ is the language modeling or next-token loss on motion descriptions and $\mathcal{L}_{FM}$ is the flow matching loss for action generation. During this stage, only the action-query embeddings and the action LoRA adapter on the LLM backbone are tuned, totaling about **650M parameters**. The resulting model is termed the **Expert** [2507.17520].

The second stage is the main contribution. The pretrained action expert is frozen, and the system adds a new language LoRA adapter and a scale head for **mixture-of-experts (MoE)** gating. Only the MoE adaptation modules are trainable in this stage, about **220M parameters**. Training jointly mixes **standard multimodal VLM corpora**, **manipulation datasets**, and the **curated VLA-IT dataset**. The product of this stage is the **Generalist** model [2507.17520].

This staged design is intended to preserve broad multimodal competence while making it useful for control. The paper explicitly argues that robot manipulation should not be trained in isolation. Mixing standard VLM corpora retains OCR, visual grounding, commonsense reasoning, captioning, and QA skills, while the manipulation corpora align those retained semantics with action generation. A plausible implication is that VLA-IT uses curriculum structure to reduce interference between general VLM behavior and control-specific adaptation [2507.17520].

## 3. Architecture, adaptation mechanism, and action generation

InstructVLA uses a **single VLM backbone** plus a separate **flow-based action expert**. The named architectural components are **Eagle2-2B** as the VLM backbone, **learnable action queries**, **MoE LoRA adapters**, a **DINOv2 visual encoder** inside the action expert, **FiLM modulation** to condition visual features on latent actions, and a **flow-matching decoder** to produce final actions [2507.17520].

The MoE adaptation mechanism is given by

$$
h = W_0x + \sum_{i=0}^{K} B_i A_i x \cdot \alpha_i \cdot \lambda_i,
$$

where $x$ is the input hidden state, $W_0$ the original backbone weight, $A_i$ and $B_i$ the LoRA down- and up-projections for expert $i$, $\alpha_i$ the LoRA scaling factor, and $\lambda_i$ the gating coefficient predicted by the scale head. This mechanism allows the model to emphasize reasoning experts during language or multimodal understanding and action experts during control generation, without switching to a separate network [2507.17520].

Action generation is learned with **flow matching** over action chunks $\mathbf{A} \in \mathbb{R}^{H \times 7}$, using

$$
\mathcal{L}_{FM} = \mathbb{E}\left[\left\| V_{\theta}(\mathbf{A}^{\tau}, q_t) - (\epsilon - \mathbf{A}) \right\|^2\right],
$$

where $\tau \in [0,1)$, $\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$, $\mathbf{A}^{\tau} = \tau \mathbf{A} + (1-\tau)\epsilon$, and $q_t$ is the conditioning signal from DINOv2 image features and latent action $C$. At inference,

$$
\mathbf{A}^{\tau + 1/N} = \mathbf{A}^{\tau} + \frac{1}{N} V_{\theta}(\mathbf{A}^{\tau}, q_t),
$$

with **$N=10$ denoising steps** [2507.17520].

The generation pipeline is described in three stages: **asynchronous autoregressive reasoning** by the VLM, **latent action generation**, and **action decoding** by the flow expert. Several inference-time mechanisms are introduced to make textual reasoning operational for control: **think-before-act decoding**, **language response caching**, **latent action caching**, **dual-frequency inference**, and **test-time thinking**. The paper reports a **36.1% gain** from enabling thinking in the generalist model over direct instruction execution, especially on situated reasoning tasks [2507.17520].

## 4. The VLA-IT dataset and annotation strategy

The **VLA-IT dataset** contains **650K human-robot interaction samples** with rich annotations. It is built from large-scale manipulation datasets such as **BridgeData** and **RT-1/Fractal-like data**. The dataset is designed to address the claim that existing robot datasets are overly templated, with simple commands, repetitive verbs, and narrow vocabulary [2507.17520].

The dataset includes four annotation types:

| Annotation type | Function |
|---|---|
| Scenario captioning | Describes the scene in natural language |
| Question answering | Reinforces visual understanding |
| Command rewriting | Adds diverse paraphrases |
| Context creation | Generates implicit or long-horizon user intents |

The **command rewriting** component includes synonyms, attribute-based references, relational expressions, multilingual expressions, and negation. The **context creation** component is presented as especially important because it forces the model to infer missing intent from context. Taken together, these annotation types are intended to add linguistic diversity, situated reasoning, instruction paraphrases, commonsense QA, and grounded captions [2507.17520].

The annotation pipeline uses **GPT-4o**, but with a notable constraint: GPT-4o is given the **ground-truth instruction** from a human expert during annotation. The paper reports **95.4% with ground-truth instruction** versus **45.0% without ground-truth instruction**. Without the true instruction, GPT-4o often ignores vision context, reverses temporal order, and hallucinates objects. This is important because it shows that the dataset is not a purely automatic synthetic corpus; its accuracy depends on human-guided instruction grounding [2507.17520].

A plausible implication is that the dataset functions not only as additional supervision for action generation but also as an anti-forgetting mechanism: it re-exposes the model to the kinds of captions, questions, paraphrases, and contextual descriptions that large VLMs handle well before robot-specific tuning.

## 5. Empirical performance and ablations

The evaluation spans **multimodal benchmarks**, **SimplerEnv**, **SimplerEnv-Instruct**, and **real-world experiments**. On in-domain SimplerEnv tasks, InstructVLA achieves a **30.5% improvement over SpatialVLA**. On **SimplerEnv-Instruct**, a benchmark introduced by the paper with **80 tasks** and **1.1K trials**, it reports a **92% improvement over fine-tuned OpenVLA** and a **29% improvement over GPT-4o-assisted action expert** [2507.17520].

The manipulation results also distinguish between variants. On SimplerEnv in-domain settings, the paper reports averages of **52.9** for **Expert**, **49.4** for **Generalist**, and **55.4** for **Generalist(S.)**. On SimplerEnv-Instruct, the corresponding numbers are **15.6**, **46.0**, and **46.1**. This is used to support a central claim of the paper: the generalist configuration substantially improves reasoning-heavy instruction following even when a specialist expert can remain competitive on more atomic control [2507.17520].

On multimodal benchmarks, the generalist model reportedly outperforms the co-trained baseline **Magma**, the base VLM **Eagle2**, and **Bunny** across several tasks. Reported examples include **44.8 on MMMU**, compared with **38.8 for Magma** and **43.1 for Eagle2**, and **64.4 on RWQA**, compared with **56.5 for Magma**. The paper therefore presents InstructVLA not as a trade-off that simply sacrifices general VLM ability for control, but as a system that can improve both domains under the proposed co-training regime [2507.17520].

The real-world results are reported as a **23.3% improvement** over OpenVLA in direct instruction, **41.7% improvement** in few-shot reasoning tasks, and **46.7% improvement** in zero-shot reasoning tasks. Key ablations further support the architectural choices: low-level language motion labels improve success by **10.5%**; a latent action query count of **64** is described as a good trade-off; removing **DINOv2** causes a **50% performance drop**; adding **FiLM** gives an additional **15.3% improvement**; and inference-time thinking particularly helps tool use, commonsense grounding, and subtask identification [2507.17520].

The paper also states an explicit reasoning-control trade-off. Preserving language ability can slightly reduce pure atomic manipulation relative to a specialist expert in some settings, but the **generalist model** gains robustness and instruction-following power. This framing is significant because it treats multimodal competence not as incidental overhead but as a directly useful component of embodied control [2507.17520].

## 6. Limitations, interpretations, and later developments

The paper explicitly notes several limitations: current experiments use only **single-image input** and **instruction text**; no depth or tactile sensing is used; some failures are due to **real-to-sim gaps** and depth estimation issues; and the system still struggles more with action execution than with language understanding. Future work is stated to require richer sensory modalities and broader real-world deployment [2507.17520].

Later works position themselves relative to this instruction-following VLA paradigm in distinct ways. A lightweight adaptation of OpenVLA via synthetic instruction augmentation is described as **InstructVLA-style tuning**: instead of redesigning the architecture, it synthesizes diverse paraphrases for BridgeData V2 trajectories with an LLM and applies LoRA to improve linguistic generalization. In that study, the proposed LoRA plus synthetic instruction augmentation changes performance from **6.62%** to **5.09%** in Top-1 accuracy while improving **5-Bin accuracy** from **40.76%** to **42.47%**, suggesting greater robustness to semantically broad instruction variation at the cost of exact token matching [2603.16044].

A different response appears in **VP-VLA**, which is introduced partly as a reaction to the weaknesses of instruction-following VLA systems such as **InstructVLA-style end-to-end policies**. VP-VLA argues that the “black-box” mapping from language and observations directly to actions can underperform on spatial precision and out-of-distribution robustness, and proposes instead a dual-system framework with a **System 2 Planner**, a **System 1 Controller**, and a structured visual prompting interface. It reports improvements of **5.0%** on RoboCasa-GR1-Tabletop and **8.3%** on SimplerEnv over QwenOFT, emphasizing explicit grounding and phase decomposition [2603.22003].

In autonomous driving, **Vega** extends the broader instruction-following VLA logic into a different embodiment regime. It introduces **InstructScene**, with around **100,000 instruction-annotated scenes**, and a unified **Vision-Language-World-Action** model that combines autoregressive multimodal understanding with diffusion-based future prediction and trajectory generation. Although this is not presented as an InstructVLA derivative, a plausible implication is that it shares the same underlying shift: actions are not learned as isolated low-dimensional outputs, but as consequences of instruction-conditioned reasoning tied to richer supervision—in Vega’s case, **future image prediction** and **trajectory generation** [2603.25741].

Taken together, these later developments suggest that InstructVLA is best understood not merely as a single model but as a reference point in a broader research transition. That transition treats embodied control as an instruction-following problem, seeks to preserve pretrained VLM semantics during robot tuning, and explores multiple interfaces between reasoning and action: latent MoE adaptation, synthetic instruction augmentation, structured visual prompting, and joint world modeling [2507.17520].

Source: https://www.emergentmind.com/topics/instructvla