VLA-IT: Vision-Language-Action Tuning

• VLA-IT is a family of methods that align visual, language, and action outputs to enhance robotic instruction-following under diverse, out-of-distribution inputs.
• These techniques leverage synthetic instruction augmentation, multimodal parameter-efficient tuning, and flatness-preserving optimization to mitigate issues like instruction blindness and catastrophic forgetting.
• VLA-IT systems utilize varied interfaces such as text, speech, and interleaved modalities to preserve pretrained multimodal competence while improving action prediction robustness.

Vision-Language-Action Instruction Tuning (VLA-IT) denotes a family of post-pretraining adaptation procedures for embodied policies that map visual observations and task instructions to robot actions. In its narrowest usage, the term refers to fine-tuning OpenVLA with synthetically augmented instructions in order to improve linguistic generalization on Bridge Dataset V2 (Shin, 17 Mar 2026). Across related work, however, the same label also covers instruction-tuning regimes that broaden the instruction channel to speech or interleaved image-text inputs, preserve instruction adherence during limited-data finetuning, strengthen visual grounding, or reformulate low-level actions as language-like outputs (Zhao et al., 19 Feb 2025, Fan et al., 4 May 2025, Zhang et al., 22 Jun 2026, Chen et al., 4 Feb 2026, Hancock et al., 26 Sep 2025, Yang et al., 23 Jul 2025). The unifying objective is not a single architecture, but more reliable conditional control under variation in language, scene content, and task specification.

1. Problem formulation and conceptual scope

Recent VLA-IT work begins from a shared diagnosis: large pretrained vision-language or vision-language-action models can execute many manipulation skills, yet they often degrade when instructions depart from narrow finetuning distributions. In the OpenVLA synthetic-instruction setting, the core issue is brittleness to linguistic variation despite strong zero-shot physical task execution (Shin, 17 Mar 2026). In flatness-preserving finetuning, the same failure is named instruction blindness, formalized by the approximate invariance $\pi(a\mid o,l')\approx \pi(a\mid o,l)$ for semantically distinct instructions $l'\neq l$ after limited-data finetuning (Zhang et al., 22 Jun 2026). Other papers emphasize adjacent but non-identical pathologies: catastrophic forgetting caused by distribution mismatch between internet-scale pretraining and robotics finetuning (Hancock et al., 26 Sep 2025), and vision-action misalignment in which action predictions exhibit weak dependence on the current visual state (Chen et al., 4 Feb 2026).

This yields a broader interpretation of VLA-IT than paraphrase augmentation alone. The literature uses the term for procedures that modify the instruction distribution, the optimization objective, the parameterization of actions, or the modality mix seen during finetuning. A recurring theme is that instruction following in embodied control is not reducible to language modeling in isolation; it depends on preserving multimodal representations while adapting them to small, action-labeled robot datasets (Yang et al., 23 Jul 2025).

A useful synthesis is that VLA-IT targets the conditional structure $p(a\mid o,l)$, but different papers intervene at different points in that conditional. Some enlarge or diversify $l$; some regularize the adaptation of the parameters governing $p$; some change the representation of $a$ so that action prediction is less distributionally alien to the pretrained backbone. This suggests that VLA-IT is best understood as a family of alignment strategies between instruction semantics and control generation rather than a single finetuning recipe.

2. Architectural substrates and action interfaces

Most VLA-IT systems are built on pretrained multimodal backbones and differ mainly in how instructions and actions are represented at finetuning time. OpenVLA, which anchors several of these studies, combines a SigLIP vision encoder with a Llama-2 backbone whose vocabulary is extended by replacing its 256 least frequent tokens with discrete action tokens; pretraining on the Open X-Embodiment dataset (approximately 900 K trajectories) minimizes cross-entropy over discrete action-token prediction conditioned on image frames and language prompts (Shin, 17 Mar 2026). By contrast, VLAS integrates raw speech into the policy itself: log-mel speech features are encoded by Whisper, RGB observations by CLIP-ViT, both are projected into language space, and an autoregressive decoder predicts actions conditioned on speech, retrieved text, and vision (Zhao et al., 19 Feb 2025).

Interleave-VLA changes the instruction interface rather than the core transformer. It allows the prompt to be an ordered sequence of alternating text and image tokens, with text segments embedded by a language encoder and instruction images by a vision encoder; the resulting interleaved sequence is concatenated with observation embeddings and decoded into continuous actions (Fan et al., 4 May 2025). VLM2VLA moves further by eliminating special action vocabularies entirely: subtasks, motion plans, and low-level action chunks are all expressed as natural-language strings, so training and inference remain in the same token space as ordinary VLM reasoning (Hancock et al., 26 Sep 2025). InstructVLA introduces a two-headed design in which a pretrained VLM retains an autoregressive language head while a latent-action head extracts query-conditioned intent embeddings that drive a separately parameterized flow-matching action expert (Yang et al., 23 Jul 2025).

System	Instruction/action interface	Core architectural move
OpenVLA + synthetic instruction augmentation (Shin, 17 Mar 2026)	Text instructions; discrete action tokens	SigLIP + Llama-2; 256 least frequent tokens replaced by action tokens
VLAS (Zhao et al., 19 Feb 2025)	Speech, retrieved text, RGB	Whisper and CLIP-ViT projected into language space
Interleave-VLA (Fan et al., 4 May 2025)	Alternating text and image tokens	Interleaved multimodal prompt with minimal backbone changes
VLM2VLA (Hancock et al., 26 Sep 2025)	Subtasks, motion plans, actions as language	Natural-language action representation on Gemma-3-12B-IT
InstructVLA (Yang et al., 23 Jul 2025)	Free-form reasoning text plus latent actions	LoRA-MoE VLM backbone plus frozen flow-matching action expert

The architectural diversity reflects different answers to the same question: how much of the pretrained multimodal substrate should be preserved, and how should action generation be made compatible with it? Systems such as VLM2VLA and InstructVLA explicitly prioritize retention of pretrained VLM capabilities, whereas OpenVLA-centered work more often starts from a VLA backbone and seeks to repair specific instruction-following or grounding failures during downstream adaptation (Hancock et al., 26 Sep 2025, Yang et al., 23 Jul 2025, Chen et al., 4 Feb 2026).

3. Data construction and instruction-space expansion

A major branch of VLA-IT is data-centric. Rather than collecting entirely new robot demonstrations, these methods relabel or augment existing trajectories so that the model encounters a broader instruction manifold. In the synthetic augmentation study, Bridge Dataset V2 originally contains 53,896 trajectories across 24 environments, but the reported experiment uses a curated subset of 100 scripted trajectories of approximately 25 frames each, yielding 2,500 image-action pairs (Shin, 17 Mar 2026). An LLM is prompted with three key frames and task metadata to “Synthesize exactly 5 distinct natural language instructions,” with varied sentence structures and vocabulary synonyms; manual curation retains one set of five high-quality paraphrases per trajectory, producing 500 unique instruction-trajectory sets and 12,500 triplets $(o,l,a)$ (Shin, 17 Mar 2026). During training, random pairing assigns one of the five instructions to the trajectory at each epoch, explicitly forcing decoupling between surface syntax and underlying intent.

Other VLA-IT datasets target richer instruction modalities. Interleave-VLA converts text-only instructions from the 11 constituent datasets of Open X-Embodiment into interleaved image-text prompts using a three-stage pipeline: keyword extraction with Qwen2.5, open-vocabulary detection with OWLv2, and verification or refinement with Qwen2.5-VL and Segment Anything (Fan et al., 4 May 2025). The resulting corpus contains 210,000 episodes, approximately 13 million frames, and 3,500 unique objects. InstructVLA constructs a 650 K-sample VLA-IT dataset from RT-1/Fractal and Bridge, with each annotation instance containing scenario captioning from three frames, question-answer pairs, command rewriting, and context creation; GPT-4o is prompted with the ground-truth instruction plus three key frames, followed by filtering and manual quality checks (Yang et al., 23 Jul 2025).

Speech-oriented VLA-IT requires yet another data curriculum. VLAS uses LibriSpeech-360 for Stage I speech alignment, SQA with 185K speech-image-text answer triples plus 665K LLaVA VQA samples and LibriSpeech-100 for Stage II multimodal reasoning, and CSI with 194K instruction-to-trajectory pairs for Stage III robot manipulation fine-tuning (Zhao et al., 19 Feb 2025). VLM2VLA likewise relies on data relabeling: Gemini 2.5 segments teleoperated Bridgev2 trajectories into subtasks, motion plans, and action chunks, then coalesces deltas smaller than 2.5 cm per dimension so that textual action descriptions remain perceptible (Hancock et al., 26 Sep 2025).

These pipelines indicate that instruction tuning in robotics is frequently a relabeling and representation problem. A plausible implication is that VLA-IT depends as much on the structure of the supervision signal as on the choice of optimizer: one can enlarge the set of admissible instructions, inject missing modalities, or bring action targets closer to the statistical form of the pretrained backbone without changing the underlying robot trajectories.

4. Optimization regimes and parameter-efficient adaptation

Despite their heterogeneity, VLA-IT methods often share a preference for parameter-efficient or regularized adaptation. In the OpenVLA synthetic-instruction setting, the base model is pretrained with action-token cross-entropy,

$\mathcal{L}_{\mathrm{vla}} = -\sum_{t=1}^T \sum_{c=1}^C \mathds{1}[a_t=c]\log P_\theta(a_t=c\mid o_{1:t},l),$

and downstream VLA-IT applies LoRA to every query, key, value, and output projection in the self-attention layers, using

$W = W_0 + \Delta W = W_0 + BA,$

with rank $r=32$ and scaling $l'\neq l$0; optimization uses AdamW with learning rate $l'\neq l$1 and weight decay $l'\neq l$2 on a single A100 40 GB GPU (Shin, 17 Mar 2026). The primary objective remains discrete cross-entropy over 256 action bins, with regularization applied through decoupled weight decay and no auxiliary loss terms (Shin, 17 Mar 2026).

Flatness-preserving VLA finetuning argues that limited-data instruction failures are fundamentally optimization failures. It replaces ordinary supervised updates with sharpness-aware minimization (SAM), solving the bilevel objective

$l'\neq l$3

and reports that global SAM over all parameters outperforms selective variants such as language-only or component-specific SAM (Zhang et al., 22 Jun 2026). The practical guideline reported for $l'\neq l$4 on LIBERO is AdamW with learning rate $l'\neq l$5, batch size 16, SAM radius $l'\neq l$6, approximately 30 K finetune steps, and weight decay $l'\neq l$7 (Zhang et al., 22 Jun 2026).

VISTA introduces a two-stage optimization strategy aimed at visual conditioning. Stage 1 performs Direct Preference Optimization on a surrogate track-following task built from BootsTAPNext tracks, using in-batch preferred and dispreferred action chunks. Stage 2 then returns to the original instruction-following data and combines standard supervised action loss with a latent distillation term matching student and teacher multimodal backbone outputs, with default $l'\neq l$8; LoRA adapters of rank 32 are applied throughout (Chen et al., 4 Feb 2026). InstructVLA pursues a different separation of concerns: Stage 1 jointly trains language-motion prediction and a flow-matching action expert, while Stage 2 freezes the action expert, adds LoRA adapters and a scale head, and interleaves standard VLM corpora with the 650K VLA-IT dataset under a 1:7 multimodal:manipulation ratio (Yang et al., 23 Jul 2025).

VLM2VLA occupies a distinctive point in this space. It trains only LoRA parameters on a frozen Gemma-3-12B-IT backbone and uses ordinary cross-entropy language modeling over concatenated subtask, motion-plan, and action-text sequences, explicitly reporting that no extra regularization term is used because the combination of data alignment and parameter efficiency is intended to prevent forgetting (Hancock et al., 26 Sep 2025). Taken together, these approaches show that VLA-IT optimization spans three broad strategies: preserve the backbone through low-rank updates, regularize the loss landscape to retain instruction sensitivity, or impose auxiliary alignment objectives that bias the model toward stronger visual or semantic grounding.

5. Evaluation protocols and reported empirical behavior

Evaluation in VLA-IT is notably fragmented. Some papers assess token-level action prediction, some measure robotic success rate on counterfactual instructions, and some track retention of generic multimodal reasoning. In the synthetic OpenVLA study, held-out Bridge V2 zero-shot evaluation uses Top-1 accuracy, defined as k-Bin accuracy with $l'\neq l$9, and 5-Bin tolerance accuracy with $p(a\mid o,l)$0 (Shin, 17 Mar 2026). The reported outcome is a decrease in Top-1 from 6.62% to 5.09% but an increase in 5-Bin accuracy from 40.76% to 42.47%, indicating greater tolerance-band robustness under linguistic variation despite reduced exact token matching (Shin, 17 Mar 2026).

Approach	Benchmark	Reported outcome
Synthetic-instruction LoRA (Shin, 17 Mar 2026)	Held-out Bridge V2	Top-1 6.62% → 5.09%; 5-Bin 40.76% → 42.47%
SAM finetuning (Zhang et al., 22 Jun 2026)	LIBERO-PRO / LangGap / LIBERO-CF	26.6% → 42.6%; 24.5% → 41.7%; 13.2% → 47.8% SR
Interleave-VLA (Fan et al., 4 May 2025)	SIMPLEREnv OOD average	39.7 → 63.4; with co-train 66.5
VLAS (Zhao et al., 19 Feb 2025)	CALVIN / customization benchmark	94.2% 1-step and 3.70 Len; 86.5% avg with RAG
VLM2VLA (Hancock et al., 26 Sep 2025)	Multilingual pick-up / “Ash Ketchum”	65% / 60%; OpenVLA 5% / 0%
VISTA (Chen et al., 4 Feb 2026)	LIBERO mean SR / CALVIN OFT average	76.3% → 79.4%; 3.87 → 4.02
InstructVLA (Yang et al., 23 Jul 2025)	SimplerEnv-Instruct overall	46.0 vs 24.0 OpenVLA FT and 35.6 OpenVLA + GPT-4o

Across papers, the strongest gains usually appear under counterfactual or out-of-domain evaluation rather than ordinary in-distribution manipulation. SAM-based finetuning reports over 60% improvement in instruction following without additional data or architectural modification, and further gains of +17.8% success rate on LIBERO-PRO when combined with classifier-free guidance (Zhang et al., 22 Jun 2026). Interleave-VLA reports 2–3× improvement in semantic out-of-domain generalization on SIMPLEREnv and analogous 2–3× gains on low-data real-robot tasks by exposing the policy to heterogeneous instruction images, including Internet images and user-provided sketches (Fan et al., 4 May 2025). VISTA reports more modest but consistent gains in mean success rate, together with visual-conditioning curves in which Stage 1 preference optimization doubles or triples the visual-conditioning metric for later tokens (Chen et al., 4 Feb 2026).

A second empirical theme is preservation of pretrained multimodal competence. VLM2VLA reports retention of more than 85% of base VQA performance while improving multilingual and open-world robotic generalization, including 65% average success on multilingual pick-up and 60% on the task “Pick up item above ‘Ash Ketchum’” (Hancock et al., 26 Sep 2025). InstructVLA explicitly mixes VLM corpora and manipulation data and reports that its 2B-parameter generalist model outperforms co-trained Magma-8B and larger 8B VLMs in average multimodal benchmark score, while also improving situated manipulation reasoning; it further reports an additional +36% gain on Situated Reasoning tasks when textual reasoning is interleaved at test time (Yang et al., 23 Jul 2025). VLAS shows that end-to-end speech alignment narrows the gap between spoken and textual instructions on CALVIN—94.2% versus 95.5% on 1-step evaluation—while greatly improving customized tasks when voice retrieval-augmented generation is enabled (Zhao et al., 19 Feb 2025).

6. Limitations, disagreements, and open directions

The VLA-IT literature does not converge on a single failure mechanism for poor instruction following. One line of work attributes the problem to narrow linguistic coverage and surface-form overfitting (Shin, 17 Mar 2026). Another attributes it to sharp high-curvature minima induced by limited-data finetuning, diagnosing representation collapse through sharpness measures and top Hessian eigenvalues (Zhang et al., 22 Jun 2026). A third emphasizes catastrophic forgetting from distribution mismatch between internet-scale pretraining and robot control data (Hancock et al., 26 Sep 2025), while VISTA foregrounds weak visual dependence in the action predictor and treats improved visual conditioning as the operative variable (Chen et al., 4 Feb 2026). These explanations are not mutually exclusive, but they imply different interventions and evaluation priorities.

Trade-offs are explicit in the reported results. Synthetic instruction augmentation improves 5-Bin robustness but reduces exact Top-1 accuracy, which the paper interprets as a possible redistribution of probability mass across multiple valid action bins; the same study also scales only to 100 trajectories, leaving performance on the full 53 K Bridge V2 corpus and on multi-stage tasks unvalidated (Shin, 17 Mar 2026). SAM preserves instruction following without extra data, but incurs approximately 2× compute and memory because each update requires two forward-backward passes, and absolute success rates still remain below 50% on several counterfactual benchmarks (Zhang et al., 22 Jun 2026). Interleave-VLA improves zero-shot behavior yet increases compute cost because longer input sequences contain image tokens as well as text tokens (Fan et al., 4 May 2025). VLM2VLA preserves reasoning but reports autoregressive inference latency with median approximately 6.1 s per cycle and maximum approximately 49 s, and presently handles coarse translational-only control rather than dexterous or rotational action spaces (Hancock et al., 26 Sep 2025).

Several future directions recur across papers. Automated quality control is proposed to replace manual filtering of synthetic instructions, and extension to multi-step or hierarchical task descriptions is repeatedly identified as necessary (Shin, 17 Mar 2026). Visual-grounding approaches suggest richer surrogate tasks, more sophisticated negative sampling, and possible integration into large-scale pretraining rather than finetuning only (Chen et al., 4 Feb 2026). Interleaved-instruction work points toward token compression and active selection of the most informative instruction images (Fan et al., 4 May 2025), while speech-based VLA-IT highlights user-specific knowledge injection through retrieval and the importance of preserving non-semantic voice information (Zhao et al., 19 Feb 2025). A plausible implication is that future VLA-IT systems will combine multiple strands now studied separately—data relabeling, flatness-preserving optimization, multimodal instruction interfaces, and representation-preserving parameter-efficient finetuning—because current evidence suggests that no single intervention fully resolves brittleness in embodied instruction following.