Qwen-VLA-Instruct: Unified Vision-Language Model

• Qwen-VLA-Instruct is an instruction-tuned unified vision-language(-action) model that integrates advanced cross-modal fusion and flow-matching diffusion for continuous action prediction.
• It employs a multi-stage training pipeline—combining pretraining, supervised fine-tuning, and reinforcement learning—to achieve robust multimodal reasoning and embodied task performance.
• The model’s prompt-based design enables seamless adaptation to diverse robotic control, manipulation, and navigation tasks without requiring architectural modifications.

Qwen-VLA-Instruct denotes the instruction-tuned variant of the Qwen-VLA family: unified vision-language(-action) foundation models that enable multi-modal reasoning, perception, and embodied interaction via a single architecture. It combines the Qwen vision-language backbone, advanced cross-modal fusion, and a flow-matching diffusion action head, trained through a staged pipeline culminating in supervised fine-tuning and reinforcement learning alignment. The model extends earlier Qwen-VL “Instruct” protocols to support open-ended robotic control, navigation, and general multimodal instruction following, and demonstrates strong empirical results across a spectrum of vision-language and embodied benchmarks (Bai et al., 2023, Wang et al., 28 May 2026).

1. Model Architecture and Key Components

Qwen-VLA-Instruct builds upon the Qwen3.5-4B backbone, employing early vision-language fusion via spatially-merged Vision Transformer (ViT) patch embeddings interleaved with text tokens. The multimodal transformer alternates gated linear attention and grouped-query softmax, ensuring global context propagation. Visual tokens are generated by a ViT (e.g., ViT-bigG) and position-aware VL adapter, with cross-attention compressing the patch sequence to a fixed token set, which is then concatenated with text tokens.

Beyond perception, Qwen-VLA-Instruct incorporates an action expert module: a DiT-based (Diffusion Transformer) continuous action decoder with 16 blocks (~1.15B parameters). This DiT action head operates by flow-matching, predicting denoised action trajectories conditioned on VLM features, multimodal observations, instructions, and an embodiment prompt, using AdaLN for timestep encoding and multi-section RoPE positional embeddings. Prompt-based embedding informs the head of robot morphology and action space, supporting broad transfer across platforms without architectural branching.

No additional modifications are made for instruction-tuning; Qwen-VLA-Instruct shares its underlying architecture with Qwen-VLA-Base, ensuring direct comparability and stability (Wang et al., 28 May 2026).

2. Training Pipeline and Instruction-Tuning Protocols

Qwen-VLA-Instruct is produced via a multi-stage pipeline:

1. Pretraining (Stage I–II):
   • T2A (Stage I): Text-to-action pretraining; VLM frozen, only DiT action expert is updated using synthetic and teleop trajectories.
   • Continued Pretraining (CPT, Stage II): Joint loss over vision-language modeling (captioning, VQA, grounding, driving VQA, etc.) and action prediction. Corpora include real and synthetic manipulation, navigation, simulation, egocentric human demos, and multimodal auxiliary data. All model parameters are updated.
2. Supervised Fine-Tuning (SFT, Stage III):
   • Multi-task, multi-embodiment data mixture incorporating VQA, spatial grounding, embodied action captions, manipulation demos (H=16 timesteps), and navigation (H=8).
   • Joint objective: next-token cross-entropy (weight 0.1) and flow-matching loss on actions (weight 1.0).
   • Cosine-decay learning rate, balanced batch sampling.
3. Reinforcement Learning (Stage IV):
   • PPO (Proximal Policy Optimization) using a lightweight value head, acting on simulation rollouts with sparse binary reward, GAE ($\gamma=0.99, \lambda=0.95$).

Instruction templates (embodiment prompts) encode robot control types, morphology, and task instructions, ensuring the model generalizes across platforms and action conventions without explicit code changes (Wang et al., 28 May 2026).

For vision-language “Instruct” variants (pure VLMs), a three-stage pretraining pipeline is utilized: frozen LLM vision-language pretraining, multi-task vision-language and grounding pretraining with all weights trainable, and supervised instruction-tuning using ChatML-format multimodal dialogues (Bai et al., 2023).

3. Embodiment-Aware Prompting and Multimodal I/O

Qwen-VLA-Instruct utilizes a unified input-output interface for both visual and embodied tasks:

• Image tokens: ViT features compressed to a fixed token set, wrapped as <img>...</img>.
• Bounding boxes/grounded regions: Tokenized coordinate strings, bracketed by <box>...</box>, with referenced phrases bracketed as <ref>...</ref>.
• Dialogue turns: ChatML-style turn markers (<im_start>user ... <im_end>) for instruction-following chat.
• Embodiment-aware prompts: Natural language templates specifying robot type, control frequency, arms/base/waist, action chunk size, and original instruction.
• Action outputs: Model predicts “chunks” of continuous control actions (e.g., velocity, position), formatted to match platform control conventions.

This design enables seamless adaptation to new robots and tasks by editing only prompt templates, without architectural or codebase changes. Conditioning via prompt tokens makes the DiT action expert robust to differing action dimensionalities, semantics, and horizons (Wang et al., 28 May 2026).

4. Empirical Performance and Benchmarking

Qwen-VLA-Instruct demonstrates strong results across simulated and real-world embodied tasks and multimodal benchmarks:

Task	Score (Qwen-VLA-Instruct)	Baseline Comparison
LIBERO Manipulation	97.9%	Exceeds ABot-M0; +7–21pp SFT+RL over Base
Simpler-WidowX	73.7%	+9.4pp over Pretrain-Base
RoboTwin (Easy/Hard)	86.1% / 87.2%	-
VLN-CE (R2R OS/SR/SPL)	69.0 / 57.5 / 51.2	Standard protocols
RxR (SR/SPL)	59.6 / 47.8	-
DOMINO zero-shot (Dynamic Manip.)	26.6%	Manip.Score: 39.5
Real-World ALOHA Bimanual (OOD)	76.9%	-

Further, vision-language “Instruct” models (Qwen-VL-Chat) outperform or match leading vision-language and instruction-tuned baselines on standard datasets:

• Zero-shot Captioning: Nocaps CIDEr 121.4; Flickr30K 85.8 (exceeding Flamingo-80B score of 67.2 with only 7B params).
• VQA and Text-Reading: VQAv2 79.5, OKVQA 58.6, ScienceQA-Img 67.1, TextVQA 63.8, DocVQA 65.1, OCR-VQA 75.7, significantly surpassing BLIP-2, InstructBLIP, mPLUG-DocOwl on OCR-centric tasks by 10+ points in some cases.
• Grounding: RefCOCO val/testA/testB 89.36/92.26/85.34, outperforming Shikra-13B.
• Instruction Following: TouchStone (English) 645.2 (vs mPLUG-Owl 605.4), SEED-Bench All: 58.2 (Bai et al., 2023).

Ablations confirm the contributions of staged pretraining, vision-language co-training, specific flow-matching schedules, and RL refinement; specialist models are not required to exceed task-by-task expert performance (Wang et al., 28 May 2026).

5. Adaptation, Fine-Tuning, and Shadow-FT

A distinct protocol for updating Qwen-VLA-Instruct leverages the Shadow-FT framework (Wu et al., 19 May 2025), which exploits the near-isomorphism between “Base” and “Instruct” models. The process is as follows:

1. Fine-tune the Base model (Qwen-VLA-Base) on downstream multimodal or action objectives (using full or LoRA PEFT).
2. Compute the parameter update $\Delta\theta_b = \theta_b' - \theta_b$.
3. Graft $\Delta\theta_b$ directly onto the Instruct model: $\theta_i^{\text{new}} = \theta_i + \Delta\theta_b$.

This approach introduces no extra parameters or inference cost, and does not disturb the pre-existing alignment and instruction-following priors of the Instruct model. No reapplication of layer-norm statistics or special scaling is required; learnable parameters (e.g., LayerNorm $\gamma$, $\beta$) and any vision adapters, QKV/Proj layers, and image modules are updated elementwise. Empirically, Shadow-FT is robust to LoRA rank, avoids overfitting present in conventional LoRA-fine-tuned Instruct models, and yields monotonically improving performance as tuning capacity increases.

Evaluation protocols recommend benchmarking on VQA, GQA, ChartQA, ScienceQA, M3Exam, and COCO Captioning, using accuracy, exact match, BLEU/METEOR scores, and multi-modal reasoning sets. Gains of +1.4 to +3.5 points are reported for ChartQA in comparable models, with stability for $\sigma$-gap $< 2\%$ (Wu et al., 19 May 2025). A plausible implication is that Shadow-FT enables rapid domain adaptation for Qwen-VLA-Instruct in safety-critical or resource-constrained applications.

6. Practical Deployment and Extensibility

Qwen-VLA-Instruct is designed for prompt-centric deployment:

• Embodiment Update: Change robot description in prompt string (e.g., “Franka_Panda with single arm. Control frequency 20 Hz...”) to switch between platforms.
• Task Generalization: Free-form task instructions are accepted and generalized via model’s multitask and multimodal SFT alignment.
• Adapting to New Tasks/Embodiments: For improved performance in novel domains, collect demonstration data in the standard prompt format and continue SFT or RL.
• Real-World Usage: Model rollouts yield control trajectories or navigation waypoints directly from natural language and visual input, and are adaptable via prompt tokens alone.

No code changes are required for new robot formats, action horizons, or prompt structures, supporting rapid prototyping and extensibility.

7. Significance and Context

Qwen-VLA-Instruct substantiates the hypothesis that unified foundation models with large-scale staged pretraining and prompt-based conditioning can achieve broad generalization across vision, language, and action domains. Its architectural and training choices eliminate the need for bespoke task/head designs for manipulation, navigation, visual reasoning, or dialog, and maintain competitive performance to specialist systems. The instruction tuning protocols and the Shadow-FT methodology offer reliable pathways for continual domain adaptation without sacrificing instruction-following fidelity or multimodal alignment (Bai et al., 2023, Wu et al., 19 May 2025, Wang et al., 28 May 2026).