---
title: 'JARVIS-VLA: Vision-Language-Action Agent'
url: https://www.emergentmind.com/topics/jarvis-vla
type: topic
---

# JARVIS-VLA: Vision-Language-Action Agent

JARVIS-VLA is a Vision-Language-Action agent paradigm that advances decision-making in open-world environments by post-training large-scale vision-language models (VLMs) with visual and linguistic supervision prior to imitation learning. The result is agents that integrate perception, world knowledge, and action at a level beyond that achieved by pure trajectory-based policy learning. JARVIS-VLA is demonstrated within the Minecraft environment, where it sets state-of-the-art benchmarks for following natural human instructions across over 1,000 distinct atomic tasks, including crafting, smelting, cooking, mining, and entity interaction [2503.16365].

## 1. Training Pipeline: Visual Language Post-Training

JARVIS-VLA implements a three-stage pipeline integrating sequential off-trajectory post-training before action imitation:

1. **Stage I – Language Post-Training**: The language transformer is unfrozen and trained on domain-specific text (e.g., Minecraft wiki, Reddit posts) using supervised next-token prediction:
   $$
   \mathcal{L}_\mathrm{SFT} = -\sum_{i=1}^N \log p_\theta(x_i | x_{<i}, x_v, x_\text{ins})
   $$
   where $x_v$ encodes visual context, $x_\text{ins}$ the textual instruction, and $x_i$ the token in the response or action sequence. The ViT and image adapter remain frozen at this stage.

2. **Stage II – Vision–Language Alignment & Spatial Grounding**: All model parameters (ViT, image adapter, language transformer) are trainable. The dataset combines image captioning, visual question answering (VQA), and spatial grounding (point and bounding-box annotation) tasks using the same $\mathcal{L}_\mathrm{SFT}$ objective. This stage enhances both semantic alignment and localization capacity.

3. **Stage III – Imitation Learning on Trajectories**: The agent is exposed to human and agent-generated action trajectories. For control, the tokenizer’s reserved tokens are repurposed: 22 for mouse actions, 29 for keyboard inputs (mu-law discretization for continuous actions, discrete tokens for key presses and GUI slots). Only the language transformer is fine-tuned:
   $$
   \mathcal{L}_\mathrm{IL} = -\sum_{t=1}^T \log \pi_\theta(a_{t:t+\tau} | o_t, x_\text{ins})
   $$
   This modular approach sequentially enhances world knowledge, vision–language alignment, and action capacity [2503.16365].

## 2. Model Architecture and Tokenization for Action Decoding

The core architecture builds on open-source VLMs such as Llava-Next or Qwen2-VL:

- **Vision Transformer (ViT)**: Processes raw RGB frames or UI snapshots into patch embeddings.
- **Image Adapter**: A lightweight two-layer MLP projects ViT output into the language model’s embedding space.
- **Autoregressive Transformer Language Model**: Text tokens and image tokens (via special "<img>" tokens) are jointly processed. For long-horizon or partially observable settings, multiple historical frames are concatenated in the prompt.
- **Action Decoder**: Discrete actions (keyboard, mouse) and camera motions are retokenized. Mu-law discretization translates continuous movements into reserved language tokens (21 bins/axis).

No specialized attention or grounding heads are introduced. The unified next-token objective enables the VLM to output both descriptions (narrative, world knowledge) and low-level action commands.

## 3. Pretraining and Data Augmentation Protocols

The post-training corpus combines three sources:

- **World Knowledge QA**: 277K entries from Wikipedia and community forums, formatted as multi-turn dialog (1.5K–2K tokens per item), augmenting classical LLM pretraining.
- **Vision–Language Alignment**: 35K annotated keyframes from YouTube agent rollouts, with 15K captions and 20K VQA pairs, curated via advanced VLMs (GPT-4o, Claude 3.5, Molmo) and filtered by Llama 3.1.
- **Spatial Grounding**: 404K instances, including 236K embodied frames (localized by SAM2) and 168K GUI inventory slot examples.

Extensive image data augmentations are applied, including random hue/saturation/brightness/contrast shifts, translation, rotation, scaling, shearing, and flipping (notably during Stage I/II). Training utilized a cosine learning-rate schedule, AdamW optimizer, bfloat16 precision, and large-scale distributed GPU resources (32×A800-80GB, total 640 GPU-hours).

## 4. Evaluation in Minecraft: Task Success and Benchmarks

Evaluation is performed using the MCU benchmark on over 1,000 atomic subtasks:

- **Mining**: e.g., "Mine iron ore with stone pickaxe"
- **Killing**: e.g., "Kill a sheep/spider"
- **Crafting**: e.g., "Craft diamond sword"
- **Smelting**: e.g., "Smelt iron ingot"
- **Cooking**: e.g., "Cook beef"
- **GUI Tasks**: Crafting and inventory manipulation via discrete slot selection

Each task is evaluated over at least 30 runs, averaging success rates. The results are summarized as follows:

| Model               | Mine Blocks | Kill Entities | Craft Items | Smelt Items |
|---------------------|-------------|--------------|-------------|-------------|
| VPT-BC (248M)       | 0.33        | 0.44         | 0.41        | 0.05        |
| STEVE-1 (248M)      | 0.54        | 0.38         | 0.57        | 0.33        |
| MineDreamer         | 0.55        | 0.39         | 0.42        | 0.30        |
| Qwen2-VL (IL)       | 0.75        | 0.86         | 0.65        | 0.29        |
| –Qwen2-VL (Posttr.) | **0.88**    | **0.95**     | **0.77**    | **0.70**    |

JARVIS-VLA, denoted by "–Qwen2-VL," achieves a ~40% improvement over the best agent baseline in key categories. In GUI-intensive tasks, benefits from vision-language post-training are especially pronounced.

## 5. Ablation Studies and Scaling Laws

To disentangle the effects of each modality, three ablations are performed using Qwen2-VL as the base:

- **World Knowledge Only**: Improves reasoning, but less impact than visual signals.
- **Vision–Language Alignment Only**: Major gains in grounding and captioning.
- **Spatial Grounding Only**: Yields the largest improvement in long-horizon embodied tasks.

Increasing the number of imitation trajectories reduces $\mathcal{L}_\mathrm{IL}$ below ≈0.22, beyond which task success rates sharply rise. Lower evaluation losses in Stage II are linearly correlated with higher downstream success rates, establishing scaling laws for vision-language post-training analogous to those found in pure-text LLMs.

## 6. Contributions, Limitations, and Prospects

**Contributions**:
- Introduction of first VLA agents in Minecraft compliant with free-form human instruction over >1,000 atomic subtasks.
- Definition and empirical validation of the "Visual Language Post-Training" paradigm; self-supervised vision-language signals boost decision-making by ~40% over best imitator baselines.
- Scaling laws for post-training dataset size and model size are demonstrated across Llava-Next and Qwen2-VL backbones.
- Resources (code, checkpoints, datasets) publicly available at https://craftjarvis.github.io/JarvisVLA.

**Limitations**:
- Inference throughput remains suboptimal (≈55 FPS on 4×A800s; below human 40 Hz target).
- Performance gap persists to top human players (≈90% success).
- Further work is needed to optimize inference with sparse MoE, scale non-trajectory corpora, and generalize to other open-world settings.

A plausible implication is that this post-training paradigm can be generalized to other domains where active decision-making under vision-language guidance is critical. The staged training mode enhances sample efficiency, reduces trajectory annotation requirements, and yields multifaceted agents with advanced world knowledge and spatial reasoning [2503.16365].

Source: https://www.emergentmind.com/topics/jarvis-vla