VIMA-Bench: Multimodal Robot Simulation Benchmark

• VIMA-Bench is a large-scale simulation benchmark that unifies diverse robot tabletop manipulation tasks using multimodal (text and visual) prompts.
• It procedurally generates thousands of unique task instances and categorizes tasks into six families to rigorously test generalization.
• Empirical results show VIMA-Bench achieves up to 2.9× higher success rates for novel tasks with significantly less data compared to baselines.

VIMA-Bench is a large-scale simulation benchmark aimed at unifying diverse robot tabletop manipulation problems under a single “multimodal-prompt → policy” paradigm. Inspired by prompt-based learning in NLP, VIMA-Bench introduces a procedurally generated suite of tasks that covers a wide range of instruction modalities and cognitive demands, leveraging multimodal—textual and visual—input prompts for specifying tasks. The benchmark includes a substantial expert-generated demonstration dataset, and an evaluation protocol that systematically tests various dimensions of generalization, serving as a cornerstone for research in prompt-driven, generalist robot manipulation (Jiang et al., 2022).

1. Objectives and Task Categorization

VIMA-Bench is designed to promote a unified approach to robotic manipulation by expressing a broad spectrum of task formats as multimodal prompts. The benchmark’s objective is to bridge disparate manipulation paradigms—including language instruction following, visual imitation, and goal-conditioned rearrangement—within a single framework. Specifically, it categorizes manipulation tasks into six core families, instantiated as 17 distinct prompt templates:

1. Simple object manipulation: e.g., “put the red block into the green bowl.”
2. Visual goal reaching: rearrangement to match a target image, potentially requiring distractor removal.
3. Novel concept grounding: bootstrapping compositional symbol grounding by defining dummy nouns, adjectives, or verbs via in-prompt visual samples (“This is a blicket”, image).
4. One-shot video imitation: reproducing manipulation motions from a sequence of exemplar frames.
5. Visual constraint satisfaction: completing tasks with explicit visual constraints (“sweep 2 rings into the box without crossing the red line”).
6. Visual reasoning: higher-level appearance-based or relational tasks, including “restore to original position” or comparative logic.

All tasks use the Ravens physics simulator with a UR5 robotic arm, a 0.5 × 1 m workspace, two end-effectors (suction cup and spatula), and low-level manipulation primitives (“pick-and-place,” “wipe”) (Jiang et al., 2022).

2. Procedural Task Instance Synthesis

Each prompt template comprises placeholders parameterized by object types, textures, quantifiers, or scene images. To generate an episode:

• Object shapes $O$ and textures $T$ are sampled.
• Each “{object}” placeholder is instantiated with an $O \times T$ pairing.
• For “scene” images, objects (and distractors) are randomly placed with pose sampled as $x, y \sim \mathrm{Uniform}$(workspace bounds), $\theta \sim \mathrm{Uniform}[0,2\pi)$.
• Distractor slots are stochastically filled (e.g., $p=0.5$).
• Comparative tasks use invented adjectives (e.g., “dax”, “blicket”), each mapped to an underlying semantic (height, color intensity), selected at random per instance on the basis of examples such as “Obj₁ is daxer than Obj₂.” This enforces meta-learning of prompt-specific semantics.

This proceduralization yields thousands of unique episodes per template, supporting systematic, large-scale experimentation and robust measurement of generalization.

3. Multimodal Prompt Encoding Pipeline

Tasks are specified by a sequence prompt $P = [x_1, ..., x_l]$, where each $x_i$ is either text or an image (object crop or full scene frame). The encoding pipeline is as follows:

• Text tokens: Tokenized via a pre-trained T5 tokenizer; embedded in 768 dimensions.
• Image tokens:
  • Single-object: Cropped, resized, encoded by a Vision Transformer (ViT), yielding a 768-d vector.
  • Full-scene: Objects detected by a Mask R-CNN (fine-tuned on 22K frames); each object’s bounding box $b_i = (x_c, y_c, h, w)$ is normalized to $[0, 1]$, then passed through a 2-layer MLP; the object crop is encoded by a small ViT. Concatenation followed by a linear projection produces a 768-d “object token.”
• Tokens are interleaved in the original prompt order with learned absolute position embeddings and passed through a frozen T5 encoder (last two layers fine-tuned).
• At each time step, the observation $o_t$ (dual RGB views, one-hot end-effector state) is similarly parsed to generate a variable-length sequence of object tokens.
• Past actions $a_1,\ldots,a_{t-1}$ are encoded with a 2-layer MLP.

The pipeline supports compositional prompts of arbitrary text-image complexity, enabling flexible specification of manipulation goals and constraints.

4. Expert Demonstrations: Dataset Composition

VIMA-Bench includes over 600,000 scripted demonstration trajectories, precisely 50,000 per task template for the 17 tasks. These demonstrations are generated by scripted oracle policies using privileged simulator state and low-level primitives (pick/place/wipe), ensuring perfect adherence to prompt intent whenever feasible. Of the total 850,000 generated episodes, 650,000 successful ones are retained for behavioral cloning.

Strategies for robust dataset coverage include per-episode randomization of objects and textures as well as prompt sampling. Demonstrations are capped at $T \leq 20$ steps and labeled with a strict binary success criterion (no partial credit). The dataset also supports subsampling to probe sample efficiency, with regimes using 0.1%, 1%, 10%, or the full 100% of the data.

5. Systematic Four-Level Evaluation Protocol

Generalization is evaluated at four increasingly challenging levels, each defined by a different held-out variable:

Level	Held-out Variable	Test Specification
L1	Initial object poses	Same prompts as train, but test with unseen $(x, y, \theta)$
L2	Object/texture combinations	Verbatim prompts, novel remixes of seen objects/textures
L3	Object shapes or textures	Prompts/scenes with objects/textures never seen during training
L4	Prompt template (“zero-shot”)	Entire task types (4 of 17) not encountered during training

The evaluation metric at each level $i$ is the mean episode-level binary success rate: $$L_i = (1/|Tasks_i|)\sum_{\tau \in Tasks_i} \mathbb{1} \{\text{policy solves } \tau\}$$.

This hierarchy comprehensively probes a policy’s ability to generalize across geometric, compositional, semantic, and task-structural shifts.

6. Summary of Empirical Findings

When tested with the standard T5-Base encoder (111M params) and varying controller capacities, the VIMA agent achieves strong scaling with both model size and data:

Controller Size	L1	L2	L3	L4
2M	76.5%	77.1%	77.3%	25.7%
20M	77.1%	77.6%	77.3%	48.8%
92M	79.3%	80.1%	81.9%	49.6%
200M	81.5%	81.5%	78.7%	48.6%

For level 4 “novel task generalization”, VIMA yields up to $2.9\times$ higher success rates than the best baseline (e.g., VIMA-Gato at 200M achieves ~12% vs. VIMA’s 49% at 92M). Competing architectures such as VIMA-Flamingo and VIMA-GPT plateau below 15% on L4, highlighting the importance of VIMA’s object-centric, cross-attention architecture (Jiang et al., 2022).

With only 1% of training data, VIMA matches or surpasses alternatives trained on 10$\times$ more data, attaining 17% success on L4 compared to $\sim$10% for VIMA-Gato. This attests to VIMA’s data efficiency and highlights the value of the compositional, multimodal prompt approach in generalist robotic policy learning.

7. Context and Implications

VIMA-Bench provides a scalable, procedurally defined, and multimodally prompted tabletop manipulation environment, facilitating rigorous study of prompt-driven robotic generalization across multiple axes. Its strict binary success metric, perpetual novelty via instance synthesis, and compositional prompt design distinguish it from prior robotic manipulation testbeds. A plausible implication is that future work on prompt- or instruction-conditioned RL and imitation learning can leverage VIMA-Bench to dissect generalization barriers and evaluate compositional reasoning in embodied agents under unified metrics (Jiang et al., 2022).