ClevrSkills: Compositional Robotics Benchmark

• ClevrSkills is a benchmark and dataset designed for evaluating compositional language and visual reasoning in robot manipulation, integrating natural language with continuous control.
• It comprises 33 tasks across varying complexity levels, leveraging 330,000 annotated trajectories and predicate-based grammars to test skill composition and memory-dependent manipulation.
• The benchmark emphasizes real-world challenges by combining low-level motor control with high-level reasoning, revealing limitations in current vision-language-action models under non-Markovian constraints.

Searching arXiv for the primary ClevrSkills paper and recent follow-up work that uses or extends it. arxiv_search: query: "ClevrSkills compositional language and visual reasoning in robotics" max_results: 5 arxiv_search: query: "\"Notes-to-Self\" Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks ClevrSkills" max_results: 5 ClevrSkills is a simulation benchmark and dataset for compositional language and visual reasoning in robot manipulation. It is built on top of ManiSkill2 and consists of an environment suite of 33 manipulation tasks together with approximately 330,000 oracle trajectories annotated with language, visual prompts, rewards, and step-level metadata. Its central research question is whether a model that has acquired low-level manipulation capabilities such as pick, place, push, throw, rotate, touch, and trace can compose them in novel ways to solve higher-level tasks under natural-language and multimodal instructions, in a closed-loop continuous-control setting (Haresh et al., 2024). Subsequent work has also used ClevrSkills as the primary controlled testbed for memory-dependent manipulation, introducing a split of tasks that explicitly violate the Markov assumption and probe temporal and spatial memory in vision-language-action systems (Haresh et al., 24 Feb 2026).

1. Scope, motivation, and conceptual framing

ClevrSkills was introduced to test compositional generalization in robotics rather than isolated motor competence. The benchmark targets settings in which task specifications compose objects, attributes, relations, order constraints, and, in some cases, invented adjectives and nouns whose semantics must be inferred from examples. In this formulation, “compositional” spans linguistic composition, skill composition, visual reasoning, and temporal-logical composition. The benchmark therefore couples high-level reasoning with low-level continuous control, rather than abstracting manipulation into oracle object-pose actions or reducing the task to question answering (Haresh et al., 2024).

The benchmark is explicitly positioned against several adjacent paradigms. Relative to CLEVR and CLEVRER, it retains CLEVR-style compositional reasoning over objects, attributes, relations, and novel words, but grounds that reasoning in physical action. Relative to ManiSkill2, it adds language, multimodal prompts, and a curriculum organized around compositional task design. Relative to VIMA, it uses end-effector delta actions rather than object-pose commands, so planning and control are not factored by an external manipulation oracle. Relative to CALVIN-style long-horizon evaluation, it emphasizes richer combinatorial structure, including predicate composition and combined constraints such as swap-plus-rotate or sort-by-throwing (Haresh et al., 2024).

This design suggests that ClevrSkills is intended less as a benchmark of raw dexterity than as a benchmark of structured policy formation under compositional task semantics. A plausible implication is that failures on the suite are informative about reasoning, memory, grounding, and subgoal management even when low-level controllers are adequate.

2. Environment, embodiment, and task formalization

ClevrSkills is developed on ManiSkill2. The default embodiment is a UFACTORY xArm 6 with a vacuum gripper, and Franka Emika Panda is also supported. Oracle motion planning for “Move” sub-solvers uses MPLib, described as a Python wrapper around OMPL’s RRT for collision-free joint-space paths. The benchmark configuration uses two RGB cameras: a hand or end-effector camera that provides a first-person view, and a base camera that provides a third-person view of the table (Haresh et al., 2024).

The action space is ManiSkill2’s delta end-effector controller, represented as a 7D continuous action vector comprising a 6DOF pose delta and a 1D gripper scalar. This choice is central to the benchmark’s difficulty: the policy must generate continuous motor commands rather than select symbolic object-level actions. The paper formulates the environment in standard continuous-control terms, with simulator state $s_t$, observation $o_t$, action $a_t$, reward $r_t$, and transition dynamics given by the physics engine:

$$(s_{t+1}, r_t) \sim p(s_{t+1}, r_t \mid s_t, a_t), \quad o_t = g(s_t).$$

Tasks are specified through a predicate-based grammar. Physical predicates include EEAtPos, EEAtPose, AtPos, AtPose, OnTop, Inside, Touch, Hit, and ToppleStructure. Logical predicates compose them through Set(p_1,\dots,p_k), Sequence(p_1,\dots,p_k), and Once(p). Physical predicates define dense reward functions, typically with distance-to-goal shaping, while logical predicates aggregate sub-predicate rewards according to order or unordered satisfaction constraints. This predicate layer simultaneously defines success conditions, enables dense reward shaping, and supplies a modular grammar for constructing new tasks as compositions of primitive conditions (Haresh et al., 2024).

The environment also ships with scripted oracle policies, referred to as solvers. These solvers can run end-to-end or be interleaved with learned policies. That structure makes the suite usable not only for benchmark evaluation but also for imitation learning, offline RL, DAgger-style data aggregation, and hierarchical policy learning.

3. Curriculum of tasks and compositional structure

ClevrSkills defines 33 tasks across three levels of compositional complexity. The curriculum begins with primitive manipulation skills, proceeds to short-horizon compositions, and culminates in long-horizon tasks that combine multiple skills and conceptual operators. Median episode length grows substantially from L0 to L2, and the mean number of solvers used rises from roughly 1–2 at L0 to roughly 9 at L1 and roughly 11 at L2, indicating progressively deeper subtask structure (Haresh et al., 2024).

Level	Task count	Characterization
L0	12	Primitive motor skills
L1	15	Short-horizon compositions
L2	6	Long-horizon complex compositions

L0 contains the primitive tasks Match pose, Move without hitting, Pick, Place, Push, Rotate, Throw, Throw topple, Touch, Touch push, Touch topple, and Trace. These tasks require object discrimination, geometric targeting, contact control, and action constraints such as toppling without grasping or touching without moving the object more than 3 cm. Rotate, for example, requires correct direction, angle within 5°, and translation less than 5 cm; Push requires movement toward a goal object within $\pm 45^\circ$ and at least 30% reduction in distance; Trace requires ordered traversal of green waypoint spheres that change state once touched (Haresh et al., 2024).

L1 contains 15 composed tasks: Simple manipulation, Follow order, Follow order and restore, Neighbour, Novel Adjective, Novel Noun, Novel Adjective and Noun, Rearrange, Rearrange and restore, Rotate and restore, Rotate symmetry, Stack, Stack reversed, Sort, and Swap. These tasks introduce relational reasoning over initial layouts, restoration to the initial state, pseudo-word concept learning, texture-based grouping, list reversal, and multi-step spatial rearrangement. Several of them already require memory of prior states, even though the original benchmark is framed around compositionality rather than explicit non-Markovianity (Haresh et al., 2024).

L2 contains six long-horizon tasks: Balance, Sort Stack, Stack topple, Swap with push, Swap and rotate, and Throw sort. These explicitly superimpose constraints. Throw sort, for instance, combines texture-based assignment with a reachability constraint that makes throwing the only viable strategy. Swap with push preserves the semantics of Swap while prohibiting grasping. Stack topple requires ordered sequential construction followed by destruction of the completed structure (Haresh et al., 2024).

A later derivative benchmark, ClevrSkills-Mem, extracts five memory-dependent tasks from the environment: Touch-Reset-Pick, Place-Next-To and Restore, Stack-and-Topple, Swap, and Rotate-Restore. These tasks are designed to probe temporal memory, spatial memory, spatio-temporal memory, and fine-grained continuous phase tracking. They are motivated by the observation that tasks such as restoration and swapping can render the current observation insufficient, because original poses or phase information are no longer visible after earlier actions (Haresh et al., 24 Feb 2026).

4. Dataset, annotations, and prompt modalities

The ClevrSkills dataset contains approximately 330,000 trajectories, corresponding to 33 tasks with 10,000 trajectories per task. These trajectories are generated by scripted oracle policies using motion planning plus task-specific logic. Diversity arises from random initial object positions, 59 object meshes, 61 textures, and varying numbers of distractor objects and obstacles (Haresh et al., 2024).

Each trajectory includes multi-camera RGB videos, an $N \times 7$ array of delta end-effector actions, dense rewards per step, success flags per step, episode metadata, prompt assets, keystep images, and camera intrinsics and extrinsics. Additional annotations include textual action labels for each step, bounding boxes, and visibility flags for each object at each timestep. Per-episode directories contain files such as actions.npy, rewards.npy, success.npy, action_labels.npy, ep_info.json, info.json, and prompt_assets.npy (Haresh et al., 2024).

Language supervision is available at three granularities: task or predicate level, sub-task level, and step level. This multi-level annotation supports language-conditioned imitation learning, hierarchical decomposition, and alignment between natural language and motor primitives. Prompting is multimodal by default. Following the VIMA style, task specifications interleave text with image placeholders such as {obj:object}, {tex:object}, and {ks:keystep}. These placeholders refer, respectively, to object images, texture swatches, and visual goal states such as poses or scenes. For many tasks, a language-only version can be constructed by replacing image placeholders with textual descriptions, although inherently visual keystep tasks remain exceptions (Haresh et al., 2024).

The benchmark also defines train/test splits over objects and textures, so evaluation can probe out-of-distribution generalization to unseen combinations. Initial placement seeds are likewise varied between training and evaluation. In the later ClevrSkills-Mem split, evaluation is performed on unseen starting positions of the same objects seen during training, emphasizing continuous spatial generalization and long-horizon memory rather than novel object categories (Haresh et al., 24 Feb 2026).

5. Evaluation protocol and empirical findings

The original evaluation protocol is designed to test both basic skill acquisition and compositional transfer. Models are trained on L0 tasks and evaluated on unseen seeds and unseen objects or textures within L0. They are then tested zero-shot on L1 without L1 finetuning, and, in a separate setting, fine-tuned on L1 and evaluated on held-out seeds and objects. An analogous protocol applies from L0+L1 training to L2 zero-shot and L2 finetuned evaluation. Reported metrics are success rate, average reward, and reward per step (Haresh et al., 2024).

The baseline set comprises JAT, Octo, RoboFlamingo, and StreamRoboLM. JAT is modified so that multimodal prompt tokens precede sequences of state and action embeddings, and it is fine-tuned with MSE loss on actions. Octo is extended to take the base camera, hand camera, and prompt images. RoboFlamingo uses a Flamingo-style VLM with an LSTM policy head; the VLM is frozen and only the policy head is trained. StreamRoboLM, introduced as a new baseline, uses an OPT-1B or Llama3.2-3B LLM backbone, a ViT image encoder, Flamingo-style cross-attention, and an LSTM policy head, with LoRA used to fine-tune the LLM while preserving general language ability (Haresh et al., 2024).

On L0, Oracle achieves 100% success. JAT reaches approximately 23.75% success on unseen seeds and approximately 24.16% on unseen objects and textures. RoboFlamingo reaches approximately 35.4% and approximately 27.9%, respectively. Octo reaches approximately 34.2% and approximately 26.3%. StreamRoboLM with an OPT backbone reaches 62.91% on unseen seeds and 41.66% on unseen objects and textures, while StreamRoboLM with a Llama3 backbone reaches 62.5% on unseen seeds and 55.41% on unseen objects and textures. Despite these L0 gains, zero-shot success on L1 and L2 is essentially 0% across all models. Even after finetuning, L1 success remains low—0.60% for JAT, 1.0% for RoboFlamingo, 5.0% for Octo, and 3.33% for StreamRoboLM—and L2 success remains similarly low, with Octo best at 5.83% and StreamRoboLM between 3.33% and 4.16% (Haresh et al., 2024).

The paper also reports that language-only prompts often outperform multimodal prompts on L0 for several baselines. RoboFlamingo, for example, improves from approximately 35.4% to approximately 57.5% success when multimodal prompts are replaced by language-only descriptions. On L1 and L2, however, zero-shot success remains approximately 0% regardless of prompt type. This indicates that current open-source VLM-based robot policies are weak not only at compositional transfer but also at interpreting image-rich prompts (Haresh et al., 2024).

The memory-focused follow-up yields a more targeted diagnosis. On ClevrSkills-Mem, adding a language scratchpad to a transformer VLA improves performance by 68% on Touch-Reset-Pick, 72% on Swap, 68% on Place-Next-To and Restore, and 30% on Stack-and-Topple, for an average performance gain of 48.8% across the five tasks. For a recurrent VLA, scratchpad training still yields an average improvement of 11%. Rotate-Restore remains unsolved, suggesting that textual scratchpads are more effective for discrete subgoal and spatial memory than for very fine-grained continuous-state tracking such as cumulative angle estimation (Haresh et al., 24 Feb 2026).

6. Research significance, limitations, and subsequent use

ClevrSkills contributes a benchmark environment, a large richly annotated dataset, an explicit predicate-based task grammar, and a systematic compositional evaluation protocol. Its main empirical conclusion is that moderate success on primitive tasks does not translate into systematic composition of those skills in novel settings. Failure modes reported in the benchmark include selecting the wrong object among distractors, losing track of identities after objects have moved, violating temporal order constraints, and failing on tasks that require memory of the initial configuration or deferred subgoal execution (Haresh et al., 2024).

The benchmark has several methodological strengths. It provides low-level continuous control rather than abstract action interfaces; exposes both language-only and multimodal prompt regimes; supports train/test splits over objects, textures, and seeds; and makes task construction extensible through physical and logical predicates. Because it includes oracle solvers and rich annotation, it is suitable for imitation learning, offline RL, hierarchical control, language grounding, subgoal supervision, and memory-augmented policy design. The later ClevrSkills-Mem split sharpens one aspect of this utility by isolating non-Markovian structure and showing that stateless vision-language-action models fail precisely when historical information must be retained (Haresh et al., 24 Feb 2026).

Its limitations are equally clear. The underlying world consists of simple objects manipulated with a vacuum gripper in simulation, so the benchmark is controlled rather than photorealistically diverse. Although it contains 33 tasks, the space of real-world object categories, contact-rich dynamics, and open-ended task semantics is much broader. The original results also show that multimodal prompts are not automatically beneficial for current baselines, which complicates the interpretation of prompt-format effects. In the memory derivative, Rotate-Restore exposes a remaining gap for continuous-valued memory even when explicit scratchpads are available (Haresh et al., 2024, Haresh et al., 24 Feb 2026).

Within robotics research, ClevrSkills occupies a specific niche: it is a benchmark for asking whether a policy can “compose what it knows” under natural-language and multimodal task descriptions while still solving the low-level control problem. That niche has made it useful not only for benchmarking VLM-based robot policies but also for studying explicit memory, phase tracking, and the interaction between symbolic or textual structure and continuous control.