RoboWits: Robotic Reasoning Benchmark

• RoboWits is a benchmark that measures bi-manual robotic manipulation through 208 tasks focusing on cognitive reasoning, creative tool use, and adaptation under unexpected conditions.
• The benchmark employs an automated multi-agent pipeline that generates seed tasks and applies pivot, trap, and add mutations to test geometry, material, and assembly reasoning.
• Empirical evaluations reveal significant performance gaps between oracle-state planners and end-to-end policies, highlighting the challenge of integrating high-level reasoning with robust embodied action.

RoboWits is a bi-manual robotic manipulation benchmark for evaluating whether robots can reason, adapt, and creatively solve manipulation problems under unexpected conditions, rather than merely execute a known motor skill. It was introduced to measure cognitive reasoning, creative tool use, and robustness to unexpected conditions in settings where the obvious strategy fails, an object is deceptive, or a small scene change blocks the original solution. The benchmark contains 30 seed tasks expanded into 208 tasks with mutations and graded difficulty across geometry, material, and assembly-based reasoning, and it is instantiated in Genesis with a dual-arm tabletop robot comprising two 7-DoF Marvin arms, parallel grippers, three cameras—two wrist cameras and one base camera—and a 16-dimensional configuration and action space (Lin et al., 28 May 2026).

1. Benchmark rationale and scope

RoboWits was proposed against a benchmark landscape in which systems such as RLBench, LIBERO, RoboTwin, RoboEval, and VLABench primarily evaluate skill execution in well-specified settings. In that framing, a robot is typically tested on whether it can grasp, place, push, or use a tool under nominal conditions. RoboWits instead asks whether a robot can infer hidden constraints, revise a failed strategy, distinguish deceptive from functional objects, reason about geometry, materials, or assembly, and remain robust when the environment changes in subtle but consequential ways.

The benchmark’s motivating examples are deliberately adversarial in an embodied sense. A useful-looking object may be fixed to the table; a lever-like board may be soft and unusable; a container may not be liftable in the expected way; two objects may look identical but differ in hidden material properties; or a small scene mutation may invalidate the original solution path. These are treated not as outliers but as central test cases for real-world robotic intelligence.

Accordingly, RoboWits shifts evaluation from skill execution to reasoning-guided problem solving, from fixed solution paths to adaptation under blocked or deceptive conditions, and from nominal scenes to unexpected scene mutations. The benchmark is explicitly organized around cognitive reasoning, creative tool use, robustness to unexpected conditions, and strategy adaptation under deceptive or constrained environments (Lin et al., 28 May 2026).

2. Formal task structure and embodied setting

Each RoboWits task is defined as

$T=(\mathcal{I, O, C, M}, s)$

where $\mathcal{I}$ is a natural language instruction, $\mathcal{O}$ is an object list, $\mathcal{C}$ is an initial 3D scene configuration, $\mathcal{M}$ is an evaluation metric, and $s \in \{1,2,3,4,5\}$ is a difficulty score. The instruction is intentionally goal-only and does not reveal the solution strategy. In the paper’s example, the robot may be told to “pinch the bank card” rather than being told how to expose the card edge and grasp it.

The metric $\mathcal{M}$ includes both a binary success indicator and a continuous progress score function. Difficulty is defined to reflect both physical execution difficulty and reasoning complexity. The verifier prompt formalizes difficulty on a 1–5 scale; for example, 1 corresponds to at most two simple single-arm steps, while 4 can require up to four steps in which earlier actions build a system of objects that must function together.

The embodied setup is deliberately bi-manual. That choice is structurally important because many tasks require one hand to stabilize an object while the other manipulates another object, or require multi-object composition that is awkward or impossible with a single arm. All tasks are instantiated in Genesis, which allows common evaluation conditions across rigid, soft, and fluid-like interactions. The benchmark also includes demonstrations for the 30 seed tasks, and the main training protocol uses 50 human-teleoperated demonstrations for 10 of the seed tasks (Lin et al., 28 May 2026).

3. Reasoning taxonomies and representative tasks

RoboWits organizes its tasks into three major reasoning families: geometry-based reasoning, material-based reasoning, and assembly reasoning. Across the benchmark, appendix statistics report 264 object instances, 13 object categories, 14 spatial relations, 32 manipulation primitives, and >75 reasoning patterns described as 30 Seed + 45 Pivot.

In the geometry-based family, the robot must reason about fit, alignment, support, or reorientation rather than merely execute local grasping. In Align Blocks, the challenge is to align three cubes in a straight line; the intended reasoning is that a ruler can serve as a straightedge constraint. In Gap Retrieve, a lemon lies in a narrow channel too tight for the gripper, so the environment must first be modified by moving one boundary block. In Stand Bulb, a tall object with a curved base is stabilized by placing a ring around the base to provide lateral support. These tasks probe whether the robot can treat objects as geometric constraints rather than only as targets for pick-and-place.

In the material-based family, the relevant variable is often hidden and must be inferred through interaction. Pinch Card requires exploiting the friction and deformability of an eraser to push a flat bank card until it becomes graspable. Hold Cup requires using cloth as a high-friction interface so that a cup can remain on a sloped target area. Water into Mug requires using a heavy object to displace water from a pitcher into a mug, explicitly invoking Archimedes’ principle. Differentiate Cubes asks the robot to separate visually similar cubes by buoyancy after adding water. These tasks test whether policies can exploit friction, softness, buoyancy, compressibility, or fluid displacement when those properties are not directly visible.

In the assembly family, success depends on building a functional multi-object system. Ball onto Tower requires stacking rings so that the top ring protrudes and forms a hollow cup-like surface. Separate Marbles and Sand requires assembling a colander over a bowl so that gravity separates the contents. Raise Platform requires constructing an elevated platform from supports and a deck, then placing a book on it. These tasks convert object manipulation into compositional physical reasoning.

Several seed tasks were designed so that a natural first strategy exists but can later be invalidated by mutation. Retrieve Cube is exemplary: the seed solution is to lift a narrow container and pour the cube out, but later variants can fix the container to the table or alter the environment so that the robot must use a spoon or water-based strategy instead. In this sense, the task taxonomy is inseparable from the benchmark’s mutation logic (Lin et al., 28 May 2026).

4. Automated task generation and mutation pipeline

A defining contribution of RoboWits is its automated task generation pipeline formulated as a multi-agent cooperative framework. The pipeline comprises five agents: Seed Task Generator, Task Verifier, Metric Generator, Task Mutator, and Scene Generator. This design was introduced to make large-scale generation of reasoning-centric unexpected scenarios feasible without hand-authoring each benchmark instance.

The Seed Task Generator proposes initial tasks within the benchmark’s three taxonomies and outputs a natural-language instruction, a structured object list, a potential solution used only for internal verification, and a short explanation of why reasoning is required. Objects are represented by functional roles rather than only by specific names; an object schema includes object_name, appearance_attribute, functional_attribute, and potential_instances. That abstraction is meant to preserve reasoning patterns while permitting multiple concrete instantiations.

The Task Verifier is a tool-augmented, simulator-aware agent that evaluates each proposed task along three axes: simulatability, solution feasibility, and solution efficiency / shortcut avoidance. A task is rejected if it cannot be instantiated in the simulator, requires capabilities beyond a dual-arm robot with parallel grippers, or admits a significantly easier bypass solution. The verifier also assigns an operational difficulty score $s_o$.

The Task Mutator is the mechanism that produces unexpected challenges. It applies three mutation strategies: pivot, trap, and add. A pivot mutation blocks the original solution and introduces a different valid strategy. A trap mutation introduces deceptive objects that look useful but are subtly ineffective, such as a spoon that is too large to fit or a soft board that resembles a lever. An add mutation inserts distractor objects that increase clutter without changing the task goal. The mutator and verifier operate iteratively for up to three rounds so that invalid or trivial mutations are revised rather than simply discarded.

The benchmark’s mutation statistics are informative. Pivot mutations number 45 with mean $\Delta$ difficulty 0.26; Trap mutations number 62 with mean $\Delta$ difficulty 0.42; and Add mutations number 70 with mean $\mathcal{I}$0 difficulty 0.09. Trap mutations therefore increase difficulty the most on average, while pivot mutations most strongly alter the task structure and can either raise or lower difficulty depending on the alternate solution.

The Scene Generator instantiates the task as a physically plausible 3D scene by retrieving or generating assets, assigning physical properties such as material type, density, and friction, placing objects, checking stability, rendering multiple views, and verifying post-step stability. The Metric Generator produces executable evaluation code for each seed task, including binary success and progress score. A crucial scaling decision is that all mutations of a seed task preserve the same goal and therefore reuse the same metric. Only 30 unique scripts therefore require verification to support the full 208-task benchmark (Lin et al., 28 May 2026).

Mutation type	Mechanism	Mean $\mathcal{I}$1 difficulty
Pivot	Blocks the current solution and requires a different valid strategy	0.26
Trap	Adds deceptive but ineffective objects	0.42
Add	Inserts distractors and clutter	0.09

5. Evaluation protocol and empirical findings

The benchmark evaluates three system classes: a transformer-based imitation policy ACT; pre-trained VLAs $\mathcal{I}$2 and $\mathcal{I}$3; and modular VLM-based systems, including VLM Planner$\mathcal{I}$4 with oracle access to object states and perfect action primitives, and VLM Controller$\mathcal{I}$5, which also uses GPT-4o for planning but relies on a human-scripted low-level controller. ACT is trained for 100,000 steps with chunk size 50 and batch size 256. The two VLA models are each fine-tuned for 100,000 steps with effective batch size 64.

Training uses 50 human demonstrations on 10 seed tasks. Evaluation is conducted on the seed tasks for 50 trials each and on each mutation for 5 trials each, with randomized object positions and rotations. The main metrics are SR for success rate and PS for average progress score.

At the aggregate level, the central result is a sharp gap between reasoning under oracle state and embodied action. VLM Planner$\mathcal{I}$6 achieves 60.8% seed SR and 44.4% mutation SR. By contrast, ACT achieves 11.0% seed SR and 4.8% mutation SR; $\mathcal{I}$7 achieves 12.2% and 6.5%; and $\mathcal{I}$8 achieves 12.6% and 9.8%. The corresponding average progress scores also drop from seed to mutation: ACT 0.43 → 0.31, $\mathcal{I}$9 0.46 → 0.32, and $\mathcal{O}$0 0.45 → 0.31. The benchmark therefore isolates not only a reasoning gap but also a large embodiment gap between high-level task inference and reliable low-level execution.

Model	Seed SR	Mutation SR
VLM Planner$\mathcal{O}$1	60.8%	44.4%
ACT	11.0%	4.8%
$\mathcal{O}$2	12.2%	6.5%
$\mathcal{O}$3	12.6%	9.8%

Task-level results show that the failure modes are structured rather than uniform. Retrieve Cube is solved by the oracle-state planner at 94% on the seed task but only 20% on mutations, while the learned robot policies are near zero. Pinch Card is difficult even for the planner, with 24% seed SR and 0% mutation SR, and all robot policies fail. Water Mug is solved by none of the reported systems. Hold Cup is an outlier in the opposite direction: the planner achieves 100% on both seed and mutation, while the robot and VLA methods remain essentially in the 0–10% range. Dominos is notable because the learned policies can perform relatively well on the seed task—ACT 82%, $\mathcal{O}$4 92%, $\mathcal{O}$5 80%—yet still show substantial drops under mutation.

Ablations reinforce the benchmark’s emphasis on online adaptation. Closed-loop replanning improves over open-loop planning by 27.4 percentage points on seed tasks and 29.3 percentage points on mutation tasks. Mutation-specific analysis shows that pivot is the hardest mutation type, followed by trap; add mainly degrades performance through clutter. For example, average success on pivot mutations is 1.7% for ACT, 2.0% for $\mathcal{O}$6, and 3.7% for $\mathcal{O}$7. Demonstration scaling helps but does not remove the reasoning bottleneck: for $\mathcal{O}$8, increasing demonstrations from 50 to 200 raises Align Blocks from 44% to 54% on the seed task and from 24% to 33% on mutations, and raises Dominos from 70% to 94% on the seed task and from 47% to 52% on mutations. The appendix also reports that replacing GPT-4o with Gemini 3 Pro in the planner yields 69.6% seed SR and 47.3% mutation SR (Lin et al., 28 May 2026).

6. Significance, limitations, and research trajectory

RoboWits is significant because it operationalizes a distinction that had often remained anecdotal in robot manipulation: the distinction between executing a familiar motor pattern and creatively solving a physical problem under unexpected constraints. In its benchmark comparison, RoboWits is characterized by bi-manual tasks, an automatic generation pipeline, 208 tasks, success + score evaluation, and explicit support for diverse materials including fluids and soft bodies, creative tool use, and strategy adaptation. The mutation framework is more functionally disruptive than ordinary domain randomization because it alters latent affordances rather than only textures, poses, or language.

The benchmark also exposes a specific fault line in current systems. The relatively strong performance of oracle-state VLM planning suggests that some high-level reasoning is already accessible when symbolic state and perfect action primitives are available. The low performance of end-to-end robot policies and fine-tuned VLAs shows that mapping such reasoning into robust embodied behavior remains difficult. The benchmark’s failure cases—persistently manipulating immovable objects, chasing deceptive lookalikes, or failing to exploit hidden material properties—suggest that present systems rely heavily on superficial affordance cues and weakly grounded strategy priors.

The current framework remains simulation-based, and the paper explicitly notes the sim-to-real gap. It also notes that rich physical phenomena such as soft bodies or fluids increase computational cost, and that true zero-shot reasoning-heavy manipulation remains largely out of reach, which is why evaluation still relies on fine-tuning and limited demonstrations. Future directions implied by the benchmark include developing reasoning-driven VLA models, improving robustness under mutation-style shifts, strengthening closed-loop replanning, scaling the task-generation pipeline further, and using RoboWits to localize whether failure originates in perception, action, planning, material reasoning, or geometric inference (Lin et al., 28 May 2026).