RoboWits: Unexpected Challenges for Robotic Creative Problem Solving

Abstract: The ability to reason, adapt, and creatively solve problems under unexpected challenges is essential for robots operating in real-world environments. However, current robotic benchmarks primarily emphasize skill-level execution and provide limited insight into such cognitive reasoning capabilities. We introduce RoboWits, a bi-manual robotic benchmark designed to systematically evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions. To enable scalable construction of high-quality reasoning-centric unexpected scenarios, we propose an automated task generation pipeline formulated as a multi-agent cooperative framework, comprising agents for seed task generation and verification, metric generation, scene generation, and task mutation. Using the pipeline, we curated 30 diverse seed tasks and 208 tasks with mutations and graded difficulty across geometry, material, and assembly-based reasoning. We benchmark popular robot policies, pre-trained VLAs, and oracle-state planners. Our results reveal a significant performance gap: while pre-trained VLAs exhibit preliminary success on seed tasks after single-task fine-tuning, they struggle to perform on mutated tasks, implying their brittleness in manipulation tasks requiring reasoning, strategy adaptation, and robustness to deceptive or constrained environments. Project page is available at https://umass-embodied-agi.github.io/RoboWits.

• The paper presents a novel automated benchmark that systematically challenges robots with reasoning-centric and mutation-based bi-manual tasks.
• It employs a multi-agent pipeline to generate, verify, and diversify tasks across geometry, material, and assembly reasoning, ensuring physical realism and cognitive demand.
• Experimental results reveal that current vision-language and planning models struggle with mutated tasks, highlighting the need for adaptive, closed-loop strategies.

RoboWits: A Benchmark for Robotic Creative Problem Solving Under Unexpected Challenges

Motivation and Problem Setting

The authors of "RoboWits: Unexpected Challenges for Robotic Creative Problem Solving" (2605.30326) address the conspicuous gap in current robotic manipulation benchmarks: the lack of systematic evaluation for cognitive reasoning, creative tool use, and robustness in the face of unexpected challenges. While Vision-Language-Action (VLA) models and related advances have exhibited strong motor-level proficiency on well-specified, single-solution tasks, there remains limited understanding of their ability to generalize strategies or adapt creatively when faced with deceptive cues, occlusions, or perturbations that invalidate naïve approaches. This deficiency restricts the diagnosis of failure modes and obscures the core reasoning shortcomings that hinder progress toward robust, reasoning-driven embodied agents.

Benchmark Design and Automated Task Generation

To address scalable creation of diverse, cognitively demanding tasks, RoboWits introduces an automated, multi-agent pipeline wherein foundation model-driven agents collaboratively generate, verify, and instantiate bi-manual manipulation tasks that necessitate non-trivial physical reasoning. The pipeline's modular agents are tasked with: (1) seed task generation, (2) simulation-aware verification, (3) evaluation metric synthesis, (4) scene construction with high physical realism, and (5) iterative task mutation to produce minimally edited variants (e.g., blocking naive solutions, inserting “trap” distractors, and increasing clutter).

Figure 1: The multi-agent automated task generation pipeline, where specialized agents iteratively generate, verify, and diversify reasoning-centric task specifications and instantiate them in simulation.

Seed tasks span three broad reasoning categories: geometry-based, material-based, and assembly-based, and are instantiated in a high-fidelity physics simulator with diverse 3D assets and materials. The mutation workflow systematizes the generation of more challenging and cognitively taxing variants by either removing key tools (pivot), introducing misleading alternatives (trap), or augmenting scene complexity with irrelevant distractors (add).

Figure 2: Mutation process on a seed task, illustrating pivot, trap, and add operations to systematically increase reasoning complexity and diversify solution space.

Figure 3: Mutation trees for selected tasks, showing the combinatorial expansion of task variants and the associated increase in difficulty and solution diversity.

The final benchmark, RoboWits, comprises 30 seed tasks and 208 mutated variants, each assigned a difficulty rating capturing both execution and reasoning load.

Figure 4: The gallery of 30 bi-manual manipulation tasks constituting the seed set, covering geometry, material, and assembly reasoning.

Figure 5: Left: Distribution of reasoning types in seed tasks. Right: Histogram of difficulty across the 208-task benchmark.

Figure 6: Ego-centric views of the 30 seed tasks to illustrate scene diversity and manipulation requirements.

Task Formalization and Evaluation Metrics

Each RoboWits task specifies a natural language instruction devoid of solution hints, a structured and abstracted object list, a 3D scene configuration, a binary and continuous progress-based evaluation metric, and a difficulty score. All evaluation metrics are programmatically defined and amortized across task mutation families, allowing efficient and objective success/failure assignment.

Importantly, the automated verification pipeline enforces that only tasks requiring genuine reasoning are retained: solutions reliant on high-precision motor skills or the exploitation of simulator artifacts are rejected, and tasks permitting simpler bypass solutions are filtered out, ensuring that each variant genuinely tests creative adaptation rather than rote execution.

Experimental Evaluation and Baseline Performance

The empirical study rigorously benchmarks a suite of policies: from pre-trained VLAs (e.g., $\pi_0$, $\pi_{0.5}$), from-scratch imitation models (ACT), to hierarchical planners utilizing vision-LLMs (VLMs) with oracle state information. Training uses 50 human teleoperation demonstrations per seed task, with models tested on both seen seeds and unseen mutations under randomized object placements.

The numerical results highlight several critical findings:

• VLA models (e.g., $\pi_0$, $\pi_{0.5}$) demonstrate only limited success even on seed tasks after fine-tuning, with performance collapsing on mutated variants (e.g., progress scores halved and success rates dropping from ~12% to ~7% on average).
• VLM-based planners equipped with oracle state access outperform end-to-end policies in adaptability, but still suffer significant performance drops on mutations, underscoring systemic brittleness.
• The open-loop execution paradigm is severely deficient compared to closed-loop, reactive planning, with closed-loop VLM-planners outperforming open-loop variants by 27 points on seed tasks and 29 on mutations.
• Scaling demonstration data improves performance on tasks with single dominant strategies (e.g., Dominos from 70% to 94%), but improvements do not transfer to mutation variants, indicating a reasoning rather than data bottleneck.

Analysis, Implications, and Future Work

RoboWits establishes a new standard for diagnosing reasoning and adaptation in robotic learning. The systematic and scalable generation of challenging, multi-step, bi-manual tasks reveals the fundamental brittleness of current VLAs and modular planners when their learned priors are invalidated or when the environment presents deceptive affordances not in training data. The benchmark demonstrates explicitly that fine-tuning and increased demonstration count alone do not overcome reasoning failures—the ability to flexibly replan, recognize failure, and creatively exploit new affordances is lacking in current architectures.

Figure 7: Contrasting robot behaviors under escalating task difficulty; standard policies fail by repeating invalid actions, while creative solvers dynamically replan and adapt.

These findings necessitate future research focus on:

• Inductive bias and architecture modifications for improved closed-loop, multi-step causal reasoning and tool-use induction.
• Integration of model-based physical prediction and counterfactual simulation to reason under novel constraints.
• Real-to-sim transfer for reasoning-centric benchmarks, addressing the sim-to-real gap present even in state-of-the-art simulation environments.
• Development of diagnostic probes targeting specific reasoning failure modes (e.g., bias toward spurious correlations, inability to detect traps, rigid plan persistence).
• Scalable methodology for unbiased, automated task generation that can extend beyond tabletop bi-manual settings.

Conclusion

RoboWits offers a rigorous, scalable, and reasoning-centric benchmark for robotic manipulation under unexpected challenges. The results provide compelling evidence for the limits of current VLA and VLM-driven models in robust problem-solving, despite substantial proficiency in well-specified domains. RoboWits marks a crucial step toward embodied agents that generalize beyond demonstration distributions and execute creative, adaptive problem-solving in the physical world. Its automated task generation pipeline and comprehensive evaluation suite set a new foundation for the development and diagnosis of next-generation reasoning-driven robot policies.