RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (2506.06677v1)

Abstract: Recent advances in vision-LLMs (VLMs) have enabled instruction-conditioned robotic systems with improved generalization. However, most existing work focuses on reactive System 1 policies, underutilizing VLMs' strengths in semantic reasoning and long-horizon planning. These System 2 capabilities-characterized by deliberative, goal-directed thinking-remain under explored due to the limited temporal scale and structural complexity of current benchmarks. To address this gap, we introduce RoboCerebra, a benchmark for evaluating high-level reasoning in long-horizon robotic manipulation. RoboCerebra includes: (1) a large-scale simulation dataset with extended task horizons and diverse subtask sequences in household environments; (2) a hierarchical framework combining a high-level VLM planner with a low-level vision-language-action (VLA) controller; and (3) an evaluation protocol targeting planning, reflection, and memory through structured System 1-System 2 interaction. The dataset is constructed via a top-down pipeline, where GPT generates task instructions and decomposes them into subtask sequences. Human operators execute the subtasks in simulation, yielding high-quality trajectories with dynamic object variations. Compared to prior benchmarks, RoboCerebra features significantly longer action sequences and denser annotations. We further benchmark state-of-the-art VLMs as System 2 modules and analyze their performance across key cognitive dimensions, advancing the development of more capable and generalizable robotic planners.

• The paper proposes a novel benchmark that shifts robotic manipulation evaluation from reactive System 1 to deliberative System 2 reasoning.
• It introduces a hierarchical framework integrating vision-language models with action controllers to facilitate structured long-horizon task planning.
• Extensive simulation data and statistical analysis reveal enhanced planning accuracy and robust performance in complex, temporally rich tasks.

RoboCerebra: Benchmarking Long-horizon Robotic Manipulation

The paper "RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation" introduces a novel benchmarking platform aimed at evaluating long-horizon reasoning in robotic manipulation tasks. RoboCerebra addresses the deficiencies in existing benchmarks, particularly in capturing System 2 capabilities pertinent to long-term planning and high-level reasoning.

Benchmark Overview

RoboCerebra represents a transition from the reactive System 1 approach to robotic imitation learning towards an emphasis on deliberative System 2 reasoning. Emphasizing tasks composed of extended subtask sequences, it provides a comprehensive benchmark that features significantly longer trajectories compared to existing benchmarks—approximately six times longer—with denser annotations. This is achieved through a substantial simulation dataset that focuses on household environments, capturing extended task horizons and diverse subtask sequences.

Hierarchical Framework

The paper proposes a hierarchical framework that integrates both high-level vision-LLMs (VLMs) and low-level vision-language-action (VLA) controllers. This framework allows for structured interaction between System 1 and System 2, facilitating evaluations of planning, reflection, and memory within long-horizon tasks. The hierarchical design aims to enhance semantic reasoning while ensuring precise control—a crucial requirement for dynamic long-horizon tasks.

Data Generation and Analysis

A top-down data generation pipeline forms the backbone of RoboCerebra, leveraging GPT to produce sophisticated task instructions and subtask decompositions. Human operators execute these subtasks within simulations to generate high-quality trajectories, introducing dynamic object variations to ensure semantic diversity. The resulting dataset offers extensive temporal richness and detailed annotations, ultimately supporting comprehensive evaluations of robotic planning and reasoning capabilities. Statistical analyses reveal a broader distribution of trajectory lengths and task types, capturing the complexity of real-world manipulation tasks.

Evaluation Protocol

The paper presents a multidimensional evaluation protocol that goes beyond traditional binary success metrics. Several evaluation dimensions are considered, including task success rate, planning accuracy, planning efficiency, and action completion accuracy. The authors test various System 2 models, including pretrained and supervised fine-tuned VLMs. Performance analyses highlight differences among these models' reasoning capabilities, emphasizing the utility of the hierarchical framework to improve task success rates in complex scenarios.

Implications and Future Directions

RoboCerebra enables investigations into advanced robotic planning and execution strategies, promoting the development of more adaptable and capable robotic planners. The benchmark's implications extend both practically and theoretically by offering a rigorous methodology for evolving System 2 reasoning in robots. Looking ahead, the exploration of bidirectional communication between high-level reasoning and low-level control could further enhance the interpretability and robustness of autonomous systems. Extending these tasks into real-world settings would validate long-horizon reasoning capabilities under realistic conditions, offering fertile grounds for future research in AI-driven robotics.

In summary, RoboCerebra represents a significant step towards advancing the capabilities of robotic systems in handling long-horizon tasks. By focusing on high-level reasoning, structured planning, and the dynamic interaction between System 1 and System 2, it opens new avenues for the exploration of AI in complex, temporally rich environments.