OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation (2104.11446v2)

Abstract: In this paper, we propose a cloud-based benchmark for robotic grasping and manipulation, called the OCRTOC benchmark. The benchmark focuses on the object rearrangement problem, specifically table organization tasks. We provide a set of identical real robot setups and facilitate remote experiments of standardized table organization scenarios in varying difficulties. In this workflow, users upload their solutions to our remote server and their code is executed on the real robot setups and scored automatically. After each execution, the OCRTOC team resets the experimental setup manually. We also provide a simulation environment that researchers can use to develop and test their solutions. With the OCRTOC benchmark, we aim to lower the barrier of conducting reproducible research on robotic grasping and manipulation and accelerate progress in this field. Executing standardized scenarios on identical real robot setups allows us to quantify algorithm performances and achieve fair comparisons. Using this benchmark we held a competition in the 2020 International Conference on Intelligence Robots and Systems (IROS 2020). In total, 59 teams took part in this competition worldwide. We present the results and our observations of the 2020 competition, and discuss our adjustments and improvements for the upcoming OCRTOC 2021 competition. The homepage of the OCRTOC competition is www.ocrtoc.org, and the OCRTOC software package is available at https://github.com/OCRTOC/OCRTOC_software_package.

• The paper presents OCRTOC, a novel cloud-based benchmark platform offering standardized remote testing on real robots for evaluating robotic grasping and manipulation algorithms.
• Observations from the OCRTOC competition highlight persistent difficulties in robust manipulation in cluttered scenes and significant performance gaps in motion planning with increasing complexity.
• OCRTOC promotes equitable, accessible robotics research through its cloud-based platform, enabling remote testing and offering potential for integrating domain adaptation and real-world variability.

Overview of OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation

The paper presents the OCRTOC benchmark, a novel cloud-based platform designed for evaluating robotic grasping and manipulation algorithms, specifically focusing on table organization tasks. This benchmark was created in response to challenges faced in establishing standardized performance assessment tools for robotic manipulation, a field that has seen substantial advancements in related areas yet lacks robust tools for fair comparison across varied implementations.

Key Contributions

1. Standardized Remote Testing: The OCRTOC benchmark provides identical robotic setups and standardized tasks, enabling researchers to experiment remotely with real robots. Code submissions are executed on a central server, allowing for consistent evaluation across all solutions.
2. Simulation Environment: Alongside the physical robot tests, OCRTOC includes a simulation platform to facilitate development and preliminary testing of solutions. This dual approach ensures robustness and reproducibility in research efforts.
3. Cloud-Based Accessibility: By utilizing a cloud-based infrastructure, the OCRTOC benchmark lowers the barrier for participation and allows researchers worldwide to access and utilize the robotic setups without needing their own hardware, thereby democratizing robotics research.
4. Data-Driven Results: The benchmark evaluates manipulation solutions based on their performance in transforming initial object configurations to target ones. This performance is quantified through metrics such as 3D Euclidean distance errors between estimated and target poses, with upper bounds set to prevent excessive penalization from error outliers.

Observations from 2020 Competition

The benchmark hosted a competition during IROS 2020, attracting 59 teams globally. Despite the simplicity of the challenge, results highlighted the persistent difficulty in achieving robust manipulation in cluttered scenes. Teams generally relied on hybrid approaches integrating both traditional and deep learning methods for pose estimation and planning. However, significant performance gaps in motion planning were evident, particularly as scene complexity increased. These findings underscore the need for more sophisticated planning strategies in the robotic manipulation domain.

Improvements for OCRTOC 2021

Acknowledging feedback and observations from the inaugural competition, OCRTOC is introducing several enhancements for the 2021 edition:

• Flexible Submission Schedule: The competition format has shifted to ongoing monthly submissions throughout the year, accommodating participation amidst varying schedules and promoting sustained engagement.
• Provision of Baseline Solutions: A baseline solution, incorporating planning and perception modules, is provided to guide participants, notably aiding those new to the field in developing their systems without starting entirely from scratch.
• Expanded Dataset and Better Hardware: Updates include labeled real-world datasets for training and transition to 7-axis manipulators with force feedback for enhanced control and motion planning capabilities.

Implications and Future Research Directions

OCRTOC represents a promising advancement in robotic benchmarking, offering new opportunities for developing and testing manipulation algorithms in standardized settings. The cloud-based model notably shifts the paradigm of robotics research towards more equitable and accessible experimentation. Future plans to integrate domain adaptation, real-world variable scenarios, and reinforcement learning techniques into the benchmark may catalyze further progress in achieving adaptable and autonomous robotic systems capable of complex manipulation tasks. Moreover, increasing the repository of labeled data will be crucial in improving learning models, thus bridging the gap between simulated and real-world performance.