COMPARE Ecosystem for Robot Manipulation
- COMPARE Ecosystem is a community-driven framework enhancing modularity and standardization for robot manipulation software.
- It integrates component and pipeline-level benchmarks to support reproducible evaluations in grasping and pick-and-place tasks.
- The ecosystem curates repositories and implements ROS 2-based pipelines to simplify open-source product interoperability.
Searching arXiv for papers on the COMPARE Ecosystem and related robot manipulation benchmarking/modularity. The COMPARE Ecosystem is a community-driven ecosystem for improving the compatibility and benchmarking of open-source products for robot manipulation, with an initial emphasis on robot grasping and pick-and-place pipelines. It was created in response to fragmentation in robot manipulation software, where most researchers contribute a single novel subsystem while the remaining hardware and software stack is assembled from lab-specific, often nonstandard components, making true side-by-side comparisons difficult. In this setting, COMPARE aims to increase cohesion between open-source products, promote modularity and interoperability, and support reproducible benchmarking through repositories, standards, guidelines, and validation procedures (Zhao et al., 26 Sep 2025).
1. Origin, motivation, and scope
The central motivation for COMPARE is the structure of contemporary robot manipulation research. Manipulation systems require many interacting subsystems, but most research contributions focus on only one of them. The remaining perception, planning, execution, and integration code is often inherited from local infrastructure, frequently built with ROS, and not designed for direct substitution or comparative evaluation. This makes it difficult to perform true side-by-side comparisons across methods, even when nominally similar tasks are being studied (Zhao et al., 26 Sep 2025).
Workshops and surveys of the robot manipulation community identified three recurring limitations that shaped the ecosystem: difficulty integrating open-source products, a lack of truly modular software, and limited support for both component-level and holistic system evaluations. COMPARE is explicitly linked to broader reproducibility concerns in robotics code and experimentation. Its goals therefore include organized repositories of open-source products and benchmarking assets, standards and guidelines for software development, benchmarking, and reporting, and test and evaluation procedures to validate both individual products and their compatibility within larger pipelines (Zhao et al., 26 Sep 2025).
The scope described in the current work is deliberately focused. The reported activity is a work-in-progress centered on standards and guidelines for robot grasping and manipulation pipelines, initially targeting object grasping and pick-and-place. This focus gives the ecosystem a concrete domain in which interoperability requirements, benchmarking procedures, and software interfaces can be specified in detail without treating “robot manipulation” as an undifferentiated whole (Zhao et al., 26 Sep 2025).
2. Architectural view of manipulation within COMPARE
COMPARE distinguishes two levels of standardization: the component level and the pipeline level. At the component level, the concern is the modularization and characterization of individual open-source software modules such as perception, grasp planning, and motion planning. At the pipeline level, the concern is the integration of these modules into task-complete chains that execute robot capabilities and support benchmarking (Zhao et al., 26 Sep 2025).
This two-level distinction is foundational. Component-level standardization is intended to make a module reusable independently of any particular pipeline. Pipeline-level standardization is intended to make multiple modules interoperable, so that components can be swapped in and out with minimal engineering overhead. A plausible implication is that COMPARE treats reproducibility not merely as replication of an entire stack, but as controlled recombination of compatible subsystems under shared interfaces.
The paper identifies a representative manipulation pipeline consisting of perception, grasp planning, motion planning, execution, and benchmarking actions. Benchmarking is integrated into the operational pipeline rather than appended informally after the fact. An example benchmarking procedure includes actions such as lifting and shaking the object to verify that the grasp is firm. This indicates that, within COMPARE, benchmarking is conceived as a first-class pipeline function rather than a separate reporting layer (Zhao et al., 26 Sep 2025).
The same logic appears in the ecosystem’s stated aim that components be “easily swapped in and out.” In practical terms, this requires agreed input/output data formats, common interfacing techniques, compatible ROS service architectures, and disciplined execution semantics. The standards effort is therefore not limited to style guidance or repository hygiene; it is directed at the technical conditions required for interchangeable manipulation modules (Zhao et al., 26 Sep 2025).
3. Repositories and characterization of open-source products
One major activity in the ecosystem is the construction of repositories of open-source products on Robot-Manipulation.org. These repositories were initially created from literature reviews of prominent software contributions and are characterized by parameters that are useful both for users selecting software and for identifying common structural roles within manipulation pipelines (Zhao et al., 26 Sep 2025).
Repositories explicitly mentioned include Grasp Planning, Motion Planning, Perception, Learning, and Simulation. Their purpose is not only cataloging. By recording shared salient characteristics, the repositories provide an empirical basis for deciding which standards are needed. For example, if several grasp planners consume point clouds and output 6-DoF grasps, that convergence suggests a candidate interface standard. This suggests that COMPARE uses repository curation as an evidence-generating activity for standard formation, rather than as a purely descriptive index (Zhao et al., 26 Sep 2025).
The paper gives Grasp Planning as a concrete example of component characterization. The axes used for grasp planners include planning method, end-effector hardware, input data, and output pose format. For grasp datasets, similar axes are recorded together with metrics such as number of grasps and number of objects. These attributes are explicitly integration-relevant: input data format constrains upstream perception modules, while end-effector hardware constrains the interpretation of output poses and downstream execution (Zhao et al., 26 Sep 2025).
The following table summarizes the grasp-planner characteristics explicitly listed in the paper.
| Grasp Planner | Input Data | Output Pose |
|---|---|---|
| 6-DoF GraspNet | point cloud | 6-DoF grasp pose |
| Contact-GraspNet | point cloud | 6-DoF grasp pose |
| DeepRLManip | depth image | grasp policy |
| GP-Net | point cloud | 6-DoF grasp pose |
| GraspIt! | 3D object model mesh | 6-DoF grasp pose and force closure metrics |
| GraspSAM | RGB image | 2D grasp rectangle |
| ICG-Net | point cloud | 6-DoF grasp pose |
| PointNetGPD | point cloud | 6-DoF grasp pose |
| UniGrasp | point cloud | grasp points |
| VGN | depth image | 6-DoF grasp pose |
The paper also records Camera PoV for these planners, including eye-in-hand, overhead, oblique, multi-view, and agnostic configurations (Zhao et al., 26 Sep 2025). The significance of these metadata is methodological: compatibility in manipulation pipelines depends not only on algorithmic quality but also on representational assumptions. A planner that emits a 2D grasp rectangle is not interchangeable with one that emits a 6-DoF grasp pose unless an explicit translation layer exists. COMPARE’s repositories are therefore an instrument for revealing hidden incompatibilities that would otherwise frustrate reproducible comparison.
4. Standards and guidelines at component and pipeline levels
At the component level, the proposed guidelines are meant to apply to individual open-source modules. The paper states that these guidelines will include recommendations for specifying dependencies, resolving compatibility issues, Dockerizing and deployment practices, mitigating CUDA and graphics/rendering issues, and general open-source development practices (Zhao et al., 26 Sep 2025).
These concerns are narrowly technical and reflect common failure modes in manipulation software deployment. Dependency specification and compatibility resolution are prerequisites for reproducible builds. Dockerization and deployment practices address environment portability. CUDA and graphics/rendering issues are singled out because vision-based manipulation pipelines often depend on GPU-accelerated inference and rendering stacks that are difficult to reproduce across machines. This suggests that COMPARE defines software quality partly in terms of integrability under realistic deployment constraints, not solely in terms of algorithmic correctness.
At the pipeline level, the standards focus on how modules connect and execute collectively. The paper says these standards will specify common interfacing techniques between components, input/output data formats, ROS service architectures, execution of robot capabilities, and benchmarking procedures (Zhao et al., 26 Sep 2025). The paper provides a representative example of the intended normative form:
“all grasp planners that output a 6-DoF grasp pose shall do so using X specified format as a standard input to the motion planner in the pipeline.” (Zhao et al., 26 Sep 2025)
This example is important because it captures the functional role of COMPARE standards. The objective is not to prescribe internal algorithms but to standardize the interface contract at the system boundary where one component hands control to another. If grasp planners emit a common pose representation, motion planners can be reused across pipelines and planners can be benchmarked more fairly. A plausible implication is that COMPARE is organized around interoperability by contract: modules may differ internally, but they must conform externally if they are to be comparable within a shared benchmark framework.
5. Existing modular pipelines as reference implementations
A second major activity is the investigation of existing modular pipelines in order to extract best practices. Two examples are highlighted: GRASPA and SceneReplica (Zhao et al., 26 Sep 2025).
GRASPA provides a grasp-planning benchmarking protocol in a reproducible Docker container, with three grasp planners. Its role in the paper is as an existence proof that containerized, reproducible benchmarking can be operationalized for grasp planning. However, the more important reference system for COMPARE is SceneReplica, because it supports modular, interchangeable components across perception, grasp planning, motion planning, and control rather than benchmarking only a single stage (Zhao et al., 26 Sep 2025).
SceneReplica offers two perception-to-action pipelines. The model-based grasping pipeline is for known objects and uses image-based pose estimation and offline grasp optimization. The model-free grasping pipeline is for unknown objects and uses online segmentation and point-cloud-based grasp detection. In the model-based path, example modules include pose estimation with PoseCNN or PoseRBPF, grasp processing with GraspIt!, and motion planning/execution. In the model-free path, example modules include segmentation with UOIS or MSMFormer, grasp sampling with 6D-GraspNet or Contact-GraspNet, grasp filtering, and motion planning/execution (Zhao et al., 26 Sep 2025).
The authors replicated SceneReplica locally and studied services and nodes per stage, component parameterization, publish/subscribe communication, runtime control logic, data flow and coordination, and compatibility problems, especially around CUDA and graphics inside Docker (Zhao et al., 26 Sep 2025). This replication activity is methodologically significant because it treats an existing pipeline as a source of concrete interoperability knowledge. COMPARE does not posit modularity abstractly; it derives draft standards partly from the detailed failure points and interface conventions observed in a working multi-stage system.
The paper also notes that SceneReplica will serve as a benchmarking reference in future COMPARE evaluation, with the intent to follow the SceneReplica protocol and update the SceneReplica leaderboard on Robot-Manipulation.org (Zhao et al., 26 Sep 2025). This indicates that COMPARE is positioned not as a competitor to prior benchmark infrastructures, but as an ecosystem layer that can systematize and extend them.
6. New infrastructure: the ROS 2/FlexBE modular pipeline
The third major activity is the development of new modular pipelines that improve on prior work while following the proposed standards. The new infrastructure is based on ROS 2, the FlexBE behavior engine, a modular service/action architecture, and support for ROS 1 components via ros1_bridge (Zhao et al., 26 Sep 2025).
The goal is a community-usable infrastructure in which users can use existing ROS packages, create their own components, “drop in” components that comply with COMPARE standards, and make minimal changes to the infrastructure. The paper further states that the infrastructure is intended to be hardware-agnostic, provided the hardware is ROS compatible and has a driver (Zhao et al., 26 Sep 2025). This hardware agnosticism is conditional rather than absolute: it does not eliminate hardware dependencies, but it relocates them to the ROS driver boundary.
The architectural core is a FlexBE-based ROS 2 pipeline organized around hierarchical finite state machines and single-purpose states. Each FlexBE state requests a single service, each state performs a single task, states can be modified via drag-and-drop, timers and checks can be added for debugging and benchmarking, and states can be templated with boilerplate code for easier development (Zhao et al., 26 Sep 2025). The emphasis on single-purpose states mirrors the component philosophy of the ecosystem more broadly: discrete software units should have explicit responsibilities and interfaces.
The paper makes a conceptual distinction between user data variables and parameters. User data variables are shared resources inside the behavior and can be read or updated by states if they are declared as input/output keys; parameters behave like ROS parameters, being shared and accessible but not modifiable during runtime (Zhao et al., 26 Sep 2025). The authors stress that data is not literally passed between states. Instead, each state accesses live data, performs a service or action request, then may update user data before completion triggers the next state. This execution model matters because it clarifies the semantics of state interaction and avoids an overly informal notion of pipeline “passing” that could obscure the actual data dependencies.
Service and action server nodes follow the same modular discipline. Each node waits for a request and performs one task, such as sending MoveIt commands, processing a point cloud, or starting or stopping recording software (Zhao et al., 26 Sep 2025). To support ROS 1 packages such as GPD, the architecture includes a second machine with both ROS 1 and ROS 2 workspaces bridged by ros1_bridge (Zhao et al., 26 Sep 2025). This bridging arrangement reveals a practical design principle within COMPARE: compatibility is to be extended where possible to legacy components rather than requiring a clean-sheet ROS 2 ecosystem.
A concrete example is given for a FlexBE pipeline implementing model-free grasping followed by pick and place. Example perception states include GetPointCloud, VoxelGridFilter, OutlierRemoval, PlaneSegmentation, and ClusterExtraction. Example grasping states include DetectGrasps, FilterCandidates, and CalculateGraspPoses. Example pick-and-place task states include MoveToApproach, PerformGrasp, and MoveToDropoff (Zhao et al., 26 Sep 2025). Inputs and outputs are explicitly named, including camera_topic, leaf_size, radius_max, full_cloud, filtered_cloud, plane_indices, cluster_indices, grasp_candidates, grasp_poses, tool_width, outcome, step_size, and dropoff_pose. This explicitness is precisely what the standards effort seeks to normalize.
7. Benchmarking logic, evaluation plans, and significance
COMPARE is intended to support both component-level comparisons and system-level comparisons. The next phase described in the paper is to use the infrastructure to integrate multiple open-source components, conduct side-by-side benchmarking for pick-and-place, follow the SceneReplica protocol, and update the SceneReplica leaderboard on Robot-Manipulation.org (Zhao et al., 26 Sep 2025). Initially, motion planning is expected to remain fixed, using something like OMPL in MoveIt, while perception and grasp planning components vary (Zhao et al., 26 Sep 2025). This reflects a controlled experimental design in which one subset of the pipeline is held constant while another is compared.
The planned physical setup includes a Universal Robots UR5e arm, a Robotiq 2F-85 end-effector, a wrist-mounted Intel RealSense camera, and YCB objects in cluttered scenes (Zhao et al., 26 Sep 2025). Performance will be measured using grasp execution success rates, pick-and-place success rates, and physical trials ordered near-to-far (Zhao et al., 26 Sep 2025). These details establish the benchmark as embodied and task-grounded rather than purely simulated.
The broader significance of the ecosystem lies in its attempt to transform open-source manipulation software from isolated lab-specific stacks into reusable, comparable building blocks. The paper states that this should enable easier integration of new perception or grasping methods, more reproducible benchmarks, clearer comparison of algorithms under shared interfaces, reduced engineering cost for assembling experimental pipelines, and stronger community adoption of standardized open-source software (Zhao et al., 26 Sep 2025).
Two misconceptions are usefully avoided by the work. First, COMPARE is not presented as a finished standard; it is explicitly a work-in-progress centered on repositories, draft standards, and new modular pipelines (Zhao et al., 26 Sep 2025). Second, the ecosystem does not claim that all manipulation components are already interchangeable. On the contrary, the paper’s detailed attention to data formats, camera viewpoints, end-effector assumptions, Docker/CUDA issues, and ROS bridging underscores that interchangeability is currently difficult and must be engineered deliberately. The practical value of COMPARE is therefore not that it eliminates complexity, but that it makes the conditions of modularity, comparability, and reproducibility explicit within a shared ecosystem framework (Zhao et al., 26 Sep 2025).
