COSMO-Bench: Multi-Robot C-SLAM Benchmark

• The paper introduces COSMO-Bench, an open benchmark offering 24 datasets derived from real LiDAR traversals to evaluate distributed optimization in multi-robot C-SLAM.
• It details a comprehensive pipeline integrating SLAM front-end processing, realistic loop closure detection, and communication modeling to simulate dynamic, noisy environments.
• The benchmark fosters reproducible research and fair evaluation by providing standardized, synchronized trials under varying communication conditions (Wi-Fi and Pro-Radio).

The Collaborative Open-Source Multi-robot Optimization Benchmark (COSMO-Bench) is an openly available, rigorously constructed suite of datasets for benchmarking distributed optimization algorithms in Collaborative Simultaneous Localization and Mapping (C-SLAM) with multi-robot teams. By deriving datasets from state-of-the-art LiDAR measurements and explicit communication modeling, COSMO-Bench serves as a standardized foundation for reproducible and realistic assessment of C-SLAM back-end algorithms, directly addressing the lack of dedicated multi-robot benchmark datasets in the robotics research community (McGann et al., 22 Aug 2025).

1. Purpose and Rationale

COSMO-Bench was created to fulfill the need for standardized, open benchmark datasets in multi-robot SLAM research. Existing practices often relied on partitioned single-robot datasets, simulated environments, or closed-source testbeds, which fail to capture the intertwined dynamics and communication complexities of concurrent robot deployments. COSMO-Bench directly targets:

• Realistic inter-robot interactions, including measurement distributions and concurrent communication patterns
• Dynamic environments reflected in measurement noise, outlier rates, and trajectory structure
• The need for reproducible, cross-institutional comparisons of distributed optimization algorithms in C-SLAM

The benchmark suite thus enables researchers to rigorously evaluate and compare methods under unified, real-world conditions.

2. Dataset Composition and Structure

COSMO-Bench comprises 24 benchmark datasets constructed from real-world LiDAR traversals in campus-scale environments. The data generation pipeline involves:

• Use of original data from the Multi-Campus Dataset (MCD) and CU-Multi Dataset
• Synchronized multi-robot deployment simulation via temporal offsets: a reference trial is selected with start time $t^*$, and additional robots are initialized at $t_i = t^* + \Delta_i$, with $\Delta_i$ drawn from a normal distribution
• Realistic scenario construction with both intra-robot loop closures (within single trajectories) and inter-robot loop closures (across different robots)

Each dataset entry encodes metadata: sequence name, dataset source, number of trials, total traversal duration and distance, counts and rates of intra-robot loop-closure (LC) and inter-robot loop-closure (IRLC) measurements, and overall measurement statistics. To paper the impact of communication on algorithm performance, each dataset is released in two versions based on simulated inter-robot communication models (“Wi-Fi” and “Pro-Radio”), resulting in 24 total datasets.

3. Benchmarking Methodology

The COSMO-Bench pipeline consists of three fundamental components:

A. SLAM Front-End Processing

• Odometry is obtained using LOAM (LiDAR Odometry and Mapping), yielding high-frequency poses.
• Odometry is subsampled into keyframes: a new keyframe is created when the robot traverses $d_{kf} = 2\,\mathrm{m}$.
• Intra-robot loop closure detection employs ScanContext descriptors (and the associated RingKey) computed per keyframe for localization.
• Inter-robot loop-closure detection simulates robots sharing raw LiDAR scans, which are processed to find matching scans across robots and compute relative transformations using KISS-Matcher.
• The procedure automatically produces realistic levels of measurement inliers and outliers.

B. Communication Modeling

• Bandwidth sharing is modeled such that, for a maximum bandwidth $B$, robots within interference distance $d_{intf}$ split resources evenly. Effective bandwidth for a robot pair $(a, b)$ is $B_{ab} = B/I_{ab}(t, \Theta^*)$, where $I_{ab}$ is the local interference count.
• Connectivity follows a probabilistic model $\varphi(d_{ab})$, where $d_{ab}(t, \Theta^*)$ is the reference separation. A Bernoulli random variable $$C_{ab}(t, \Theta^*) \sim \mathrm{Bern}(\varphi(d_{ab}(t, \Theta^*)))$$ is sampled per timestep.
• Throughput is computed over a timestep $\delta$ as $$T_{ab}(t) = (B/I_{ab}(t, \Theta^*)) \cdot C_{ab}(t, \Theta^*) \cdot \delta$$.
• Initialization ($d_{init}$) and timeouts are implemented to manage connection lifetimes.

C. Noise and Outlier Modeling

• Reference solutions (computed via survey-grade mapping or LiDAR-Inertial systems) enable measurement covariance modeling in $\mathfrak{se}(3)$ tangent space:

$$Q = \sum_m (r_m r_m^\top), \quad r_m = \log(m^{-1} \circ m^*)$$

where $m$ is a relative pose measurement and $m^*$ the ground truth.

• Measurements with weighted residuals above the 95% $\chi^2$ threshold, given $Q$, are classified as outliers.

This pipeline ensures that benchmarking incorporates realistic data structures, exchange patterns, and measurement ambiguities encountered in multi-robot SLAM.

4. Applications and Use Cases

COSMO-Bench is engineered for a range of evaluation and development tasks, including:

• Comparative benchmarking of C-SLAM back-end algorithms under realistic data, loop closure distributions, and communication constraints
• Algorithmic evaluation in terms of both estimation accuracy (pose error, map consistency) and time efficiency (runtime under limited communication)
• Exploration of communication model impacts, algorithm robustness to different channel conditions (e.g., Wi-Fi vs. Pro-Radio bandwidth/connectivity)
• Reproducible research: datasets can be referenced permanently via DOI (https://doi.org/10.1184/R1/29652158) and are openly available for download at https://cosmobench.com
• Extension to new algorithms and environments, as the datasets are sufficiently varied to represent multi-campus traversals with diverse loop closure and noise statistics

The suite serves as a standard testbed, enabling fair comparisons and accelerating the iterative development cycle in distributed, collaborative SLAM research.

5. Data Access, Format, and Sharing

COSMO-Bench datasets are provided in JSON Robot Log (JRL) format, offering both human readability and platform independence. Each file encodes keyframe indices, loop closure descriptors, pose measurements, noise models, and all relevant metadata for downstream analysis. The hosting arrangement is designed for persistent access (described as “Hosted in perpetuity”) and the open-source ethos enables integration and usage across institutions and toolchains.

Researchers are instructed to check licensing details on the repository to confirm usage permissions, but the intent and permanent DOI signal support for wide, ongoing academic use.

6. Context and Community Significance

By drawing from state-of-the-art SLAM front-end and real-world LiDAR data, and by explicitly simulating inter-robot communication, COSMO-Bench provides a technical bridge from traditional single-robot SLAM benchmarking to the layered complexities of distributed multi-robot C-SLAM. Existing benchmarks have proven transformative in single-agent SLAM research; COSMO-Bench is positioned to provide similar standardization and rigor for collaborative optimization, fostering robust research practices and efficient progression in the field. The benchmark supports the principled evaluation of novel distributed algorithms, the paper of communication constraints, and the advancement of reproducible robotics research.

7. Summary Table: Dataset Key Characteristics

Dataset Source	# Datasets	Communication Model	Data Type
Multi-Campus/CU-Multi	24	Wi-Fi, Pro-Radio	Real LiDAR, JRL

Each dataset includes the synchronized trials, keyframe sampling, loop closure measurements, noise classification, and metadata, reflecting both intra- and inter-robot SLAM conditions.

---

COSMO-Bench establishes a comprehensive, open, and realistic standard for evaluating multi-robot collaborative SLAM optimization algorithms, enabling the rigorous advancement, benchmarking, and reproducibility that is foundational for progress in multi-robot systems research (McGann et al., 22 Aug 2025).