QBIT: Cloud-Based Benchmark for Robotic Insertion

• QBIT is a quality-aware framework that evaluates robotic insertion using metrics such as force energy, smoothness, and completion time.
• It integrates large-scale MuJoCo simulations with Kubernetes orchestration and ROS2-based real-robot validation for robust benchmarking.
• The framework emphasizes reproducibility and practical insight by using randomized trials and comparing trade-offs beyond mere success rates.

QBIT is a quality-aware, cloud-based benchmarking framework for robotic insertion tasks that was introduced to evaluate insertion methods beyond success rate by incorporating force energy, force smoothness, and completion time, while also emphasizing statistical significance, sim-to-real transfer, extensibility, and reproducibility (2503.07479). It targets contact-rich assembly operations such as peg-in-hole and connector insertion, where continuous interaction with the environment, perceptual uncertainty, and nonlinear contact dynamics make nominally successful executions potentially unsuitable for production use. Its design combines large-scale MuJoCo-based simulation, Kubernetes-orchestrated microservices, and ROS2-based physical-robot validation into a single benchmarking workflow.

1. Motivation and problem framing

QBIT was proposed in response to a limitation in prior evaluation practice for robotic insertion: benchmarking had typically focused almost exclusively on success rate, and sometimes on completion time, even though industrial assembly also depends on whether the insertion is gentle, smooth, repeatable, robust, and reproducible (2503.07479). Insertion tasks are difficult because they involve continuous contact dynamics, perception errors in pose estimation and calibration, nonlinear contact behavior that is hard to model accurately, and nontrivial risk and cost when experiments are transferred to physical systems.

Within this framing, QBIT treats two insertion algorithms with the same success rate as potentially very different methods if they expose parts to different force levels, exhibit different force transients, or require substantially different execution times. The framework therefore expands benchmarking from a binary notion of task completion to a multidimensional assessment of execution quality. This suggests a shift from “whether insertion succeeds” to “how insertion succeeds,” which is particularly relevant for manufacturing settings in which excessive contact force can damage parts and unstable motion can signal poor process quality.

2. Architecture and execution model

QBIT is organized as a modular, extensible, microservice-oriented framework with two execution modes: simulation-based benchmarking at scale and real-robot validation through standardized robot interfaces (2503.07479). The architectural principle is strict component isolation: insertion algorithms, simulation environments, task and object definitions, hardware interfaces, and orchestration components are each packaged in their own containers. This containerized decomposition is used to improve deployment portability, reusability, and reproducibility.

For large-scale simulation, QBIT uses a Kubernetes cluster that can run on local edge servers or public cloud infrastructure. Benchmark execution is specified through a YAML file that defines the algorithm container image, simulation instances, randomized simulation parameters, and starting-pose uncertainty. A batch component communicates with the Kubernetes API server to spawn jobs, a shuffle service generates task queues, and the algorithm is exposed as a server while simulation instances act as clients communicating through gRPC. The paper emphasizes an asymmetric resource model in which a small number of inference containers can serve many simulation containers, such as 2 inference instances and 20 simulation instances, thereby decoupling simulation scaling from model-inference scaling.

For physical validation, QBIT adopts ROS2 as the primary robotics middleware, together with ros2_control, MoveIt2, and containerized robot hardware interfaces. The robot interface is standardized around abstractions such as joint position or torque and end-effector position or velocity. The paper notes that direct torque control would be useful for insertion but is not widely available on many robots, so the framework is structured to remain extensible rather than assuming a single control modality.

3. Evaluation metrics and experimental formalism

QBIT evaluates insertion along four main dimensions: success rate, completion time, force energy, and force smoothness (2503.07479). It also separates force-based quality measures into the insertion direction $z$ and the orthogonal plane $xy$, allowing the benchmark to distinguish axial insertion effort from lateral contact behavior.

The insertion begins from an initial pose $H_s \in \mathbb{R}^6$. During execution, the end-effector wrench is measured as

$$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$

where $\mathbf{F} \in \mathbb{R}^3$ is force and $\mathbf{T} \in \mathbb{R}^3$ is torque. For statistical significance, each experiment is repeated $K$ times, yielding wrench sequences

$$\mathbf{w}_i(n) = \begin{bmatrix} \mathbf{F}_i(n), \mathbf{T}_i(n) \end{bmatrix}^T, \quad n \in [0,N].$$

The paper states that torque is omitted from the proposed quality metrics because it is a linear mapping of force and would be redundant for the chosen quality measures.

Perceptual and calibration uncertainty are introduced through a Gaussian start-pose model,

$H_{s\Delta} \sim \mathcal{N}(H_s,\,\Sigma_H).$

Success rate is defined as

$R = \frac{1}{K}\sum_{i=1}^{K} b_i \in [0,1],$

where $xy$0 is the binary success indicator for trial $xy$1, and mean completion time as

$xy$2

Force energy is defined by

$xy$3

The framework computes this separately as $xy$4 in the insertion direction and $xy$5 in the orthogonal plane. Force smoothness is defined as the standard deviation of the derivative of force, again computed separately for $xy$6 and $xy$7. In real experiments, the force signal is low-pass filtered before smoothness is computed in order to reduce sensor noise.

Metric	Definition in QBIT	Notes
Success rate	$xy$8	Binary trial outcome
Mean completion time	$xy$9	Averaged over repeated runs
Force energy	$H_s \in \mathbb{R}^6$0	Reported as $H_s \in \mathbb{R}^6$1 and $H_s \in \mathbb{R}^6$2
Force smoothness	Standard deviation of $H_s \in \mathbb{R}^6$3	Reported as $H_s \in \mathbb{R}^6$4 and $H_s \in \mathbb{R}^6$5

These definitions make repeated randomized trials a first-class part of the benchmark rather than a supplementary check. A plausible implication is that QBIT treats robustness as a distributional property of an insertion policy, not as performance on a single nominal setup.

4. Simulation realism and sim-to-real methodology

A major design objective of QBIT is reduction of the sim-to-real gap for contact-rich insertion (2503.07479). The framework uses MuJoCo as its physics engine because it supports detailed contact modeling while remaining efficient enough for large-scale experimentation. To avoid evaluating algorithms only on a single idealized environment, QBIT randomizes contact parameters and pose uncertainty across repeated trials.

The randomized contact parameters include stiffness, impedance, friction coefficients, and MuJoCo contact options such as o_solref, o_solimp, and geometry sliding friction. The reported ranges are o_solref in $H_s \in \mathbb{R}^6$6, o_solimp in $H_s \in \mathbb{R}^6$7, and sliding friction in $H_s \in \mathbb{R}^6$8. The framework also injects Gaussian white noise into force-torque data in simulation to mimic sensor noise. Taken together with the Gaussian pose model $H_s \in \mathbb{R}^6$9, this means that benchmark scores are produced over a distribution of plausible deployment conditions rather than over a single deterministic contact model.

QBIT also studies contact realism through mesh decomposition. The compared approaches are VHACD, COACD, and a sphere-based decomposition proposed by the authors. The sphere-based method is described as being able to create many contact points and to simulate surface roughness by translating spheres along surface normals. The paper reports that mesh decomposition has a significant effect on contact-force prediction, especially in tight-tolerance insertion, and that the sphere-based decomposition produces more realistic contact behavior than standard convex decomposition methods, albeit with increased simulation runtime. It also notes that convex decomposition methods are highly sensitive to mesh scaling, which makes them difficult to tune reliably for precise insertion.

5. Benchmarked tasks, baselines, and reported findings

QBIT is evaluated on both simulated and physical setups using three insertion approaches: geometric-based, force-based, and learning-based (2503.07479). The real-robot platform uses a UR5e, with the peg fixed to the robot in order to remove grasp uncertainty, and a SCHUNK FTN-AXIA80 six-axis force-torque sensor. The task set includes peg-in-hole insertion at high tolerance $$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$0, low tolerance $$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$1, and smaller connector-like insertions such as USB plugs.

The geometric or position-based approach is a plain position-control baseline that ignores external force feedback and is intended as a simple reference compatible with most robots. The force-based approach uses the control law

$$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$2

$$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$3

where $$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$4 is the actual wrench, $$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$5 the desired wrench, and $$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$6, $$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$7, and $$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$8 are diagonal mass, damping, and stiffness matrices. The learning-based method is InsertionNet (IN), a multimodal force-and-vision model that predicts a residual Cartesian pose correction,

$$\mathbf{w} = \begin{bmatrix} \mathbf{F}, \mathbf{T} \end{bmatrix}^T,$$9

The benchmark’s central empirical result is that success rate alone is insufficient: methods can all achieve successful insertion while differing substantially in force energy, force smoothness, and completion time. The paper reports that the position-controlled baseline is simple but incurs large forces and therefore performs poorly on force-based metrics; the force-controlled method generally minimizes force best but takes longer; and InsertionNet achieves relatively low forces while retaining the flexibility of a learned multimodal policy, though it also tends to have longer completion times than pure force control. QBIT therefore exposes trade-offs that would be hidden by a success-only evaluation.

The paper also reports that randomized simulation better approximates the distribution of real maximum forces than a fixed-contact model, and that cloud-parallelized execution on Kubernetes yields large speedups relative to sequential execution. These findings support the framework’s claim that statistically meaningful insertion benchmarking requires both randomized environment modeling and scalable infrastructure.

6. Reproducibility, availability, and limitations

QBIT’s reproducibility strategy combines containerized software components, standardized ROS2 interfaces, YAML-based benchmark descriptions, repeated trials, explicit randomization distributions, and Kubernetes-managed execution (2503.07479). The framework is designed so that simulation and real-hardware workflows share closely related control abstractions, thereby reducing integration barriers when moving from laboratory evaluation to physical deployment. The paper explicitly presents this as a way to accelerate the transition from laboratory to real-world applications.

The implementation stack comprises MuJoCo for simulation, Kubernetes for orchestration, Docker-style containerization for modular deployment, gRPC for communication between algorithm servers and simulation clients, and ROS2 with ros2_control and MoveIt2 for real-robot integration. The authors also state that the code is available on GitHub at https://github.com/djumpstre/Qbit.

The paper identifies several limitations. Support is currently restricted to robots available in the authors’ laboratory, built-in rendering is limited, and additional contact-rich tasks remain to be added in future work. These constraints indicate that QBIT is best understood as an extensible benchmarking substrate rather than a closed benchmark suite. Even so, the framework’s combination of quality-aware metrics, randomized contact simulation, and cloud-native execution establishes a benchmarking model in which insertion policies are compared not only by whether they finish a task, but by the physical quality, stability, and reproducibility of the process by which they do so.