RMIS Benchmarking in Robotic Surgery

• RMIS Benchmark is a framework that defines evaluation protocols for robotic minimally invasive surgery, encompassing skill assessment, error detection, force feedback, and control metrics.
• It integrates diverse benchmark layers such as dry-lab tasks, public datasets like JIGSAWS, and multimodal acquisition to capture clinically relevant failure modes.
• It establishes reproducible protocols and quantitative metrics across kinematics, pose estimation, and constrained control to drive innovation in surgical robotics.

Searching arXiv for recent and foundational papers on RMIS benchmarking, JIGSAWS, and related benchmark-style evaluations. arXiv search query: RMIS benchmark JIGSAWS robotic minimally invasive surgery error detection pose estimation multimodal dataset “RMIS Benchmark” most commonly denotes the family of benchmarks, benchmark-style evaluations, datasets, and protocols used to assess methods for robot-assisted minimally invasive surgery rather than a single universally standardized benchmark. In the literature, RMIS benchmarking spans kinematic skill assessment on JIGSAWS, frame-level surgical error detection, force- and feedback-sensitive training protocols, zero-shot 6-DoF surgical tool pose estimation, platform-agnostic multimodal data acquisition, segmentation-aware visual question answering, and constrained inverse kinematics under a remote center of motion constraint. This suggests a heterogeneous benchmark ecosystem in which training, perception, sensing, and control are evaluated with different tasks, modalities, and metrics rather than through one monolithic leaderboard (Zia et al., 2017, Rai et al., 16 May 2025).

1. Benchmark scope and conceptual structure

Within surgical robotics, RMIS benchmarking is organized around repeated dry-lab tasks, annotated surgical datasets, phantom-based perception datasets, multimodal acquisition systems, and benchmark-ready control formulations. The resulting landscape is broad enough that “benchmark” may refer either to a public dataset such as JIGSAWS, to a benchmark-style comparative study such as SurgPose, or to a reproducible evaluation protocol for a narrowly defined capability such as force regulation or constrained inverse kinematics.

Benchmark layer	Representative resource	Principal outputs
Skill assessment	JIGSAWS	skill labels, modified-OSATS, GRS
Error detection	JIGSAWS with frame-level error labels	binary error labels
Force and haptics	ring rollercoaster / ring-on-wire tasks	time, RMS force, TPE, RPE
Perception	SurgPose	ADD, 2D Projection, AP, AR
Multimodal acquisition	MiDAS	synchronized video, hand sensing, pedals
Control	constrained IK with RCM	RCM error, pose error, manipulability, solve time

A recurring design feature is that benchmark validity depends on whether the protocol exposes the clinically relevant failure mode. In training studies, the key issue is frequently the speed–accuracy tradeoff rather than task completion alone. In perception, the decisive nuisance factors are occlusion, reflections, texture scarcity, and unreliable depth. In control, the central benchmark constraint is often the trocar or remote center of motion requirement rather than unconstrained end-effector tracking. These benchmark emphases appear explicitly across the haptics, pose-estimation, multimodal, and constrained-IK literature (Machaca et al., 2022, Weerasinghe et al., 12 Feb 2026, Colan et al., 2024).

2. JIGSAWS and the canonical skill-assessment regime

JIGSAWS is the publicly available benchmark repeatedly used for RMIS skill assessment and error-detection studies. It contains kinematics and video data from 8 participants performing Suturing, Knot Tying, and Needle Passing, and the kinematic time series has dimensionality $D=76$ in the 2017 holistic-feature study. It also provides supervision used in benchmarking through self-proclaimed skill level, modified-OSATS scores, and GRS, and the standard evaluation protocols are LOSO and LOUO (Zia et al., 2017).

Two benchmark traditions emerge from this dataset. The earlier tradition uses handcrafted kinematic summaries and classical classifiers. The 2016 study on suturing alone used completion time, path length, depth perception, speed, smoothness and curvature, with logistic regression and SVM, reporting 85.7% overall accuracy under LOSO and 72.1% overall accuracy under LOUO for binary expert-versus-novice classification. The paper is benchmark-relevant because it explicitly contrasts the easier trial-held-out protocol with the harder subject-held-out protocol and shows that performance drops substantially under subject-independent evaluation (Fard et al., 2016).

The later JIGSAWS skill-assessment study broadened the benchmark targets from classification to score prediction. It evaluated Sequential Motion Texture (SMT), Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT), and Approximate Entropy (ApEn) from robot kinematics, used a nearest neighbor classifier for skill classification and linear support vector regression for exact score prediction, and reported that weighted late fusion achieved up to 0.61 average Spearman correlation coefficient for GRS under LOSO. The same work found that ApEn was strongest for skill classification under the harder LOUO setting, whereas DCT was the strongest individual feature for score prediction and DCT+DFT+ApEn was the best average fused predictor (Zia et al., 2017).

JIGSAWS has also become a benchmark for real-time error detection rather than only skill scoring. “Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos” formulates frame-wise online binary error detection on JIGSAWS using only video, merges all error types into a single normal-versus-erroneous label, uses LOSO, and reports 72.3 frame-level F1, 68.3 frame-level Accuracy, and 56.8 frame-level Jaccard, with 74.6 / 69.8 / 59.8 at window level and 6.69 milliseconds average processing time per frame. A common misconception is that RMIS benchmarking on JIGSAWS is restricted to kinematics and skill labels; this result shows that the same benchmark lineage now supports end-to-end contextual video reasoning for error detection (Shao et al., 2024).

3. Force, haptic feedback, and training benchmarks

Force-sensitive RMIS benchmarking addresses a long-standing platform deficiency: current robotic minimally invasive surgery systems often provide no haptic feedback of tool–environment interaction. The 2022 wrist-squeezing study isolates this issue in a da Vinci S training setup with a ring rollercoaster task, an ATI Mini40 six-axis force/torque sensor, a 50 Hz force-recording pipeline, and a randomized novice study. Its two principal dependent variables are task completion time and root-mean-square force, with RMS force used as an operational measure of accuracy. Participants receiving feedback applied approximately $0.67\,\text{N}$ less force than the no-feedback group across the experiment, while the initial time penalty disappeared over training; average per-trial completion-time reduction was 7.68% with feedback versus 5.26% without. The benchmark implication is direct: force minimization, learning rate, and speed–accuracy coupling should be treated as joint endpoints rather than isolated metrics (Machaca et al., 2022).

The same paper is methodologically important because it standardizes several protocol elements that are benchmarkable in their own right: a commercially recognizable dry-lab task, fixed start and end positions, a required hand-off, prohibited camera movement and clutching, synchronized force and video acquisition, and mixed-effects analysis over repeated trials. It also reports that NASA-TLX-like subjective ratings did not differ significantly between groups for mental demand, physical demand, perceived success, naturalness, frustration, or concentration, suggesting that objective force benefits need not coincide with increased reported burden (Machaca et al., 2022).

A closely related training benchmark concerns feedback timing rather than merely feedback presence. The 2025 virtual ring-on-wire study compared real-time error feedback, trial replay with error feedback, and no error feedback in 42 surgical novices, each completing 42 total trials. Performance was evaluated with Trial time, Translational Path Error (TPE), and Rotational Path Error (RPE). Whole-path TPE did not differ significantly between groups, but RPE improvement did: the real-time group improved significantly relative to control, and segment-wise analysis showed that tightly curved sections were more discriminative for translational accuracy while long straight sections were more discriminative for orientation. This benchmark demonstrates that feedback timing is itself a first-class protocol variable and that spatially resolved scoring can reveal differences hidden by whole-task averages (Gale et al., 25 Aug 2025).

Force-aware benchmarking is also enabled by hardware. The open-source 3-DoF jaw-mounted force sensor paper reports a modular distal sensor with range $\pm 5\,\text{N}$ axially and $\pm 3\,\text{N}$ laterally, average RMSEs below 0.15 N in all directions, and 0.156 N grip-force RMSE during teleoperated mock tissue manipulation. Because it supports bimanual distal force sensing on a dVRK-compatible large needle driver, it supplies the measurement infrastructure needed for benchmarks in haptic feedback, force-control baselines, and force-based skill assessment that cannot be supported by under-phantom force plates alone (Chua et al., 2022).

4. Perception, multimodal acquisition, and scene-understanding benchmarks

Perception benchmarking in RMIS has recently expanded from segmentation and tracking toward pose estimation, multimodal telemetry substitution, and language-conditioned scene understanding. “SurgPose” is best described as a benchmark-style comparative evaluation for zero-shot RGB-D 6-DoF surgical instrument pose estimation. It uses three datasets: Dataset A with 1027 non-occluded stereo images, Dataset B with 797 stereo images containing partial or full occlusion, and Dataset C with 1489 real images plus 1000 synthetic images for segmentation training. The evaluated methods include FoundationPose, SAM-6D, OVE-6D, and MegaPose, and the benchmark reports AP, AR, ADD, and 2D Projection. A critical clarification in the paper is that the evaluation is zero-shot with respect to the pose-estimation architecture, but it is centered on one target instrument category, the EndoWrist Large Needle Driver, so it is not yet a large-scale unseen-category benchmark in the stricter split-design sense (Rai et al., 16 May 2025).

The SurgPose results show how RMIS-specific nuisance factors reshape benchmark outcomes. On the occlusion-heavy Dataset B, the enhanced SAM-6D with Mask R-CNN reaches 49.06% ADD@5 mm and 98.87% 2D Projection@50 px, while FoundationPose reaches 6.02% ADD@5 mm and 28.73% 2D Projection@50 px. The paper repeatedly attributes the gains to stereo-derived depth via RAFT-Stereo and, more strongly, to replacing the default SAM segmentation front-end with a fine-tuned Mask R-CNN. This establishes occlusion, reflectivity, and segmentation quality as benchmark-critical axes in RMIS pose estimation (Rai et al., 16 May 2025).

MiDAS addresses a different perception benchmark problem: access to proprietary robot telemetry. It introduces an open-source, platform-agnostic acquisition system combining electromagnetic hand tracking, RGB-D hand tracking, foot pedal sensing, and surgical video on both Raven-II and da Vinci Xi. The released datasets include 15 Raven-II peg transfer trials and 17 da Vinci Xi hernia-repair suturing trials, with the latter comprising 212 minutes and 1,724 gesture samples. On Raven-II, EmHT nearly matches internal MTM/PSM kinematics for gesture recognition with F1 0.86 versus 0.87 for PSM/MTM under MTRSAP. A common misconception is that external sensing can only provide weak proxy data; MiDAS suggests that non-invasive signals can be benchmark-competitive for downstream gesture recognition, while still not fully replacing precise orientation or grasper telemetry (Weerasinghe et al., 12 Feb 2026).

RoboSurg-VQA pushes RMIS benchmarking into multimodal scene understanding. It repurposes public surgical segmentation datasets into a segmentation-aware visual question answering benchmark in which each frame is paired with 9 fixed questions spanning procedure context, anatomy, modality, artefacts, quality, visibility, and spatial attributes. The reported instantiation contains 11,480 frames, split into 8,745 training and 2,735 test frames across three sites, and uses Macro-F1 as the primary metric because of substantial class imbalance. The benchmark also imposes explicit consistency rules such as Q4.smoke=no $\Rightarrow$ Q9=none. This benchmark shows that RMIS evaluation is moving beyond mask accuracy toward structured clinical querying under smoke, glare, blur, and occlusion (Zhang et al., 21 May 2026).

5. Control-oriented RMIS benchmarks and benchmark-ready metrics

RMIS benchmark design is not limited to datasets and perception tasks; it also includes benchmark-ready formulations of control problems. The 2024 constrained inverse-kinematics paper studies a redundant surgical robot with a hard remote center of motion (RCM) constraint, joint limits, and a lower-priority manipulability maximization objective in a hierarchical quadratic programming (HQP) framework. In the paper’s benchmark framing, the required inputs are the current joint configuration, robot kinematic model, desired end-effector trajectory, trocar geometry, Jacobians, and joint limits, while the outputs include joint velocities, pose error, RCM error, manipulability, and solve time (Colan et al., 2024).

The reported evaluation uses two redundant kinematic chains: KC-1, a 7-DOF manipulator + 3-DOF surgical tool, and KC-2, a 7-DOF manipulator + 5-DOF surgical tool. In the constrained RCM scenario, manipulability improves from 0.270 to 0.329 for KC-1 and from 0.437 to 0.483 for KC-2 with optimization, while average RCM error remains on the order of 0.0056–0.0070 mm and mean solve time remains under 1 ms. In the unconstrained scenario, the redundancy effect is much larger: KC-2 average manipulability improves from 0.334 to 0.824, which the paper highlights as approximately 146%. These results identify a benchmark regime in which dexterity, singularity avoidance, trocar safety, and real-time feasibility can be evaluated jointly rather than through end-effector error alone (Colan et al., 2024).

The same paper is explicit about which metrics should become benchmark endpoints: RCM error, end-effector pose error, manipulability index, and solver time per cycle. It also notes that feasibility rate and joint-limit proximity would be natural additions even though they are not tabulated explicitly. This is a significant shift in RMIS benchmarking because it moves evaluation from observational skill metrics toward control-theoretic safety and dexterity measures grounded in surgical kinematics (Colan et al., 2024).

6. Interpretation, limitations, and emerging directions

A central interpretive point is that the RMIS benchmark landscape remains fragmented. Several papers explicitly contribute benchmark-style evaluations or benchmark-ready protocols rather than a mature standalone benchmark with a codified public leaderboard, hidden test server, or broad multi-institutional governance. SurgPose states this explicitly for zero-shot pose estimation; MiDAS is better understood as a benchmark-enabling infrastructure than as a definitive benchmark; the constrained-IK work offers benchmark metrics without defining a public benchmark; and the haptics papers provide highly reusable protocols without claiming benchmark standardization (Rai et al., 16 May 2025, Weerasinghe et al., 12 Feb 2026).

The literature also exposes repeated limitations that narrow generalization. Training studies often use novices, dry-lab tasks, virtual ring tasks, or short repeated-practice sessions rather than residents or expert surgeons and clinically realistic tissue interaction. Pose-estimation studies may center on one target instrument category and phantom scenes rather than diverse unseen tools or in vivo imagery. Multimodal acquisition studies may have modest trial counts and incomplete split reporting. Control studies may be simulation-only and assume accurate kinematics without tissue interaction or calibration drift. These limitations do not invalidate the benchmarks, but they restrict what benchmark scores can legitimately support.

Several common misconceptions follow from this fragmentation. One is that RMIS benchmarking is equivalent to timing or success-rate measurement; the force-feedback and feedback-timing studies show that force, orientation accuracy, learning trajectory, and speed–accuracy coupling are often the more sensitive endpoints. Another is that “zero-shot” automatically means benchmarked unseen-category generalization; SurgPose explicitly does not yet provide a broad held-out instrument taxonomy. A third is that external sensing simply cannot substitute for proprietary telemetry; MiDAS shows that this is too strong, even though precise orientation and grasper states remain harder to recover (Machaca et al., 2022, Gale et al., 25 Aug 2025, Rai et al., 16 May 2025, Weerasinghe et al., 12 Feb 2026).

The next benchmark directions are comparatively clear in the surveyed work. The pose-estimation literature points toward broader multi-instrument datasets with explicit seen/unseen splits, public stereo-based benchmark protocols, and evaluation under stronger occlusion and reflectivity variation. The training literature points toward longer-horizon retention studies, transfer beyond canonical dry-lab tasks, and benchmark reporting that preserves segment-level difficulty structure. Multimodal acquisition work points toward broader cross-platform data collection that weakens dependence on proprietary telemetry. VQA-style work points toward fixed-schema multimodal evaluation under degraded operative views rather than segmentation-only scoring. Control-oriented benchmarking points toward standardized reporting of RCM error, manipulability, feasibility, and solve-time distributions rather than pose tracking alone (Rai et al., 16 May 2025, Gale et al., 25 Aug 2025, Weerasinghe et al., 12 Feb 2026, Zhang et al., 21 May 2026, Colan et al., 2024).

The acronym itself is also overloaded outside surgical robotics. In separate arXiv contexts, “RMIS” can refer to Representation of M5 Industrial Signals in the FISHER benchmark, while “RMIS Benchmark” has also been discussed under a microservice-management interpretation in work on MicroServo; these are unrelated to robotic minimally invasive surgery and should not be conflated with the surgical literature summarized here (Fan et al., 22 Jul 2025, Sun et al., 2024). Within surgical robotics, however, RMIS benchmarking denotes a layered evaluation regime in which skill assessment, error detection, force-aware training, perception, multimodal sensing, and constrained control are benchmarked as complementary parts of the same system problem rather than as isolated algorithmic subfields.