---
title: Automated Surgical Skill Assessment
url: https://www.emergentmind.com/topics/automated-surgical-skill-assessment-ssa
type: topic
---

# Automated Surgical Skill Assessment

Automated Surgical Skill Assessment (SSA) is the algorithmic estimation of technical performance from surgical activity recordings. In published systems, the target may be a categorical skill label, a continuous Objective Structured Assessment of Technical Skill (OSATS)-style score, a GOALS- or GEARS-aligned rating, or a proxy variable such as clearness of operating field, inferred from robot kinematics, endoscopic video, accelerometers, or multimodal streams. The topic spans dry-lab robotic training tasks, bench-top phantoms, and in-vivo clinical procedures, and is motivated by the need for objective, scalable assessment for training, credentialing, and quality control [2103.05113, 2509.09327, 2603.00108, 2008.11954].

## 1. Definitions and assessment targets

SSA is not tied to a single output formulation. In JIGSAWS-style robotic training studies, the task is often three-class skill classification—novice, intermediate, expert—either at the interval level or at the trial level [1806.05798, 1712.08604, 1610.07245, 2312.10208]. Other works formulate SSA as binary classification, such as proficient versus expert in robot-assisted radical prostatectomy suturing, or low-performing versus high-performing GOALS efficiency in laparoscopic cholecystectomy [2509.09327, 2207.02247]. A separate line of work treats SSA as regression of global rating scores, with performance evaluated by rank correlation rather than classification accuracy [2106.01035, 2008.11954, 2603.00108].

When OSATS-style annotation is available, the Global Rating Score can be defined explicitly as
\[
\text{GRS} = \sum_{d=1}^{6} s_d,
\]
where \(s_d\) is the OSATS score for dimension \(d\) [2509.09327]. Other scoring instruments appear in procedure-specific settings: GOALS efficiency for Calot Triangle Dissection [2207.02247], modified OSATS for pituitary simulation [2409.17025], NOMAT-aligned dimensions for microanastomosis [2512.24411], and M-GEARS for clinical robotic hysterectomy and prostatectomy [2603.00108].

The target of assessment may also be indirect. “Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field” identifies clearness of operating field (COF) as a proxy for overall surgical skills because of its strong correlation with overall skills and high inter-annotator consistency [2008.11954]. In unstructured septoplasty, skill is operationalized through interpretable motion descriptors—Stroke Curvature Consistency, Stroke Duration Consistency, and Coverage Rate—rather than a single global visual score [1412.6163]. This heterogeneity of targets is characteristic of SSA: the field contains direct global scoring, indirect proxy-based scoring, and gesture- or action-conditioned subskill assessment.

## 2. Modalities and operating environments

Kinematic SSA uses robot-side motion streams or other sensor signals as primary input. JIGSAWS-based works operate on da Vinci motion sequences sampled at 30 Hz, including raw multi-channel kinematics and sliding windows of robotic motion [1806.05798, 1712.08604, 1610.07245]. Video and accelerometer fusion has also been used for bench-top suturing and knot tying, where accelerometers provide 6D motion time series and video provides motion-class sequences derived from spatio-temporal interest points [1702.07772].

Video-based SSA covers a broader procedural range. Existing systems use endoscopic or laparoscopic video from simulated tasks, benchtop phantoms, and real clinical cases, including gastrectomy [2008.11954], laparoscopic cholecystectomy [2207.02247], septoplasty in the operating room [1412.6163], robot-assisted radical prostatectomy suturing [2509.09327], microanastomosis [2512.24411], pituitary bench-top simulation [2409.17025], and general laparoscopic cholecystectomy or EndoVis-derived settings [2308.13073]. This expansion from simulator-only RMIS data to clinical video is a major structural change in the literature [2103.05113].

A distinct recent trend is multimodal visual SSA. “SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment” uses RGB frames, optical flow, and tool segmentation masks as three modalities, specifically to address clinical robotic-assisted surgery where camera motion and tissue motion introduce substantial complexities [2603.00108]. A related, but pedagogically oriented, direction uses video with interpretable pose-derived proxies and gesture segmentation to generate user-specific feedback rather than only a score [2508.02593]. The modality choice therefore determines not only accuracy but also what kind of explanation or feedback can be produced.

## 3. Methodological families

Early SSA methods were predominantly feature-based. Shape-based gesture recognition in robotic minimally invasive surgery uses Dynamic Time Warping (DTW) on raw kinematic time series and \(k\)-Nearest Neighbors, avoiding the manual feature engineering and parameter tuning typical of Hidden Markov Model pipelines [1610.07245]. The DTW recurrence is written as
\[
\gamma(i,j) = d(a_i,b_j) + \min\{\gamma(i-1,j-1),\, \gamma(i,j-1),\, \gamma(i-1,j)\},
\]
which aligns sequences with variable local speed [1610.07245]. Holistic kinematic descriptors such as Sequential Motion Texture, Discrete Cosine Transform, Discrete Fourier Transform, Approximate Entropy, and Cross-Approximate Entropy were later used for JIGSAWS skill classification and score prediction, with entropy-based descriptors quantifying regularity and synchronization in motion [1712.08604, 1702.07772]. In unstructured OR settings, handcrafted descriptors have also been defined from clinically meaningful task decompositions, such as brushing away from the septal plane and coverage along the septal plane [1412.6163].

Deep learning introduced end-to-end sequence models for raw motion and video. “SATR-DL” learns joint skill and task labels from raw da Vinci kinematics through a parallel 1D CNN plus GRU architecture with a shared representation and multi-output classifiers [1806.05798]. Video-based methods expanded this pattern to CNN-LSTM, CNN-GRU, temporal convolution, and transformer architectures [2103.05113]. “Surgical Skill Assessment via Video Semantic Aggregation” argues that a common CNN-LSTM practice spatially pools short-term CNN features before temporal modeling, which neglects differences among semantic concepts such as tools, tissues, and background; ViSA instead discovers semantic parts and aggregates them across spatiotemporal dimensions, while also providing explanatory visualization and compatibility with auxiliary information [2208.02611].

A second deep-learning family models multiple skill aspects jointly. “Towards Unified Surgical Skill Assessment” defines visual, tool, proxy, and event paths, encodes each temporally, computes per-frame path scores, and combines them through a path dependency module [2106.01035]. Its video-level prediction is
\[
q = \frac{1}{4} \sum_{m} \sum_{i=0}^{L} S_{m,i} W_{m,i},
\]
with \(W_m\) learned from all paths jointly [2106.01035]. This formalizes SSA as multi-aspect regression rather than single-stream classification.

Recent architectures have specialized further around tracking, probabilistic classification, graphs, and multimodal fusion. “Video-based Surgical Skills Assessment using Long term Tool Tracking” uses tracking-by-detection with re-identification and a self-attention transformer over tool trajectories [2207.02247]. “Video-based Surgical Skill Assessment using Tree-based Gaussian Process Classifier” combines a representation flow CNN with a tree-based Gaussian process classifier, including a noisy-input extension and new kernels [2312.10208]. “SurGNN” represents procedures as graphs whose nodes correspond to surgical phases and whose node features encode instrument motion statistics, then applies graph attention for supervised or self-supervised SSA [2308.13073]. “SurgFusion-Net” fuses RGB, optical flow, and segmentation masks through Cross-Stage Fusion Blocks, Dynamic Fusion Blocks, and Divergence Regulated Attention (DRA) [2603.00108]. In parallel, few-shot SSA now exploits self-supervised video pre-training: “Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment” uses VideoMAEv2, linear evaluation, and Temporal Convolutional Networks to study how domain similarity affects downstream few-shot performance [2509.09327].

## 4. Datasets, protocols, and evaluation practice

JIGSAWS remains the canonical benchmark for RMIS training tasks. It contains suturing, needle passing, and knot tying performed by 8 surgeons, with kinematics, synchronized video, and skill labels, and it is commonly evaluated with Leave-One-Supertrial-Out (LOSO) and Leave-One-User-Out (LOUO) protocols [1806.05798, 1712.08604, 1610.07245]. In this setting, deep and feature-based systems have reported very high LOSO results, including trial-level accuracies of 0.960 for skill assessment and 1.000 for task recognition in SATR-DL [1806.05798], and up to \(98\%\)–\(99\%\) LOSO accuracy in the systematic review of DNN-based SSA [2103.05113]. The same review also notes that LOUO is a stricter test of generalization and typically lowers performance [2103.05113].

Clinical and quasi-clinical datasets have diversified the benchmark landscape. These include 57 in-vivo laparoscopic gastrectomies with six-surgeon ratings and COF labels [2008.11954], 80 Cholec80 cholecystectomy videos with GOALS efficiency labels for the last three minutes of Calot Triangle Dissection [2207.02247], 48 septoplasty cases with expert and novice trials derived from tracked Cottle elevator motion [1412.6163], SAR-RARP50 with 33 OSATS-annotated videos plus 21 unlabeled videos for self-supervised pre-training [2509.09327], 37 robot-assisted hysterectomy videos and 33 robot-assisted radical prostatectomy videos with M-GEARS scores and multimodal annotations [2603.00108], and a public endoscopic pituitary phantom dataset with dense instrument annotations and mOSATS labels [2409.17025]. These datasets vary markedly in duration, label granularity, and modality availability.

Evaluation metrics also differ by formulation. Classification papers report accuracy, precision, recall, F1-score, Cohen’s \(\kappa\), and occasionally task-balanced statistics [1806.05798, 2207.02247, 2409.17025]. Regression papers emphasize Spearman’s rank correlation coefficient, Pearson correlation, Kendall’s tau, and MAE [2106.01035, 2008.11954, 2603.00108]. A few representative numbers illustrate the current spread: the unified multi-path framework improved JIGSAWS state of the art from 0.71 Spearman’s correlation to 0.80 [2106.01035]; COF-based clinical SSA achieved 0.55 Spearman’s correlation with overall technical skill [2008.11954]; few-shot robotic SSA reached accuracies of 60.16\%, 66.03\%, and 73.65\% in 1-, 2-, and 5-shot settings, respectively [2509.09327]; transformer-based cholecystectomy efficiency assessment reached 0.83 accuracy and \(\kappa = 0.63\) with the proposed tracker [2207.02247]; pituitary bench-top SSA reached 87\% novice-versus-expert accuracy using tracking-derived features [2409.17025]; and multimodal SurgFusion-Net reached SCC \(0.8294\) on RAH-skill and \(0.8486\) on RARP-skill [2603.00108]. These results are not directly interchangeable, but they show the coexistence of high-accuracy simulator benchmarks and moderate-correlation clinical regression settings.

## 5. Explainability, proxy variables, and feedback

Explainability in SSA takes several distinct forms. One form is visual attribution. ViSA explicitly states that discovery of semantic parts provides an explanatory visualization and that the revised manuscript includes GradCAM-based explanation analysis [2208.02611]. The broader DNN review reports class activation maps and attention visualizations as a recurrent mechanism for temporal or spatial localization of skill-relevant evidence in kinematic and video models [2103.05113].

A second form is semantically grounded proxy assessment. COF is treated as a visually grounded proxy for overall skill because of its strong correlation with overall skills and high inter-annotator consistency [2008.11954]. In septoplasty, Stroke Curvature Consistency, Stroke Duration Consistency, and Coverage Rate were designed to provide personalized, actionable feedback about wrist motion and coverage patterns, rather than only a scalar expert-versus-novice label [1412.6163]. SurGNN makes procedure graphs and embedding clusters interpretable at the level of phases and motion statistics, arguing that graph structure exposes which components of a procedure contribute to skill predictions [2308.13073].

A third form is direct educational feedback. “Explainable AI for Automated User-specific Feedback in Surgical Skill Acquisition” decomposes suturing into primitive gestures using MSTCN++, extracts hand-orientation and thumb–index-distance proxies, compares each trainee to expert reference values, and selects the top three proxy–gesture deviations for individualized explanation [2508.02593]. The study reports improved cognitive load and confidence post-intervention, but no differences emerged between XAI-guided feedback and traditional video-based coaching in reducing performance gaps or practice adjustments; trends in the XAI group nevertheless revealed desirable effects where participants more closely mimicked expert practice [2508.02593]. This is a useful corrective to a common misconception: interpretability does not automatically translate into demonstrably superior short-horizon learning outcomes, even when it increases specificity and actionability.

## 6. Limitations, controversies, and future directions

The dominant limitations are recurrent across almost all settings. Labels remain costly, subjective, and often sparse; several studies rely on single-procedure cohorts, small samples, or simplified binary targets because fine-grained scoring would leave too few examples per class [2103.05113, 2509.09327, 2008.11954]. Simulator and dry-lab performance does not eliminate the domain gap to real surgery, where camera motion, tissue deformation, occlusion, and workflow variability degrade motion cues and challenge direct transfer [2103.05113, 2603.00108, 2207.02247, 2409.17025]. Video-only pipelines that depend on tracking inherit tracking errors; in pituitary simulation, tracking achieved 71.9\% Multiple Object Tracking Precision at 22 FPS, which was sufficient for SSA but still left clear room for improvement [2409.17025].

Another recurring issue is that apparently strong benchmark numbers can mask narrow validity. JIGSAWS LOSO performance is near saturation in several studies, yet LOUO and clinical transfer remain substantially harder [2103.05113]. Clinical regression studies report lower but arguably more realistic correlations, and even within clinical robotic SSA the value of pre-training depends sharply on domain match: small but domain-relevant datasets can outperform large-scale, less aligned sources, and incorporating procedure-specific data into pre-training improved downstream performance when the auxiliary domain was highly similar but could degrade performance when the source was less similar [2509.09327]. This suggests that, under label scarcity, domain relevance is at least as consequential as scale.

Current research directions therefore emphasize larger and more diverse public datasets, multimodal fusion, stronger self-supervised pre-training, and finer-grained feedback loops. Proposed extensions include multiple real-world surgical datasets from different centers, broader skill spectra enabling multi-class or continuous SSA, richer modalities such as optical flow, tool masks, audio, text, force, or physiological signals, and real-time deployment for simulation and intraoperative analytics [2103.05113, 2509.09327, 2603.00108, 2409.17025, 2508.02593]. A plausible implication is that future SSA systems will be judged less by single benchmark accuracy and more by cross-domain robustness, rating reliability, and whether their outputs can support adaptive training rather than only retrospective scoring.

Source: https://www.emergentmind.com/topics/automated-surgical-skill-assessment-ssa