Automated Surgical Skill Assessment

• Automated Surgical Skill Assessment is an algorithmic approach that quantifies surgical performance using sensor data, video, and kinematic streams.
• It integrates traditional feature-based methods and deep learning models such as CNNs, LSTMs, and transformers to objectively rate technical skills.
• Emerging trends focus on multimodal fusion, real-time feedback, and enhanced interpretability to improve training, credentialing, and quality control.

Automated Surgical Skill Assessment (SSA) is the algorithmic estimation of technical performance from surgical activity recordings. In published systems, the target may be a categorical skill label, a continuous Objective Structured Assessment of Technical Skill (OSATS)-style score, a GOALS- or GEARS-aligned rating, or a proxy variable such as clearness of operating field, inferred from robot kinematics, endoscopic video, accelerometers, or multimodal streams. The topic spans dry-lab robotic training tasks, bench-top phantoms, and in-vivo clinical procedures, and is motivated by the need for objective, scalable assessment for training, credentialing, and quality control (Yanik et al., 2021, Anastasiou et al., 11 Sep 2025, He et al., 18 Feb 2026, Liu et al., 2020).

1. Definitions and assessment targets

SSA is not tied to a single output formulation. In JIGSAWS-style robotic training studies, the task is often three-class skill classification—novice, intermediate, expert—either at the interval level or at the trial level (Wang et al., 2018, Zia et al., 2017, Fard et al., 2016, Rezaei et al., 2023). Other works formulate SSA as binary classification, such as proficient versus expert in robot-assisted radical prostatectomy suturing, or low-performing versus high-performing GOALS efficiency in laparoscopic cholecystectomy (Anastasiou et al., 11 Sep 2025, Fathollahi et al., 2022). A separate line of work treats SSA as regression of global rating scores, with performance evaluated by rank correlation rather than classification accuracy (Liu et al., 2021, Liu et al., 2020, He et al., 18 Feb 2026).

When OSATS-style annotation is available, the Global Rating Score can be defined explicitly as

$\text{GRS} = \sum_{d=1}^{6} s_d,$

where $s_d$ is the OSATS score for dimension $d$ (Anastasiou et al., 11 Sep 2025). Other scoring instruments appear in procedure-specific settings: GOALS efficiency for Calot Triangle Dissection (Fathollahi et al., 2022), modified OSATS for pituitary simulation (Das et al., 2024), NOMAT-aligned dimensions for microanastomosis (Meng et al., 30 Dec 2025), and M-GEARS for clinical robotic hysterectomy and prostatectomy (He et al., 18 Feb 2026).

The target of assessment may also be indirect. “Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field” identifies clearness of operating field (COF) as a proxy for overall surgical skills because of its strong correlation with overall skills and high inter-annotator consistency (Liu et al., 2020). In unstructured septoplasty, skill is operationalized through interpretable motion descriptors—Stroke Curvature Consistency, Stroke Duration Consistency, and Coverage Rate—rather than a single global visual score (Poddar et al., 2014). This heterogeneity of targets is characteristic of SSA: the field contains direct global scoring, indirect proxy-based scoring, and gesture- or action-conditioned subskill assessment.

2. Modalities and operating environments

Kinematic SSA uses robot-side motion streams or other sensor signals as primary input. JIGSAWS-based works operate on da Vinci motion sequences sampled at 30 Hz, including raw multi-channel kinematics and sliding windows of robotic motion (Wang et al., 2018, Zia et al., 2017, Fard et al., 2016). Video and accelerometer fusion has also been used for bench-top suturing and knot tying, where accelerometers provide 6D motion time series and video provides motion-class sequences derived from spatio-temporal interest points (Zia et al., 2017).

Video-based SSA covers a broader procedural range. Existing systems use endoscopic or laparoscopic video from simulated tasks, benchtop phantoms, and real clinical cases, including gastrectomy (Liu et al., 2020), laparoscopic cholecystectomy (Fathollahi et al., 2022), septoplasty in the operating room (Poddar et al., 2014), robot-assisted radical prostatectomy suturing (Anastasiou et al., 11 Sep 2025), microanastomosis (Meng et al., 30 Dec 2025), pituitary bench-top simulation (Das et al., 2024), and general laparoscopic cholecystectomy or EndoVis-derived settings (Khalid et al., 2023). This expansion from simulator-only RMIS data to clinical video is a major structural change in the literature (Yanik et al., 2021).

A distinct recent trend is multimodal visual SSA. “SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment” uses RGB frames, optical flow, and tool segmentation masks as three modalities, specifically to address clinical robotic-assisted surgery where camera motion and tissue motion introduce substantial complexities (He et al., 18 Feb 2026). A related, but pedagogically oriented, direction uses video with interpretable pose-derived proxies and gesture segmentation to generate user-specific feedback rather than only a score (Gomez et al., 4 Aug 2025). The modality choice therefore determines not only accuracy but also what kind of explanation or feedback can be produced.

3. Methodological families

Early SSA methods were predominantly feature-based. Shape-based gesture recognition in robotic minimally invasive surgery uses Dynamic Time Warping (DTW) on raw kinematic time series and $k$-Nearest Neighbors, avoiding the manual feature engineering and parameter tuning typical of Hidden Markov Model pipelines (Fard et al., 2016). The DTW recurrence is written as

$$\gamma(i,j) = d(a_i,b_j) + \min\{\gamma(i-1,j-1),\, \gamma(i,j-1),\, \gamma(i-1,j)\},$$

which aligns sequences with variable local speed (Fard et al., 2016). Holistic kinematic descriptors such as Sequential Motion Texture, Discrete Cosine Transform, Discrete Fourier Transform, Approximate Entropy, and Cross-Approximate Entropy were later used for JIGSAWS skill classification and score prediction, with entropy-based descriptors quantifying regularity and synchronization in motion (Zia et al., 2017, Zia et al., 2017). In unstructured OR settings, handcrafted descriptors have also been defined from clinically meaningful task decompositions, such as brushing away from the septal plane and coverage along the septal plane (Poddar et al., 2014).

Deep learning introduced end-to-end sequence models for raw motion and video. “SATR-DL” learns joint skill and task labels from raw da Vinci kinematics through a parallel 1D CNN plus GRU architecture with a shared representation and multi-output classifiers (Wang et al., 2018). Video-based methods expanded this pattern to CNN-LSTM, CNN-GRU, temporal convolution, and transformer architectures (Yanik et al., 2021). “Surgical Skill Assessment via Video Semantic Aggregation” argues that a common CNN-LSTM practice spatially pools short-term CNN features before temporal modeling, which neglects differences among semantic concepts such as tools, tissues, and background; ViSA instead discovers semantic parts and aggregates them across spatiotemporal dimensions, while also providing explanatory visualization and compatibility with auxiliary information (Li et al., 2022).

A second deep-learning family models multiple skill aspects jointly. “Towards Unified Surgical Skill Assessment” defines visual, tool, proxy, and event paths, encodes each temporally, computes per-frame path scores, and combines them through a path dependency module (Liu et al., 2021). Its video-level prediction is

$$q = \frac{1}{4} \sum_{m} \sum_{i=0}^{L} S_{m,i} W_{m,i},$$

with $W_m$ learned from all paths jointly (Liu et al., 2021). This formalizes SSA as multi-aspect regression rather than single-stream classification.

Recent architectures have specialized further around tracking, probabilistic classification, graphs, and multimodal fusion. “Video-based Surgical Skills Assessment using Long term Tool Tracking” uses tracking-by-detection with re-identification and a self-attention transformer over tool trajectories (Fathollahi et al., 2022). “Video-based Surgical Skill Assessment using Tree-based Gaussian Process Classifier” combines a representation flow CNN with a tree-based Gaussian process classifier, including a noisy-input extension and new kernels (Rezaei et al., 2023). “SurGNN” represents procedures as graphs whose nodes correspond to surgical phases and whose node features encode instrument motion statistics, then applies graph attention for supervised or self-supervised SSA (Khalid et al., 2023). “SurgFusion-Net” fuses RGB, optical flow, and segmentation masks through Cross-Stage Fusion Blocks, Dynamic Fusion Blocks, and Divergence Regulated Attention (DRA) (He et al., 18 Feb 2026). In parallel, few-shot SSA now exploits self-supervised video pre-training: “Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment” uses VideoMAEv2, linear evaluation, and Temporal Convolutional Networks to study how domain similarity affects downstream few-shot performance (Anastasiou et al., 11 Sep 2025).

4. Datasets, protocols, and evaluation practice

JIGSAWS remains the canonical benchmark for RMIS training tasks. It contains suturing, needle passing, and knot tying performed by 8 surgeons, with kinematics, synchronized video, and skill labels, and it is commonly evaluated with Leave-One-Supertrial-Out (LOSO) and Leave-One-User-Out (LOUO) protocols (Wang et al., 2018, Zia et al., 2017, Fard et al., 2016). In this setting, deep and feature-based systems have reported very high LOSO results, including trial-level accuracies of 0.960 for skill assessment and 1.000 for task recognition in SATR-DL (Wang et al., 2018), and up to $98\%$–$99\%$ LOSO accuracy in the systematic review of DNN-based SSA (Yanik et al., 2021). The same review also notes that LOUO is a stricter test of generalization and typically lowers performance (Yanik et al., 2021).

Clinical and quasi-clinical datasets have diversified the benchmark landscape. These include 57 in-vivo laparoscopic gastrectomies with six-surgeon ratings and COF labels (Liu et al., 2020), 80 Cholec80 cholecystectomy videos with GOALS efficiency labels for the last three minutes of Calot Triangle Dissection (Fathollahi et al., 2022), 48 septoplasty cases with expert and novice trials derived from tracked Cottle elevator motion (Poddar et al., 2014), SAR-RARP50 with 33 OSATS-annotated videos plus 21 unlabeled videos for self-supervised pre-training (Anastasiou et al., 11 Sep 2025), 37 robot-assisted hysterectomy videos and 33 robot-assisted radical prostatectomy videos with M-GEARS scores and multimodal annotations (He et al., 18 Feb 2026), and a public endoscopic pituitary phantom dataset with dense instrument annotations and mOSATS labels (Das et al., 2024). These datasets vary markedly in duration, label granularity, and modality availability.

Evaluation metrics also differ by formulation. Classification papers report accuracy, precision, recall, F1-score, Cohen’s $\kappa$, and occasionally task-balanced statistics (Wang et al., 2018, Fathollahi et al., 2022, Das et al., 2024). Regression papers emphasize Spearman’s rank correlation coefficient, Pearson correlation, Kendall’s tau, and MAE (Liu et al., 2021, Liu et al., 2020, He et al., 18 Feb 2026). A few representative numbers illustrate the current spread: the unified multi-path framework improved JIGSAWS state of the art from 0.71 Spearman’s correlation to 0.80 (Liu et al., 2021); COF-based clinical SSA achieved 0.55 Spearman’s correlation with overall technical skill (Liu et al., 2020); few-shot robotic SSA reached accuracies of 60.16\%, 66.03\%, and 73.65\% in 1-, 2-, and 5-shot settings, respectively (Anastasiou et al., 11 Sep 2025); transformer-based cholecystectomy efficiency assessment reached 0.83 accuracy and $s_d$0 with the proposed tracker (Fathollahi et al., 2022); pituitary bench-top SSA reached 87\% novice-versus-expert accuracy using tracking-derived features (Das et al., 2024); and multimodal SurgFusion-Net reached SCC $s_d$1 on RAH-skill and $s_d$2 on RARP-skill (He et al., 18 Feb 2026). These results are not directly interchangeable, but they show the coexistence of high-accuracy simulator benchmarks and moderate-correlation clinical regression settings.

5. Explainability, proxy variables, and feedback

Explainability in SSA takes several distinct forms. One form is visual attribution. ViSA explicitly states that discovery of semantic parts provides an explanatory visualization and that the revised manuscript includes GradCAM-based explanation analysis (Li et al., 2022). The broader DNN review reports class activation maps and attention visualizations as a recurrent mechanism for temporal or spatial localization of skill-relevant evidence in kinematic and video models (Yanik et al., 2021).

A second form is semantically grounded proxy assessment. COF is treated as a visually grounded proxy for overall skill because of its strong correlation with overall skills and high inter-annotator consistency (Liu et al., 2020). In septoplasty, Stroke Curvature Consistency, Stroke Duration Consistency, and Coverage Rate were designed to provide personalized, actionable feedback about wrist motion and coverage patterns, rather than only a scalar expert-versus-novice label (Poddar et al., 2014). SurGNN makes procedure graphs and embedding clusters interpretable at the level of phases and motion statistics, arguing that graph structure exposes which components of a procedure contribute to skill predictions (Khalid et al., 2023).

A third form is direct educational feedback. “Explainable AI for Automated User-specific Feedback in Surgical Skill Acquisition” decomposes suturing into primitive gestures using MSTCN++, extracts hand-orientation and thumb–index-distance proxies, compares each trainee to expert reference values, and selects the top three proxy–gesture deviations for individualized explanation (Gomez et al., 4 Aug 2025). The study reports improved cognitive load and confidence post-intervention, but no differences emerged between XAI-guided feedback and traditional video-based coaching in reducing performance gaps or practice adjustments; trends in the XAI group nevertheless revealed desirable effects where participants more closely mimicked expert practice (Gomez et al., 4 Aug 2025). This is a useful corrective to a common misconception: interpretability does not automatically translate into demonstrably superior short-horizon learning outcomes, even when it increases specificity and actionability.

6. Limitations, controversies, and future directions

The dominant limitations are recurrent across almost all settings. Labels remain costly, subjective, and often sparse; several studies rely on single-procedure cohorts, small samples, or simplified binary targets because fine-grained scoring would leave too few examples per class (Yanik et al., 2021, Anastasiou et al., 11 Sep 2025, Liu et al., 2020). Simulator and dry-lab performance does not eliminate the domain gap to real surgery, where camera motion, tissue deformation, occlusion, and workflow variability degrade motion cues and challenge direct transfer (Yanik et al., 2021, He et al., 18 Feb 2026, Fathollahi et al., 2022, Das et al., 2024). Video-only pipelines that depend on tracking inherit tracking errors; in pituitary simulation, tracking achieved 71.9\% Multiple Object Tracking Precision at 22 FPS, which was sufficient for SSA but still left clear room for improvement (Das et al., 2024).

Another recurring issue is that apparently strong benchmark numbers can mask narrow validity. JIGSAWS LOSO performance is near saturation in several studies, yet LOUO and clinical transfer remain substantially harder (Yanik et al., 2021). Clinical regression studies report lower but arguably more realistic correlations, and even within clinical robotic SSA the value of pre-training depends sharply on domain match: small but domain-relevant datasets can outperform large-scale, less aligned sources, and incorporating procedure-specific data into pre-training improved downstream performance when the auxiliary domain was highly similar but could degrade performance when the source was less similar (Anastasiou et al., 11 Sep 2025). This suggests that, under label scarcity, domain relevance is at least as consequential as scale.

Current research directions therefore emphasize larger and more diverse public datasets, multimodal fusion, stronger self-supervised pre-training, and finer-grained feedback loops. Proposed extensions include multiple real-world surgical datasets from different centers, broader skill spectra enabling multi-class or continuous SSA, richer modalities such as optical flow, tool masks, audio, text, force, or physiological signals, and real-time deployment for simulation and intraoperative analytics (Yanik et al., 2021, Anastasiou et al., 11 Sep 2025, He et al., 18 Feb 2026, Das et al., 2024, Gomez et al., 4 Aug 2025). A plausible implication is that future SSA systems will be judged less by single benchmark accuracy and more by cross-domain robustness, rating reliability, and whether their outputs can support adaptive training rather than only retrospective scoring.