SurgSync: Synchronized Multi-Modal Surgical Data

• SurgSync is a time-synchronized multi-modal data framework for surgical robotics that integrates calibrated stereo imaging, robot kinematics, and capacitive contact sensing.
• It employs dual-mode synchronization, offering both real-time online data usability and offline low-latency, regular sampling for diverse research applications.
• Its comprehensive post-processing toolbox standardizes calibration, depth estimation, optical flow, and annotation, supporting tasks like 3D reconstruction, skill assessment, and autonomous guidance.

Searching arXiv for the cited SurgSync paper and closely related surgical synchronization work to ground the article in current literature. SurgSync is a time-synchronized multi-modal data collection framework and dataset for surgical robotics, implemented on the da Vinci Research Kit (dVRK) to support both offline dataset construction and online inference-time synchronization (Zhou et al., 6 Mar 2026). It combines dual-mode synchronized recorders, a modern stereo endoscope, a side-view RGB-D camera, robot kinematics, and a capacitive contact sensor, and it is paired with a post-processing toolbox for depth estimation, optical flow, kinematic reprojection, and annotation. In its reported form, SurgSync is less a single algorithm than an integrated acquisition and synchronization stack whose purpose is to supply tightly aligned visual, kinematic, and contact signals for perception, skill assessment, and autonomy research.

1. Problem setting and design rationale

SurgSync is motivated by a structural limitation in surgical robotics data: most current robotic surgery systems remain human-in-the-loop, yet higher-level control, supervised autonomy, and perception-driven assistance require large amounts of high-quality, time-aligned multi-modal data (Zhou et al., 6 Mar 2026). The framework explicitly targets perception, skill assessment, supervised autonomy, and interaction modeling, and it does so under the observation that existing datasets often suffer from poor or inconsistent time alignment between video and kinematics, legacy imaging, limited sensing modalities, and sparse post-processing support.

The framework therefore treats synchronization as a primary systems requirement rather than a downstream convenience. This emphasis is consistent with adjacent lines of research in surgical video alignment and heterogeneous camera synchronization. Kinematics-based Dynamic Time Warping has been used to align surgical videos for comparative review (Fawaz et al., 2019), while millisecond-accurate synchronization across RGB and IR camera systems has been validated in large-scale surgical recordings involving more than 25 heterogeneous cameras (Meyer et al., 18 Nov 2025). Within that broader landscape, SurgSync addresses the specific case of synchronized data acquisition on a surgical robot, with the dVRK as the reference platform.

A common misconception is to view surgical synchronization as only a timestamping problem. The framework instead couples timestamping, sensing, calibration, recorder design, and post-processing. This suggests that, in surgical robotics, temporal alignment is inseparable from hardware integration and modality-specific data handling.

2. Framework architecture and dual-mode synchronization

At the systems level, SurgSync consists of a ROS-based stack built around the dVRK Classic and dVRK-Si platforms, with an Endoscopic Camera Manipulator for the stereo endoscope and Patient-Side Manipulators for tools (Zhou et al., 6 Mar 2026). Its principal components are a modern stereo chip-on-tip endoscope, a side-view RGB-D camera, robot kinematics, a capacitive tool–tissue contact sensor, dual-mode synchronized recorders, and a post-processing toolbox.

The distinguishing systems feature is the separation between online synchronization and offline synchronization. The online-matching recorder is intended for real-time or interactive use. It buffers incoming images and kinematic/contact messages, uses image timestamps as reference times, searches each modality buffer for the closest sample within a fixed tolerance, and emits a synchronized packet only when all required modalities satisfy that tolerance. It then writes images as PNGs and kinematics as JSON. This yields data that are immediately usable, but the stream can become temporally irregular because unmatched samples are dropped.

The offline-matching recorder separates capture from synchronization. During acquisition, video is written at a fixed frame rate and kinematic/contact topics are logged to binary files without matching. In post-processing, video is decomposed into timestamped frames, binary kinematics are converted to JSON, and the closest kinematic sample is assigned to each frame with $k = 1$. This produces uniformly sampled sequences and lower latency at the cost of deferred synchronization.

Recorder mode	Mean latency	Recording frequency
Online-matching	$6.36 \pm 4.72$ ms	$4.04 \pm 1.69$ Hz
Offline-matching	$1.35 \pm 0.81$ ms	10 Hz

The same comparison also reports median latencies of 5.58 ms for online matching and 1.33 ms for offline matching (Zhou et al., 6 Mar 2026). The architectural implication is straightforward: online synchronization prioritizes immediate availability, whereas offline synchronization prioritizes lower latency and regular sampling. The framework is explicit that both modes are necessary, because training-set construction and live inference impose different constraints.

3. Sensors, modalities, and calibration

SurgSync records multiple synchronized modalities. Its primary visual sensor is a modern chip-on-tip stereo endoscope mounted on the dVRK-Si ECM, producing 1080p stereo images at 60 Hz, later subsampled for the synchronized dataset (Zhou et al., 6 Mar 2026). A side-view Intel RealSense RGB-D camera provides 1080p RGB with depth at 30 Hz. For robot state, the framework records measured and setpoint joint states, Cartesian poses, and jaw/gripper positions for the PSMs and ECM.

A notable hardware contribution is the imaging upgrade relative to legacy dVRK optics. The framework reports a Laplacian variance of focus of 529.48 for the modern endoscope versus 16.93 for the legacy dVRK-Si scope, with lower CCU brightness at 30% versus 100% (Zhou et al., 6 Mar 2026). This suggests that synchronization quality alone is not sufficient; image quality materially affects the utility of the synchronized corpus for vision tasks.

The capacitive contact sensor is designed to provide tool–tissue contact ground truth. It uses an Arduino UNO Rev3, the Capacitive Sensing Library, a 2 M$\Omega$ resistor, and custom wiring to the tool connectors or internal joints. The raw capacitance signal is thresholded at 205 to obtain binary contact/non-contact, and the resulting bit is fed through the dVRK FPGA digital input and published as a ROS topic aligned with the remaining modalities (Zhou et al., 6 Mar 2026). Reported instance-level accuracies are 99.1% for tissue manipulation, 74.3% for dissection, and 45.2% for suturing, with humidity changes and short-circuit events identified as failure modes. Manual correction through the annotation GUI is therefore part of the final ground-truth construction.

Calibration is equally central. Stereo camera intrinsics and extrinsics are included, and hand–eye calibration is performed at JHU to relate the PSM tool frames to camera coordinates. This calibration underpins the reprojection pipeline. For a tool point $p_{\text{base}}$ in robot coordinates, the camera-frame point is

$$p_{\text{cam}} = T_{\text{cam} \leftarrow \text{base}} \cdot p_{\text{base}}.$$

With intrinsic matrix $K$, projection to image coordinates is

$$\begin{pmatrix} u_p \ v_p \ 1 \end{pmatrix} = \frac{1}{z_p} K \begin{pmatrix} x_p \ y_p \ z_p \end{pmatrix}.$$

SurgSync then represents the projected location with a Gaussian heatmap

$$G(p_x, p_y) = \exp\left( -\left( \frac{(p_x - u_p)^2}{\sigma_x^2} + \frac{(p_y - v_p)^2}{\sigma_y^2} \right) \right),$$

and forms an attention-weighted grayscale image through

$6.36 \pm 4.72$0

The stated rationale is that even calibrated dVRK kinematics remain imperfect, so the Gaussian heatmap provides a smooth uncertainty-aware projection target (Zhou et al., 6 Mar 2026).

4. Post-processing toolbox and dataset composition

The post-processing toolbox standardizes calibration, stereo rectification, resizing, depth estimation, optical flow, kinematic reprojection, and annotation (Zhou et al., 6 Mar 2026). For stereo depth, it uses FoundationStereo. Given focal length $6.36 \pm 4.72$1, baseline $6.36 \pm 4.72$2, and disparity $6.36 \pm 4.72$3, depth is computed by

$6.36 \pm 4.72$4

For motion, it uses RAFT to estimate dense optical flow, followed by a magnitude-aware filter that removes low-magnitude noise. These steps make the synchronized dataset immediately usable for tasks such as 3D reconstruction, instrument localization, and tissue-motion analysis.

The annotation layer is implemented in a PyQt-based GUI that supports frame-by-frame playback, contact-label correction, and phase/event annotation. This matters because several modalities are explicitly designed as weak or noisy signals that become ground truth only after human correction. The framework therefore couples sensor design with annotation tooling, rather than treating annotation as an independent stage.

The reported dataset was collected through a user study involving 13 participants: 4 novices, 5 experienced dVRK users, and 4 surgeons (Zhou et al., 6 Mar 2026). Tasks were performed on phantoms and ex-vivo tissues including chicken breast, chicken heart, beef, and pork. The canonical tasks are peg transfer, single interrupted suturing and knot tying, tissue manipulation, and dissection following a trace. The dataset contains 214 validated instances across these tasks.

Suturing and knot tying account for 104 instances, peg transfer for 18, tissue manipulation for 21, and dissection for 71 (Zhou et al., 6 Mar 2026). The corpus includes 102 offline-matching instances, with 96 of those recorded inside an Intuitive abdominal dome phantom. There are also 9 instances from UBC acquired with the same recorder design but without hand–eye calibration and without the contact sensor.

These design choices indicate that SurgSync is not merely a synchronized dump of sensor streams. It is a curated dataset with modality-specific processing and manual correction, intended to be a reusable research substrate rather than a raw logging system.

5. Validation and downstream utility

Validation in SurgSync is both systems-level and task-level. On the systems side, synchronization quality is quantified by the latency between left camera timestamps and the remaining topics across four instances per recorder, yielding the millisecond-level results reported above (Zhou et al., 6 Mar 2026). The authors attribute outliers to dVRK control-loop latencies, inverse-kinematics behavior near singularities, and CPU scheduling.

On the task side, the paper evaluates the collected data with a multi-path skill assessment framework from Liu et al. (2021) (Zhou et al., 6 Mar 2026). The experiment uses 43 instances from 8 users, segmented into suturing and knot-tying phases. Each sample combines 14D kinematic features, 2048D visual features from a ResNet-101 encoder, and 14D one-hot gesture labels. Ground-truth skill labels are Global Rating Scores in $6.36 \pm 4.72$5, derived from six rubric categories: respect for tissue, suture/needle handling, time and motion, flow of operation, quality of final product, and overall performance.

Using 4-fold stratified cross-validation, the framework reports Spearman rank correlation coefficients of $6.36 \pm 4.72$6 for suturing and $6.36 \pm 4.72$7 for knot tying (Zhou et al., 6 Mar 2026). The paper interprets these as evidence that the synchronized visual, kinematic, and gesture signals are coherent enough to support nontrivial downstream learning.

This utility aligns with adjacent task formulations in the literature. Streaming workflow understanding has been addressed with hierarchy-aware vision-LLMs that emit online phase and step transition tokens (He et al., 20 May 2026), while SfM-free surgical scene reconstruction has been formulated as progressive geometry-constrained Gaussian Splatting from monocular endoscopic video (Zheng et al., 6 Jun 2025). SurgSync does not implement these models, but it supplies the type of aligned multi-modal substrate they require.

6. Relation to the broader synchronization ecosystem and open challenges

SurgSync sits within a broader family of synchronization problems in surgical AI. One branch concerns visual–geometric synchronization: SurGSplat reconstructs a temporally coherent 3D Gaussian scene and camera trajectory from monocular endoscopic video, which could serve as a reconstruction and pose backbone for synchronized navigation overlays (Zheng et al., 6 Jun 2025). Another branch concerns workflow synchronization: SurgOnAir produces streaming hierarchical narrations and explicit <State> transitions that mark current phase and step online, effectively creating machine-readable synchronization points for downstream assistance modules (He et al., 20 May 2026). A third branch concerns sensor-level temporal synchronization, where RocSync achieves a residual error of 1.34 ms RMSE across heterogeneous RGB and IR camera systems and has been validated in surgical recordings with more than 25 cameras (Meyer et al., 18 Nov 2025).

Seen against that background, SurgSync occupies the acquisition and data-integration layer. It does not itself solve every synchronization problem in the operating room. Instead, it organizes robot state, stereo vision, side-view sensing, contact signals, and derived modalities into a common time-aligned substrate on which reconstruction, tracking, workflow analysis, and safety monitoring can be built. This interpretation is consistent with other components already studied in the literature, including context-aware unsafe event detection from robot kinematics (Yasar et al., 2020), language-conditioned surgical grounding (Ashraf et al., 1 Mar 2026), and kinematics-based temporal alignment of multi-video surgical recordings (Fawaz et al., 2019).

Its current limitations are explicit. The dataset scale is modest at 214 instances; the experiments are performed on ex-vivo tissues and phantoms rather than in-vivo clinical cases; task diversity is restricted to canonical training exercises; the contact sensor is single-mode and affected by environmental conditions; and the framework remains dVRK-specific (Zhou et al., 6 Mar 2026). The planned directions are to collect more data using the offline recorder, add new tasks and instruments, extend toward in-vivo or cadaver experiments, integrate additional sensors such as force/torque or higher-fidelity depth, and further tighten real-time integration.

A plausible implication is that SurgSync is best understood as an infrastructure layer for surgical data synchronization rather than a finalized autonomy stack. Its technical contribution lies in making synchronization operational: dual-mode recording, calibrated sensing, modality-aware post-processing, and validated downstream use within a single reproducible framework (Zhou et al., 6 Mar 2026).