EgoSurg: Egocentric Surgical Vision

• EgoSurg is a research domain in egocentric surgical vision that focuses on capturing and reconstructing the surgeon’s viewpoint using both wearable and ambient OR sensors.
• It integrates diverse datasets, including real and synthetic open-surgery videos, to enable tasks such as phase recognition, hand–tool detection, and multimodal ego–exo analysis.
• Advanced methods like gaze-guided masked autoencoders and stereo-based egocentric replay enhance workflow analysis and support applications in augmented surgical guidance and training.

Searching arXiv for papers on EgoSurg and closely related egocentric surgical vision resources. EgoSurg denotes a line of research in egocentric surgical vision centered on what surgeons and other operating-room personnel see, how their hands and instruments move, and how workflow can be inferred from first-person or reconstructed first-person visual streams. In the current literature, the term encompasses real and synthetic egocentric datasets for open surgery, marker-less pose estimation for augmented guidance, ego–exo operating-room benchmarks, and a specific framework named EgoSurg that reconstructs dynamic egocentric replays for any operating-room staff directly from wall-mounted fixed-camera video (Fujii et al., 2024, Wang et al., 2023, Doughty et al., 2022, Özsoy et al., 30 May 2025, Zhang et al., 6 Oct 2025). Across these usages, the common objective is operating-room informatics from perspective-aligned visual evidence rather than from fixed, room-level summaries.

1. Scope, terminology, and problem setting

Within this literature, EgoSurg is not restricted to a single dataset or model. “EgoSurg—egocentric surgical vision—centers on what the surgeon sees and how their hands and instruments move in real time,” particularly for mixed or augmented reality guidance, navigation, and skill assessment (Wang et al., 2023). Related work on open-surgery workflow understanding frames egocentric surgical video captured from a surgeon’s viewpoint as “a natural substrate for understanding workflow, recognizing high-level phases, and enabling operating room informatics” (Fujii et al., 2024). HMD-based tracking work uses the same egocentric premise to align virtual overlays to the surgeon’s line of sight for context-aware augmented guidance (Doughty et al., 2022).

The problem setting differs materially from minimally invasive surgery. Prior work has concentrated on MIS, where fixed endoscopic views and public datasets have catalyzed progress, whereas open surgery remains underexplored because egocentric open-surgery video is harder to collect at scale and poses elevated privacy concerns (Fujii et al., 2024). The visual regime is also harder: wide field-of-view, surgeon head motion, lighting variability, occlusions, non-informative regions beyond the surgical field, hand–tool occlusions, reflective metallic tools, texture-poor or blood-stained gloves, and frequent room-level clutter are repeatedly identified as core difficulties (Fujii et al., 2024, Wang et al., 2023).

A common misconception is that egocentric surgical perception necessarily requires wearable capture. The specific “EgoSurg” framework introduced later in 2025 reconstructs person-aligned egocentric replay directly from wall-mounted fixed-camera video, explicitly targeting egocentric viewpoints “without intervention to clinical workflow” (Zhang et al., 6 Oct 2025). This suggests that the term now spans both direct egocentric sensing and retrospective egocentric reconstruction from ambient infrastructure.

2. Principal datasets and annotation regimes

The empirical basis of EgoSurg research is a small but diverse set of datasets spanning real open surgery, synthetic egocentric hand–tool scenes, and multimodal ego–exo operating-room recordings.

Resource	Primary supervision	Scale
EgoSurgery-Phase	9 surgical phases, video, gaze	21 videos, 15 hours after pre-processing, 27,694 labeled frames
EgoSurgery-Tool	15 tool categories, 4 hand categories	15,437 images, 49,652 tool boxes, 46,320 hand boxes
POV-Surgery	RGB-D, 3D/2D hand-object pose, segmentation masks	53 sequences, 88,329 frames
EgoExOR	Ego–exo multimodal recordings, scene graphs	94 minutes, 84,553 frames, 568,235 triplets

EgoSurgery-Phase introduced 21 egocentric videos of real open surgeries recorded with surgeon head-mounted Tobii cameras, spanning 10 distinct open-surgery types performed by 8 surgeons and totaling 28 hours of raw video; after privacy-preserving pre-processing, 15 hours remain (Fujii et al., 2024). The public release subsamples to 0.5 fps, blurs patient faces, and removes frames containing other personally identifiable information. Frames are manually labeled at frame level into 9 phases—Disinfection, Design, Anesthesia, Incision, Dissection, Hemostasis, Irrigation, Closure, and Dressing—and the dataset includes per-frame gaze heatmaps aligned to the subsampled video (Fujii et al., 2024). The paper states that, to the authors’ knowledge, it is the first publicly available real egocentric open-surgery dataset for phase recognition that provides both video and gaze.

EgoSurgery-Tool extends the same head-mounted real open-surgery corpus with dense bounding boxes for 15 surgical tool categories and 4 hand categories, distinguishing the camera wearer’s left and right hands from other surgeons’ left and right hands (Fujii et al., 2024). It annotates 15 videos, 15,437 images, 49,652 tool boxes, and 46,320 hand boxes, with a markedly long-tailed tool distribution: Tweezers has 10,012 instances, whereas Skewer has 344 (Fujii et al., 2024). Scene density is high for this domain: the dataset averages 3.2 tool instances per frame, with 8,803 images containing 6–10 instances and 506 images containing 11–15 (Fujii et al., 2024).

POV-Surgery supplies a complementary synthetic regime rather than real OR footage. It comprises 53 sequences and 88,329 high-resolution RGB-D frames rendered from an egocentric camera mounted on the head of a simulated surgeon, with accurate 3D and 2D annotations for hand-object pose and 2D hand-object segmentation masks (Wang et al., 2023). The dataset focuses on three orthopedic instruments—scalpel, friem, and diskplacer—and emphasizes glove diversity, bloodstains, temporal continuity, and realistic occlusion structure (Wang et al., 2023).

EgoExOR expands the scope from single-operator egocentric video to synchronized ego–exo operating-room capture. It is described as the first OR dataset and benchmark to fuse first-person and third-person perspectives, spanning 94 minutes and 84,553 frames at 15 FPS across two emulated spine procedures (Özsoy et al., 30 May 2025). It integrates egocentric RGB, gaze, hand tracking, and audio from wearable glasses; exocentric RGB-D; ultrasound imagery; and dense scene graph annotations with 36 entities, 22 relations, and 568,235 triplets (Özsoy et al., 30 May 2025).

Taken together, these resources show that EgoSurg is methodologically heterogeneous. Some datasets prioritize real open-surgery workflow and privacy-preserving release, some emphasize dense detection and hand–tool interaction, some supply synthetic scale for pose learning, and some extend egocentric reasoning into multi-actor room context.

3. Workflow phase recognition in egocentric open surgery

Phase recognition is a central EgoSurg task because it underpins real-time assistance, context-aware decision support, resource scheduling, quality and safety monitoring, automated video indexing for documentation and teaching, and analytics for process optimization (Fujii et al., 2024). In EgoSurgery-Phase, evaluation uses macro-averaged precision, recall, and Jaccard index across the 9 phases to counter class imbalance (Fujii et al., 2024). The paper notes that the phase distribution is imbalanced and applies resampling during fine-tuning.

The principal method introduced with EgoSurgery-Phase is the gaze-guided masked autoencoder (GGMAE), built on VideoMAE with a ViT-Small backbone (Fujii et al., 2024). The motivation is specific to egocentric open surgery: frames often contain large non-informative areas, including regions outside the surgical field and dark or overexposed zones due to surgical lamps, so random masking may waste capacity on low-information tokens (Fujii et al., 2024). GGMAE uses gaze as an empirical semantic richness prior. For each token $x_i$, gaze intensity is accumulated as

$d_i = \sum_{j \in \Omega_i} G(j),$

converted to masking probabilities

$\pi^t = \mathrm{Softmax}(d^t / \tau),$

and optimized during masked reconstruction with

$$L = \frac{1}{|\Omega|} \sum_{p \in \Omega} |I(p) - \hat{I}(p)|^2.$$

The model is pretrained for 800 epochs with masking ratio $\rho = 0.90$, then fine-tuned for phase classification for 100 epochs with a cross-entropy loss (Fujii et al., 2024).

Quantitatively, GGMAE achieves macro Precision 51.7, Recall 45.6, and Jaccard 33.9 on EgoSurgery-Phase (Fujii et al., 2024). The best prior phase-recognition baseline, NETE, reports Precision 43.7, Recall 35.2, and Jaccard 27.5, so GGMAE improves Jaccard by 6.4 absolute points (Fujii et al., 2024). Relative to MAE-style pretraining baselines, supervised ViT-S from scratch yields Jaccard 27.1, VideoMAE 29.8, VideoMAE V2 30.8, SurgMAE 27.8, and GGMAE 33.9 (Fujii et al., 2024).

The ablations are unusually informative for the semantics of egocentric video. At $\rho = 0.90$, random masking yields Jaccard 30.6, tube masking 29.8, and gaze-guided masking 33.9; for the temperature parameter, $\tau = 0.5$ gives the best result, while $\tau = 1.0$ gives 30.1 and $\tau = 0.25$ gives 27.2 (Fujii et al., 2024). Qualitative examples show that gaze-guided masks concentrate on the surgical field, whereas random masks often select dark or overexposed regions outside the field. This supports the paper’s semantic-richness interpretation rather than merely a generic regularization effect.

A second misconception addressed by this line of work is that open-surgery phase recognition should behave like MIS phase recognition. The data argue otherwise: open surgery introduces wide scene composition changes, non-scope viewpoints, head motion, and privacy-driven subsampling to 0.5 fps, all of which distinguish the problem from endoscopic phase modeling (Fujii et al., 2024).

4. Hand, tool, and pose understanding

EgoSurg research treats hands and instruments not as ancillary cues but as primary state variables. EgoSurgery-Tool formalizes this with dense hand and tool detection in real open surgery (Fujii et al., 2024). Using the COCO evaluation protocol, the strongest reported detector for surgical tool detection is VarifocalNet with AP 45.8, [email protected] 63.3, and [email protected] 51.1; for hand detection, VarifocalNet again leads in AP with 59.4 and [email protected] with 65.3, while DINO is close with hand AP 58.8 and [email protected] 65.6 (Fujii et al., 2024). The authors attribute errors to similar geometries and textures between tools, heavy occlusion, poor lighting, and cases where varied classes such as gauze and retractors are mistaken for background (Fujii et al., 2024). Domain transfer is particularly weak: Faster R-CNN trained solely on EgoHands yields hand AP 8.9 on EgoSurgery-Tool, whereas training on EgoSurgery-Tool yields AP 55.3 (Fujii et al., 2024).

Pose estimation forms a parallel strand. HMD-EgoPose is a single-shot, marker-less framework that uses monocular RGB from a Microsoft HoloLens 2 to jointly estimate the rigid 6DoF pose of an orthopedic drill and the 3D hand skeleton of the user manipulating it (Doughty et al., 2022). The architecture uses EfficientDet-D0 with subnetworks for axis–angle rotation, translation via image center and depth, and a 21-joint 3D hand skeleton (Doughty et al., 2022). On real data, after pretraining on SynColibri and fine-tuning on RealColibri, it achieves tool ADD of $11.00 \pm 7.27$ mm, drill tip error $d_i = \sum_{j \in \Omega_i} G(j),$0 mm, drill bit direction error $d_i = \sum_{j \in \Omega_i} G(j),$1 deg, and hand ADD $d_i = \sum_{j \in \Omega_i} G(j),$2 mm (Doughty et al., 2022). The deployment pipeline streams HoloLens 2 RGB at 896×504 px and 30 FPS to a workstation for TensorRT inference, returning pose estimates with a measured average pixel-to-photon latency of $d_i = \sum_{j \in \Omega_i} G(j),$3 ms; on-device CPU inference on the HoloLens 2 is approximately $d_i = \sum_{j \in \Omega_i} G(j),$4 ms per frame (Doughty et al., 2022).

POV-Surgery addresses the corresponding data bottleneck for egocentric hand and tool pose learning (Wang et al., 2023). It is a large-scale synthetic RGB-D dataset specifically designed around surgical gloves, blood patterns, reflective metallic tools, and temporal manipulation sequences. Methods fine-tuned on POV-Surgery improve sharply over their original checkpoints. For example, METRO improves from 95.11 px to 30.49 px in 2D reprojection error and from 77.46 to 14.90 in MPJPE after fine-tuning; SEMI improves from 77.91 px to 13.42 px in 2D reprojection error and from 115.67 to 15.14 in MPJPE; HandOccNet improves from 64.70 px to 13.80 px in 2D reprojection error and from 95.19 to 14.35 in MPJPE (Wang et al., 2023). The paper also reports a 2D control-point error baseline of 41.56 pixels for object pose after fine-tuning SEMI (Wang et al., 2023).

Across these works, the methodological pattern is consistent. EgoSurg models benefit when hand–tool interaction is represented jointly rather than as separate detection problems, and when egocentric learning is trained on data that explicitly reproduces surgical occlusion, glove appearance, tool reflectance, and line-of-sight geometry.

5. Ego–exo fusion and ambient egocentric replay

A major extension of EgoSurg is the move from single-surgeon egocentric capture to multi-actor operating-room perception. EgoExOR is explicitly positioned as a response to the fact that modern ORs involve multiple agents with distinct attentional foci and that many critical cues are visible only from task-centric, first-person views, while other cues require broader room context (Özsoy et al., 30 May 2025). The dataset records Project Aria glasses worn by the head surgeon, assistant, circulator, and anaesthetist, plus wearable cameras on a microscope and OR light, exocentric Azure Kinect RGB-D cameras, ultrasound monitor feeds, gaze, hand pose, audio, and point clouds (Özsoy et al., 30 May 2025). For scene graph generation, the authors compare ORacle, MM2SG, and a new dual-branch multimodal baseline. On macro F1 over predicates, ORacle achieves 0.63 overall, MM2SG 0.67, and the EgoExOR baseline 0.72; ablations show ego-only 0.68 and exo-only 0.42 (Özsoy et al., 30 May 2025). The authors interpret the largest gains as arising from explicit dual-branch modeling of ego and exo views together with gaze and hand signals.

The specific framework titled “EgoSurg” takes a different route to egocentricity (Zhang et al., 6 Oct 2025). Instead of relying on wearable first-person cameras, it reconstructs dynamic egocentric replays for any OR staff directly from wall-mounted fixed cameras, using four wall-mounted stereo RGB cameras, hardware Precision Time Protocol synchronization, stereo depth via FoundationStereo, and geometry-driven 3D Gaussian Splatting initialized from stereo-derived point clouds (Zhang et al., 6 Oct 2025). Around each physical camera, $d_i = \sum_{j \in \Omega_i} G(j),$5 auxiliary viewpoints are sampled within radius $d_i = \sum_{j \in \Omega_i} G(j),$6 m, rendered from the current 3DGS, refined with a single-step image-conditioned diffusion model following DiFiX3D, and distilled back into the 3DGS optimization (Zhang et al., 6 Oct 2025). Virtual egocentric cameras are anchored at head-level positions associated with team members and oriented toward task-relevant targets using a look-at formulation, without requiring explicit head-pose estimation (Zhang et al., 6 Oct 2025).

The empirical results define the current state of ambient egocentric replay. Against wearable PoV, EgoSurg reports PSNR $d_i = \sum_{j \in \Omega_i} G(j),$7, SSIM $d_i = \sum_{j \in \Omega_i} G(j),$8, and LPIPS $d_i = \sum_{j \in \Omega_i} G(j),$9, compared with depth reprojection at PSNR $\pi^t = \mathrm{Softmax}(d^t / \tau),$0, SSIM $\pi^t = \mathrm{Softmax}(d^t / \tau),$1, LPIPS $\pi^t = \mathrm{Softmax}(d^t / \tau),$2, baseline 3DGS at PSNR $\pi^t = \mathrm{Softmax}(d^t / \tau),$3, SSIM $\pi^t = \mathrm{Softmax}(d^t / \tau),$4, LPIPS $\pi^t = \mathrm{Softmax}(d^t / \tau),$5, and EgoSurg without diffusion at PSNR $\pi^t = \mathrm{Softmax}(d^t / \tau),$6, SSIM $\pi^t = \mathrm{Softmax}(d^t / \tau),$7, LPIPS $\pi^t = \mathrm{Softmax}(d^t / \tau),$8 (Zhang et al., 6 Oct 2025). For novel-view fidelity on Site A, EgoSurg reports PSNR $\pi^t = \mathrm{Softmax}(d^t / \tau),$9, SSIM $$L = \frac{1}{|\Omega|} \sum_{p \in \Omega} |I(p) - \hat{I}(p)|^2.$$0, LPIPS $$L = \frac{1}{|\Omega|} \sum_{p \in \Omega} |I(p) - \hat{I}(p)|^2.$$1, CLIP $$L = \frac{1}{|\Omega|} \sum_{p \in \Omega} |I(p) - \hat{I}(p)|^2.$$2, temporal PSNR $$L = \frac{1}{|\Omega|} \sum_{p \in \Omega} |I(p) - \hat{I}(p)|^2.$$3, temporal SSIM $$L = \frac{1}{|\Omega|} \sum_{p \in \Omega} |I(p) - \hat{I}(p)|^2.$$4, and temporal LPIPS $$L = \frac{1}{|\Omega|} \sum_{p \in \Omega} |I(p) - \hat{I}(p)|^2.$$5 (Zhang et al., 6 Oct 2025). Scene reconstruction takes approximately 2 minutes, and rendering exceeds 150 fps in static scenes (Zhang et al., 6 Oct 2025).

This branch of work changes the meaning of “egocentric” in surgical vision. In earlier EgoSurg work, egocentricity is sensor-defined: a head-mounted camera or HMD records what the wearer sees. In the ambient-replay formulation, egocentricity becomes reconstructive: the viewpoint is synthesized retrospectively from room-scale sensors. A plausible implication is that future operating-room perception systems may combine both paradigms rather than treating them as alternatives.

6. Applications, limitations, and open directions

The application space across EgoSurg studies is broad but coherent. Phase recognition is tied to real-time assistance, context-aware decision support, scheduling, resource allocation, automated event logging, retrospective indexing, documentation, teaching, competency assessment, and quality and safety monitoring (Fujii et al., 2024). HMD-EgoPose targets augmented surgical guidance, planned trajectories, safety margins, and context-aware overlays aligned with the surgeon’s line of sight (Doughty et al., 2022). EgoExOR frames multimodal ego–exo fusion as support for action recognition, human/tool-centric perception, scene graph generation, team coordination, and context-aware robotic or augmented-reality guidance (Özsoy et al., 30 May 2025). The EgoSurg replay framework emphasizes safety adjudication, immersive role-specific training, and counterfactual workflow optimization such as testing alternative stances or room layouts (Zhang et al., 6 Oct 2025).

The limitations are equally consistent. EgoSurgery-Phase remains modest in scale relative to large MIS benchmarks, its inter-rater reliability is not reported, its phase taxonomy is generic, and gaze accuracy or surgeon-specific viewing habits can affect the quality of the semantic prior (Fujii et al., 2024). EgoSurgery-Tool inherits severe long-tail imbalance and heavy occlusion, while some quantitative nuisance statistics, such as occlusion and truncation rates, are not reported (Fujii et al., 2024). POV-Surgery is synthetic and currently limited to three tools and simulated orthopedic activities, even though it demonstrates substantial synthetic-to-real benefit (Wang et al., 2023). EgoExOR is simulated in a high-fidelity simulation center rather than recorded in live surgery, and its scope is limited to two spine procedures and a specific OR configuration (Özsoy et al., 30 May 2025). HMD-EgoPose depends on accurate CAD geometry, camera calibration, and reliable low-latency networking, and its tracking can drop when the tool leaves the image center (Doughty et al., 2022). The ambient EgoSurg framework reconstructs each timestamp independently, so it does not provide explicit spatiotemporal continuity and can exhibit minor drift across adjacent frames; very close viewpoints and fully unobserved regions remain challenging despite diffusion enhancement (Zhang et al., 6 Oct 2025).

The research directions proposed in these papers are notably convergent. They include scaling datasets to more procedures, surgeons, hospitals, and specialties; incorporating additional team members and multiple egocentric roles; combining video with gaze, audio, instrument tracking, kinematics, ultrasound, and point clouds; pursuing real-time and privacy-preserving deployment; extending self-supervision with motion cues, semantic segmentation of the surgical field, or instrument detections; and improving adaptation across hospitals, camera systems, and OR layouts (Fujii et al., 2024, Özsoy et al., 30 May 2025, Fujii et al., 2024). From the evidence available, EgoSurg is evolving from isolated first-person datasets into a broader operating-room perception program in which viewpoint, attention, manipulation, and workflow are modeled jointly, whether the visual field is sensed directly or reconstructed after the fact.