---
title: 'TrackOR: Long-Term OR Staff Tracking'
url: https://www.emergentmind.com/topics/trackor
type: topic
---

# TrackOR: Long-Term OR Staff Tracking

TrackOR is a framework for long-term multi-person tracking and re-identification in the operating room (OR), designed to maintain a persistent identity state over entire surgical procedures rather than only within short visible intervals. It is formulated as an end-to-end “tracking by detection” system with an online tracking stage and an offline global recovery stage, and its central technical claim is that 3D geometric signatures derived from segmented human point clouds are more reliable than conventional RGB appearance cues in the OR, where staff exhibit severe occlusions, crowded interactions, homogeneous attire, and prolonged exits and re-entries. On MM-OR, TrackOR reports **82.3% Association Accuracy** and improves the strongest baseline by about **+11 percentage points**, while also enabling downstream analyses such as temporal pathway imprints for staff-centric OR intelligence [2508.07968].

## 1. Problem setting and formalization

TrackOR addresses operating-room scene understanding at the level of specific staff members over long time spans. The paper distinguishes between a **tracklet**, which is a single continuous visible segment of one person, and the person’s full **trajectory**, which may contain multiple separated visible episodes because OR personnel can leave and later re-enter the room. This distinction is formalized as
\[
\text{trk}_m^a = \{s_t^m\}_{t=b_m^a}^{e_m^a},
\]
where \(s_t^m\) is the state of person \(m\) at time \(t\), and \(b_m^a, e_m^a\) are the start and end frame indices for the \(a\)-th visible period. The complete trajectory is then
\[
\tau_m = \{ \text{trk}_m^1, \text{trk}_m^2, ..., \text{trk}_m^{A_m}\},
\]
where \(A_m\) is the number of separate appearances of person \(m\) [2508.07968].

The OR setting combines several classical failure modes of multi-object tracking. The paper explicitly emphasizes **severe occlusions**, **crowded scenes**, **homogeneous attire**, **long procedures**, and the “**revolving door**” reality of surgery, in which personnel such as a circulating nurse may leave the OR and later return. In this regime, short-term tracking is insufficient because a tracker must solve not only frame-to-frame association but also re-identification after long absences. TrackOR is therefore motivated by a shift from **role-based** understanding to **staff-centric** understanding: personalized OR intelligence requires distinguishing which specific nurse, technician, or surgeon is present, not merely inferring an abstract role [2508.07968].

## 2. End-to-end framework and sensing configuration

TrackOR uses **multi-view RGB-D data from ceiling-mounted cameras**. In the MM-OR dataset used for evaluation, each frame includes **three annotated RGB views** and a corresponding **3D point cloud**. These inputs are used in a complementary manner: multi-view RGB is used for **3D human pose detection**, while the 3D point cloud is segmented per person and converted into depth-based representations for re-identification [2508.07968].

The framework has two major stages. The first is an **online tracking stage** that performs framewise association while attempting to preserve identity as robustly as possible. The second is an **offline global recovery stage** that links fragmented tracklets into person-level trajectories and produces **analysis-ready global trajectories**. The detection component follows prior “3D-first” OR approaches and directly detects **3D human poses** from multi-view RGB input using **VoxelPose**. The paper summarizes VoxelPose as predicting 2D heatmaps in each camera view, aggregating them into a unified 3D voxel volume, using a 3D CNN to predict each person’s root location, and then using a second 3D CNN to regress the detailed 3D pose [2508.07968].

After pose detection, the full scene point cloud is segmented to isolate each human, and each 3D pose is associated with its corresponding **person point cloud**. TrackOR then renders each segmented 3D human point cloud into **8 virtual viewpoints** arranged equidistantly in a circle around the person. These projections produce **2D depth maps**, which are processed by a re-identification network to yield a feature tensor
\[
\mathcal{F}_t^{i} \in \mathbb{R}^{8 \times C},
\]
where \(C\) is the feature dimension. This feature is the system’s central identity representation and is explicitly geometric rather than texture-based [2508.07968].

## 3. 3D geometric signatures and online association

The main technical contribution of TrackOR is the use of **3D geometric signatures** for person re-identification in the OR. A geometric signature is constructed from a person’s **segmented 3D point cloud**, rendered from 8 virtual camera viewpoints, converted into depth maps, and encoded by a ReID network. The intended identity cues are person-specific 3D structural properties such as body shape, proportions, silhouette, and multi-view geometric structure. The paper’s claim is that these cues are more robust than RGB appearance in the OR because gowns, masks, and caps suppress color and texture variation [2508.07968].

Framewise detections are denoted
\[
D_t = \{d_t^1, d_t^2, \dots, d_t^{N_t}\},
\]
and the target output trajectories are
\[
\mathcal{T} = \{\tau_1, \tau_2, ..., \tau_M\}.
\]
For online tracking, each detection and each active trajectory is represented by a **ReID feature vector** and a **3D bounding box**. TrackOR computes a cost matrix \(\mathbf{C}\) whose entries combine a **shape cost** based on the **cosine dissimilarity** of the ReID features and a **spatial cost** based on **3D Generalized IoU (GIoU)** between 3D boxes. Assignment is solved with the **Hungarian algorithm**, and any match whose final cost exceeds a threshold \(\gamma\) is rejected. Matched detections update existing tracks, unmatched detections initialize new tracks, and unmatched trajectories are marked as **lost** [2508.07968].

This design is structurally close to a classical linear-assignment tracker, but its identity cue is specific to the OR. The paper explicitly states that TrackOR’s own method relies on **3D positional overlap and ReID geometry**, rather than on a Kalman filter or an explicit motion model. The role of **3D GIoU** is especially important under ceiling-mounted multi-view sensing, because 2D overlap can be ambiguous when people occlude one another around the operating table, whereas 3D position and extent are more physically meaningful [2508.07968].

## 4. Offline global trajectory recovery

TrackOR supplements online association with a separate **offline recovery** stage designed to repair fragmentation and long-term identity errors. This stage operates at the **tracklet** level rather than the frame level. For each tracklet, TrackOR stores a sequence of framewise descriptors of shape
\[
\mathbb{R}^{l \times 8 \times C},
\]
where \(l\) is the tracklet length. These descriptors are aggregated by **temporal max-pooling** to obtain one descriptor per virtual view. An **SVM-Gallery** classifier then predicts the tracklet identity, and the paper states that the final identity is assigned by **majority vote over the 8 view-specific feature vectors**. All tracklets assigned the same identity are then grouped into the complete trajectory of that person [2508.07968].

The offline stage is intended to repair precisely the long-term errors that matter in surgical workflows: fragmentation caused by occlusion, identity switches introduced during difficult interactions, and failures to reconnect a person who disappears and later re-enters the room. The paper characterizes this as a “straightforward and effective offline global trajectory recovery.” It is not presented as a graph-optimization formulation; instead, it is a tracklet classification-and-grouping procedure that depends on the stability of the geometric identity features [2508.07968].

This separation between online and offline outputs is operationally important. The online stage provides immediate framewise tracks and tracklets, while the offline stage produces **global, merged, analysis-ready trajectories** that span all appearances of each individual. A plausible implication is that TrackOR is designed not only for live scene understanding but also for retrospective workflow analysis over whole procedures.

## 5. Experimental evaluation on MM-OR

TrackOR is evaluated on the **MM-OR dataset**, which the paper argues is more appropriate for long-term identity tracking than MVOR or 4D-OR. MM-OR contains the challenging conditions emphasized by the method: frequent occlusions, multiple clinicians in tight spaces, homogeneous attire, and realistic OR dynamics. The experiments use all MM-OR takes with segmentation labels, yielding **20 takes total** and **23,442 frames**, split into **13 takes train (62%)**, **2 takes validation (16%)**, and **5 takes test (22%)**. Each take corresponds to a **single surgery** [2508.07968].

All methods are evaluated in **2D image space** so that 2D and 3D trackers can be compared fairly. For 3D trackers, 3D poses are projected into each 2D image plane to produce boxes. Reported metrics include **HOTA**, **AssA**, **DetA**, **IDF1**, **IDSW**, **MOTA**, count metrics, and **FPS**. Because boxes derived from projected 3D poses differ geometrically from silhouette-derived 2D ground truth, the HOTA threshold range is modified to
\[
\alpha = 0.05 - 0.5.
\]
Baselines include 2D RGB trackers such as OC-Sort, ByteTrack, StrongSort, BoostTrack++, Deep OC-Sort, and BoT-SORT, as well as 3D pose-based trackers such as KSP Tracker, Nearest-Neighbor, and Kalman Filter [2508.07968].

From **Table 1**, TrackOR achieves:

- **HOTA = 82.216**
- **AssA = 82.300**
- **DetA = 83.685**
- **IDF1 = 76.362**
- **IDSW = 125**
- **FPS = 17**

The strongest baseline in association is **BoT-SORT** with **AssA = 71.309**, so the reported improvement is
\[
82.300 - 71.309 = 10.991
\]
percentage points. Other comparisons in the same table include **Deep OC-Sort: AssA 66.348**, **Nearest-Neighbor (3D): AssA 65.813**, and **Kalman Filter (3D): AssA 62.441**. The pattern described by the paper is that **3D-only spatial tracking** helps but is not enough, **2D appearance-ReID** helps but is limited by OR visual homogeneity, and **3D geometry + ReID** gives the strongest identity association [2508.07968].

The paper also notes a trade-off. Some 2D trackers achieve better detection-oriented metrics because their detectors were fine-tuned directly on ground-truth boxes, whereas TrackOR’s 3D pose detector is trained with the cited **self-supervised approach** because MM-OR does not have 3D pose annotations. The authors therefore argue that older detection-heavy metrics such as **MOTA** favor 2D methods, while TrackOR’s main strength lies in maintaining identity correctly over long OR sequences [2508.07968].

## 6. Applications, limitations, and subsequent context

TrackOR is positioned as an enabling technology for **personalized intelligent operating rooms**. Once persistent staff identities are available, the system can support individualized workflow analysis, personalized feedback, safety monitoring at the level of specific staff, longitudinal study of movement strategies, and tailored assistance for the surgical team. The paper’s concrete downstream example is **temporal pathway imprints**, which are generated by taking a person’s trajectory root positions, projecting them onto the **X-Y floor plane**, visualizing them from a **bird’s-eye view**, and adding context such as **regions of interest** or **sterility zones**. In **Figure 4**, pathway imprints for the **same robot technician** in two surgeries show that the technician’s main workspace was the **MPS station** in both surgeries, that the technician interacted with the robot **twice** in Surgery 1 and **once** in Surgery 2, and that in Surgery 2 the non-sterile technician came into **close proximity with the sterile patient table** [2508.07968].

The framework also has clear operational constraints. TrackOR runs at **17 FPS** on a **single RTX 2080 Ti**. It requires **ceiling-mounted multi-view RGB-D cameras**, calibrated multi-view geometry, 3D point cloud generation, and person segmentation in 3D. Its performance depends on reliable **3D pose estimation**, sufficiently accurate **point cloud segmentation**, and enough geometric completeness in the observed person cloud. The paper also notes that 3D-based methods show lower **DetA** than the strongest 2D methods in its experiments, and that online tracking can still fragment, which motivates the offline recovery stage. A further practical issue mentioned briefly is that **privacy concerns** make public OR datasets scarce, especially from ceiling-mounted cameras [2508.07968].

A later development in the same research area, **“Geometry OR Tracker: Universal Geometric Operating Room Tracking”**, shifts emphasis from long-term staff re-identification to the geometric reliability of multi-view OR tracking itself. That work argues that shared-frame OR tracking is often limited by bad calibration and RGB-D registration, and reports that a rectification front-end reduces cross-view depth disagreement on MM-OR by more than **30×** compared to raw calibration while improving world-frame tracking [2603.00560]. This suggests a broader trajectory in OR tracking research: TrackOR establishes persistent staff-centric identity tracking through 3D geometric signatures and offline recovery, while subsequent work emphasizes that world-scale, metrically consistent OR tracking also depends critically on cross-view geometric consistency.

Source: https://www.emergentmind.com/topics/trackor