Instrument–Tissue Interaction Tracking

Updated 8 June 2026

Instrument–tissue interaction tracking is the real-time mapping of surgical tools and tissues, capturing their spatial and temporal relationships.
It integrates dense video tracking, force sensing, and advanced biomechanical models to enhance scene understanding and operational safety.
Key applications include context-aware automation, precise outcome assessment, and improved safety in robotic-assisted surgical procedures.

Instrument–tissue interaction tracking refers to the simultaneous, continuous localization and association of surgical tools (instruments) and biological tissues, including their mutual relationships, physical contacts, and the spatiotemporal evolution of these interactions during interventional or robotic-assisted surgical procedures. This capability underpins scene understanding, safety, context-aware automation, quantitative outcome assessment, and task-driven robotics. The field encompasses methods ranging from dense point-wise video tracking, structured triplet detection and segmentation, affordance heatmaps, biomechanical modeling, force sensing, to advanced learning-based fusion frameworks for the real-time recovery and analysis of instrument–tissue contact events.

1. Foundational Concepts and Task Definitions

Instrument–tissue interaction tracking tasks extend conventional perception (instrument detection, tissue segmentation) by explicitly reasoning about their dynamic spatial and functional relationships, often at fine granularity. The core challenges include:

Multi-object and dense point tracking: Jointly localizing moving instruments and deforming, often texture-less, tissue surfaces under severe occlusion, specular glare, and fast motion.
Instance-level interaction grounding: Associating actions to specific instrument-tissue pairs with spatial precision, overcoming ambiguity in multi-tool scenes and minimizing misassociation errors.
Contact event inference: Detecting, localizing, and classifying physical contact points and complex interaction types (grasp, cut, cauterize, retract, etc.), while maintaining robust temporal consistency and visibility state tracking (occluded, out-of-view).
Real-time constraints and safety: Delivering feedback, enforcing safety policies, and enabling closed-loop control in time-critical surgical settings.

Task definitions formalize outputs as either trajectory sets (dense or sparse), relation tuples/quintuples (such as ⟨instrument, verb, tissue, action, bounding boxes⟩), or pixel-wise triplet segmentations—enabling both frame-level and continuous event reasoning (Zhan et al., 2024, Lin et al., 2024, Alabi et al., 1 Nov 2025).

2. Dense and Sparse Tracking Methodologies

Dense, Long-Term Point Tracking: The SurgMotion pipeline exemplifies state-of-the-art in long-term, dense 2D trajectory recovery within real surgical video (Zhan et al., 2024). The architecture builds on a canonical volume representation, where each query pixel is lifted to 3D and mapped across frames via invertible Real-NVP networks. A NeRF-style MLP volumetrically renders color and density for differentiable tracking. Three surgical-specific priors enhance performance:

Tool-mask module: Ensures instrument points remain within segmented tool regions.
As-rigid-as-possible (ARAP) constraint: Enforces rigidity within instrument clusters, maintaining geometric integrity under rapid or occluded motion.
Long-range LoFTR-guided matching: Recovers correspondences across distant frames, mitigating drift and identity switches in challenging scenarios.

The loss function integrates photometric, optical-flow, mask-consistency, ARAP rigidity, and long-term feature-matching terms, with schedule-controlled weights.

Sparse Landmark and Feature Tracking: Prior paradigms, such as TAP-Vid, PIPs, and SENDD (Schmidt et al., 2023, Chen et al., 2024), use sparse keypoint detection, descriptor-based matching (across stereo or temporal pairs), and graph neural network (GNN) architectures for both depth and motion estimation. SENDD, for instance, detects 1280~ sparse landmarks per frame and refines 3D flow via GNN-based message passing and coordinate-based attention, supporting arbitrary query points and integration with instrument pose for local interaction analysis.

Occlusion Handling & Uncertainty Estimation: The A-MFST framework augments SENDD by combining multi-flow trajectory candidates evaluated via forward–backward consistency and instrument segmentation with SAM2. Candidate points under instrument occlusion are robustly masked and excluded, reducing mean endpoint errors under occlusion by ~45% relative to the baseline (Chen et al., 2024).

3. Structured Interaction Representation: Action Triplets and Segmentation

Triplet and Quintuple Detection: Multiple works formalize interaction tracking as the detection of structured relation tuples—most notably ⟨instrument class, verb/action, tissue/target⟩. Tripnet (Nwoye et al., 2020) employs a three-headed network with weakly-supervised Class Activation Guides and learns a 3D interaction tensor, achieving robust frame-level triplet detection. MCIT-IG (Sharma et al., 2023) improves spatial association by first generating instrument- and target-aware embeddings (via transformers), then connecting detected objects via a bipartite dynamic graph to associate instrument instances with both verbs and targets using mixed (strong and pseudo) supervision.

Spatially Grounded Triplet Segmentation: TargetFusionNet (Alabi et al., 1 Nov 2025) unifies instance segmentation and triplet assignment. Extending Mask2Former, it fuses weak anatomy priors with transformer decoder queries, computing per-instrument instance masks and their corresponding triplet labels. This avoids the ambiguity of class activation maps or bounding-box associations and enables pixel-level, instance-specific triplet reasoning. On the CholecTriplet-Seg dataset, TargetFusionNet achieves mAP ${}_{IVT}^{seg}$ of 13.47%, outperforming prior Mask2Former-Triplet baselines (12.23%).

Table: Triplet Segmentation Benchmarks (CholecTriplet-Seg test set)

| Method | mAP $_{IVT}^{seg}$ | mAP $_{IVT}^{det}$ | mAP $_{IVT}^{rec}$ | |--------------------------|------------------:|------------------:|------------------:| | RDV+Mask2Former | 8.73% | 8.29% | 29.51% | | Mask2Former-Triplet | 12.23% | 12.92% | 31.26% | | TargetFusionNet (ours) | 13.47% | 13.45% | 34.23% |

4. Affordance and Safety-Aware Interaction Prediction

Beyond pure detection, AffordTissue (Maksutova et al., 1 Apr 2026) addresses the critical need for instrument–action–specific affordance region prediction on tissue surfaces, central to safe automation. The architecture uses a temporal video encoder (Video Swin Transformer), language conditioning (SigLIP-2 on prompt ⟨surgery, tool, action⟩), and a DiT/AdaLN-style decoder to predict dense affordance heatmaps aligned to annotated safe interaction polygons. The predicted tissue regions inform explicit safety policies: a simple tip-position check against the affordance heatmap can trigger a safety stop in real time at 24 fps/40 ms latency.

Performance is quantified by Average Symmetric Surface Distance (ASSD)—AffordTissue achieves 20.6 px versus 60.2 px for state-of-the-art VLMs (Molmo-VLM), indicating ~3× improvement in affordance localization.

5. Biomechanical, Force, and Physical Interaction Modeling

Beyond purely vision-based frameworks, integration of force estimation and biomechanical modeling is essential for both high-fidelity simulation and objective measurement of tool–tissue interactions.

Sensor-Based Force Tracking: Modular instrument frameworks with wrist-mounted 6-DOF force/torque sensors (Shaker et al., 30 Apr 2026) provide haptic feedback by real-time compensation for bias and gravity, coordinate transformations across multiple tool frames, and nonlinear perceptual scaling for operator safety. This configuration achieves <0.1 N RMS accuracy and maintains <5 ms control latency, doubling success rates and reducing force-regulation errors in user studies.

Parametric FE and Elastic Modeling: For needle–tissue interactions, validated parametric models embed high-fidelity FE simulations of hyperelastic, frictional tissue contact in polynomial form for real-time haptic feedback at >10 kHz update rates (Bora et al., 2022).

Image/Model-Based Force Estimation: In robotic endovascular navigation, image-based estimation of tool-vessel wall contact uses continuous nonlinear beam FEM—parameterized by spatially varying flexural rigidity—to solve an online inverse boundary value problem and reconstruct distributed, multi-point contact forces directly from intravascular video (Razban et al., 2020). Local contact forces (e.g., ~0.1–0.3 N peaks in cannulation tasks) and stress maps are resolved with 1.3–3.1 mm RMSE shape accuracy for procedural safety analysis.

6. Probabilistic and Context-Aware Relational Models

Probabilistic task parameterization, as in TP-GMM frameworks (Wang et al., 14 Apr 2025), combines sparse keypoint tracking of tissue and tool landmarks with local frame adaptation (via PCA clustering of tissue keypoints) and dynamic relative pose encoding to build Gaussian mixture models over task-conditioned tool–tissue poses. This statistical representation yields data-driven prediction of safe and context-appropriate interactions, integrating expert priors and scaling to complex scenes with deformable environments.

Complementary simulation platforms (Han et al., 2020) use position-based dynamics (PBD) for 2D tissue tracking, integrating constraint-based mesh deformation with fast collision queries, to support real-time interactive surgical planning and controller energy-based feedback.

7. Evaluation Protocols, Datasets, and Benchmarking

New, richly annotated datasets enable robust benchmarking:

SurgMotion Surgical Tracking Dataset (Zhan et al., 2024): 1,200 frames, 30,000 annotated correspondences, 25 points/frame across tissue and instruments, with occlusion and out-of-view flags.
CholecTriplet-Seg (Alabi et al., 1 Nov 2025): >30,000 frames, 49,866 spatially grounded triplets via aligned instrument instance and triplet labels.
AffordTissue Benchmark (Maksutova et al., 1 Apr 2026): 15,638 clips from 103 cholecystectomy cases, with dense tool–action polygon affordance labels.

Metrics include endpoint error (< $\delta_{avg}^x$ ), average Jaccard (AJ), occlusion accuracy (OA), symmetric surface distance (ASSD), and full/partial mAP on instance and triplet detection or segmentation.

Recent advances demonstrate significant improvements—e.g., SurgMotion attains AJ = 63.0%, $<\delta_{avg}^x$ = 74.2%, OA = 92.3% on tools in challenging RAMIS, with a 10–15% drift reduction over prior TAP methods (Zhan et al., 2024).

References

(Zhan et al., 2024) Tracking Everything in Robotic-Assisted Surgery
(Maksutova et al., 1 Apr 2026) AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction
(Alabi et al., 1 Nov 2025) Grounding Surgical Action Triplets with Instrument Instance Segmentation
(Chen et al., 2024) A-MFST: Adaptive Multi-Flow Sparse Tracker for Real-Time Tissue Tracking Under Occlusion
(Shaker et al., 30 Apr 2026) An Experimental Modular Instrument With a Haptic Feedback Framework for Robotic Surgery Training
(Nwoye et al., 2020) Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets
(Sharma et al., 2023) Surgical Action Triplet Detection by Mixed Supervised Learning
(Bora et al., 2022) Parametric modelling of needle-tissue interaction using finite element analysis
(Razban et al., 2020) Image-based Intraluminal Contact Force Monitoring in Robotic Vascular Navigation
(Schmidt et al., 2023) SENDD: Sparse Efficient Neural Depth and Deformation for Tissue Tracking
(Wang et al., 14 Apr 2025) Probabilistic Task Parameterization via Sparse Landmarks Tracking in Robotic Surgery
(Han et al., 2020) A 2D Surgical Simulation Framework for Tool-Tissue Interaction
(Lin et al., 2024) Instrument-tissue Interaction Detection Framework for Surgical Video Understanding