ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding (2407.19435v1)

Abstract: Surgical instrument segmentation is crucial in surgical scene understanding, thereby facilitating surgical safety. Existing algorithms directly detected all instruments of pre-defined categories in the input image, lacking the capability to segment specific instruments according to the surgeon's intention. During different stages of surgery, surgeons exhibit varying preferences and focus toward different surgical instruments. Therefore, an instrument segmentation algorithm that adheres to the surgeon's intention can minimize distractions from irrelevant instruments and assist surgeons to a great extent. The recent Segment Anything Model (SAM) reveals the capability to segment objects following prompts, but the manual annotations for prompts are impractical during the surgery. To address these limitations in operating rooms, we propose an audio-driven surgical instrument segmentation framework, named ASI-Seg, to accurately segment the required surgical instruments by parsing the audio commands of surgeons. Specifically, we propose an intention-oriented multimodal fusion to interpret the segmentation intention from audio commands and retrieve relevant instrument details to facilitate segmentation. Moreover, to guide our ASI-Seg segment of the required surgical instruments, we devise a contrastive learning prompt encoder to effectively distinguish the required instruments from the irrelevant ones. Therefore, our ASI-Seg promotes the workflow in the operating rooms, thereby providing targeted support and reducing the cognitive load on surgeons. Extensive experiments are performed to validate the ASI-Seg framework, which reveals remarkable advantages over classical state-of-the-art and medical SAMs in both semantic segmentation and intention-oriented segmentation. The source code is available at https://github.com/Zonmgin-Zhang/ASI-Seg.

• The paper introduces a novel framework that leverages audio commands and multimodal fusion to segment surgical instruments based on real-time surgeon intentions.
• The methodology integrates intention-oriented multimodal fusion and a contrastive learning prompt encoder to achieve superior IoU scores on EndoVis datasets.
• The framework enhances surgical safety by reducing visual cognitive load and enabling dynamic, intention-aware instrument segmentation during procedures.

An Overview of ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding

The paper "ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding" presents a novel approach to surgical instrument segmentation by leveraging audio commands from surgeons to guide segmentation tasks. Traditional segmentation methods, which detect pre-defined instrument categories from images, fail to consider the dynamic and intention-focused nature of surgical procedures. ASI-Seg addresses this limitation by integrating auditory inputs to ascertain the surgeon's real-time focus and preferences during operations.

The authors propose a comprehensive framework called ASI-Seg that accurately segments relevant surgical tools according to the surgeon’s spoken commands. Central to this framework is the intention-oriented multimodal fusion that interprets audio instructions, integrating audio, text, and visual inputs to refine instrument segmentation. The framework further enhances its capabilities with a contrastive learning prompt encoder, which differentiates essential instruments from irrelevant ones.

Methodology

The ASI-Seg framework involves several key components:

• Intention-Oriented Multimodal Fusion: This module encapsulates audio intention recognition alongside text and visual fusion. It leverages an audio intention recognition module to convert spoken words into actionable segmentation intents and then utilizes both text and visual features to enrich the segmentation framework.
• Contrastive Learning Prompt Encoder: This element utilizes contrastive learning to bolster the discernment between relevant and irrelevant instruments by emphasizing different features. It applies a cross-attention mechanism to facilitate precise feature extraction, thereby supporting the segmentation process with meaningful prompts.

Experimental Results

The experimental evaluation of ASI-Seg demonstrates its significant advantages in both semantic segmentation and intention-oriented segmentation tasks. Using the EndoVis2018 and EndoVis2017 datasets, ASI-Seg outperformed contemporary state-of-the-art approaches, evidenced by its superior Intersection over Union (IoU) metrics. On EndoVis2018, ASI-Seg achieved a Challenge IoU of 82.37%, compared to 80.33% by the SurgicalSAM, solidifying its efficacy even under constrained experimental setups.

Implications and Future Work

The presented framework offers a pivotal advancement in the field of surgical scene understanding by reducing visual cognitive load during surgeries, thereby enhancing procedural safety and outcomes. From a practical perspective, the reliance on audio inputs aligns the segmentation process more closely with the operational demands of the surgical workflow, rather than static pre-defined image-based inputs.

The paper demonstrates how auditory-driven intention recognition can dynamically adapt segmentation tasks in real time, paving the way for more interactive and responsive surgical assistant systems. The ASI-Seg framework contributes an innovative direction towards integrating auditory and multimodal inputs into medical AI, suggesting potential expansions into more complex multimodal surgical assistant systems.

For future developments in the AI domain, such a methodology could extend beyond surgical applications to any field where user intent can be phonetically expressed and dynamically implemented in tasks relying on real-time decision-making. Researchers could explore further refinements in audio processing and intention recognition, potentially integrating additional sensory inputs to further enhance the system's adaptive capabilities.

In conclusion, ASI-Seg represents a meaningful stride in bridging the gap between surgeon intentions and automated instrument segmentation, while facilitating a more intuitive interaction pathway within computer-assisted surgery systems.