nnInteractive: Redefining 3D Promptable Segmentation (2503.08373v1)

Abstract: Accurate and efficient 3D segmentation is essential for both clinical and research applications. While foundation models like SAM have revolutionized interactive segmentation, their 2D design and domain shift limitations make them ill-suited for 3D medical images. Current adaptations address some of these challenges but remain limited, either lacking volumetric awareness, offering restricted interactivity, or supporting only a small set of structures and modalities. Usability also remains a challenge, as current tools are rarely integrated into established imaging platforms and often rely on cumbersome web-based interfaces with restricted functionality. We introduce nnInteractive, the first comprehensive 3D interactive open-set segmentation method. It supports diverse prompts-including points, scribbles, boxes, and a novel lasso prompt-while leveraging intuitive 2D interactions to generate full 3D segmentations. Trained on 120+ diverse volumetric 3D datasets (CT, MRI, PET, 3D Microscopy, etc.), nnInteractive sets a new state-of-the-art in accuracy, adaptability, and usability. Crucially, it is the first method integrated into widely used image viewers (e.g., Napari, MITK), ensuring broad accessibility for real-world clinical and research applications. Extensive benchmarking demonstrates that nnInteractive far surpasses existing methods, setting a new standard for AI-driven interactive 3D segmentation. nnInteractive is publicly available: https://github.com/MIC-DKFZ/napari-nninteractive (Napari plugin), https://www.mitk.org/MITK-nnInteractive (MITK integration), https://github.com/MIC-DKFZ/nnInteractive (Python backend).

Overview of nnInteractive for Medical Image Segmentation

The paper presents nnInteractive, a framework for promptable medical image segmentation, which addresses several shortcomings of existing solutions in medical imaging. Unlike traditional papers that introduce methodological novelty in network architectures, this work focuses on a comprehensive solution that significantly improves the state-of-the-art (SOTA) in segmentation performance through a well-executed configuration and training process.

Background

Foundation models, like the Segment Anything Model (SAM), have significantly improved image segmentation across various domains, including natural and medical images. While successful in some areas of medical imaging, existing solutions mainly cater to 2D images and have limited capabilities in handling 3D volumes. Their training is often constrained to specific imaging modalities, such as CT, and lacks support for open-set segmentation. Additionally, interactive features in current models are restricted to basic prompts like points and bounding boxes, with few options for advanced refinement.

Contributions

nnInteractive addresses critical limitations in existing promptable models concerning interaction types, positive/negative prompting, usability, and speed. Notable contributions include:

• Interactive Segmentation: The paper introduces the first 3D interactive open-set segmentation model supporting a variety of prompts—positive/negative points, scribbles, bounding boxes, and lasso prompts—across multiple imaging modalities (CT, MR, PET, etc.).
• Scalability: The framework is trained on over 120 3D datasets, surpassing competitors in dataset size and modality range. This scale offers improved zero-shot capabilities and segmentation accuracy across diverse scenarios.
• User-Friendly Interaction: nnInteractive supports intuitive 2D interactions for efficient 3D annotation, bridging the gap between user-friendly 2D prompts and precise 3D segmentation.
• Early Prompting Strategy: The framework adopts an early prompting strategy, incorporating user inputs during the initial stages of feature extraction to preserve spatial relationships, leveraging the strength of convolutional models.

Results and Implications

Extensive benchmarking highlights nnInteractive's superior accuracy, adaptability, and usability compared to existing methods. The scalable architecture and interactive design establish a new benchmark for user-friendly medical segmentation tools that integrate seamlessly into clinical workflows.

The results imply significant practical advancements, enabling more precise and efficient medical image annotation, and improving diagnostic accuracy. Theoretically, it paves the way for further exploration in user interactions for complex segmentation tasks. The introduction of holistic prompt interactions and the scalability of data sets mark a considerable advancement in the generalizability and robustness of medical imaging models.

Future Directions

Future research may focus on expanding the interactive capabilities of nnInteractive, exploring additional prompt types, or enhancing real-time usability in clinical contexts. The model's emphasis on user-friendly design and extensive dataset integration opens avenues for its application in various medical imaging challenges, potentially leading to broader adoption and adaptation in complex diagnostic scenarios.

In summary, nnInteractive represents a substantial step forward in interactive medical image segmentation, setting new standards for adaptability, accuracy, and clinical usability, all implemented through a strategic, well-grounded approach to configuration and training.