MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation (2303.14444v2)

Abstract: The medical imaging community generates a wealth of datasets, many of which are openly accessible and annotated for specific diseases and tasks such as multi-organ or lesion segmentation. Current practices continue to limit model training and supervised pre-training to one or a few similar datasets, neglecting the synergistic potential of other available annotated data. We propose MultiTalent, a method that leverages multiple CT datasets with diverse and conflicting class definitions to train a single model for a comprehensive structure segmentation. Our results demonstrate improved segmentation performance compared to previous related approaches, systematically, also compared to single dataset training using state-of-the-art methods, especially for lesion segmentation and other challenging structures. We show that MultiTalent also represents a powerful foundation model that offers a superior pre-training for various segmentation tasks compared to commonly used supervised or unsupervised pre-training baselines. Our findings offer a new direction for the medical imaging community to effectively utilize the wealth of available data for improved segmentation performance. The code and model weights will be published here: [tba]

• The paper introduces a novel framework that decouples overlapping class predictions using Sigmoid activations to effectively leverage multiple datasets.
• It proposes a dataset-adaptive loss function that mitigates class imbalance by computing Dice and Binary Cross-Entropy losses across images.
• Extensive experiments across 13 public CT datasets demonstrate improved dice scores and efficiency, outperforming single-dataset approaches.

Analysis of "MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation"

The paper entitled "MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation" introduces a novel methodology for leveraging multiple partially labeled CT datasets to train a comprehensive model for medical image segmentation. This work addresses a significant challenge in medical imaging: the underutilization of diverse annotated data available across different datasets. By harnessing the wealth of existing information, the proposed method seeks to improve segmentation outcomes, particularly in complex tasks such as lesion detection and multi-organ segmentation, which often utilize disparate, conflicting class definitions.

Methodological Advancements

The central contribution of this research is the MultiTalent framework, which accommodates variability in class definitions across datasets by maintaining individual class labeling properties. It achieves this via:

• Decoupled Segmentation Outputs: The network employs Sigmoid activations allowing for overlapping class predictions, circumventing the limitations typical of Softmax activations.
• Dataset and Class Adaptive Loss Function: By tailoring a loss function strategy that accounts for annotations only present within each specific dataset, issues arising from class imbalance are mitigated. This nuanced approach involves calculating the Dice and Binary Cross-Entropy losses across all images in a batch rather than on a per-image basis.

To assure general applicability, MultiTalent was tested across three different network architectures—3D U-Net, Resenc U-Net, and SwinUNETR—to establish its robustness in handling varying network topologies.

Experimental Evaluation

Comprehensive experiments conducted over 13 public abdominal CT datasets, encompassing 47 classes with 1477 3D images, demonstrate that MultiTalent generally outperforms single-dataset state-of-the-art methods. It is especially noteworthy in improving dice scores for challenging and clinically important classes, such as those involving tumors. The method also shows considerable training and inference time efficiency, being a single model as opposed to an ensemble of individually trained models.

Additionally, MultiTalent displayed significant performance in transfer learning scenarios. When tested over datasets like BTCV, AMOS, and KiTS19, it outperformed both unsupervised and other supervised pre-training baselines, indicating its strength in generalizing features across various tasks.

Competitive Assessment

The MultiTalent approach outstrips previous attempts in multi-dataset learning by effectively addressing the challenge of inconsistent class labels across datasets without compromising performance on par with models trained explicitly for specific datasets. On the BTCV leaderboard, MultiTalent's superior performance against other multi-dataset approaches illustrates its potential as a robust framework for foundational model pre-training in medical image segmentation.

Implications and Future Work

This work's implications span practical improvements in medical imaging workflows, where better segmentation can assist in diagnostic accuracy and therapeutic interventions. MultiTalent sets a precedent for utilizing partially labeled datasets more effectively and could catalyze further research in combining diverse datasets into unified models without sacrificing individual dataset label integrity.

Future research avenues could explore extending the MultiTalent framework to integrate other modalities beyond CT, such as MRIs, or expanding its applicability to other domains within medical diagnostics that similarly suffer from diverse yet incomplete data annotations. Another potential development could focus on enhancing the architecture to better support transformer-based models, as the current paper indicates that convolutional architectures benefited more from the MultiTalent paradigm.

In summary, MultiTalent represents a significant step forward in exploiting the synergistic potential of available medical imaging data, achieving not only improved performance metrics but also reshaping the approach to training and deploying segmentation models across multiple, disparate datasets.