Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation

Abstract: Automatic parsing of anatomical objects in X-ray images is critical to many clinical applications in particular towards image-guided invention and workflow automation. Existing deep network models require a large amount of labeled data. However, obtaining accurate pixel-wise labeling in X-ray images relies heavily on skilled clinicians due to the large overlaps of anatomy and the complex texture patterns. On the other hand, organs in 3D CT scans preserve clearer structures as well as sharper boundaries and thus can be easily delineated. In this paper, we propose a novel model framework for learning automatic X-ray image parsing from labeled CT scans. Specifically, a Dense Image-to-Image network (DI2I) for multi-organ segmentation is first trained on X-ray like Digitally Reconstructed Radiographs (DRRs) rendered from 3D CT volumes. Then we introduce a Task Driven Generative Adversarial Network (TD-GAN) architecture to achieve simultaneous style transfer and parsing for unseen real X-ray images. TD-GAN consists of a modified cycle-GAN substructure for pixel-to-pixel translation between DRRs and X-ray images and an added module leveraging the pre-trained DI2I to enforce segmentation consistency. The TD-GAN framework is general and can be easily adapted to other learning tasks. In the numerical experiments, we validate the proposed model on 815 DRRs and 153 topograms. While the vanilla DI2I without any adaptation fails completely on segmenting the topograms, the proposed model does not require any topogram labels and is able to provide a promising average dice of 85% which achieves the same level accuracy of supervised training (88%).

• The paper proposes a novel unsupervised domain adaptation framework using a task driven GAN to leverage labeled DRRs for effective X-ray segmentation.
• It integrates a Dense Image-to-Image network with cycle-GAN based style transfer to maintain segmentation accuracy without annotated X-ray images.
• Experimental results demonstrate an 85% dice score, closely matching supervised benchmarks and underscoring its potential for clinical application.

Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation

Introduction

Medical image segmentation, particularly the parsing of anatomical structures in X-ray images, plays an essential role in clinical applications such as surgical planning and treatment evaluation. However, annotating X-ray images is resource-intensive and requires substantial clinical expertise due to complex anatomical overlaps and indistinct boundaries inherent to such images. The paper proposes an innovative framework utilizing a Task Driven Generative Adversarial Network (TD-GAN) to address these challenges by leveraging labeled CT-derived Digitally Reconstructed Radiographs (DRRs) for the task of X-ray image segmentation.

Methodology

Overview of the Proposed Framework

The core proposal is a novel unsupervised domain adaptation framework for X-ray image segmentation using DRRs. The framework consists of a Dense Image-to-Image (DI2I) network trained on labeled DRRs, followed by a TD-GAN that performs both style transfer and segmentation on real X-ray images via a cycle-GAN substructure. Critically, the approach eliminates the need for labeled X-ray images while achieving robust segmentation through domain adaptation.

Figure 1: Overview of the proposed task driven generative model framework.

Dense Image-to-Image Network (DI2I)

The DI2I network is deployed using an encoder-decoder structure enriched with dense blocks, enhancing gradient flow and feature extraction capabilities. It is trained to perform multi-organ segmentation on DRRs, including structures like the lung, heart, liver, and bone, by utilizing a customized loss function tailored for binary classifications.

Task Driven Generative Adversarial Network (TD-GAN)

TD-GAN extends the cycle-GAN architecture to incorporate task-specific generative modeling. It employs two generators for style translation between DRRs and X-ray images and two discriminators for adversarial training. Uniquely, TD-GAN integrates pre-trained DI2I for segmentation consistency, leveraging both conditional adversarial training and cycle-consistent losses to ensure effective segmentation while maintaining domain-specific features.

Figure 2: Proposed TD-GAN architecture. Real X-ray images and DRRs are passed through 4 different paths for simultaneous synthesis and segmentation.

Experimental Results

The proposed methodology was validated on a substantial dataset of 815 labeled DRRs and 153 X-ray topograms. When directly tested, the pre-trained DI2I failed on X-ray images. However, using TD-GAN, notable performance gains were observed with the segmentation results closely matching supervised methods achieving an average dice score of 85%, comparable to the supervised benchmark of 88%.

Figure 3: Visualization of segmentation results on topograms (bottom) against direct application of DI2I (top). The red curves stand for the boundary of the ground truth. The colored fill-in parts are the predictions by TD-GAN and DI2I.

Discussion and Conclusion

The TD-GAN framework represents a significant advancement in unsupervised domain adaptation for medical image analysis, specifically X-ray segmentation without requiring ground truth labels. It leverages the structural regularity from DRRs and cycle-consistent adversarial training to substantially improve segmentation accuracy on challenging X-ray modalities. Future work can explore expanding this model to other medical imaging tasks such as lesion detection and classification, showcasing its adaptability and potential in diverse diagnostic applications.

In conclusion, the proposed methodology demonstrates a compelling application of generative modeling and domain adaptation, offering a valuable tool for medical imaging research with profound implications for clinical practice.