TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays

Abstract: Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reports can be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI).

• The paper introduces TieNet, a joint CNN-RNN model with multi-level attention to enhance chest X-ray classification and reporting.
• The approach improved AUC by about 6% over state-of-the-art methods, demonstrating significant gains in diagnostic accuracy.
• The system effectively integrates visual and textual data to generate coherent reports that assist radiologists in clinical decision-making.

Evaluation of TieNet for Chest X-ray Disease Classification and Reporting

This paper presents a new approach, TieNet, which integrates text and image data for the classification and reporting of thorax diseases in chest X-rays. The system addresses significant challenges in medical imaging: the limited availability of large-scale annotated datasets and the distinct model reasoning required to align with the expertise of trained radiologists.

The proposed framework introduces a Text-Image Embedding Network that combines Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) to process and integrate both visual and textual data. By utilizing multi-level attention mechanisms, TieNet enhances image features and text representations from clinical reports, yielding salient embeddings that bolster the classification process.

Architecture and Methodology

Key components of TieNet include:

• End-to-End CNN-RNN Architecture: This model captures complex interactions between image regions and textual descriptions. The CNN extracts spatial features from the X-ray images, while the RNN processes sequences derived from radiological reports.
• Multi-Level Attention Models: These components weigh both salient image regions and text portions, effectively highlighting crucial diagnostic information.
• Joint Learning Framework: The network is trained to perform two interconnected tasks: classifying diseases in chest X-rays and generating coherent textual reports.

Experimental Results

The efficacy of TieNet is demonstrated through experiments conducted on the ChestX-ray14 dataset and validated across additional datasets, including a hand-labeled subset and the OpenI repository. Notably, the framework improved the area under the receiver operating characteristic curve (AUC) by an average of 6% over state-of-the-art methods on unseen data.

The study reports high accuracy in disease classification, reflecting an AUC greater than 0.9 in several categories. In addition, the TieNet-generated reports achieved higher BLEU scores compared to traditional image captioning approaches.

Implications and Future Directions

The integration of text embeddings, alongside image features, underscores the paper's contribution to bridging text-mining and image-processing methodologies within medical CAD systems. Providing enhanced diagnostic and reporting capabilities, TieNet represents a step toward more efficient automated systems that can assist radiologists by offering preliminary analyses.

Looking forward, expanding the system's capabilities to encompass a broader range of diagnostic attributes and refining its report generation utility hold promise. Further, adapting the network for other diagnostic imaging tasks could open avenues for extensive applications in automated medical diagnostics, propelling advancements in AI-assisted healthcare solutions.

In conclusion, while the current implementation showcases significant improvements in chest X-ray classification, continued exploration of multi-modal embedding networks augments the horizon for AI's transformative potential in medical imaging.