AutoRG-Brain: Grounded Report Generation for Brain MRI (2407.16684v3)

Abstract: Radiologists are tasked with interpreting a large number of images in a daily base, with the responsibility of generating corresponding reports. This demanding workload elevates the risk of human error, potentially leading to treatment delays, increased healthcare costs, revenue loss, and operational inefficiencies. To address these challenges, we initiate a series of work on grounded Automatic Report Generation (AutoRG), starting from the brain MRI interpretation system, which supports the delineation of brain structures, the localization of anomalies, and the generation of well-organized findings. We make contributions from the following aspects, first, on dataset construction, we release a comprehensive dataset encompassing segmentation masks of anomaly regions and manually authored reports, termed as RadGenome-Brain MRI. This data resource is intended to catalyze ongoing research and development in the field of AI-assisted report generation systems. Second, on system design, we propose AutoRG-Brain, the first brain MRI report generation system with pixel-level grounded visual clues. Third, for evaluation, we conduct quantitative assessments and human evaluations of brain structure segmentation, anomaly localization, and report generation tasks to provide evidence of its reliability and accuracy. This system has been integrated into real clinical scenarios, where radiologists were instructed to write reports based on our generated findings and anomaly segmentation masks. The results demonstrate that our system enhances the report-writing skills of junior doctors, aligning their performance more closely with senior doctors, thereby boosting overall productivity.

• The paper presents an innovative two-step approach that first segments brain MRI regions and then uses visual prompts to generate detailed radiology reports.
• It leverages a modified nnU-Net model and the RadGenome-Brain MRI dataset to enhance accuracy in identifying brain structures and anomalies.
• Clinical evaluations show that AutoRG-Brain achieves performance comparable to experienced radiologists, reducing errors and improving workflow efficiency.

AutoRG-Brain: Grounded Report Generation for Brain MRI

Introduction

The paper discusses a sophisticated approach to generating radiology reports from brain MRI images called AutoRG-Brain. This system addresses challenges in radiological reporting by integrating automatic region of interest (ROI) generation with visual prompting to create detailed and accurate reports. By incorporating various components such as segmentation modules and report generation mechanisms, AutoRG-Brain aims to optimize the workflow for radiologists, potentially reducing errors in report generation and enhancing the quality of reports generated by less experienced clinicians.

System Design and Architecture

AutoRG-Brain's architecture consists of two primary components: an ROI generation module and a visual prompting guided report generation module. The system uses a segmentation module, built on a modified nnU-Net framework, to automatically generate segmentations for brain structures and anomalies across multimodal MRI inputs (Figure 1). These serve as grounded visual prompts for the subsequent report generation process.

Figure 1: The architecture of our proposed AutoRG-Brain. The segmentation module ($\Phi_\text{SEG}$) produces global brain structure and anomaly segmentations.

The prompting component enhances report specificity by directing the focus on auto-segmented regions or user-specified areas, achieving a granularity that enables contextually detailed reports. This two-pronged approach—segment first, prompt and narrate second—sets it apart from conventional systems that rely solely on global image features.

Dataset Construction

A novel dataset termed RadGenome-Brain MRI was developed specifically for this system, consisting of 3,408 region-report pairs encompassing five diseases and six MRI modalities (Figure 2). The dataset provides pixel-level segmentation and associated findings, aimed at bolstering the development and benchmarking of similar AI-assisted reporting tools.

Figure 2: Overview of our contributions including the RadGenome-Brain MRI dataset and its utility in refining report generation.

Evaluation Process

Quantitative assessments were conducted to evaluate the system's efficacy in brain structure segmentation and anomaly localization, demonstrating significant improvement over state-of-the-art (SOTA) approaches. AutoRG-Brain was integrated into real clinical settings, showing enhancements in report writing efficiency comparable to reports generated by seasoned radiologists (Figure 3). The system's outcomes were validated through metrics like Dice Similarity Coefficient (DSC), Precision, and Sensitivity, with AutoRG-Brain delivering competitive results.

Figure 3: The comparison results illustrating the accuracy and quality of reports generated by AutoRG-Brain compared to unaided and AI-assisted radiologists' reports.

Implementation Details

AutoRG-Brain operates through a pipeline initiated by a self-supervised training stage, utilizing synthetic data to enhance segmentation accuracy. It is followed by semi-supervised training incorporating publicly available datasets with authentic human-annotated anomalies. The segmentation network employs modality-specific encoders with a shared decoder, integrated within a multitask training regime to optimize its performance across varied MRI modalities.

Report Generation

Evaluation on report generation on datasets like SSPH and RadGenome-Brain MRI showcases AutoRG-Brain's proficiency in generating clinically useful reports, integrating both qualitative and quantitative analyses, including human evaluations (Figure 4).

Figure 4: Evaluation metrics across four dimensions assessing omission, false prediction, location errors, and lesion description correctness.

Conclusion

AutoRG-Brain exemplifies advancement in automated medical imaging analysis, specifically tailored towards brain MRI interpretations. By blending precise segmentation with robust visual prompt-based report generation, it significantly narrows the performance gap between junior and senior radiologists, thereby augmenting clinical workflow efficiency. Future endeavors could focus on scaling the system across different imaging modalities and enhancing its integration with electronic health record systems for broader clinical applicability. The release of RadGenome-Brain MRI further provides an open platform for fostering ongoing research in AI-driven medical imaging fields.