An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation

Abstract: Radiological services are experiencing unprecedented demand, leading to increased interest in automating radiology report generation. Existing Vision-LLMs (VLMs) suffer from hallucinations, lack interpretability, and require expensive fine-tuning. We introduce SAE-Rad, which uses sparse autoencoders (SAEs) to decompose latent representations from a pre-trained vision transformer into human-interpretable features. Our hybrid architecture combines state-of-the-art SAE advancements, achieving accurate latent reconstructions while maintaining sparsity. Using an off-the-shelf LLM, we distil ground-truth reports into radiological descriptions for each SAE feature, which we then compile into a full report for each image, eliminating the need for fine-tuning large models for this task. To the best of our knowledge, SAE-Rad represents the first instance of using mechanistic interpretability techniques explicitly for a downstream multi-modal reasoning task. On the MIMIC-CXR dataset, SAE-Rad achieves competitive radiology-specific metrics compared to state-of-the-art models while using significantly fewer computational resources for training. Qualitative analysis reveals that SAE-Rad learns meaningful visual concepts and generates reports aligning closely with expert interpretations. Our results suggest that SAEs can enhance multimodal reasoning in healthcare, providing a more interpretable alternative to existing VLMs.

• The paper introduces SAE-Rad, a framework that leverages sparse autoencoders to generate interpretable radiology reports from X-ray images.
• It details a hybrid autoencoder architecture that achieves high reconstruction fidelity and sparsity, enabling mechanistic interpretability without extensive fine-tuning.
• Experimental results on the MIMIC-CXR dataset highlight SAE-Rad’s competitive CheXpert F1 scores and efficient multimodal reasoning in clinical applications.

Interpretable Radiology Report Generation with SAE-Rad

The paper "An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation" presents SAE-Rad, a novel framework that leverages sparse autoencoders (SAEs) to automate radiology report generation. This approach seeks to address several challenges faced by existing Vision-LLMs (VLMs), such as hallucinations, lack of interpretability, and the necessity of costly fine-tuning.

Methodology Overview

SAE-Rad introduces a unique architecture that employs sparse autoencoders to decompose latent representations from a pre-trained vision transformer into human-interpretable features. These interpretable features are used to generate radiology reports by:

1. Sparse Autoencoder Architecture: SAE-Rad employs a hybrid autoencoder architecture that combines elements from gated SAEs with unconstrained decoder norms. This design achieves both high reconstruction fidelity and sparsity in the learned representations.
2. Interpretability Through Text Descriptions: Using a pre-trained LLM, the authors distill ground-truth reports into feature descriptions. These descriptions are compiled into a full report by identifying the most relevant features for a given radiographic image.
3. Multimodal Reasoning: The framework represents the first known application of mechanistic interpretability techniques for downstream multimodal reasoning tasks in radiology. The authors demonstrate this capability using the MIMIC-CXR dataset.

Experimental Results

SAE-Rad was evaluated on the MIMIC-CXR dataset, where it demonstrated competitive performance in radiology-specific metrics. Notably, it achieved strong results in the CheXpert F1 score, signaling its capability to capture clinically relevant content. Moreover, SAE-Rad was shown to be more computationally efficient compared to state-of-the-art systems like MAIRA-2, using significantly fewer computational resources.

Qualitative Analysis

The authors provide qualitative evidence that SAE-Rad captures meaningful visual features, such as instrumentation and pathological conditions, by showcasing monosemantic features learned by the SAE. The system was able to generate reports aligned closely with expert interpretations, demonstrating not only accuracy but also enhanced interpretability.

Implications and Future Directions

The paper suggests that SAEs can be a valuable addition to multimodal reasoning in healthcare, providing a more interpretable alternative to traditional VLMs. By using mechanistic interpretability techniques, the framework enhances trust in automated systems, making it particularly suitable for sensitive fields like healthcare.

Looking ahead, further exploration into improving human-like language generation styles without sacrificing interpretability could be a key area for future development. Additionally, expanding SAE-Rad to encompass a wider range of imaging modalities and integrating more complex multimodal reasoning tasks could significantly impact the field.

Conclusion

By innovatively combining SAEs with mechanistic interpretability, this research contributes to developing more transparent and interpretable radiology report generation systems, offering insights that could extend beyond radiology into broader healthcare applications. The modularity of the approach also affords flexibility, allowing for enhancements and adaptations with future advancements in language and vision models.