Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

I-AI: A Controllable & Interpretable AI System for Decoding Radiologists' Intense Focus for Accurate CXR Diagnoses (2309.13550v4)

Published 24 Sep 2023 in cs.CV and cs.AI

Abstract: In the field of chest X-ray (CXR) diagnosis, existing works often focus solely on determining where a radiologist looks, typically through tasks such as detection, segmentation, or classification. However, these approaches are often designed as black-box models, lacking interpretability. In this paper, we introduce Interpretable Artificial Intelligence (I-AI) a novel and unified controllable interpretable pipeline for decoding the intense focus of radiologists in CXR diagnosis. Our I-AI addresses three key questions: where a radiologist looks, how long they focus on specific areas, and what findings they diagnose. By capturing the intensity of the radiologist's gaze, we provide a unified solution that offers insights into the cognitive process underlying radiological interpretation. Unlike current methods that rely on black-box machine learning models, which can be prone to extracting erroneous information from the entire input image during the diagnosis process, we tackle this issue by effectively masking out irrelevant information. Our proposed I-AI leverages a vision-LLM, allowing for precise control over the interpretation process while ensuring the exclusion of irrelevant features. To train our I-AI model, we utilize an eye gaze dataset to extract anatomical gaze information and generate ground truth heatmaps. Through extensive experimentation, we demonstrate the efficacy of our method. We showcase that the attention heatmaps, designed to mimic radiologists' focus, encode sufficient and relevant information, enabling accurate classification tasks using only a portion of CXR. The code, checkpoints, and data are at https://github.com/UARK-AICV/IAI

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Big self-supervised models advance medical image classification. In CVPR, pages 3478–3488, 2021.
  2. Sam3d: Segment anything model in volumetric medical images. arXiv preprint arXiv:2309.03493, 2023.
  3. Bias in radiology: the how and why of misses and misinterpretations. Radiographics, 38(1):236–247, 2018.
  4. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In WACV, pages 839–847. IEEE, 2018.
  5. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
  6. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306, 2021.
  7. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290–1299, 2022.
  8. Kunio Doi. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Computerized medical imaging and graphics, 31(4-5):198–211, 2007.
  9. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  10. Medical deep learning—a systematic meta-review. Computer methods and programs in biomedicine, 221:106874, 2022.
  11. Self-supervised deep convolutional neural network for chest x-ray classification. IEEE Access, 9:151972–151982, 2021.
  12. Knowledge-augmented contrastive learning for abnormality classification and localization in chest x-rays with radiomics using a feedback loop. In WACV, pages 2465–2474, 2022.
  13. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  14. Point-unet: A context-aware point-based neural network for volumetric segmentation. In MICCAI, pages 644–655. Springer, 2021.
  15. Densely connected convolutional networks. In CVPR, pages 4700–4708, 2017.
  16. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
  17. Creation and validation of a chest x-ray dataset with eye-tracking and report dictation for ai development. Scientific data, 2021.
  18. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, pages 2668–2677. PMLR, 2018.
  19. Reflacx: Reports and eye-tracking data for localization of abnormalities in chest x-rays, 2021.
  20. Narrow band active contour attention model for medical segmentation. Diagnostics, 11(8):1393, 2021.
  21. Relevance-cam: Your model already knows where to look. In CVPR, pages 14944–14953, 2021.
  22. Thoracic disease identification and localization with limited supervision. In CVPR, pages 8290–8299, 2018.
  23. Self-supervised mean teacher for semi-supervised chest x-ray classification. In Machine Learning in Medical Imaging: 12th International Workshop, MLMI 2021, pages 426–436. Springer, 2021.
  24. Align, attend and locate: Chest x-ray diagnosis via contrast induced attention network with limited supervision. In CVPR, pages 10632–10641, 2019.
  25. Semi-supervised medical image classification with relation-driven self-ensembling model. IEEE transactions on medical imaging, 39(11):3429–3440, 2020.
  26. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  27. A scoping review of transfer learning research on medical image analysis using imagenet. Computers in biology and medicine, 128:104115, 2021.
  28. Pip-net: Patch-based intuitive prototypes for interpretable image classification. In CVPR, pages 2744–2753, 2023.
  29. Neural prototype trees for interpretable fine-grained image recognition. In CVPR, pages 14933–14943, 2021.
  30. 3d-ucaps: 3d capsules unet for volumetric image segmentation. In MICCAI, pages 548–558. Springer, 2021.
  31. Learning hierarchical attention for weakly-supervised chest x-ray abnormality localization and diagnosis. IEEE transactions on medical imaging, 40(10):2698–2710, 2020.
  32. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017.
  33. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241. Springer, 2015.
  34. Localization with limited annotation for chest x-rays. In Machine Learning for Health Workshop, pages 52–65. PMLR, 2020.
  35. Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215, 2019.
  36. Interpretable image classification with differentiable prototypes assignment. In ECCV, pages 351–368. Springer, 2022.
  37. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD, pages 1420–1430, 2021.
  38. Integrated grad-cam: Sensitivity-aware visual explanation of deep convolutional networks via integrated gradient-based scoring. In ICASSP, pages 1775–1779. IEEE, 2021.
  39. Grad-cam: Visual explanations from deep networks via gradient-based localization. In CVPR, pages 618–626, 2017.
  40. Swinchex: Multi-label classification on chest x-ray images with transformers. arXiv preprint arXiv:2206.04246, 2022.
  41. 3dconvcaps: 3dunet with convolutional capsule encoder for medical image segmentation. In ICPR, pages 4392–4398. IEEE, 2022.
  42. Masayuki Tsuneki. Deep learning models in medical image analysis. Journal of Oral Biosciences, 64(3):312–320, 2022.
  43. This looks like it rather than that: Protoknn for similarity-based classifiers. In The Eleventh International Conference on Learning Representations, 2022.
  44. Probabilistic integration of object level annotations in chest x-ray classification. In CVPR, pages 3630–3640, 2023.
  45. Neural architecture search for medical image applications. In Meta-Learning with Medical Imaging and Health Informatics Applications, pages 369–384. Elsevier, 2023.
  46. Score-cam: Score-weighted visual explanations for convolutional neural networks. In CVPRW, pages 24–25, 2020.
  47. Interpretable image recognition by constructing transparent embedding space. In CVPR, pages 895–904, 2021.
  48. Side adapter network for open-vocabulary semantic segmentation. In CVPR, pages 2945–2954, 2023.
  49. Weakly supervised deep learning for thoracic disease classification and localization on chest x-rays. In ACM international conference on bioinformatics, computational biology, and health informatics, pages 103–110, 2018.
  50. Weakly supervised medical diagnosis and localization from multiple resolutions. arXiv preprint arXiv:1803.07703, 2018.
  51. Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304.13785, 2023.
  52. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915, 2023.
  53. Learning deep features for discriminative localization. In CVPR, pages 2921–2929, 2016.
  54. Deep reinforcement learning in medical imaging: A literature review. Medical image analysis, 73:102193, 2021.
  55. Multi-task unet: Jointly boosting saliency prediction and disease classification on chest x-ray images. arXiv preprint arXiv:2202.07118, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Trong Thang Pham (4 papers)
  2. Jacob Brecheisen (1 paper)
  3. Anh Nguyen (157 papers)
  4. Hien Nguyen (33 papers)
  5. Ngan Le (84 papers)
Citations (9)
View on Semantic Scholar