Exploring Radiologists' Expectations of Explainable Machine Learning Models in Medical Image Analysis

Abstract: In spite of the strong performance of ML models in radiology, they have not been widely accepted by radiologists, limiting clinical integration. A key reason is the lack of explainability, which ensures that model predictions are understandable and verifiable by clinicians. Several methods and tools have been proposed to improve explainability, but most reflect developers' perspectives and lack systematic clinical validation. In this work, we gathered insights from radiologists with varying experience and specialties into explainable ML requirements through a structured questionnaire. They also highlighted key clinical tasks where ML could be most beneficial and how it might be deployed. Based on their input, we propose guidelines for designing and developing explainable ML models in radiology. These guidelines can help researchers develop clinically useful models, facilitating integration into radiology practice as a supportive tool.

• The paper provides a radiologist-centric evaluation of explainable ML, showing that trust is built on clinically meaningful and transparent explanations.
• It uses a structured questionnaire among 46 radiologists, highlighting a preference for ML support in routine workflow tasks over rare case diagnosis.
• The study outlines design guidelines for developing multi-modal, self-explainable models that integrate visual heatmaps and SRTCS-based explanations into clinical practice.

Radiologist-Centric Assessment of Explainable Machine Learning for Medical Imaging

Introduction

This paper presents a structured investigation of radiologists’ expectations and requirements regarding explainable machine learning (XAI) in medical image analysis, with a focus on supporting alignment between model design and clinical usability (2604.11700). The study employs a comprehensive questionnaire distributed among 46 radiologists, encompassing both experienced specialists and trainees, from multiple hospitals. The analysis uniquely emphasizes multidisciplinary imaging practice, capturing domain-specific perspectives across diverse clinical/radiological foci and modalities.

Participant Demographics and Clinical Context

The participant pool consists of a broad spectrum of experience levels and clinical interests, enabling a nuanced understanding of XAI needs. Notably, the dominant clinical focuses are Abdominal and Neuroimaging divisions, with MRI and CT being the most common modalities in main practice areas.

Figure 2: Distribution of main clinical/radiological focus among survey participants, highlighting participation across subspecialties and modalities.

This diversity strengthens the representativeness of the findings. Most respondents report only basic familiarity with machine learning, emphasizing the need for intuitive, clinician-aligned explanatory methods. The survey further demonstrates that explainability is prioritized not by technical background, but rather by the requirements imposed by clinical decision-making and workflow integration.

Identified Use Cases for ML in Radiology

Radiologists predominantly envision ML as a supportive tool rather than a primary diagnostic agent. Key anticipated roles include workflow management, triage of urgent/emergent findings, and automation of routine, high-throughput diagnostic tasks. There is a lower preference for ML assistance in rare/difficult case diagnosis—contrary to much of the prior research emphasis. Open-ended responses reinforce the demand for efficiency (e.g., expediting normal case reporting, segmentation, and report preparation), corroborating that clinical utility is closely tied to routine rather than edge-case support.

Explainability Requirements and Model Acceptance

A critical insight is that virtually all respondents require ML explainability mainly to foster trust and enable validation of model decisions. The most desired model behaviors for explainability include:

• Focusing on radiomics or other interpretable, clinically relevant quantitative features.
• Providing visual heatmaps (attention maps) indicating regions of importance in images.
• Generating explanations grounded in Standardized Radiological Terminology and Classification Systems (SRTCS), especially for senior radiologists.
• Supplying textual rationale to accompany predictions.

Importantly, there is heterogeneity between junior (trainees) and senior radiologists: senior radiologists exhibit greater expectation for SRTCS mapping and broader requirements for model transparency. Counterfactual explanations are recognized but considered less essential.

Explainability is regarded as inseparable from trust—42 out of 46 respondents indicate explainability as mandatory for establishing trust in ML outputs. In contradiction scenarios, the majority do not outright reject the model, but seek further evidence, including reviewing model explanations or consulting colleagues/literature. This approach underscores a measured but open attitude towards algorithmic assistance, conditional on transparency.

Model Development and Evaluation Considerations

Radiologists express several major concerns regarding clinical deployment of ML, particularly data/training bias, insufficient accuracy, and lack of explainability. Senior radiologists are especially focused on potential bias, but only half are convinced that explainability alone is sufficient for bias detection. The evaluation of ML models for clinical integration is largely multifactorial: task performance, confidence estimation, and quality of explanations are all deemed critical.

A notable fraction of respondents are willing to accept ML models with performance comparable to radiologists in the workflow, even in the absence of full explainability—a pragmatic stance but one that comes with qualification (i.e., other requirements must also be met). However, nearly two-thirds stipulate that ML tools should be embedded into familiar radiology software (such as PACS) for practical use, even though this is not associated with explainability per se.

Design Guidelines and Theoretical Implications

The study yields empirically grounded design recommendations across the ML pipeline:

• Problem Framing and Data Preprocessing: Collaboration with radiologists is necessary from project inception. Data heterogeneity across patient populations and imaging modalities must be addressed. Preprocessing transparency is crucial, as is clear motivation for case/method selection.
• Model Training: Training should integrate domain knowledge (e.g., radiology reports, SRTCS), and facilitate interpretability through design. Multimodal models capable of both visual (attention maps, segmentation) and textual explanatory outputs are strongly preferred. Post-hoc explanations (e.g., Grad-CAM) are viewed as less reliable relative to inherently self-explainable models (e.g., attention mechanisms, prototype-based methods), but even these demand further investigation regarding their true explanatory power.
• Explainability Evaluation: Explanations must be aligned with clinical evidence, exhibit high fidelity (perturbation analysis), robustness to real-world variations, and demonstrable positive impact on radiologist performance. Both alignment with expert-provided annotations and empirical effect-on-decision-making studies are recommended.
• Model/Explanation Refinement: Iterative refinement with radiologist-in-the-loop interaction is essential. Feedback loops enable continuous improvement of both model decisions and explanatory behaviors.

Contradictory and Strong Claims

The study finds that, contrary to frequent assumptions in the research literature, radiologists are less interested in ML for “difficult” or rare cases, and more interested in augmenting efficiency for routine, high-volume tasks. Another strongly articulated claim is that post-hoc explanations, while ubiquitous in the technical literature, are not sufficient for clinical trust or actionable insight—explanations must be both semantically meaningful and closely connected to underlying model mechanisms.

The emphasis on SRTCS as a critical explanatory modality among senior radiologists is also in tension with common academic practice, which rarely prioritizes radiologist-standard lexicons in explanation pipelines.

Implications and Future Directions

Practically, the findings suggest ML research for radiology should prioritize the design of multi-modal, self-explainable, and workflow-integrated models, coupled with evaluation protocols that reflect real clinical utility, not just technical interpretability metrics. Theoretically, these results highlight the gap between academic perceptions of explainability and the nuanced, context-dependent definition required by front-line clinical users. Further empirical work is needed to establish rigorous protocols for bias detection via explainability, and to validate the correlation between explanation fidelity and the reduction of model-induced errors.

Conclusion

This work provides a radiologist-centered assessment of XAI needs for medical imaging ML, revealing both clear requirements and significant design gaps in prevailing research practice. The emphasis on SRTCS, workflow efficiency, and trust-dependent explainability, along with skepticism regarding the sufficiency of post-hoc explanations, defines a new benchmark for clinically relevant ML/XAI research in radiology. The proposed design guidelines offer a pathway for alignment between ML technology development and real-world radiology practice, but also delineate multiple avenues for ongoing theoretical and empirical inquiry.