Evaluating Physician-AI Interaction for Cancer Management: Paving the Path towards Precision Oncology (2404.15187v1)

Abstract: We evaluated how clinicians approach clinical decision-making when given findings from both randomized controlled trials (RCTs) and ML models. To do so, we designed a clinical decision support system (CDSS) that displays survival curves and adverse event information from a synthetic RCT and ML model for 12 patients with multiple myeloma. We conducted an interventional study in a simulated setting to evaluate how clinicians synthesized the available data to make treatment decisions. Participants were invited to participate in a follow-up interview to discuss their choices in an open-ended format. When ML model results were concordant with RCT results, physicians had increased confidence in treatment choice compared to when they were given RCT results alone. When ML model results were discordant with RCT results, the majority of physicians followed the ML model recommendation in their treatment selection. Perceived reliability of the ML model was consistently higher after physicians were provided with data on how it was trained and validated. Follow-up interviews revealed four major themes: (1) variability in what variables participants used for decision-making, (2) perceived advantages to an ML model over RCT data, (3) uncertainty around decision-making when the ML model quality was poor, and (4) perception that this type of study is an important thought exercise for clinicians. Overall, ML-based CDSSs have the potential to change treatment decisions in cancer management. However, meticulous development and validation of these systems as well as clinician training are required before deployment.

• The paper demonstrates the integration of ML models with RCT data to enhance cancer treatment decision-making in multiple myeloma.
• It employs a multi-tiered clinical decision support system to measure how additional ML insights affect physician confidence and choices.
• Results indicate that when ML recommendations align or contrast with RCT data, clinicians adjust their trust, highlighting the need for rigorous model validation and targeted training.

Evaluating Physician-AI Interaction for Cancer Management in the Context of Precision Oncology

Introduction

The paper at hand explores the integration of ML models and randomized controlled trials (RCTs) in clinical decision-making for cancer management, specifically focusing on multiple myeloma (MM). The evolving landscape of cancer treatments, marked by rapid advancements in molecular sequencing and targeted therapies, underlines the potential for ML to augment decision-making, especially in conditions like MM where clinical trials cannot keep pace with the rate of therapeutic developments.

Study Design and Methods

The research team developed a clinical decision support system (CDSS) displaying survival curves and adverse event data from both a synthetic RCT and an ML model covering 12 patient scenarios. An interventional paper was conducted using this CDSS, where oncologists made treatment decisions at different tiers of information availability:

1. Tier I: Data from the RCT alone.
2. Tier II: Addition of ML model outcomes.
3. Tier III: Additional details on ML model training and validation.
4. Tier IV: Information on replication experiments using observational data to mirror RCT results.

Throughout the paper, participants were asked to select a treatment option and rate their confidence and the perceived reliability of the ML model. This multi-tiered approach aimed to discern how additional ML insights impact clinical choices.

Key Findings

Quantitative Outcomes

• The integration of ML model results generally increased physicians' confidence when these were concordant with RCT data. However, when ML results contradicted RCT outcomes, there was a notable shift, often favoring the ML-driven recommendation.
• The perceived reliability of the ML model significantly improved when the training and validation process of the model was disclosed to the participants.
• Remarkably, in instances where the ML model data contradicted RCT findings (and where the ML model was appropriately trained), physicians still showed a propensity to trust the ML recommendations, suggesting an intrinsic value placed on personalized data insights.

Qualitative Insights

• Four prominent themes emerged from follow-up interviews: variability in decision-making criteria among physicians, perceived superiority of ML models in specific settings, challenges due to lack of training on similar patients, and the acknowledgment of such studies as valuable for clinical practice.
• Overall, findings indicate a careful yet optimistic incorporation of ML insights into clinical decision-making, underscoring the necessity for thorough validation and clinician education on these new tools.

Implications and Future Directions

The interaction between AI and physicians in precision oncology promises enhancements in treatment personalization. However, this paper highlights the importance of comprehensive clinician training on AI tools and the rigorous development and validation of AI models before practical implementation. Future studies should investigate larger and more diverse physician groups to validate these findings and explore interfaces that effectively present ML data to clinicians without overwhelming them. Additionally, exploring strategies to improve model explainability and clinicians' ability to critically evaluate AI tools in clinical settings could further integrate AI into routine medical practice effectively.

Overall, this paper reflects a methodical evaluation of AI's emerging role in clinical cancer care, suggesting that with careful integration, AI could significantly impact decision-making in complex cases where traditional clinical trials lag behind rapid therapeutic advancements.