Can Artificial Intelligence Generate Quality Research Topics Reflecting Patient Concerns? (2411.14456v1)

Abstract: Patient-centered research is increasingly important in narrowing the gap between research and patient care, yet incorporating patient perspectives into health research has been inconsistent. We propose an automated framework leveraging innovative NLP and AI with patient portal messages to generate research ideas that prioritize important patient issues. We further quantified the quality of AI-generated research topics. To define patient clinical concerns, we analyzed 614,464 patient messages from 25,549 individuals with breast or skin cancer obtained from a large academic hospital (2013 to 2024), constructing a 2-staged unsupervised NLP topic model. Then, we generated research topics to resolve the defined issues using a widely used AI (ChatGPT-4o, OpenAI Inc, April 2024 version) with prompt-engineering strategies. We guided AI to perform multi-level tasks: 1) knowledge interpretation and summarization (e.g., interpreting and summarizing the NLP-defined topics), 2) knowledge generation (e.g., generating research ideas corresponding to patients issues), 3) self-reflection and correction (e.g., ensuring and revising the research ideas after searching for scientific articles), and 4) self-reassurance (e.g., confirming and finalizing the research ideas). Six highly experienced breast oncologists and dermatologists assessed the significance and novelty of AI-generated research topics using a 5-point Likert scale (1-exceptional, 5-poor). One-third of the AI-suggested research topics were highly significant and novel when both scores were lower than the average. Two-thirds of the AI-suggested topics were novel in both cancers. Our findings demonstrate that AI-generated research topics reflecting patient perspectives via a large volume of patient messages can meaningfully guide future directions in patient-centered health research.

• The paper demonstrates that AI-driven NLP and topic modeling can generate novel, patient-centered research questions in oncology.
• It employs a two-stage method using BERT and BIRCH on over 600K patient messages to extract key clinical concerns.
• Expert evaluations rated one-third of the generated topics as highly significant, underscoring their potential impact on oncology research.

Analyzing AI's Potential to Generate Patient-Centered Research Topics in Oncology

The paper presents a paper exploring the viability of using AI combined with NLP to develop research topics centered on patients' perspectives. It focuses on oncology, specifically involving breast and skin cancer, and seeks to bridge a crucial gap between existing health research and patient-reported concerns. By employing innovative NLP techniques on a large dataset of patient portal messages, the paper aims to identify prevalent issues in patients’ communications with clinicians and to propose research topics addressing those issues using AI models.

Methods and Framework

The researchers utilized a robust dataset consisting of over 600,000 de-identified messages from patients diagnosed with breast or skin cancer. A two-stage topic modeling technique, applying Bidirectional Encoder Representations from Transformers (BERT) and the Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) algorithm, was utilized to extract significant clinical concerns from these messages. Subsequent AI-generated research topics were created through the application of a version of ChatGPT, aided by meticulous prompt-engineering approaches. The tasks involved included interpretation of NLP-derived topics, knowledge generation, self-reflection for novelty verification via database searches, and final affirmation of research ideas.

Evaluation Metrics

A panel of six experienced oncologists and dermatologists evaluated the AI-suggested research topics to determine their significance and novelty on a 5-point Likert scale. Interestingly, the paper found that one-third of AI-generated topics were evaluated as highly significant and novel, particularly for skin cancer topics, which scored higher in significance and novelty compared to those for breast cancer.

Results and Implications

The research found that approximately two-thirds of the AI-generated research topics were novel. For tasks related to certain aspects, such as developing patient-centered digital tools or specialized care protocols, the AI-driven insights were deemed particularly promising. The experts noted a significant occurrence of new, unexplored issues that AI was able to spotlight, leveraging patient feedback which traditional research methodologies might overlook due to resource constraints.

This work uncovers the scalable potential of integrating AI with patient data to improve the feedback loop between patients' voiced concerns and research priorities. While it acknowledges the current challenges and limitations in employing AI models, including the need to exclude non-scientific patient service issues and potential bias due to single-institution input, it suggests a new avenue for creating patient-informed, objectively significant research agendas.

Future Directions

Looking ahead, the paper recommends widening the scope beyond the specific oncological contexts examined, potentially applying this framework across various medical specialties to harness a broader spectrum of patient experiences. Furthermore, multi-institutional collaboration is crucial to reduce the likelihood of scoring bias and to broaden the consensus on what constitutes important research areas. Importantly, soliciting direct patient input on AI-generated topics could enhance the alignment of research endeavors with community needs.

In conclusion, the paper signifies a pertinent leap in utilizing AI not merely as an analytical tool but as a generative partner capable of shaping future patient-centered research landscapes. As AI technology evolves, its use could be transformative in tailoring healthcare research closely aligned with real-world patient concerns, thus potentially improving healthcare outcomes and patient satisfaction. Future developments should focus on refining AI methods for greater contextual understanding and exploring collaborative, inclusive strategies that prioritize patient involvement at every stage of the research process.