BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT

Abstract: LLMs have performed well in providing general and extensive health suggestions in single-turn conversations, exemplified by systems such as ChatGPT, ChatGLM, ChatDoctor, DoctorGLM, and etc. However, the limited information provided by users during single turn results in inadequate personalization and targeting of the generated suggestions, which requires users to independently select the useful part. It is mainly caused by the missing ability to engage in multi-turn questioning. In real-world medical consultations, doctors usually employ a series of iterative inquiries to comprehend the patient's condition thoroughly, enabling them to provide effective and personalized suggestions subsequently, which can be defined as chain of questioning (CoQ) for LLMs. To improve the CoQ of LLMs, we propose BianQue, a ChatGLM-based LLM finetuned with the self-constructed health conversation dataset BianQueCorpus that is consist of multiple turns of questioning and health suggestions polished by ChatGPT. Experimental results demonstrate that the proposed BianQue can simultaneously balance the capabilities of both questioning and health suggestions, which will help promote the research and application of LLMs in the field of proactive health.

• The paper introduces BianQue, which leverages a self-constructed BianQueCorpus with balanced questioning (46.2%) and suggestions (53.8%) to enhance multi-turn health dialogues.
• BianQue is built on ChatGLM-6B and polished with ChatGPT-refined prompts, achieving superior performance on BLEU scores and a novel Proactive Questioning Ability metric.
• The model’s balanced approach fosters personalized AI-driven health consultations and paves the way for safer, more engaging multi-turn dialogue strategies in healthcare.

Overview of "BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT"

The paper "BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT" addresses a notable deficiency in current LLMs like ChatGPT: their inadequate ability to engage in multi-turn questioning during health consultations. While existing LLMs can provide general health suggestions in single-turn conversations, they fall short in personalizing advice due to insufficient multi-turn questioning capabilities. The authors propose a new model, BianQue, to enhance these capabilities, offering a balanced approach between questioning and suggesting in health dialogues.

Methodology

The BianQue model, derived from ChatGLM-6B, is fine-tuned using a self-constructed dataset, BianQueCorpus. This dataset comprises multi-turn health conversations enriched with balanced questioning (46.2%) and suggestions (53.8%). The construction of this corpus involved automatic data cleaning and the use of prompts to refine conversations with ChatGPT, ensuring a higher quality of dialogue relevant to real-world medical contexts.

Experimental Results

The performance evaluation of BianQue was conducted using multi-turn conversation datasets such as MedDialog-CN, IMCS-V2, CHIP-MDCFNPC, and MedDG. BianQue exhibited superior results compared to baseline models like ChatGLM-6B, DoctorGLM, and ChatGPT. Noteworthy metrics included BLEU scores and a newly introduced Proactive Questioning Ability (PQA) metric, indicating BianQue’s adeptness at questioning across multiple turns of dialogue.

Implications

Practically, BianQue's questioning and suggestion balance can enhance AI-driven health consultations, potentially leading to more effective and personalized patient interactions. Theoretically, this work underscores the importance of multi-turn dialogue capabilities in LLMs, suggesting a direction for future research in the proactive engagement of AI in complex conversational tasks.

The model’s development has implications for advancing LLM applications in healthcare, stressing the need for models to move beyond mere suggestion into active dialogue facilitation. This research could inspire further exploration into fine-tuning methodologies and dataset constructions that mirror real-world conversational dynamics.

Limitations and Future Directions

The authors acknowledge that the application of generative LLMs in health contexts requires careful consideration of the risks associated with privacy and diagnostic accuracy. They emphasize the necessity for mechanisms to inspect and correct AI-generated health advice, as well as incorporating privacy safeguards to prevent inappropriate question prompts.

Future research should focus on refining questioning strategies and integrating mechanisms for fine-tuned user privacy protection. Moreover, incorporating reinforcement learning with human feedback could further optimize the safety and efficiency of these models in real-world deployment.

Conclusion

The BianQue model presents a significant step forward in addressing the limitations of current LLMs concerning multi-turn interactions in health consultations. Although confined to academic research, BianQue opens avenues for developing more interactive and user-responsive AI models in healthcare, pointing toward a future where AI can contribute more meaningfully to personal health management.