HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs (2412.18925v1)

Abstract: The breakthrough of OpenAI o1 highlights the potential of enhancing reasoning to improve LLM. Yet, most research in reasoning has focused on mathematical tasks, leaving domains like medicine underexplored. The medical domain, though distinct from mathematics, also demands robust reasoning to provide reliable answers, given the high standards of healthcare. However, verifying medical reasoning is challenging, unlike those in mathematics. To address this, we propose verifiable medical problems with a medical verifier to check the correctness of model outputs. This verifiable nature enables advancements in medical reasoning through a two-stage approach: (1) using the verifier to guide the search for a complex reasoning trajectory for fine-tuning LLMs, (2) applying reinforcement learning (RL) with verifier-based rewards to enhance complex reasoning further. Finally, we introduce HuatuoGPT-o1, a medical LLM capable of complex reasoning, which outperforms general and medical-specific baselines using only 40K verifiable problems. Experiments show complex reasoning improves medical problem-solving and benefits more from RL. We hope our approach inspires advancements in reasoning across medical and other specialized domains.

Understanding HuatuoGPT-o1: Enhancing Medical Reasoning in LLMs

The paper "HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs" presents a novel approach to augment the reasoning capabilities of LLMs within the medical domain. While the progress in LLM development, notably by OpenAI's o1, has demonstrated significant advancements in tasks that require mathematical reasoning, the extension of such methodologies to specialized fields like medicine remains unexplored. This work addresses this gap by introducing a model capable of handling complex reasoning tasks in the medical field, named HuatuoGPT-o1.

Methodological Advancements

The crux of the paper is the design of a two-stage training process aimed at enhancing medical reasoning in LLMs. The two stages are:

1. Learning Complex Reasoning:
   • Data Construction: A key component of this stage is the construction of a specialized dataset consisting of 40,000 verifiable medical problems. These problems are derived from closed-set medical examination questions, transformed into open-ended queries with objective ground truths.
   • Fine-tuning with Constructed Data: The model is initially trained using strategy-based searches that generate complex reasoning trajectories, refined through a search strategy guided by verifier feedback. This process enables the model to critique and iterate on its reasoning paths, learning from the step-by-step process akin to the Chain-of-Thought (CoT) method.
2. Reinforcement Learning (RL) Enhancements:
   • Verifier-Based Rewards: Once basic reasoning capabilities are established, the model undergoes further refinement using Proximal Policy Optimization (PPO), with feedback provided by a medical verifier. The verifier checks the outputs against the correct answers and delivers binary feedback (True or False), guiding the model to explore different reasoning pathways.

Experimental Findings

The experiments demonstrate that HuatuoGPT-o1 significantly outperforms both generalist LLMs and other medical-specific models across a variety of benchmarks, such as MedQA, MedMCQA, and PubMedQA. Notably, the model achieves an 8.5-point improvement in medical benchmarks using only 40,000 data points, validating the efficacy of the two-stage training approach. The approach is noted to be particularly effective in tasks that require complex reasoning, as it simulates the processes of medical diagnosis where iterative reflection and correction are crucial.

Theoretical and Practical Implications

The findings have profound implications for the implementation of LLMs in domains requiring specialized knowledge. By developing a framework that effectively verifies and refines reasoning processes, the model demonstrates potential transferability to other domains beyond medicine, such as law and finance. This is particularly promising for applications in fields where high-stakes decision-making is frequent.

Future Directions

The paper suggests several avenues for future research. These include refining the verifier's reliability, expanding the complexity and scope of reasoning problems tackled, and exploring the scaling potential of similar methodologies to enhance cross-domain applicability. Furthermore, this approach paves the way for LLMs to autonomously enhance their reasoning capabilities based on feedback, mimicking a learning process similar to human reasoning.

In conclusion, HuatuoGPT-o1 represents an important step forward in adapting LLMs for specialized applications, making complex reasoning both feasible and verifiable within machine learning frameworks. As AI continues to evolve, such multi-stage training methodologies could become essential in bridging the gap between generalized understanding and domain-specific expertise.