Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models (2401.15269v3)

Published 27 Jan 2024 in cs.CL, cs.AI, and cs.IR
Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models

Abstract: Recent proprietary LLMs, such as GPT-4, have achieved a milestone in tackling diverse challenges in the biomedical domain, ranging from multiple-choice questions to long-form generations. To address challenges that still cannot be handled with the encoded knowledge of LLMs, various retrieval-augmented generation (RAG) methods have been developed by searching documents from the knowledge corpus and appending them unconditionally or selectively to the input of LLMs for generation. However, when applying existing methods to different domain-specific problems, poor generalization becomes apparent, leading to fetching incorrect documents or making inaccurate judgments. In this paper, we introduce Self-BioRAG, a framework reliable for biomedical text that specializes in generating explanations, retrieving domain-specific documents, and self-reflecting generated responses. We utilize 84k filtered biomedical instruction sets to train Self-BioRAG that can assess its generated explanations with customized reflective tokens. Our work proves that domain-specific components, such as a retriever, domain-related document corpus, and instruction sets are necessary for adhering to domain-related instructions. Using three major medical question-answering benchmark datasets, experimental results of Self-BioRAG demonstrate significant performance gains by achieving a 7.2% absolute improvement on average over the state-of-the-art open-foundation model with a parameter size of 7B or less. Overall, we analyze that Self-BioRAG finds the clues in the question, retrieves relevant documents if needed, and understands how to answer with information from retrieved documents and encoded knowledge as a medical expert does. We release our data and code for training our framework components and model weights (7B and 13B) to enhance capabilities in biomedical and clinical domains.

PDF HTML Abstract

Introduction

LLMs like GPT-4 have made significant contributions to fields such as biomedical text analysis. However, when tackling domain-specific problems, general retrieval-augmented generation (RAG) methods often exhibit poor generalization, resulting in incorrect document retrieval or judgment. In response, a new framework has been introduced, Self-BioRAG, aimed at improving medical reasoning through retrieval and self-reflection specifically tailored for the biomedical domain.

Background

The benchmark for Self-BioRAG's performance is established against well-known proprietary and open-source LLMs. While proprietary LLMs have advanced in instruction tuning, their limited access poses a challenge for researchers in the biomedical area. Open-source alternatives like the LLaMA family, consequently, have garnered interest, albeit with room for domain-specific improvements. Furthermore, Self-RAG, a model designed for cost-efficient retrieval with reflective tokens capable of self-assessment, was previously introduced but found inadequate for biomedical queries.

Self-BioRAG Framework

The construction of Self-BioRAG encompasses four integral components: instructional sets pertinent to biomedical and clinical texts, a specialized retriever for this domain, a self-reflection LLM which can critique its own outputs, and a domain-specific, instruction-tuned generator LLM. Instruction sets were crafted with existing biomedical tasks in mind, and a robust biomedical corpus was compiled, featuring medical textbooks, PubMed abstracts, clinical guidelines, and PMC full-text articles. The training processes for each LLM imitate that of Self-RAG, with an additional refining step using data from domain-specific components to improve model performance.

Experimental Results

Experimentation across three major medical question-answering benchmark datasets demonstrated Self-BioRAG's significant performance gains, outperforming the SOTA model average by a substantial 7.2% for languages models with parameters of 7 billion or less. The framework, tested in various configurations, indicates that factual content retrieval is most effective when done adaptively depending on the question. Furthermore, the retriever's bias towards Medical Textbooks suggests a strong relevance to USMLE-style questions, indicating a nuanced understanding of biomedical knowledge needs. The contribution of domain-specific components undeniably underpins the model's heightened performance, and the authors have made the code and data available for public use to further capabilities in this vertical.

Conclusion

Self-BioRAG represents a significant stride towards enhancing LLMs' effectiveness in the biomedical field. Its domain-specific components prove fundamental in enabling the model to autonomously appraise its generated responses and explanations, closely mimicking medical expert behavior. Limitations, such as performance drops in closed-domain datasets where excessive noise hampers the model, indicate potential areas for future refinement. By addressing these challenges, further advancements could lead to LLMs that not only generate but critique and improve upon their knowledge-based outputs in highly specialized fields such as medicine.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Asai, A. et al. (2023). Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.
  2. Bajaj, P. et al. (2016). Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
  3. Cao, M. et al. (2022). Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
  4. Chen, Z. et al. (2023). Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079.
  5. Christiano, P. F. et al. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems.
  6. Chung, H. W. et al. (2022). Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
  7. Dao, T. et al. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems.
  8. Fang, Y. et al. (2023). Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint arXiv:2306.08018.
  9. Guo, G. et al. (2003). Knn model-based approach in classification. In On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2003, Catania, Sicily, Italy, November 3-7, 2003. Proceedings.
  10. Guu, K. et al. (2020). Retrieval augmented language model pre-training. In International conference on machine learning.
  11. Hendrycks, D. et al. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  12. Izacard, G. et al. (2022a). Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299.
  13. Izacard, G. et al. (2022b). Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.
  14. Ji, Z. et al. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys.
  15. Jiang, Z. et al. (2023). Active retrieval augmented generation. arXiv preprint arXiv:2305.06983.
  16. Jin, D. et al. (2021). What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences.
  17. Jin, Q. et al. (2019). Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  18. Jin, Q. et al. (2023). Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics (Oxford, England).
  19. Kang, M. et al. (2023). Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks. arXiv preprint arXiv:2305.18395.
  20. Karpukhin, V. et al. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  21. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  22. Kitaev, N. et al. (2019). Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
  23. Kwon, W. et al. (2023). Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles.
  24. Lewis, P. et al. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems.
  25. Li, B. et al. (2023). Meddm: Llm-executable clinical guidance tree for clinical decision-making. arXiv preprint arXiv:2312.02441.
  26. Mao, Y. et al. (2021). Generation-augmented retrieval for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
  27. Nori, H. et al. (2023). Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375.
  28. OpenAI (2023a). Chatgpt.
  29. OpenAI (2023b). Openai gpt-4 technical report.
  30. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems.
  31. Pal, A. et al. (2022). Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning.
  32. Paszke, A. et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems.
  33. Rajbhandari, S. et al. (2020). Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE.
  34. Schulman, J. et al. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  35. Shao, Z. et al. (2023). Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. arXiv preprint arXiv:2305.15294.
  36. Singhal, K. et al. (2022). Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138.
  37. Taori, R. et al. (2023). Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html.
  38. Taylor, R. et al. (2022). Galactica: A large language model for science. arXiv preprint arXiv:2211.09085.
  39. Google (2023). Bard.
  40. Touvron, H. et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  41. Wang, Y. et al. (2022). Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560.
  42. Wang, Y. et al. (2023). Augmenting black-box llms with medical textbooks for clinical question answering. arXiv preprint arXiv:2309.02233.
  43. Wang, Z. et al. (2019). Multi-passage bert: A globally normalized bert model for open-domain question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  44. Wei, J. et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems.
  45. Wolf, T. et al. (2019). Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  46. Wu, Z. et al. (2023). Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693.
  47. Zhang, X. et al. (2023). Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Minbyul Jeong (18 papers)
  2. Jiwoong Sohn (6 papers)
  3. Mujeen Sung (20 papers)
  4. Jaewoo Kang (83 papers)
Citations (18)
View on Semantic Scholar