LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction

Abstract: LLMs are increasingly adopted for applications in healthcare, reaching the performance of domain experts on tasks such as question answering and document summarisation. Despite their success on these tasks, it is unclear how well LLMs perform on tasks that are traditionally pursued in the biomedical domain, such as structured information extraction. To bridge this gap, in this paper, we systematically benchmark LLM performance in Medical Classification and Named Entity Recognition (NER) tasks. We aim to disentangle the contribution of different factors to the performance, particularly the impact of LLMs' task knowledge and reasoning capabilities, their (parametric) domain knowledge, and addition of external knowledge. To this end, we evaluate various open LLMs - including BioMistral and Llama-2 models - on a diverse set of biomedical datasets, using standard prompting, Chain of-Thought (CoT) and Self Consistency based reasoning as well as Retrieval-Augmented Generation (RAG) with PubMed and Wikipedia corpora. Counter intuitively, our results reveal that standard prompting consistently outperforms more complex techniques across both tasks, laying bare the limitations in the current application of CoT, self-consistency and RAG in the biomedical domain. Our findings suggest that advanced prompting methods developed for knowledge- or reasoning-intensive tasks, such as CoT or RAG, are not easily portable to biomedical tasks where precise structured outputs are required. This highlights the need for more effective integration of external knowledge and reasoning mechanisms in LLMs to enhance their performance in real-world biomedical applications.

• The paper demonstrates that standard prompting outperforms complex reasoning methods like CoT and RAG in biomedical information extraction, achieving higher F1 scores.
• The study reveals that larger models, such as Llama-2-70B, significantly outperform smaller ones, highlighting model scale as a key driver in zero-shot settings.
• The evaluation indicates that retrieval-augmented techniques using external sources like PubMed or Wikipedia often misalign context, reducing accuracy in structured biomedical tasks.

Essay: Benchmarking LLM Performance in Biomedical Information Extraction

The paper "LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction" by Aishik Nagar et al. provides a comprehensive evaluation of the capabilities of LLMs in the context of structured information extraction tasks within the biomedical domain. The research highlights a systematic benchmarking of LLM performance in Medical Classification and Named Entity Recognition (NER) tasks, assessing the impact of various prompting techniques and external knowledge integration methods.

Summary

LLMs have achieved notable success in tasks such as question answering and document summarization in healthcare. However, this study scrutinizes their efficacy in structured information extraction, discerning how factors like task knowledge, domain-specific parametric knowledge, and external knowledge affect performance. The evaluation considers multiple open-source LLMs, including BioMistral and Llama-2 models, and employs different reasoning techniques such as Chain-of-Thought (CoT), Self-Consistency, and Retrieval-Augmented Generation (RAG) with PubMed and Wikipedia corpora.

Key Findings

Standard Prompting vs. Advanced Techniques

The paper's results reveal a counter-intuitive finding: standard prompting consistently outperforms more complex reasoning techniques like CoT, self-consistency, and RAG across both Medical Classification and NER tasks. For instance, the standard prompting method showed higher average F1 scores compared to advanced techniques:

• BioMistral-7B achieved 36.48% in classification tasks using standard prompting.
• Llama-2-70B-Chat attained 40.34% in the same context. This performance gap underscores that the advanced prompting methods often fail to transfer their success from knowledge-intensive tasks like QA to more structured tasks requiring precise outputs.

Impact of Model Scale

The study confirms that the parametric knowledge capacity, essentially the model size, is a significant driver of performance in zero-shot settings. The Llama-2-70B exhibited superior performance over smaller models in both classification and NER tasks, reaffirming that larger models inherently possess more robust internal knowledge representations. The 70B model, for instance, showed a considerable improvement, demonstrating a higher resilience and adaptability when incorporating techniques like CoT and RAG.

RAG Performance

Notably, the use of RAG did not yield performance improvements for structured prediction tasks. The integration of external knowledge from PubMed or Wikipedia was less beneficial and sometimes even detrimental, possibly introducing irrelevant information that complicates the model's decision-making process. This finding indicates a nuanced requirement for retrieving context-specific information rather than broad domain knowledge.

Implications and Future Research

The findings suggest that while LLMs excel in some healthcare applications, their current implementations reveal limitations in tasks requiring structured information extraction. This insight propels several implications for future research and practical applications:

1. Enhanced Knowledge Integration: To improve LLM performance in biomedical tasks, further work is needed to develop methods for more effective knowledge integration, potentially by designing retrieval mechanisms that prioritize highly relevant and context-specific information.
2. Model Specialization: The lower efficacy of complex reasoning techniques in structured tasks points toward the necessity for domain-specific training and fine-tuning approaches. Incorporating biomedical ontologies and more refined context modeling could enhance the models' ability to handle nuanced medical information.
3. Scalability and Accessibility: Given the constraint of computational costs and privacy concerns associated with large models, there is a need for scalable solutions that balance performance and resource utilization. Implementing efficient retrieval-augmented techniques adaptable to smaller, more accessible models could democratize the use of LLMs in healthcare settings.
4. Validation on Real-world Data: The performance disparity on public vs. private datasets underscores the importance of validating LLMs on real-world, proprietary medical data to assess their true potential and limitations in practical applications.

Conclusion

The paper by Nagar et al. delineates a crucial aspect of LLM application in healthcare, emphasizing the need for specialized approaches in biomedical information extraction. While larger models possess inherent advantages, the study highlights a pivotal requirement for better task-specific adaptations and knowledge integration methods. This research thus serves as a foundational step towards enhancing the effectiveness and reliability of LLMs in real-world biomedical applications, paving the way for future advancements in AI-driven healthcare solutions.