Beyond the Basics: Leveraging Large Language Model for Fine-Grained Medical Entity Recognition

Abstract: Extracting clinically relevant information from unstructured medical narratives such as admission notes, discharge summaries, and emergency case histories remains a challenge in clinical NLP. Medical Entity Recognition (MER) identifies meaningful concepts embedded in these records. Recent advancements in LLMs have shown competitive MER performance; however, evaluations often focus on general entity types, offering limited utility for real-world clinical needs requiring finer-grained extraction. To address this gap, we rigorously evaluated the open-source LLaMA3 model for fine-grained medical entity recognition across 18 clinically detailed categories. To optimize performance, we employed three learning paradigms: zero-shot, few-shot, and fine-tuning with Low-Rank Adaptation (LoRA). To further enhance few-shot learning, we introduced two example selection methods based on token- and sentence-level embedding similarity, utilizing a pre-trained BioBERT model. Unlike prior work assessing zero-shot and few-shot performance on proprietary models (e.g., GPT-4) or fine-tuning different architectures, we ensured methodological consistency by applying all strategies to a unified LLaMA3 backbone, enabling fair comparison across learning settings. Our results showed that fine-tuned LLaMA3 surpasses zero-shot and few-shot approaches by 63.11% and 35.63%, respectivel respectively, achieving an F1 score of 81.24% in granular medical entity extraction.

• The paper demonstrates that parameter-efficient fine-tuning significantly outperforms zero-shot and few-shot methods, achieving an F1 score up to 81.24%.
• It leverages a unified LLaMA3 backbone with standardized prompts and token-level similarity for effective handling of lexical heterogeneity in clinical texts.
• The study highlights that strategic LoRA adapter placement reduces invalid entity rates to nearly 0%, underscoring its impact over mere scaling factors.

Fine-Grained Medical Entity Recognition with LLaMA3: Unified Analysis Across Zero-Shot, Few-Shot, and Fine-Tuned Paradigms

Introduction

Extracting clinically relevant structured information from the vast expanse of unstructured medical narratives in EMRs is a central challenge in clinical NLP. Medical Entity Recognition (MER) enables downstream applications including decision support, patient cohort identification, and the construction of longitudinal patient histories from free-text Figure 1. Most recent advances rely on deep learning and, increasingly, LLMs. However, evaluation has focused on coarse entity classes, and direct, methodologically consistent comparisons between LLM learning modes for granular MER are lacking.

Figure 1: Example of fine-grained medical entity recognition from unstructured clinical text.

Dataset Construction and Annotation Framework

A new fine-grained MER dataset was constructed using 310 discharge summaries from the i2b2 corpus, annotated across 18 clinically detailed categories such as System Organ Site, Alcohol Consumption, Procedures, and temporal qualifiers, inspired by Amazon Comprehend Medical’s schema. Doccano was utilized for annotation, integrating and refining automatic ACM pre-annotations with domain expert review. The entity distribution in training (14,235 entities) and test (12,007 entities) sets is illustrated in Figure 2, highlighting substantial entity diversity and a realistic frequency landscape reflecting clinical needs.

Figure 2: Entity breakdown between train and test splits, showing coverage across 18 medical categories.

Unified Learning Paradigms with LLaMA3

The 8B instruction-tuned LLaMA3 model was used as a homogeneous backbone to directly compare three learning paradigms—zero-shot prompting, few-shot prompting with intelligent example selection, and parameter-efficient fine-tuning using LoRA.

Prompt Engineering

Prompt structures were standardized to include a task description, entity markup guidelines, and precise definitions Figure 3, with additional strict formatting instructions for ICL (zero/few-shot) settings Figure 4 to enforce output conformity and eliminate reasoning in answers.

Figure 3: Baseline prompt structure for controlled model comparison.

Figure 4: Enhanced prompt structure enforcing strict adherence to output constraints in zero/few-shot scenarios.

Few-Shot Example Selection: Embedding Similarity

For few-shot evaluation, two retrieval strategies leveraging BioBERT embeddings were deployed:

• Sentence-level similarity: Top-$k$ examples selected based on whole-sentence embedding proximity.
• Token-level similarity: Top-$k$ examples based on the average similarity of token-level embeddings across candidate and input sentences.

This embedding-based approach directly addresses the lexical and semantic heterogeneity prevalent in clinical text.

Efficient Fine-Tuning via LoRA

Fine-tuning used LoRA adapters, with comprehensive ablation over rank dimension ($r$), scaling factor ($\alpha$), and injection points across transformer modules. Training data input/output structures (Unprocessed/Processed) were controlled for consistent supervision Figure 5.

Figure 5: Structure of training dataset for model fine-tuning.

Experimental Results

Model-Level Performance

Fine-tuned LLaMA3 achieved an F1 score of 81.24%, outperforming the best zero-shot and few-shot configurations by 63.11% and 35.63%, respectively. Fine-tuned models (FT) maintained F1 within $80\% \pm 1.2$ across LoRA configurations, while few-shot (FS) models plateaued at 45.61% (token-level similarity, $k=6$) and zero-shot at 18.13%. These trends are summarized in Figure 6.

Figure 6: Overall F1, precision, and recall performance across zero-shot, few-shot, and fine-tuned paradigms with explicit LoRA configuration details.

Statistical testing confirmed that expanding LoRA rank from 8 to 64 did not yield significant increases; rather, adapter placement (all linear modules vs. just attention) was the primary determinant of performance improvement.

Invalid Entity Rate

The invalid entity count, reflecting model hallucination and category fidelity, was highest in zero-shot (7.75%), lower in few-shot (minimum 1.67% for token-level selection), and negligible (≤0.04%) in fine-tuned models Figure 7. This demonstrates the necessity of supervised adaptation for robust prompting compliance in high-stakes clinical tasks.

Figure 7: Invalid entity percentage for each learning mode and selection strategy.

Per-Entity Analysis

Entity-level F1 analysis Figure 8 revealed that FT models achieved robust scores across both common (e.g., tobacco use F1 = 95.2%) and rare classes (e.g., alcohol consumption F1 = 77.7%). Few-shot settings improved performance on high-frequency entities but underperformed on low-frequency or nuanced categories compared to fine-tuning. Zero-shot struggled universally across categories.

Figure 8: Per-entity F1 scores comparing selected zero-shot, few-shot, and fine-tuned configurations.

Implications and Future Directions

This work establishes several key points for applied research in clinical NLP:

• Unified LLM Backbone for Fair Comparison: Direct, method-consistent benchmarking of zero-shot, few-shot, and fine-tuned approaches on a shared LLaMA3 backbone avoids frequent inconsistencies and artifacts of cross-architecture studies.
• Value of Fine-Tuning Over ICL for Granular Tasks: While few-shot selection using token-level retrieval improves over zero-shot, only fine-tuning achieves the specificity and reliability required for fine-grained clinical extraction, with major gains in both F1 and invalid entity reduction.
• LoRA Adapter Placement Over Dimensionality: Adapter integration across all transformer modules is more consequential for adaptation performance than raw rank dimension scaling.
• Token-Level Example Retrieval: For token-centric NER, example selection strategies should operate at the token, not sentence, level to maximize contextual alignment.

The provided fine-grained MER dataset serves as a new benchmark with practical clinical relevance. Code and data release will support further research.

Prospective lines of inquiry include extending to clinical relation extraction, modeling entity temporal relations [alfattni_extraction_2020], and transferring models to diverse local health system corpora. The broader takeaways also inform evaluation practices in clinical LLMs and the boundary of current ICL-based approaches for extractive biomedical NLP.

Conclusion

Through a rigorously controlled experimental framework, this work demonstrates that parameter-efficient fine-tuning of LLaMA3 remains essential for high-accuracy, fine-grained medical entity recognition. Few-shot learning with embedding-based example selection partially closes the performance gap but is insufficient for tasks involving substantial entity granularity and contextual complexity. Methodological transparency and unified benchmarking ground these conclusions, providing actionable guidance for both practitioners and forthcoming research in clinical NLP (2604.17214).