OpenMed NER: Efficient Biomedical NER

• OpenMed NER is an open-source suite of transformer models specifically designed for biomedical named entity recognition, integrating domain-adaptive pre-training (DAPT) and LoRA-based fine-tuning.
• It employs three transformers (DeBERTa-v3-large, PubMedBERT-large, and BioELECTRA-large) with lightweight DAPT on 350K passages, achieving state-of-the-art micro-F₁ improvements across 10 benchmarks.
• The approach ensures computational efficiency and environmental sustainability, with up to 2–3× faster training, under 1.2 kg CO₂ emissions, and ease of compliant on-premise deployment.

OpenMed NER encompasses a set of open-source, domain-adapted transformer models and associated methodologies for biomedical named entity recognition (NER), focusing on efficient, high-accuracy extraction of structured information from clinical notes and biomedical literature. It is specifically oriented toward compliance, scalability, and accessibility, advancing state-of-the-art performance on a diverse spectrum of biomedical NER benchmarks by integrating lightweight domain-adaptive pre-training (DAPT), parameter-efficient fine-tuning (via LoRA), and standardized open-source model releases (Panahi, 3 Aug 2025).

1. Model Architecture and Domain Adaptation

OpenMed NER utilizes three transformer backbones—DeBERTa‑v3-large, PubMedBERT-large, and BioELECTRA-large—each selected for distinct architectural strengths:

• DeBERTa-v3-large employs disentangled attention, separately modeling content and positional information for improved long-range dependency handling, which is critical for processing complex clinical narratives.
• PubMedBERT-large is pretrained on ≈3.1B tokens of biomedical text, enabling specialized subword vocabulary and robust semantic discrimination within biomedical terminology.
• BioELECTRA-large uses a replaced-token detection objective, providing efficient token-level representations particularly favorable for span-based NER.

All backbones undergo lightweight domain-adaptive pre-training (DAPT) using a 350,000-passage (≈90 million tokens) corpus drawn from PubMed abstracts, biomedical arXiv metadata, de-identified MIMIC-III clinical sentences, and clinical trial descriptions. The training employs masked LLMing with dynamic 15% token masking.

Parameter-efficient adaptation is achieved by integrating Low-Rank Adaptation (LoRA). LoRA injects trainable, rank-decomposition matrices (e.g., rank=16, α=32, dropout=0.05) into the attention layers' query and value matrices, with less than 1.5% of all model parameters updated during DAPT and downstream fine-tuning. Only the adapter and output head weights are updated, reducing computational cost and carbon footprint while aligning the model representations with biomedical domain specifics.

2. Training Regimen and Task-Specific Fine-Tuning

After domain-adaptive pre-training, task-specific fine-tuning is performed for NER using a BIO-tagged token classification formulation. A new token-classification head maps each token’s final representation ($h_i$) to label logits:

$$P(y_i|x_{1:n}) = \mathrm{softmax}(W_{\text{cls}} h_i + b_{\text{cls}})$$

Fine-tuning optimizes cross-entropy loss on NER-labeled corpora, updating only the LoRA adapter weights and the output head, while all backbone parameters remain frozen. Bayesian optimization (TPE sampler) is employed for hyperparameter selection (learning rate, dropout), ensuring robust convergence across dataset types.

The complete adaptation and fine-tuning pipeline for each backbone (DAPT + NER fine-tuning) completes in under 12 hours on a single GPU, with individual NER task training requiring only 3–6 minutes per dataset.

3. Benchmarking and Performance Across Biomedical Domains

OpenMed NER is extensively evaluated on 12 established biomedical NER benchmarks, including:

• Disease and chemical corpora: BC5CDR-Disease, BC5CDR-Chem, BC4CHEMD, NCBI-Disease
• Gene/protein corpora: JNLPBA, BC2GM, BioNLP 2013 CG, FSU
• Anatomy and species: AnatEM, Linnaeus, Species-800
• Clinical: CLL

Key empirical results (test set micro-F₁ scores, as improvements over previous best where applicable):

Dataset	Micro-F₁ (%)	Δ Previous Best (pp)
BC4CHEMD	95.40	+1.37
BC5CDR-Chem	96.10	+1.88
BC5CDR-Disease	91.20	+2.70
NCBI-Disease	91.10	+1.39
JNLPBA	81.90	–0.10
BC2GM	[noted gain]	+5.39
Linnaeus	[noted gain]	+3.80
Species-800	[noted gain]	+0.92
CLL	[noted gain]	+9.72

OpenMed NER advanced the state-of-the-art on 10 of the 12 datasets, including foundational benchmarks in disease, chemical, clinical cell line, and species domains. Particularly large improvements were achieved on specialized gene/protein and cell line corpora (over +5.3 and +9.7 percentage points, respectively).

4. Computational Efficiency and Environmental Impact

The LoRA-based adaptation strategy ensures high efficiency:

• Updates less than 1.5% of total model parameters (~4–4.7M out of 330–335M), resulting in 2–3× faster training compared to full finetuning.
• Complete suite preparation (DAPT + fine-tuning) emits less than 1.2 kg CO₂-equivalent, with a full benchmark sweep across tasks and backbones totaling under 2 kg CO₂-equivalent.
• Training is feasible on commodity hardware (≤16 GB VRAM), supporting practical on-premise deployment for regulatory environments.

This efficiency does not compromise accuracy and is instrumental for compliance with emerging data protection and AI regulatory frameworks.

5. Regulatory and Practical Implications

OpenMed NER checkpoints are released permissively under Apache 2.0, facilitating transparent, local deployment in sensitive healthcare settings and streamlined compliance with evolving standards such as the EU AI Act. The modular adapter design, with small, swappable LoRA checkpoints, enables rapid, auditable model updating and easy version management. This supports use in both production clinical pipelines and research settings requiring reproducibility.

Furthermore, the pipeline’s reliance on open-source backbones, ethically sourced and de-identified datasets, and explicit minimal-parameter adaptation aligns with best practices for transparent and trustworthy biomedical AI.

6. Significance and Context Within Biomedical NLP

OpenMed NER exemplifies a trend toward open, accessible, and regulation-ready biomedical NLP models. By demonstrating that lightweight, domain-adaptive transformers—when paired with targeted DAPT and minimal-parameter fine-tuning strategies—can consistently outperform or match much larger closed-source or fully fine-tuned models, it establishes a new practical balance between computational efficiency and domain performance.

The extensive coverage across diverse entity types, robust benchmarking against established datasets, and focus on practical deployment make OpenMed NER a reference architecture for future biomedical NER research and applications. This approach supports structuring >80% of unstructured clinical data for downstream analytics, decision support, and research, without imposing excessive resource requirements or governance burdens.

7. Future Research Directions

While OpenMed NER sets new benchmarks for performance and efficiency, the following areas are highlighted for future investigation:

• Expanding the approach to multilingual and cross-domain adaptation, using similar lightweight DAPT and LoRA strategies for non-English clinical data.
• Integrating uncertainty quantification and calibration (as seen in E-NER frameworks) to improve model trustworthiness in open-ended, safety-critical clinical settings.
• Broadening the set of supported biomedical entity types and exploring zero-/few-shot adaptation using flexible label representations or synthetic annotation strategies.
• Further reducing emissions and training requirements, and optimizing LoRA adapter architectures for even more rapid, distributed fine-tuning in institutional environments.

OpenMed NER defines a foundational template for efficient, open, and compliant biomedical NER, with a focus on operational fidelity and transparency for real-world clinical and research contexts (Panahi, 3 Aug 2025).