ModernBERT-bio: A Long-Context Biomedical Transformer

Updated 4 July 2026

ModernBERT-bio is a specialized biomedical encoder leveraging long-context transformer architecture and domain-adaptive pretraining for enhanced clinical text processing.
It incorporates advanced methods like RoPE positional embeddings, Flash Attention, and dynamic token-aware masking to optimize processing of lengthy biomedical content.
Variants such as Clinical ModernBERT and BioClinical ModernBERT show significant benchmark improvements on tasks like ChemProt, i2b2, and biomedical retrieval.

ModernBERT-bio denotes a biomedical and clinical adaptation of the ModernBERT encoder family: a long-context, bidirectional transformer line specialized through continued pretraining on biomedical literature, clinical notes, and related medical text resources. Current usage suggests that the term functions less as a single canonical checkpoint than as a family designation spanning at least Clinical ModernBERT, BioClinical ModernBERT, and retrieval-oriented systems that use ModernBERT as the first-stage biomedical encoder. Across these variants, the defining characteristics are extended context length up to 8,192 tokens, domain-adaptive masked-language-model pretraining, and an emphasis on encoder efficiency for discriminative biomedical NLP and retrieval tasks (Sounack et al., 12 Jun 2025, Lee et al., 4 Apr 2025, Rivera et al., 6 Oct 2025).

1. Conceptual scope and model lineage

ModernBERT-bio emerged in a setting where encoder-based transformers remained central to biomedical and clinical NLP, but encoder development had lagged behind decoder models. BioClinical ModernBERT was introduced as “a domain-adapted encoder that builds on the recent ModernBERT release,” while Clinical ModernBERT was introduced as an encoder “pretrained on large scale biomedical literature, clinical notes, and medical ontologies,” explicitly transferring ModernBERT’s long-context architectural upgrades into biomedical and clinical use (Sounack et al., 12 Jun 2025, Lee et al., 4 Apr 2025).

Two principal released forms define the area. Clinical ModernBERT is a base-sized model initialized from the ModernBERT-base checkpoint and further pretrained on approximately 13 billion tokens from PubMed abstracts, MIMIC-IV clinical notes, and structured medical ontologies (Lee et al., 4 Apr 2025). BioClinical ModernBERT extends the same general program at larger corpus scale, releasing both base and large variants after continued pretraining on over 53.5 billion tokens, including 20 English-language clinical datasets from diverse institutions, domains, and geographic regions (Sounack et al., 12 Jun 2025).

A related but architecturally distinct extension appears in biomedical retrieval-augmented generation. In “ModernBERT + ColBERT,” ModernBERT serves as a lightweight bi-encoder for first-stage retrieval, followed by a ColBERTv2 late-interaction re-ranker; the system is optimized for biomedical IR and RAG rather than conventional classification or tagging (Rivera et al., 6 Oct 2025). This broadens the meaning of ModernBERT-bio from a pure encoder checkpoint to a reusable biomedical retrieval backbone.

Variant	Core specification	Domain focus
Clinical ModernBERT	12 layers, hidden dimension 768, 12 heads, maximum sequence length 8,192	Biomedical literature, clinical notes, medical ontologies
BioClinical ModernBERT-base	12 layers, hidden dimension 768, 12 heads, maximum input length 8,192	Biomedical and clinical NLP
BioClinical ModernBERT-large	24 layers, hidden dimension 1,024, 16 heads, maximum input length 8,192	Biomedical and clinical NLP

2. Architectural profile and long-context mechanisms

The architectural identity of ModernBERT-bio is defined primarily by long-context encoder design. BioClinical ModernBERT retains standard transformer self-attention,

$A=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$

but couples it with RoPE positional embeddings, an alternating attention pattern, and unpadding at inference. In the reported configuration, about two-thirds of layers use local sliding-window attention of size $w$ and one-third use full global attention, reducing the cost of pure global attention to roughly $O(N\!\cdot\!L\!\cdot\!w + N/3\!\cdot\!L^2)$ and enabling effective training and inference at $L=8{,}192$ (Sounack et al., 12 Jun 2025).

Clinical ModernBERT reports a closely related design vocabulary. It inherits RoPE, Flash Attention, extended context length up to 8,192 tokens, GeGLU activations in the feed-forward networks, and bias-free linear layers throughout (Lee et al., 4 Apr 2025). In that model, RoPE replaces 512-token positional schemes with a rotation-based positional mechanism, while Flash Attention reduces memory from $O(L^2)$ to near-linear in $L$ by computing attention block-wise in shared memory. Unlike Longformer or BigBird, the model description states that no sparse masking is needed to support 8,192-token contexts (Lee et al., 4 Apr 2025).

The two most explicit released configurations are the BioClinical base and large models. The base model has 12 layers, hidden dimension $d_{\text{model}}=768$ , 12 heads, and 150 M parameters; the large model has 24 layers, hidden dimension 1,024, 16 heads, and 396 M parameters (Sounack et al., 12 Jun 2025). Clinical ModernBERT uses the same “base” configuration as ModernBERT—12 layers, hidden dimension 768, 12 attention heads, feed-forward inner dimension 3,072, and maximum sequence length 8,192—while extending the tokenizer with approximately 500 clinical tokens (Lee et al., 4 Apr 2025).

A recurrent misconception is that biomedical encoder adaptation is only a matter of changing the corpus. The long-context literature around ModernBERT-bio does not support that simplification. Both Clinical ModernBERT and BioClinical ModernBERT present architectural choices—RoPE, Flash Attention or alternating attention, and inference-time efficiency measures—as integral to the biomedical specialization, not merely auxiliary engineering (Sounack et al., 12 Jun 2025, Lee et al., 4 Apr 2025).

3. Domain adaptation corpora and optimization procedures

Clinical ModernBERT and BioClinical ModernBERT differ most sharply in corpus scale and training schedule. Clinical ModernBERT continues pretraining on a composite corpus of approximately 13 billion tokens. The three named components are PubMed abstracts, MIMIC-IV clinical notes, and structured medical ontologies comprising textual serializations of ICD-9 through ICD-12 disease codes, CPT procedure codes, and medication terminologies (Lee et al., 4 Apr 2025).

Its pretraining objective is standard masked language modeling, but with two domain-aware modifications: a token-aware masking collator that prioritizes clinically salient tokens in the early phase of training, and a dynamic masking rate schedule decaying linearly from 30% to 15% of tokens masked. The reported pretraining schedule comprises 150,000 steps, large multi-GPU batches with effective batch approximately 4,096 sequences, StableAdamW, peak learning rate approximately $3\times 10^{-3}$ with linear warmup and cosine decay to $1\times 10^{-5}$ , mixed precision on NVIDIA A100s, and a sequence-length curriculum from 128 tokens to 512, then 2 K, then full 8 K (Lee et al., 4 Apr 2025).

BioClinical ModernBERT is organized around a larger two-phase continued-pretraining strategy. The total pretraining text is 53.5 billion tokens, split into 50.7 B from PubMed abstracts and PMC full-text and 2.8 B from 20 English-language clinical datasets, including MIMIC-III, MIMIC-IV Note, CheXpert Plus, ADE Corpus, and MedNLI (Sounack et al., 12 Jun 2025). Phase 1 resumes from ModernBERT’s pre-decay checkpoint and trains on the combined corpus until 160.5 B new tokens have been seen, using a masking probability of 30%. Phase 2 then trains 3 more epochs on the 2.8 B clinical tokens alone with masking 15%, explicitly adapting the model to clinical language (Sounack et al., 12 Jun 2025).

The BioClinical training setup uses 8 NVIDIA H100 SXM5 GPUs; reported training time is about 4 days for the base model and 8 days for the large model. The optimizer is AdamW, with a Warmup–Stable–Decay scheduler, and the phase-specific peak learning rates are $3\times10^{-4}$ for base and $w$ 0 for large in Phase 1 (Sounack et al., 12 Jun 2025).

This suggests that ModernBERT-bio development bifurcates into two complementary strategies: one emphasizing domain-aware masking and ontology-enriched corpora at moderate scale, and one emphasizing very large-scale continued pretraining with an explicit clinical-specialization phase.

4. Benchmark behavior on biomedical and clinical NLP

BioClinical ModernBERT is evaluated on five downstream tasks spanning relation extraction, document classification, and three NER scenarios, with models fine-tuned for up to 10 epochs with early stopping, batch size 16, weight decay $w$ 1, and median performance over 5 seeds. On ChemProt, Phenotype, COS, Social-History NER, and DEID, the BioClinical base model reports 89.9, 58.1, 95.1, 58.5, and 82.7 median F1, while the large model reports 90.8, 60.8, 95.1, 57.1, and 83.8, outperforming BioBERT, ClinicalBERT, BioMed-RoBERTa, Clinical-Longformer, Clinical-BigBird, and general ModernBERT variants on the table presented in the paper (Sounack et al., 12 Jun 2025).

The same work includes an ablation on long-context capacity. A clinical ModernBERT model fine-tuned with only 128-token contexts scores 54.9 on Phenotype and 44.4 on DEID, versus 58.1 and 82.7 for the BioClinical base model. The authors interpret the resulting +3.2 points on Phenotype and +38.3 points on DEID as highlighting “the benefit of processing full clinical notes (up to 8 192 tokens)” (Sounack et al., 12 Jun 2025).

Clinical ModernBERT reports a broader benchmark mixture. On short-context tasks, it achieves 0.9769 AUROC on EHR-Prediction, 0.9209 accuracy on PubMed-NCT, and 0.766 F1 on MedNER; in PMC-Patients retrieval it reports 0.2167 NDCG@10, 0.0552 P@10, 0.2791 R@10, and 0.1982 MAP, leading every retrieval metric listed in the table (Lee et al., 4 Apr 2025). On long-context i2b2 clinical NER, it reports F1 scores of 0.965 on i2b2 2006, 0.883 on i2b2 2010, 0.804 on i2b2 2012, and 0.966 on i2b2 2014, with the paper stating that it holds second place on 2006 and 2010 and best performance on 2012 and 2014 (Lee et al., 4 Apr 2025).

Efficiency is treated as a first-order property rather than an afterthought. BioClinical ModernBERT reports approximately 71 kTok/s on fixed-length inputs and approximately 73.7 kTok/s on variable-length inputs even at $w$ 2, while Clinical ModernBERT reports approximately 1.6× faster inference than BioClinicalBERT on 512-token inputs (Sounack et al., 12 Jun 2025, Lee et al., 4 Apr 2025).

5. Retrieval-oriented ModernBERT-bio and biomedical RAG

In the retrieval literature, ModernBERT-bio appears as a biomedical retriever rather than only as a classifier or tagger backbone. “ModernBERT + ColBERT” implements a two-stage architecture in which a siamese ModernBERT bi-encoder produces a single 768-dimensional vector from the [CLS] output for each query or passage, with document embeddings stored in Qdrant and approximate nearest-neighbor search performed over approximately 1,000,000 PubMed snippets (Rivera et al., 6 Oct 2025).

The second stage uses ColBERTv2 late interaction. Query and passage tokens are represented as token-wise contextual embeddings, and ranking uses the MaxSim operator:

$w$ 3

Only the $w$ 4 candidates from the ModernBERT retriever are re-ranked before being passed to the generator LLM (Rivera et al., 6 Oct 2025).

The training recipe uses 10,000 question–passage pairs from the PubMedQA subset of BigBio. First, ModernBERT is fine-tuned as a bi-encoder with in-batch negative sampling and a cosine-similarity retrieval objective; second, ColBERTv2 is fine-tuned on the top candidates produced by the trained retriever, mining hard negatives from its predictions. When end-to-end fine-tuning is used, the joint objective is

$w$ 5

The reported retrieval results are zero-shot ModernBERT Recall@10 approximately 37.9%, fine-tuned ModernBERT Recall@10 approximately 92.8%, and 93.8% with ColBERT re-ranking; Recall@3 improves by up to 4.2 percentage points over retrieve-only ModernBERT. In end-to-end biomedical RAG on MIRAGE, the system reports average accuracy 0.4448, above MedCPT at 0.4436, DPR at 0.4174, BM25 at 0.4369, and MedGemma at 0.4371 (Rivera et al., 6 Oct 2025).

The ablation studies are especially important for interpretation. The paper reports that the performance is critically dependent on joint fine-tuning or on sequential alignment in which the re-ranker is trained on the retriever’s hard negatives; otherwise, the re-ranker may degrade Recall@k and end-to-end accuracy (Rivera et al., 6 Oct 2025). This directly counters the assumption that adding a re-ranking stage is automatically beneficial.

6. Relation to earlier biomedical encoders, deployment patterns, and limitations

ModernBERT-bio belongs to a longer trajectory of biomedical encoder specialization. BioBERT adapted BERT BASE to biomedical corpora without architectural change, continuing pretraining on PubMed abstracts and PMC full-text; it reported gains on biomedical named entity recognition, relation extraction, and question answering, including a 12.24% MRR improvement in biomedical QA over the previous state of the art (Lee et al., 2019, Yoon et al., 2019). BioALBERT pursued parameter efficiency through factorized embedding parameterization and cross-layer parameter sharing, reaching 12 M and 16 M parameter configurations and reporting new state of the art in 17 of 20 benchmark datasets (Naseem et al., 2021). Bioformer pursued compactness within BERT-style biomedical text mining, producing 42–43 M parameter models with 60% fewer parameters than BERT Base, overall accuracy within 0.1% or 0.9% of PubMedBERT depending on variant, and 2–3 fold speed improvements (Fang et al., 2023).

Against that background, ModernBERT-bio shifts emphasis toward long-context processing and modern encoder efficiency while preserving the biomedical tradition of domain-adaptive pretraining. This suggests a change in optimization target: not only improving sentence- or abstract-scale biomedical understanding, but also enabling whole-note encoding, interactive retrieval, and high-throughput clinical deployment.

The literature also delineates several limitations. Clinical ModernBERT states that “a full error analysis” on rare diseases and cross-institution generalization remains to be done (Lee et al., 4 Apr 2025). BioClinical ModernBERT, although trained on 20 clinical datasets rather than a single source, is still evaluated on a finite suite of four downstream task families and one relation-extraction benchmark (Sounack et al., 12 Jun 2025). In retrieval, the ModernBERT+ColBERT paper shows that independent fine-tuning can be counterproductive, indicating that biomedical late interaction is sensitive to training alignment rather than only backbone quality (Rivera et al., 6 Oct 2025).

BioClinical ModernBERT addresses reproducibility directly by releasing both base and large model weights and all training checkpoints on Hugging Face, together with open-source training and fine-tuning code on GitHub (Sounack et al., 12 Jun 2025). A plausible implication is that ModernBERT-bio has become not merely a benchmark artifact, but an extensible biomedical encoder substrate for further continued pretraining, task adaptation, and retrieval integration.