ModernBERT-base-PT Overview

Updated 12 July 2026

ModernBERT-base-PT is a base-sized encoder with 149M parameters, 22 layers, and 8192-token support, designed for efficient long-context processing.
It employs masked language modeling with a 30% masking rate and integrates hardware-aware optimizations like FlashAttention and greedy sequence packing.
In the patent domain, the -PT variant utilizes a clean 31.6B-token corpus to enhance downstream performance while maintaining superior speed and memory efficiency.

ModernBERT-base-PT denotes a base-sized pretrained checkpoint in the ModernBERT lineage, but the label is not used uniformly across the 2024–2026 literature. In the original ModernBERT work, it refers to the pretrained “base” encoder: a 149M-parameter, 22-layer, 768-hidden, 12-head bidirectional Transformer trained with MLM on about 2 trillion tokens and extended to a native 8192-token context (Warner et al., 2024). In later domain papers, the same or closely related naming is reused for continued-pretraining or from-scratch domain variants, most explicitly in patent NLP, where “ModernBERT-base-PT” is the main patent-domain pretrained encoder and the “-PT” suffix denotes the Phase 2 clean patent corpus variant (Yousefiramandi et al., 18 Sep 2025).

1. Terminology and scope

Across the cited literature, the label is used in more than one sense. The common thread is a base-sized ModernBERT checkpoint used as a pretrained encoder for discriminative NLP, usually with MLM and long-context support, but the data regime and sometimes even the architectural description depend on the paper-specific setup.

Source paper	What the label refers to	Key distinction
(Warner et al., 2024)	Pretrained ModernBERT base encoder	General-domain base model
(Yousefiramandi et al., 18 Sep 2025)	ModernBERT-base-PT	Patent-domain “Phase 2 clean” model
(Antoun et al., 11 Apr 2025)	Base pretrained ModernBERT in a controlled French setup	ModernBERT-CV2 / ModernBERT-HQ
(Sounack et al., 12 Jun 2025)	“PT-style” continued-pretrained base checkpoint	Biomedical and clinical adaptation
(Lee et al., 4 Apr 2025)	ModernBERT-base-initialized clinical encoder	Clinical continued pretraining
(Laitz et al., 21 Jun 2026)	Portuguese adaptation of ModernBERT-base	Continued pretraining for Portuguese

This multiplicity of usage matters for model comparison. A statement about ModernBERT-base-PT in a patent paper is not automatically a statement about the original English base checkpoint, the French controlled-pretraining variants, or the biomedical and Portuguese derivatives.

2. General-domain ModernBERT base backbone

The original ModernBERT base model is presented as a modernized BERT-style encoder for classification, dense retrieval, multi-vector retrieval, long-context text processing, and code-related retrieval tasks (Warner et al., 2024). Its base configuration has 22 layers, 149M parameters, hidden size 768, GLU expansion 2304, 12 attention heads, and vocabulary size 50,368. The model is trained with MLM, removes Next Sentence Prediction, uses a 30% masking rate, and is trained in multiple phases: 1.719 trillion tokens in the main pretraining phase at maximum sequence length 1024, followed by an extension to 8192 tokens using an additional 250B tokens at constant learning rate $3\times 10^{-4}$ and then 50B tokens in a decay phase using a $1$-sqrt schedule (Warner et al., 2024).

Architecturally, the model updates older BERT/RoBERTa-era encoders with RoPE, GeGLU, pre-norm LayerNorm, no bias terms in most linear layers and in LayerNorm, and an alternating attention pattern in which every third layer uses global attention and the other layers use local sliding-window attention with window size 128 (Warner et al., 2024). The implementation combines FlashAttention 3 for global attention, FlashAttention 2 for local attention, unpadding, greedy sequence packing with $>99\%$ packing efficiency, and torch.compile, which is reported to produce around a 10% throughput improvement with negligible overhead. The design is explicitly hardware-aware and targets a “basket” of common GPUs including T4, A10, L4, RTX 3090, RTX 4090, A100, and H100.

The reported downstream profile of the original base checkpoint is correspondingly broad. ModernBERT-base achieves 88.4 average on GLUE, 41.6 on BEIR DPR average, and 51.3 on BEIR ColBERT average, and the paper states that it is the strongest overall model at both base and large sizes while also being the most memory-efficient base model overall (Warner et al., 2024). The same paper reports that on long context, ModernBERT processes documents 2.65× and 3× faster than the next-fastest encoder at base and large sizes, respectively.

3. Patent-domain pretraining: the explicit “ModernBERT-base-PT” model

In patent NLP, ModernBERT-base-PT is the paper’s main patent-domain pretrained encoder: a ModernBERT-based masked LLM trained from scratch on a large, cleaned corpus of patent text rather than on general web text (Yousefiramandi et al., 18 Sep 2025). The “-PT” suffix indicates the Phase 2 clean patent corpus variant, in contrast to the noisier raw “-VX” variant. The model is framed as a domain-adapted alternative to the general-purpose ModernBERT baseline, while retaining the same general ModernBERT efficiency-oriented architecture.

The pretraining corpus is built from roughly 100M patent documents, with an average length of about 700 words, drawn from original published applications plus proprietary Derwent World Patents Index material (Yousefiramandi et al., 18 Sep 2025). The final training corpus for ModernBERT-base-PT contains about 64M unique patents, totaling about 30.8B tokens, and the cleaned phase-2 corpus used for “-PT” contains approximately 31.6B tokens. The held-out test set contains 3.4M patents, or about 1.6B tokens. The text pipeline extracts the abstract and the first independent claim from each patent and appends DWPI titles, abstracts, and claims by matching publication numbers; it removes boilerplate, figure references, non-English characters, and retains only the first independent claim. Phase 1 performs extraction, normalization, and family-level deduplication, while Phase 2 adds English language identification, quality and repetition filtering, FineWeb-style heuristics, and MinHash-based near-duplicate removal. The Phase 2 filtering reduces the token count from about 47.8B to 31.6B, a 33.79% reduction.

The training recipe uses MLM only and explicitly omits Next Sentence Prediction because it has been found to contribute little (Yousefiramandi et al., 18 Sep 2025). The MLM masking rate is 30%. The schedule uses StableAdamW, a warmup-then-decay schedule, a peak learning rate of $3\times10^{-4}$ , 6% warmup, a global batch size of 4096 sequences, and maximum training sequence length 1024. The hyperparameter appendix specifies StableAdamW with betas $(0.90, 0.98)$ , $\varepsilon=10^{-6}$ , 30% MLM masking, and a 256-token sliding window for FlashAttention. The model has about 149M parameters, trains for 16 epochs on 8×H100 GPUs, and finishes in roughly 2.352 days.

The patent paper describes this checkpoint as using “FlexBERT-base” from the ModernBERT repository, with engineering choices inherited from MosaicBERT-like optimizations (Yousefiramandi et al., 18 Sep 2025). For the base configuration, it states 22 encoder blocks, hidden size 768, and 12 attention heads, with GELU activations and pre-LayerNorm. It further describes FlashAttention in self-attention, ALiBi positional bias rather than learned absolute positional embeddings, GLU-style feed-forward layers, residual scaling by 0.5, removal of dropout without harming stability, and use of the original ModernBERT tokenizer rather than a custom BPE tokenizer. This paper-specific description is narrower than the general-family account in the original ModernBERT release and is tied to the patent-pretraining implementation.

4. Patent classification, efficiency, and practical trade-offs

On downstream patent classification, ModernBERT-base-PT is evaluated on WIPO, WIPOEC, HUPD, and DatasetCLV, with mean ± standard deviation F1 across four fine-tuning runs (Yousefiramandi et al., 18 Sep 2025). The reported scores are 0.802 ± 0.011 on WIPO, 0.814 ± 0.004 on WIPOEC, 0.782 ± 0.003 on HUPD, and 0.843 ± 0.009 on DatasetCLV. Relative to the general-purpose ModernBERT baseline, it slightly underperforms on WIPO, where the baseline reaches 0.806, but improves on WIPOEC, HUPD, and DatasetCLV, where the baseline scores 0.786, 0.773, and 0.822, respectively. The central empirical claim is therefore narrow but clear: patent-domain pretraining helps on 3 of 4 patent-specific downstream datasets.

Relative to PatentBERT, the comparison is mixed rather than uniformly favorable (Yousefiramandi et al., 18 Sep 2025). ModernBERT-base-PT slightly trails PatentBERT on HUPD, 0.782 versus 0.810, which the authors attribute to PatentBERT having been pretrained on USPTO data. It is comparable or better on WIPO, 0.802 versus 0.801, and WIPOEC, 0.814 versus 0.807, but slightly lower on DatasetCLV, 0.843 versus 0.854. The paper’s stated balance is therefore one of competitiveness plus efficiency rather than across-the-board superiority.

Efficiency is one of the strongest empirical arguments for the patent model (Yousefiramandi et al., 18 Sep 2025). In inference, ModernBERT-base-PT reports throughput of 83.68 samples/sec on WIPO, 85.54 on WIPOEC, 368.42 on HUPD, and 90.99 on DatasetCLV. In training, the corresponding throughputs are 21.11, 21.48, 24.98, and 23.33 samples/sec. At the family level, the paper states that the FlashAttention-based ModernBERT models achieve 3.55× higher inference throughput and 2.32× higher training throughput than PatentBERT on average, while also being much smaller in parameter count: 149M parameters for ModernBERT-base-PT versus 346M for PatentBERT. The practical trade-off is described as minimal: a small amount of training-speed loss relative to the general-purpose ModernBERT baseline, but better downstream patent performance on most datasets while remaining far faster than PatentBERT.

5. Comparative evaluations and disputed sources of improvement

A central controversy around ModernBERT concerns whether its reported gains arise from architecture, training data, or both. A controlled French study addresses this directly by pretraining ModernBERT on the same dataset as CamemBERTaV2, a DeBERTaV3 French model, and concludes that the previous model generation remains superior in sample efficiency and overall benchmark performance, with ModernBERT’s primary advantage being faster training and inference speed (Antoun et al., 11 Apr 2025). In that setup, ModernBERT-CV2 is trained with MLM, max sequence length 1024, 30% dynamic token masking, the same tokenizer as CamemBERTaV2, 48 NVIDIA H100 80GB GPUs, PyTorch FSDP full sharding, bfloat16 mixed precision, and a Warmup-Stable-Decay learning-rate schedule. On downstream French tasks, CamemBERTaV2 beats ModernBERT-CV2 on NER F1, QA F1, QA EM, CLS accuracy, PAWS-X accuracy, and XNLI accuracy. The same paper reports that high-quality pretraining data accelerates convergence but does not significantly improve final performance, and also notes that ModernBERT variants were more sensitive to learning-rate choices and sometimes failed to converge on FQuAD.

Another comparison comes from NeoBERT, which positions itself as a next-generation encoder and treats ModernBERT-base as a key modern baseline (Breton et al., 26 Feb 2025). In that paper’s architectural table, ModernBERT-base is listed as 22 layers, 768 hidden, 12 heads, and 149M parameters, with GeGLU, RoPE, and Pre-LayerNorm. Under the authors’ uniform evaluation framework, ModernBERT-base scores 88.5 average on GLUE development, while NeoBERT $_{1024}$ scores 88.8 and NeoBERT $_{4096}$ scores 89.0. On MTEB after 2,000 steps of standardized contrastive fine-tuning, ModernBERT-base scores 45.0 in the base size class, while NeoBERT $_{4096}$ reaches 51.3. The same appendix reports that at 4,096 tokens NeoBERT processes 17.2k tokens/sec versus ModernBERT-base’s 11.7k, a 46.7% speedup. These results do not negate the original ModernBERT claim of strong efficiency relative to older encoders; they instead show that, by 2025, ModernBERT itself had become a baseline against which newer encoder designs were measured.

Taken together, these comparisons support a narrower characterization of ModernBERT-base-PT than some early family-level claims might imply. Under matched or newly standardized evaluation regimes, its main consistently documented advantages are speed, memory efficiency, and long-context capability, while downstream quality can be surpassed by DeBERTaV3-family models in controlled-data comparisons or by newer encoders such as NeoBERT under identical fine-tuning conditions (Antoun et al., 11 Apr 2025).

6. Continued-pretraining derivatives and the broader role of the base checkpoint

Later work uses the base ModernBERT checkpoint as a reusable initialization for domain and language adaptation. In biomedical and clinical NLP, BioClinical ModernBERT is developed through continued pretraining on 53.5B tokens, consisting of 50.7B biomedical tokens from PubMed abstracts plus PMC full-text articles and 2.8B clinical tokens from 20 clinical datasets (Sounack et al., 12 Jun 2025). The paper releases base and large versions with 150M and 396M parameters, respectively, and emphasizes retention of ModernBERT’s long-context and efficiency profile, including support for up to 8,192 tokens, alternating attention, and unpadding. The base model reports 89.9 on ChemProt, 58.1 on Phenotype, 95.1 on COS, 58.5 on Social History, and 82.7 on DEID.

A related but distinct biomedical paper, “Clinical ModernBERT,” explicitly initializes from the ModernBERT-base checkpoint and retains RoPE, Flash Attention, GeGLU feed-forward activations, bias-free linear layers, 8,192-token context length, and a BPE tokenizer initialized from ModernBERT-base with additional clinical tokens (Lee et al., 4 Apr 2025). Its pretraining corpus combines ~40 million PubMed abstracts, MIMIC-IV discharge summaries and radiology reports, and structured medical ontologies, for a final corpus of 13 billion tokens. The training setup uses a custom data collator with token-aware masking, a masking rate starting at 30% and linearly decaying to 15%, 150,000 steps, StableAdamW, a cosine learning-rate schedule, mixed precision, and NVIDIA A100 GPUs. The paper reports that removing token-aware masking sharply reduces MLM top-k accuracy, and it positions the model as especially strong for clinical retrieval and EHR classification.

Cross-lingual adaptation follows the same general pattern. moBERTo is a Portuguese adaptation of ModernBERT-base obtained through continued pretraining on 60 billion tokens, corresponding to 5 epochs over a 12-billion-token corpus curated from FineWeb2 and filtered with educational and STEM classifiers (Laitz et al., 21 Jun 2026). The model preserves the original architecture, including rotary positional embeddings, alternating local-global attention, flash attention, and unpadding. Its best variant, moBERTo-SWM-8k, combines a Portuguese tokenizer with subword-matching embedding transfer and long-context post-training, and achieves the highest average reranking nDCG@10 across QUATI, mMARCO-PT, and Robust04-PT, with an average of 0.5255, as well as the best result on PLUE-PT, 0.6980. The same paper states that continued pretraining is strongly preferable to training from scratch, particularly for preserving long-context capabilities.

This derivative literature suggests that the enduring significance of ModernBERT-base-PT is not limited to a single released checkpoint. A plausible implication is that the base ModernBERT checkpoint functions as a reusable encoder substrate: a compact, long-context, efficiency-oriented starting point that can be specialized for patents, French, biomedical and clinical text, or Portuguese while preserving much of the deployment profile that made the original family attractive. At the same time, comparative studies show that this substrate should not be conflated with guaranteed state-of-the-art downstream accuracy under all matched-data conditions.