ModernBERT-base-VX: Efficient Long-Context Transformer

Updated 12 July 2026

ModernBERT-base-VX is a variant of the base ModernBERT architecture that incorporates long-context support, rotary positional embeddings, and alternating local/global attention.
The model retains a core configuration of 149M parameters with 22 layers, GLU-based feed-forward blocks, and optimized tokenization strategies tailored to diverse applications.
Empirical evaluations demonstrate improvements in inference throughput and retrieval accuracy across domains such as patent, biomedical, and multilingual NLP tasks.

ModernBERT-base-VX can be understood as a base-sized checkpoint within the ModernBERT encoder family: an encoder-only, bidirectional Transformer that preserves the operational role of BERT-base while incorporating long-context support, rotary positional embeddings, alternating local/global attention, GLU-based feed-forward layers, and hardware-aware implementation choices. The suffix “VX” is not defined in the foundational ModernBERT paper, but later work uses effectively base-scale ModernBERT variants—sometimes explicitly, as in patent pretraining, and sometimes implicitly, as in continued-pretraining studies for Portuguese and biomedical NLP—so the label is best read as a versioned or domain-adapted ModernBERT-base derivative rather than a single canonical checkpoint (Warner et al., 2024, Yousefiramandi et al., 18 Sep 2025, Laitz et al., 21 Jun 2026, Sounack et al., 12 Jun 2025).

1. Family definition and model identity

In the foundational ModernBERT release, the base model is specified as a 149M-parameter encoder with 22 Transformer blocks, hidden size 768, 12 attention heads, GLU expansion dimension 2304, a 50,368-token vocabulary, and a native maximum sequence length of 8,192 tokens (Warner et al., 2024). This base configuration is deeper than classic BERT-base while remaining within the same deployment tier as base encoders used for retrieval, classification, and token labeling.

Several later studies preserve that base-scale identity. In patent NLP, ModernBERT-base-VX is described as using the FlexBERT-base architecture with 22 encoder layers, hidden size 768, 12 heads, and roughly 149M parameters, while training in that study is capped at 1,024 tokens rather than the architecture’s longer native limit (Yousefiramandi et al., 18 Sep 2025). In Portuguese adaptation, moBERTo is obtained through continued pretraining of answerdotai/ModernBERT-base and explicitly preserves the original architecture, including rotary positional embeddings, alternating local-global attention, flash attention, and unpadding, again at 149M parameters (Laitz et al., 21 Jun 2026). BioClinical ModernBERT similarly releases a base model at 150M parameters and a large model at 396M parameters, preserving the ModernBERT design while specializing the weights to biomedical and clinical text (Sounack et al., 12 Jun 2025).

This pattern makes “ModernBERT-base-VX” best understood as a family label for base-regime ModernBERT checkpoints that retain the same encoder substrate while varying in corpus, tokenizer, domain specialization, or post-training recipe. A plausible implication is that the architectural core is more stable across variants than the naming convention itself.

2. Core architecture and systems design

ModernBERT-base remains a Transformer encoder. At the level of self-attention, the family keeps the standard multi-head form

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$

while altering how positional information, sparsity, and memory behavior are realized (Yamagishi et al., 7 Mar 2025, Warner et al., 2024). The foundational model uses Pre-LayerNorm blocks, GeGLU feed-forward layers, and bias-free linear layers, with LayerNorm configured at $\epsilon = 10^{-5}$ and no bias (Warner et al., 2024). In the original base model, the feed-forward block is explicitly gated rather than using the older BERT-style dense-GELU-dense stack.

Positional encoding is handled through RoPE rather than absolute position embeddings. The base model uses a dual setting: global-attention layers use $\theta = 160{,}000$ , while local-attention layers use $\theta = 10{,}000$ (Warner et al., 2024). The attention pattern itself alternates between global and local layers, with every third layer using global attention and the remaining layers using local sliding-window attention with window size 128 tokens. This design is paired with FlashAttention-3 for global attention and FlashAttention-2 for local attention, and with whole-model unpadding so that padding tokens are removed before the embedding layer and computations proceed on a packed jagged sequence (Warner et al., 2024).

The same architectural package is preserved in moBERTo, where continued pretraining keeps rotary positional embeddings, alternating local-global attention, flash attention, and unpadding intact (Laitz et al., 21 Jun 2026). NorBERTo, a Portuguese model trained from scratch in the same family, reproduces the base-like recipe with 22 layers, hidden size 768, 12 heads, GLU expansion size 2,304, context length 8,192, and a 50,368-token vocabulary (Silva et al., 30 Apr 2026). French ModernBERT likewise uses 22 layers, hidden size 768, GeGLU with expansion to 2,304 dimensions, native sequence length 8,192, RoPE, and alternating local-global attention, although with a much larger tokenizer vocabulary of 129K tokens (Estève et al., 25 Feb 2026). Across these studies, the architectural signature is consistent: deep-and-narrow encoder stacks, RoPE, gated MLPs, and systems-level optimization for long inputs.

3. Tokenization, vocabulary, and long-context handling

Tokenizer design is a first-order variable in ModernBERT-family practice. In Japanese clinical NLP, a ModernBERT model with a 102,400-word vocabulary, explicitly including English, produces substantially more compact segmentations than a conventional Japanese BERT-base with vocabulary size 32,768. On 150 chest CT reports, ModernBERT averages 258.1 tokens per document versus 339.6 for BERT-base, a reduction of 24.0%, with all 150 reports yielding fewer tokens under ModernBERT and a Wilcoxon signed-rank test giving $p < 0.01$ (Yamagishi et al., 7 Mar 2025). The paper’s examples are morphologically concrete: “リンパ節” is one token in ModernBERT and two in BERT; “認められませんでした” becomes two tokens in ModernBERT and six in BERT; and “crazy paving appearance” is segmented into four tokens in ModernBERT and ten in BERT (Yamagishi et al., 7 Mar 2025).

Other variants explore the opposite choice: keeping the original tokenizer to preserve long-context behavior. In the patent study, ModernBERT-base-VX and ModernBERT-base-PT both use ModernBERT’s original public tokenizer, while tokenizer customization is reserved for a larger MosaicBERT variant (Yousefiramandi et al., 18 Sep 2025). In Portuguese, moBERTo shows the trade-off explicitly. Using the original ModernBERT tokenizer yields the best long-context retrieval, with MLDR@8192 of 0.6166, while tokenizer adaptation via a Portuguese tokenizer plus subword-matching embedding transfer improves PLUE-PT and LeNER-Br but lowers long-context retrieval relative to the original-tokenizer variant (Laitz et al., 21 Jun 2026). The ablation is especially sharp for the tokenizer-only baseline without embedding transfer: long-context retrieval degrades much more than in the subword-matching case (Laitz et al., 21 Jun 2026).

Tokenizer engineering is also a primary design axis in larger family members. Chinese ModernBERT uses a 32k BPE vocabulary tailored to frequent Chinese affixes and productive compounds, with chars/token reported as 1.41 in the 8k bucket and 1.35 in the 512 bucket, and with embedding parameters comprising only 9.0% of total parameters in a 377M-parameter model (Zhao et al., 14 Oct 2025). Although that configuration is not base-sized, it reinforces a family-wide principle: ModernBERT tokenization is treated as part of systems optimization, not merely lexical preprocessing.

Long context is equally central. The original ModernBERT-base is natively trained to 8,192 tokens, and later variants either preserve that capability directly or adapt back to it through post-training (Warner et al., 2024, Laitz et al., 21 Jun 2026). Patent models are trained at 1,024 tokens in the reported experiments, but the underlying architecture still derives from the long-context ModernBERT base (Yousefiramandi et al., 18 Sep 2025). Clinical and BioClinical ModernBERT both retain the 8,192-token setting for medical documents (Lee et al., 4 Apr 2025, Sounack et al., 12 Jun 2025).

4. Pretraining and continual adaptation regimes

The foundational ModernBERT-base is trained on approximately 2 trillion tokens. Its training is staged: 1.719T tokens at sequence length 1,024; then 250B tokens at sequence length 8,192 with constant learning rate $3 \times 10^{-4}$ ; then 50B tokens with a $1-\sqrt{\cdot}$ decay schedule, after a main short-context phase that uses learning rate $8 \times 10^{-4}$ with 3B warmup tokens (Warner et al., 2024). The optimizer is StableAdamW with betas $(0.90, 0.98)$ , $\epsilon = 10^{-6}$ , weight decay $\epsilon = 10^{-5}$ 0, and a masking rate of 30% (Warner et al., 2024).

Most derivatives keep MLM as the core objective while altering the corpus and, sometimes, the masking schedule. In patents, ModernBERT-base-VX is pretrained on a larger but noisier patent corpus of about 47.79B tokens, using StableAdamW, learning rate $\epsilon = 10^{-5}$ 1, 30% masking, global batch size 4096, maximum sequence length 1024, and 16 epochs, completing in 3.264 days on 8×H100 GPUs (Yousefiramandi et al., 18 Sep 2025). BioClinical ModernBERT performs continued pretraining on over 53.5B biomedical and clinical tokens, using a two-phase regimen: a joint biomedical+clinical phase and a clinical-only phase; the base model uses batch size 72 and starts from the ModernBERT pre-decay checkpoint with learning rate $\epsilon = 10^{-5}$ 2 (Sounack et al., 12 Jun 2025). Clinical ModernBERT adapts ModernBERT-base on 13B tokens from PubMed abstracts, MIMIC-IV notes, and medical ontologies, with token-aware masking and a masking schedule that decays from 30% to 15% over training (Lee et al., 4 Apr 2025).

Continued pretraining rather than training from scratch is a recurrent result. moBERTo continues pretraining of ModernBERT-base on 60B Portuguese tokens and shows that continued pretraining is strongly preferable to training from scratch, particularly for preserving long-context capabilities (Laitz et al., 21 Jun 2026). In the French diversity study, the strongest message is about data selection rather than architecture: a model pretrained for 483 hours on a diversity-driven dataset of 150M tokens can yield commensurate performance to a model pretrained for 1,775 hours on a randomly-driven dataset of 2.4B tokens, and diversity-driven sampling can produce gains of 10 points relative to randomly sampled data of the same size on some tasks (Estève et al., 25 Feb 2026). This suggests that for ModernBERT-base-VX variants, data curation and continuation strategy can be at least as consequential as nominal model size.

Domain-specialized base-scale ModernBERT variants also appear in cybersecurity. SecureBERT 2.0 uses a ModernBERT-based encoder backbone with 1,024-token sequences, AdamW, 20 epochs, MLM probability 0.10, and a domain corpus containing 13,623,037,185 text tokens and 53,387,658 code tokens (Aghaei et al., 30 Sep 2025). Although the paper does not publish the exact base configuration, it explicitly frames the backbone as a base-scale ModernBERT-style encoder specialized to long, heterogeneous security documents (Aghaei et al., 30 Sep 2025).

5. Empirical behavior across tasks and domains

Across published evaluations, ModernBERT-base-style models combine strong discriminative performance with unusually favorable throughput. In the original release, ModernBERT-base records 88.4 average on GLUE, 41.6 on BEIR DPR, 51.3 on BEIR ColBERT, 44.0 on MLDR in-domain DPR, 80.2 on MLDR out-of-domain ColBERT, 56.4 on CodeSearchNet, and 73.6 on StackOverflow QA (Warner et al., 2024). At 8,192 tokens on an RTX 4090, the same model reaches 123.7k fixed-length tokens/s and 133.8k variable-length tokens/s, with maximum batch size 98, substantially above other long-context base encoders in the reported comparison (Warner et al., 2024).

Representative applications illustrate how those properties transfer under specialization.

Setting	Model form	Selected reported results
General encoder benchmarks	ModernBERT-base	GLUE 88.4; BEIR DPR 41.6; BEIR ColBERT 51.3; MLDR_OOD ColBERT 80.2 (Warner et al., 2024)
Japanese chest CT reports	ModernBERT vs Japanese BERT-base	24.0% fewer tokens/document; 39% shorter training; exact match 74.67% vs 72.67%, $\epsilon = 10^{-5}$ 3 (Yamagishi et al., 7 Mar 2025)
Patent classification	ModernBERT-base-VX	WIPO 0.796; WIPOEC 0.801; HUPD 0.776; DatasetCLV 0.852; >3× PatentBERT inference throughput for ModernBERT variants (Yousefiramandi et al., 18 Sep 2025)
Biomedical RAG retrieval	ModernBERT-base + ColBERTv2	Recall@3 improved by up to 4.2 percentage points; MIRAGE average accuracy 0.4448 (Rivera et al., 6 Oct 2025)
Cross-encoder reranking	`answerdotai/ModernBERT-base` + Lion	NDCG@10 0.7225; MAP 0.5121; MRR@10 0.5988 (Kumar et al., 23 Jun 2025)

In Japanese radiology, the efficiency argument is unusually clean because the task is matched against a conventional Japanese BERT-base under identical fine-tuning hyperparameters. ModernBERT processes 38.82 samples/s during training and 139.90 samples/s during inference, compared with 23.58 and 84.18 for BERT-base, while remaining statistically indistinguishable in exact-match accuracy across 18 chest CT labels (Yamagishi et al., 7 Mar 2025). In biomedical retrieval-augmented generation, a 149M-parameter ModernBERT retriever indexes passages in 0.80 ms/passage, versus 6.16 for MedCPT and 39.31 for MedGemma, and when paired with a jointly aligned ColBERTv2 re-ranker reaches a MIRAGE average accuracy of 0.4448, slightly above MedCPT’s 0.4436 (Rivera et al., 6 Oct 2025).

Further domain results reinforce the same pattern. SecureBERT 2.0 reports document embedding R@1 of 88.83% and mAP of 92.39% in the bi-encoder setting, NER F1 0.945, and vulnerability-detection F1 0.616, all using a ModernBERT-based cybersecurity encoder (Aghaei et al., 30 Sep 2025). In Portuguese, moBERTo-SWM-8k achieves the highest average reranking nDCG@10 across three Portuguese retrieval benchmarks and the best PLUE-PT result in that study; concrete scores include reranking average 0.5255, PLUE-PT 0.6980, and LeNER-Br 0.8726, while the original-tokenizer variant remains the strongest long-context retriever with MLDR@8192 of 0.6166 (Laitz et al., 21 Jun 2026). BioClinical ModernBERT-base reaches 58.1 F1 on Phenotype and 82.7 F1 on DEID, while BioClinical ModernBERT-large improves those to 60.8 and 83.8 respectively, and the base model maintains roughly 71.1 kTok/s at 8,192 tokens (Sounack et al., 12 Jun 2025).

6. Comparative assessments, limitations, and outlook

ModernBERT-base-VX is not uniformly dominant across encoder research. A controlled French study that pretrains ModernBERT on the same data as a DeBERTaV3-based CamemBERTaV2 concludes that the previous model generation remains superior in sample efficiency and overall benchmark performance, while ModernBERT’s primary advantage is faster training and inference speed (Antoun et al., 11 Apr 2025). Under that matched-data setup, ModernBERT-CV2 consistently outperforms RoBERTa-style CamemBERTv2 but trails CamemBERTaV2 on NER, QA, and classification benchmarks (Antoun et al., 11 Apr 2025). NeoBERT pushes the comparison further: under identical contrastive fine-tuning on MTEB, NeoBERT $\epsilon = 10^{-5}$ 4 scores 51.3 while ModernBERT-base scores 45.0, and NeoBERT reports 17.2k tokens/s at 4,096 tokens versus 11.7k for ModernBERT-base, a 46.7% speedup in that setting (Breton et al., 26 Feb 2025).

These comparisons matter because they identify where ModernBERT’s gains originate. The original ModernBERT paper frames the model as a Pareto improvement over older encoders, and its strongest empirical case is indeed against BERT-base, RoBERTa-base, and other long-context base models in retrieval, code search, and long-sequence efficiency (Warner et al., 2024). The French comparison shows that once data are controlled tightly against DeBERTaV3, the picture becomes more architecture-specific and less universally favorable (Antoun et al., 11 Apr 2025). The NeoBERT comparison adds a further challenge by showing that a newer encoder can exceed ModernBERT on both MTEB and long-sequence throughput under its own design choices (Breton et al., 26 Feb 2025). This suggests that ModernBERT-base-VX should be regarded less as a final endpoint than as a highly competitive design point in the current encoder landscape.

The domain studies also impose clear limits on generalization. The Japanese chest CT study is restricted to a single translated dataset with only 150 test reports and single-institution conditions (Yamagishi et al., 7 Mar 2025). Patent ModernBERT-base-VX is English-only and evaluated only on four classification tasks (Yousefiramandi et al., 18 Sep 2025). In BioClinical ModernBERT, despite using 20 clinical datasets, MIMIC-III and MIMIC-IV still account for more than 95% of clinical tokens (Sounack et al., 12 Jun 2025). moBERTo shows that tokenizer adaptation can improve token-level tasks while degrading long-context retrieval, so there is no universally optimal tokenizer strategy even within the same architecture (Laitz et al., 21 Jun 2026).

A balanced reading is therefore straightforward. ModernBERT-base-VX denotes a base-scale ModernBERT checkpoint or derivative whose defining properties are long-context capability, hardware-aware attention, and efficient encoder inference. In many applied settings—clinical NLP, patents, retrieval, reranking, and multilingual adaptation—those properties translate into large practical gains without requiring decoder-scale parameter counts (Warner et al., 2024, Yamagishi et al., 7 Mar 2025, Yousefiramandi et al., 18 Sep 2025, Rivera et al., 6 Oct 2025). At the same time, comparative work indicates that absolute benchmark leadership remains contingent on the interaction between architecture, tokenizer, corpus, and training objective rather than on the ModernBERT label alone (Antoun et al., 11 Apr 2025, Breton et al., 26 Feb 2025).