GatorTron-Base: Clinical NLP Transformer

• GatorTron-Base is a BERT-style clinical NLP model with 345M parameters trained on over 90B words from diverse medical sources, combining domain-specific precision with scalable architecture.
• It employs a 24-layer transformer encoder with 1,024 hidden dimensions and 16 self-attention heads, achieving strong performance on clinical NER, MRE, STS, NLI, and MQA benchmarks.
• The model demonstrates practical utility in predictive ICU tasks with an 80.5% recall, offering efficient real-time inference suitable for resource-constrained clinical environments.

GatorTron-Base is a BERT-style large clinical LLM consisting of 345 million parameters, designed and trained specifically for medical NLP tasks on unstructured electronic health records (EHRs). Developed as part of the GatorTron model suite, it represents an effort to bridge the gap between smaller domain-specific models and the recent trend toward billion-parameter models trained on general-domain corpora. GatorTron-Base is optimized for both scale and domain specificity, leveraging over 90 billion words of mixed-domain text (predominantly de-identified clinical notes) and employing standard transformer training objectives and architectural conventions (Yang et al., 2022).

1. Model Architecture and Pretraining Objectives

GatorTron-Base utilizes a 24-layer transformer encoder architecture, following the standard BERT paradigm. Each transformer block uses a hidden size of 1,024 and 16 self-attention heads per layer, with a total parameter count of 345 million. The vocabulary is constructed from scratch via byte-pair encoding over the entire pretrained corpus; the precise vocabulary size is not reported. The architecture intentionally matches the design patterns seen in large general-domain transformers, enabling direct comparison and model scaling within the clinical domain. The GatorTron model family also includes GatorTron-Medium (3.9B parameters, 48 layers, hidden size 2,560, 40 heads) and GatorTron-Large (8.9B, 56 layers, hidden size 3,584, 56 heads), differing only in depth and width and sharing the same overall model structure (Yang et al., 2022).

Pretraining is conducted from scratch over a corpus containing more than 90 billion words, with >82 billion from de-identified UF Health clinical notes (2011–2021), 6 billion from PubMed (abstracts and full-text), 2.5 billion from English Wikipedia, and 0.5 billion from MIMIC-III clinical notes. The training objective is a composite of masked language modeling (MLM), masking and predicting 15% of the tokens per sequence, and sentence-order prediction (SOP), employing binary classification to determine whether two segments are correctly ordered. No further domain-adaptive pretraining phase is performed (Yang et al., 2022).

2. Training Infrastructure and Convergence

Training of GatorTron-Base is conducted on the UF HiPerGator-AI cluster using the Megatron-LM framework with both data and model parallelism. Hardware includes NVIDIA DGX SuperPOD infrastructure, utilizing A100 80GB GPUs, with scaling up to 992 GPUs across 124 nodes for the larger models in the suite. The GatorTron-Base model converges in approximately 10 epochs, as detected by the plateau of training and validation MLM loss. Medium and Large variants require fewer epochs (~7) due to their higher model capacity. Hyperparameters such as optimizer choice, learning rate schedule, batch size, and maximum sequence length are not disclosed (Yang et al., 2022).

3. Benchmarking on Clinical NLP Tasks

The model is evaluated on five well-established clinical NLP tasks, demonstrating state-of-the-art or highly competitive performance in most domains. Fine-tuning is conducted on standard benchmark datasets:

• Clinical Concept Extraction (NER): F1 = 0.8893 on 2010 i2b2, 0.7922 on 2012 i2b2, 0.8896 on 2018 n2c2.
• Medical Relation Extraction (MRE): F1 = 0.9599 on 2018 n2c2 drug–adverse events.
• Semantic Textual Similarity (STS): Pearson r = 0.8810 on the 2019 n2c2/OHNLP track.
• Natural Language Inference (NLI): Accuracy = 0.8670 on MedNLI.
• Medical Question Answering (MQA): Exact Match = 0.2978, F1 = 0.9543 on emrQA subsets.

Scaling experiments indicate monotonic performance improvements across NER, MRE, NLI, and MQA tasks with increased parameter count, while STS peaks at the Medium size (r=0.8903) and slightly declines for the Large model (Yang et al., 2022).

4. Application to Predictive ICU Tasks

GatorTron-Base has been benchmarked in predictive clinical settings, notably for Shock Index prediction in ICU patients using the MIMIC III corpus. In this context, the model is used as a black-box embedding provider, subjected to extensive text preprocessing (lowercasing, stopword/punctuation removal, vocabulary review, masking of explicit shock-related terms) and combined with downstream classifiers (Random Forest, XGBoost, Logistic Regression, etc.) to predict adverse hemodynamic events (Malhotra et al., 23 Dec 2025).

With minimal or no model fine-tuning, GatorTron-Base achieves a weighted recall of 80.5%, outperforming other LLMs (Llama 8B, Mistral 7B) and classic SLMs (BioBERT, Word2Vec) on this metric. However, overall metrics (precision, F1, AUROC) are comparable across all embedding models, indicating that large clinical LLMs do not inherently provide superior predictive value for future clinical events unless specifically trained or adapted for longitudinal forecasting. Fine-tuning with focal loss marginally improved recall but produced no significant F1 or AUROC gains (Malhotra et al., 23 Dec 2025).

5. Practical Deployment Considerations

The parameter count and architectural depth of GatorTron-Base position it as a trade-off point between memory/performance and resource requirements. With 345 million parameters, it incurs a substantially lower memory and inference latency penalty than its billion-parameter counterparts, rendering it suitable for near real-time EHR pipelines and deployment in resource-constrained environments (Yang et al., 2022). Inference speed per note is not explicitly reported, but practitioners can expect sub-second processing on modern GPU hardware. Intended use cases include high-accuracy adverse event extraction, computable phenotyping, pharmacovigilance, clinical decision support, and advanced document- or sentence-level inference tasks.

6. Limitations and Future Prospects

Empirical findings suggest that, while larger parameter counts and axial scaling improve canonical clinical NLP benchmarks, they do not automatically translate into real-world predictive gains on complex, longitudinal, or future-oriented clinical inference tasks. Achieving meaningful lifts in trajectory-aware applications likely requires pretraining and fine-tuning on large, longitudinal patient timelines and exploration of new architectures or objectives tailored for such settings (Malhotra et al., 23 Dec 2025). A plausible implication is that research emphasis should shift towards sequence modeling over patient timelines, alternative pretraining objectives, and increasing cohort scale for task-specific fine-tuning to stabilize parameter updates and generalization.

7. Summary Table: GatorTron-Base Architectural and Benchmark Highlights

Attribute	Value / Description	Source
Parameter count	345 million	(Yang et al., 2022)
Depth / layers	24 Transformer encoder layers	(Yang et al., 2022)
Hidden dimension	1,024 per layer	(Yang et al., 2022)
Attention heads	16 per layer	(Yang et al., 2022)
Pretraining data	>82B clinical, 6B PubMed, 2.5B Wikipedia, 0.5B MIMIC-III	(Yang et al., 2022)
Primary MLM loss	$$L(\theta) = - \sum_{i \in M} \log P(x_i | x_{-M}; \theta)$$	(Yang et al., 2022)
Clinical NER F1	0.8893 (2010 i2b2) / 0.8896 (2018 n2c2)	(Yang et al., 2022)
Clinical MRE F1	0.9599 (2018 n2c2)	(Yang et al., 2022)
ICU shock prediction recall	0.805 (ICU, Random Forest classifier)	(Malhotra et al., 23 Dec 2025)

The GatorTron-Base model stands as a reference BERT-scale transformer for clinical NLP, achieving state-of-the-art results on benchmark tasks and demonstrating solid recall performance on complex clinical prediction challenges, albeit with limitations in overall predictive superiority absent further longitudinal task-specific adaptation (Yang et al., 2022, Malhotra et al., 23 Dec 2025).