RoBERTa-Large: Optimized Transformer

• RoBERTa-Large is a transformer-based language model with 24 layers and approximately 355 million parameters that improves BERT pretraining by eliminating NSP and applying dynamic masking.
• It is pretrained on a 160GB diverse English corpus using a 50,000-unit byte-level BPE tokenizer, which enhances its robustness in masked language modeling tasks.
• Empirical evaluations on benchmarks like GLUE and SQuAD demonstrate its superior performance and adaptability to both general-purpose and language-specific applications.

RoBERTa-Large is a transformer-based LLM and a direct extension of the BERT-Large architecture, designed for robust and optimized large-scale pretraining in natural language processing. First introduced by Liu et al., RoBERTa-Large adopts the same 24-layer, 1024-dimensional hidden size architecture as BERT-Large but refines several critical aspects of pretraining, including removal of the Next Sentence Prediction objective, use of dynamic masking, expansion of pretraining corpora, and optimization of batch and learning rate schedules. These changes yield significant empirical gains—including state-of-the-art performance across a suite of downstream tasks—and have established RoBERTa-Large as a standard backbone for both general-purpose and specialized encoder models (Liu et al., 2019).

1. Model Architecture and Hyperparameters

RoBERTa-Large comprises a stack of 24 transformer encoder blocks, each with a hidden state dimensionality of 1024, a feed-forward bottleneck of 4096, and 16 self-attention heads (each of dimension 64). This architecture mirrors BERT-Large but with several pretraining-specific deviations:

• No Next-Sentence-Prediction (NSP) head is present; only the masked LLM (MLM) objective is used.
• Dropout and attention-dropout are set at a rate of 0.1 per layer.
• Layer normalization follows the “pre-norm” arrangement (LayerNorm precedes both self-attention and MLP sublayers).
• The total parameter count is approximately 355 million (Liu et al., 2019, Liu et al., 2021).

In derived models such as HalleluBERT-Large, this architecture is replicated exactly, with changes limited to data sources, vocabulary, and pretraining schedules (Scheible-Schmitt, 24 Oct 2025).

Parameter	Value	Notes
Layers	24	Transformer encoder blocks
Hidden size	1024	Per-layer
FFN dimension	4096	Intermediate MLP size
Attention heads	16	Head dim: 64
Dropout	0.1	Applied to layers, attention
Total parameters	≈355 million	RoBERTa-Large

2. Pretraining Data and Tokenization

RoBERTa-Large is pretrained on a composite English corpus totaling approximately 160 GB of raw text, including BookCorpus, English Wikipedia, CC-News, OpenWebText, and CommonCrawl-based Stories. All files are thoroughly deduplicated, HTML is stripped, and data is packed into sequences of up to 512 tokens.

Vocabulary construction departs from BERT in critical ways:

• RoBERTa employs a 50,000-unit byte-level BPE vocabulary, rather than a 30,000-unit character-based BPE as in BERT.
• Dynamic masking is applied at each epoch ("on the fly"), ensuring each token can be masked multiple times throughout training and alleviating training corpus overfitting to static masks (Liu et al., 2019).

Language-specific RoBERTa-Large derivatives (e.g., HalleluBERT-Large) adopt a similar byte-level BPE approach but train the tokenizer from scratch on the target language corpus (e.g., 52,000 subwords for Hebrew) and operate directly on raw Unicode bytes (Scheible-Schmitt, 24 Oct 2025).

3. Pretraining Objectives and Optimization Protocols

RoBERTa-Large is trained exclusively on the Masked Language Modeling (MLM) loss: $L_{MLM} = - \sum_{i \in M} \log p(x_i | x_{/i})$ where $M$ is the set of masked indices, and $x_{/i}$ is the input with the masked token replaced.

Key deviations in RoBERTa-Large pretraining protocol:

• Masked tokens are sampled dynamically at each pass.
• The optimizer is Adam with $\beta_1 = 0.9$, $\beta_2 = 0.98$, $\epsilon = 10^{-6}$, and weight decay 0.01.
• Training is performed with mixed-precision for scalability (original: 1024 V100 GPUs).
• Sequence length: 512 tokens.
• Batch size: 8,192 sequences per step.
• Learning rate reaches a peak of $4 \times 10^{-4}$ after a linear warmup over 30,000 steps, then decays linearly over 500,000 update steps (Liu et al., 2019).

In variants such as HalleluBERT-Large, pretraining is run for 100,000 updates with a global batch size of 8,000 tokens, peak LR of 0.00015, and is implemented in fairseq on TPUv4 for ~6 days without mixed precision (Scheible-Schmitt, 24 Oct 2025).

4. Empirical Performance and Evaluation

RoBERTa-Large achieves state-of-the-art results on several core NLP benchmarks:

• GLUE "Single-Task" Dev Set (median of 5 runs): MNLI-m: 90.2, QNLI: 94.7, QQP: 92.2, RTE: 86.6, SST-2: 96.4, MRPC: 90.9, CoLA: 68.0, STS-B: 92.4, WNLI: 91.3.
• GLUE Test Set (public leaderboard, ensemble): MNLI: 90.8/90.2, QNLI: 98.9, QQP: 90.2, RTE: 88.2, SST-2: 96.7, MRPC: 92.3, CoLA: 67.8, STS-B: 92.2, WNLI: 89.0, Avg: 88.5.
• SQuAD v1.1/v2.0 (Dev/Test): EM/F1 up to 88.9/94.6 and 86.8/89.8 (v2.0 Test).
• RACE (Test): Middle: 86.5, High: 81.3, Overall: 83.2 (Liu et al., 2019).

Language-specific RoBERTa-Large models, such as HalleluBERT-Large, transfer this performance lead to non-English domains. On Hebrew NER and sentiment tasks, HalleluBERT-Large outperforms both monolingual and multilingual baselines (e.g., XLM-RoBERTa-Large, AlephBERT) by 1–2 F1 points absolute, with stable improvements across 10 runs (Scheible-Schmitt, 24 Oct 2025).

5. Learning Dynamics and Probing Analyses

Probing research demonstrates that RoBERTa-Large acquires different types of knowledge at distinct phases of pretraining. According to (Liu et al., 2021):

• Structural linguistic knowledge (syntax, POS, dependency arcs) is learned rapidly, with 97% of final probe accuracy achieved within ~20% of pretraining steps.
• Factual knowledge and commonsense associations require substantially more updates and show higher sensitivity to pretraining corpus domains.
• Reasoning abilities, as gauged by oLMpics-style probes, exhibit non-monotonic performance and, in general, are not reliably acquired by standard masked language modeling alone.

These learning curves have practical implications: corpus diversity accelerates acquisition of factual and commonsense knowledge, while increasing corpus size yields only marginal gains in homogeneous settings. Monitoring probe performance across time can guide curriculum and training schedules, e.g., early stopping for syntax tasks or additional domain-adaptive training for fact-heavy applications.

6. Extensions, Variants, and Language Adaptation

The RoBERTa-Large configuration is now widely used as a template for monolingual and multilingual encoders. HalleluBERT-Large demonstrates the approach for Hebrew: deploying the canonical RoBERTa-Large architecture, a language-specific byte-level BPE (trained on deduplicated web and Wikipedia data), and the same MLM objective, the model sets a new state of the art for Hebrew NER and sentiment while confirming that fully-converged, large-scale monolingual pretraining with optimal schedules can surpass multilingual baselines despite a smaller total data footprint (Scheible-Schmitt, 24 Oct 2025).

Ablation directions proposed for such variants include increasing tokenizer vocab size, altering batch size or learning rate schedules, and applying whole-word masking for morphologically-rich languages. No current ablation results are available for these axes (Scheible-Schmitt, 24 Oct 2025).

7. Significance and Research Impact

RoBERTa-Large’s introduction resulted in a fundamental shift in pretraining paradigms, highlighting that prior large-scale encoder models (notably BERT) were substantially undertrained. RoBERTa-Large’s recipe—elimination of NSP, dynamic masking, larger and more diverse corpora, and larger batches—established a methodological standard widely adopted for both research and downstream system deployment. Its empirical success and proven extensibility to multiple languages and domains have made it a foundational model in NLP research and practice (Liu et al., 2019, Scheible-Schmitt, 24 Oct 2025).