---
title: Transformer-Based Pre-trained Language Models
url: https://www.emergentmind.com/topics/transformer-based-pre-trained-language-models-plms
type: topic
---

# Transformer-Based Pre-trained Language Models

Transformer-based pre-trained language models (PLMs) are large neural architectures obtained by training multi-layer Transformer networks on vast unlabeled text corpora with self-supervised objectives such as masked language modeling (MLM), next-sentence prediction (NSP), or autoregressive next-token prediction. Unlike traditional static embedding approaches, these models produce contextualized token representations, enabling high adaptation to specific NLP tasks with minimal labeled supervision. PLMs have set state-of-the-art across domains including general language understanding, biomedical informatics, legal text analysis, and speech recognition.

## 1. Architectural Principles and Contextualization

Transformer-based PLMs universally rely on the Transformer architecture: stacks of self-attention (SA) and feed-forward (FFN) layers, punctuated with residual connections and layer normalization [2111.01243]. Each layer comprises multi-head self-attention (which models contextual relationships between all sequence positions) and position-wise FFNs. Embedding layers combine token, positional, and, occasionally, segment or auxiliary embeddings.

A key distinction of PLMs is their capacity to yield highly context-specific representations. For token $w$ at position $i$ in a sequence, the model computes a unique vector $h_{\ell,i}(w)$ at each layer $\ell$, directly dependent on the sentential context. This yields separation between polysemous senses unattainable for static embeddings. Quantitative analyses of BERT reveal that contextualization primarily arises in the SA sub-layer, especially at middle-to-late depths (layers $\sim$9–10), with context-driven divergence highest in the multi-head self-attention block. The later output sub-layer partially reimposes token identity, making it less suitable for sense disambiguation [2312.06514].

## 2. Pre-training Objectives and Optimization

PLMs are trained on large corpora with self-supervised tasks:

- **Masked Language Modeling (MLM):** Randomly mask a subset of input tokens; the model predicts missing tokens, optimizing
  $$
  \mathcal{L}_{MLM} = -\sum_{i\in M} \log P(x_i| \tilde{x})
  $$
  where $M$ is the set of masked positions [2111.01243, 2006.04229].
- **Next Sentence Prediction (NSP):** Predict whether two segments are consecutive in the original text. Variants include sentence order prediction and segment-level contrastive tasks.
- **Autoregressive Language Modeling (AR):** Predict each token conditioned on all previous tokens for decoder-only models [2111.01243].
- **Replaced Token Detection (RTD, ELECTRA):** Discriminator classifies whether each token was replaced by a generator [2110.05896, 2112.03014].
- **Domain-specific or auxiliary objectives:** Biomedical PLMs integrate span boundary objectives, entity masking, or ontology-related supervision for richer representations [2105.00827].

Recent works often drop NSP (e.g., RoBERTa, Polish BERT [2006.04229]) and employ dynamic masking, whole-word masking, or domain-tuned masking protocols.

## 3. Scalability, Architecture Variants, and Training Regimes

Transformer-based PLMs exhibit several structural variants:

- **Encoder-only models (BERT, RoBERTa):** Bidirectional context modeling via MLM/NSP. Common in NLU tasks [2111.01243, 2112.03014].
- **Decoder-only models (GPT):** Causal language modeling, used for text generation [2111.04909, 2112.03014].
- **Encoder–decoder models (T5, BART):** Sequence-to-sequence objectives for summarization, translation, or denoising [2111.04909].
- Architectural depths and widths are critical for downstream performance. FPM systematically demonstrates existence of task-optimal depth (e.g., 70 layers for QQP), with larger models outperforming smaller ones up to an optimum—beyond which, overfitting or training instability occurs [2111.04909].
- Stable training at scale requires careful optimizer scheduling, mixed precision (when possible), gradient clipping, and in some cases, architectural modifications (e.g., sparse attention for long contexts in KoBigBird) [2111.04909, 2112.03014].
- Monolingual, domain-specific corpora consistently yield superior results for specialized domains/languages compared to multilingual baselines, as observed in Polish, Korean, Lao, and legal-domain PLMs [2006.04229, 2112.03014, 2110.05896, 2209.06049].

## 4. Adaptation Mechanisms and Task Transfer

PLMs support multiple adaptation paradigms:

- **Full fine-tuning:** Entire model updated on labeled task data with a task-specific head (classification/regression/sequence tagging) [2006.04229, 2111.01243].
- **Parameter-efficient tuning:** Methods such as prompt tuning, adapters, and BitFit freeze PLM parameters and train only small auxiliary parameters [2204.06130, 2211.07349].
- **Prompting and prefix/pre-modulation:** Tasks are re-expressed as masked or completion-style prompts, either discrete or continuous (learned vector prefixes) [2204.06130, 2111.01243]. Soft prompts can recover up to 95% of fine-tuning gains.
- **Knowledge distillation and inheritance:** Large PLMs serve as teachers to smaller or shallow students via KL-divergence (soft target) losses; this accelerates convergence and enables efficient scaling [2204.06130, 2105.13880].
- **Domain adaptation:** Continued pre-training of general or near-domain PLMs on domain-specific corpora enables transfer to new linguistic or technical domains (e.g., Indian Law [2209.06049], biomedical corpora [2105.00827]).

## 5. Internal Mechanisms and Model Interpretability

PLMs encode “skills”—task-relevant computations—in an unexpectedly sparse subset of their sub-components:

- **Skill neurons:** After prompt tuning, a small subset of feed-forward neurons achieve high predictivity for task labels when their activation on prompt tokens is thresholded. Perturbing these neurons degrades accuracy substantially; their importance is largely present at pre-training [2211.07349].
- **Contextualization locus:** Context-driven sense disambiguation predominantly arises in multi-head self-attention sub-layers of middle-to-late layers. The activation sub-layer refines this, and the output sub-layer partially restores the identity information via residual connections [2312.06514].
- **Attention head specialization:** Heads in middle layers encode disambiguating cues essential for sense separation, while some heads (“CLS attenders”) focus on global or sentence-level signals [2312.06514].

These findings inform interpretability research and methodology development for pruning (removing non-skill neurons/heads) or transferability assessment.

## 6. Domain-Specific and Multilingual Adaptation

PLMs are extensible to low-resource or domain-specific languages and technical domains:

- **Biomedical:** Domain-specific pre-training (PubMedBERT), intermediate fine-tuning, special tokenization, and ontology-enriched embeddings are standard. Robustness to domain shift, noise, and fairness issues are crucial challenges [2105.00827].
- **Legal:** Continued pre-training and vocabulary adaptation to local legal text (e.g., Indian Law) yield higher accuracy and improved alignment with expert attention [2209.06049].
- **Korean, Polish, Lao:** Monolingual PLMs (with domain-matched corpora and tokenizers) outperform multilingual models on all downstream tasks, including POS tagging, classification, paraphrase, and NER [2006.04229, 2112.03014, 2110.05896].
- **Cross-modal transfer:** Transformer-based PLMs pre-trained on text transfer to speech recognition tasks (ASR)—repurposed self-attention blocks improve word/character error rate when used as encoders, even when initially trained on text only [2409.17750].

## 7. Current Limitations and Prospects

- **Impossible Triangle:** PLMs cannot simultaneously offer (P1) moderate model size ($\leq 1$B parameters), (P2) state-of-the-art few-shot/zero-shot ability, and (P3) state-of-the-art fully supervised accuracy. Existing models occupy only two vertices (e.g., GPT-3 achieves P2+P3, not P1). Advanced knowledge distillation, data augmentation, and universal prompt learning are active approaches to bridging these gaps [2204.06130].
- **Scalability and Efficiency:** Model and memory efficiency (through targeted pruning, adapter layers, and mixture-of-experts) remain open domains [2111.04909].
- **Interpretability and Bias:** Mechanistic understanding of knowledge storage, debiasing (especially in biomedical/legal), privacy preservation, and representation probing are ongoing challenges [2211.07349, 2105.00827].
- **Task and Modality Transfer:** Automatic methods for task-optimal architecture search, cross-lingual/intermodal transfer, and synthetic data generation are under investigation [2111.01243, 2409.17750, 2204.06130].

A staged roadmap targets short-term improvements to the missing triangle vertex, medium-term per-task solutions, and eventually universal moderate-sized PLMs that generalize across tasks and modalities via unified architectures and multi-objective pre-training [2204.06130].

---

This entry integrates evidence spanning analyses of model structure, training paradigms, scalability, adaptation, interpretability, and specialization, illustrating the technical landscape and ongoing advances in transformer-based pre-trained language models [2312.06514, 2006.04229, 2204.06130, 2105.13880, 2111.04909, 2409.17750, 2209.06049, 2105.00827, 2110.05896, 2211.07349, 2112.03014, 2111.01243].

Source: https://www.emergentmind.com/topics/transformer-based-pre-trained-language-models-plms