Protein Language Models

• Protein Language Models are large-scale neural networks that learn complex sequence-to-structure and function mappings through self-supervised training on vast protein datasets.
• They integrate advanced architectures such as Transformers, augmented attention, and structured state-space models with explicit structural biases to enhance predictive accuracy.
• PLMs enable practical applications in protein structure prediction, function annotation, and controllable design, fundamentally transforming computational biology and molecular engineering.

Protein LLMs (PLMs) are large-scale neural networks trained to capture the statistical, functional, and structural properties of protein sequences, which are formalized analogously to sentences in natural language. By ingesting vast protein sequence databases, PLMs facilitate unsupervised learning of complex evolutionary constraints that govern sequence-to-structure and sequence-to-function mappings. State-of-the-art PLMs—including ESM, ProtBERT, ProGen, and Prot42—enable a spectrum of computational biology applications: protein structure prediction, function annotation, zero-shot mutational effect estimation, design of novel enzymes, controllable protein generation, and more. Architectural innovations, multi-scale datasets, and cross-modal training objectives now allow PLMs to exploit not only sequence diversity but also explicit 3D structural knowledge and biophysical priors, thus aligning computational representation learning with fundamental principles of protein biochemistry and molecular design.

1. Model Architectures and Pretraining Paradigms

PLMs operate primarily within the Transformer framework, but their architectures and objectives reflect adaptations to protein-specific sequence statistics and biological requirements:

• Encoder-only (BERT-style, MLM): Models such as ESM-1b/2, ProtBERT, and ProteinLM leverage masked language modeling, learning to predict ~15% randomly masked amino acids per sequence via bidirectional context. Typical configurations involve 30–33 transformer layers, 16–33 attention heads per layer, and hidden sizes of 1,024–2,048 dimensions. Positional encodings are injected by fixed sinusoids, learned embedding lookups, or rotary schemes (e.g., RoPE) (Ferruz et al., 2022, Pandi et al., 18 Dec 2024).
• Decoder-only (GPT-style, AR): Models like ProGen, ProGen2, Prot42, and RITA implement autoregressive next-token prediction, enabling explicit sequence generation and conditional sampling by prepending functional or taxonomic "control tags" (Madani et al., 2020, Nijkamp et al., 2022, Sayeed et al., 6 Apr 2025).
• Encoder–decoder (T5-style): Full sequence-to-sequence models (ProtT5, pAbT5, xTrimoPGLM) combine bidirectional encoders with autoregressive decoders, supporting both understanding (e.g., classification, regression) and generative tasks (e.g., motif insertion, protein translation) (Wang et al., 8 Feb 2025).
• Augmented attention and SSMs: Recent work replaces O(L²) self-attention with linear-scaling structured state-space models (SSMs, e.g., BiMamba-S in LC-PLM), achieving length extrapolation up to 8,000+ residues and improving scaling efficiency for universal protein representation (Wang et al., 29 Oct 2024).

Pretraining is most often performed on clustered, deduplicated sequence corpora spanning hundreds of millions to over a billion proteins (UniProtKB, UniRef, BFD, metagenomic MGnify, etc.), minimizing overfitting and maximizing evolutionary coverage. Model scale now ranges from tens of millions to tens of billions of parameters (Cheng et al., 4 Nov 2024).

2. Integration of Structural and Biological Knowledge

While early PLMs learned implicit structure–function constraints strictly from sequence, contemporary models systematically inject explicit structural knowledge:

• Structural bias in attention: The Protein Structure Transformer (PST) (Chen et al., 26 Jan 2024) integrates a two-layer graph neural network (GIN) as a "structure extractor" at each transformer block, encoding pairwise Cα–Cα residue proximities (<8 Å) as a graph G and injecting structural embeddings into the queries, keys, and values of self-attention (Q_s, K_s, V_s). This bias enhances accuracy for Enzyme Commission (EC) and Gene Ontology (GO) classification, with small models (e.g., 8M-150M params) benefiting disproportionately.
• Contrastive and structural token alignment: SaESM2 aligns residue embeddings from PLMs with those from pre-trained protein GNNs via an InfoNCE-style contrastive loss (pairwise within and across proteins), and further predicts discrete structural tokens (e.g., FoldSeek classes) per residue with a cross-entropy loss. A residue loss selection module curates the training signal to filter out noisy or low-quality structure annotations (Chen et al., 22 May 2025).
• Structure-informed fine-tuning: Models such as ESM-2-S are fine-tuned with remote homology detection as a fold classification task, leveraging only sequence (not explicit 3D input) and transferring fold-discriminative representations for downstream function prediction (EC, GO) (Zhang et al., 7 Feb 2024).
• Structural adapters for design: LM-Design implants lightweight cross-attention adapters between a pre-trained sequence PLM and a structure encoder, enabling efficient reprogramming for structure-conditioned design with only 0.5–2% extra parameters, supporting fast iterative refinement at inference (Zheng et al., 2023).
• Joint sequence–structure generative diffusion: DPLM-2 models the joint distribution over sequences and quantized structure tokens via a multimodal discrete diffusion transformer, with lookup-free quantizers mapping atomic 3D coordinates to tokens, and unified marginal, conditional, and joint sequence–structure sampling (Wang et al., 17 Oct 2024).

3. Pretraining Objectives, Control, and Interpretability

PLM objectives encompass both unsupervised sequence modeling and targeted manipulation:

• Autoregressive and masked modeling: The standard AR loss optimizes $L_{AR} = -\sum_{i=1}^L \log p(x_i | x_{<i})$; the MLM objective is $L_{MLM} = -\sum_{i\in M} \log p(x_i | x_{/M})$, where $M$ is the set of masked positions.
• Controllable generation: ProGen and follow-ons enable conditioning on "control tags" (e.g., taxonomic lineage, molecular function, catalytic site, subcellular location), which are prepended to sequences and trained as regular tokens, so that generation can be steered toward desired properties (Ferruz et al., 2022, Madani et al., 2020). Recent models leverage activation steering: at inference, a vector computed in hidden space from property-positive vs. negative sets is linearly combined with the activations at each layer to direct property-aware generation without weight updates (Huang et al., 1 Jul 2025).
• Multi-objective learning: PEvoLM jointly matches next-AA distributions and position-specific scoring matrices (PSSMs) from MSA, distilling evolutionary conservation profiles into bidirectional contextual embeddings at reduced parameter and inference cost (Arab, 2023).
• Latent optimization: Protein design via VAE or diffusion decoders leverages latent representations from pretrained PLMs; sampling is achieved by perturbing or interpolating in a latent space regularized by backbone structure or function labels (Pandi et al., 18 Dec 2024, Wang et al., 17 Oct 2024).

Attention-head analysis and embedding visualization tools (e.g., exBERT adaptations) enable interpretability: heads specialize in detecting local secondary-structure (α-helix, β-sheet) or long-range tertiary contacts, and embedding spaces encode or cluster by function, compartment, or taxonomic label (Ferruz et al., 2022, Sayeed et al., 6 Apr 2025).

4. Scaling Laws, Compute Efficiency, and Evaluation

The compute–performance frontier of PLMs is defined by scaling laws and dataset composition:

• Compute-optimal scaling: For a given FLOPs budget $C$, optimal model size $N$ and dataset tokens $D$ follow empirically fitted power laws (e.g., for CLM, $N(C) = 1.26×10^{-3} C^{0.578}$, $D(C) = 1.23×10^2 C^{0.422}$; for MLMs, parameter scaling dominates). Sequential CLM→MLM pretraining is ~1.3× more compute-efficient than MLM from scratch, with ~20% of budget to CLM preferable (Cheng et al., 4 Nov 2024).
• Long-context and resource-efficient architectures: State-space models (e.g., BiMamba-S, LC-PLM) enable O(S) scaling in sequence length, handling up to 8,192 tokens with stable loss, whereas quadratic self-attention limits vanilla Transformers. LoRA and similar low-rank adaptation methods achieve competitive generation and conditioning while updating only ~4% of parameters, enabling deployment on energy-efficient hardware (Wang et al., 29 Oct 2024, Shah et al., 8 Nov 2024).
• Evaluation metrics: Standard metrics include sequence or MLM perplexity, token recovery, contact-prediction precision, fold classification accuracy, protein structure (TM-score, pLDDT, RMSD), function prediction (accuracy, F_max, ROC-AUC), and zero-shot/fitness correlation (Spearman's ρ). PLMs are increasingly scrutinized on rigorous benchmarks (PEER, TAPE, ProteinShake) and new classes of downstream tasks: variant effect prediction, protein–protein interactions, motif scaffolding (Nijkamp et al., 2022, Chen et al., 26 Jan 2024, Chen et al., 22 May 2025).

5. Applications and Practical Impact

PLMs are now foundational tools in protein science, with applications spanning:

• Single-sequence structure prediction: Models such as ESMFold, LC-PLM, and DPLM-2 predict 3D coordinates from individual sequences at state-of-the-art accuracy and runtime 10× faster than MSA-based pipelines (Hu et al., 2022, Wang et al., 29 Oct 2024, Wang et al., 17 Oct 2024).
• Protein function annotation: EC number and Gene Ontology term prediction use fixed representations from PLMs with linear decoders, often outperforming MSA-dependent predictors (Chen et al., 26 Jan 2024, Zhang et al., 7 Feb 2024).
• Controllable and property-aware design: Conditional generation with tags (ProGen2, Prot42), instruction prompts, and latent steering allows synthesis of highly diverse, structurally valid, and property-targeted proteins, including enzyme classes and high-affinity binders (Sayeed et al., 6 Apr 2025, Shah et al., 8 Nov 2024).
• Atom-level and multimodal design: Fine-grained atom-level generation enables design of unnatural amino acids, protein–small-molecule conjugates, and hybrid chemotypes (Flam-Shepherd et al., 2023). DPLM-2 achieves coupled sequence–structure generation via unified diffusion modeling (Wang et al., 17 Oct 2024).
• Immunology and therapeutics: Specialized PLMs trained on BCR/TCR repertoires (e.g., AntiBERTa, pAbT5) support antigen-specificity classification, antibody structure prediction, and chain translation, with robust AUC and RMSD metrics (Dounas et al., 6 Feb 2024).
• Diagnostic and sequencing support: Peptide sequencing PLMs reconstruct full sequences from sparse Edman/click-chemistry outputs with per-amino-acid accuracy up to 90%, and TM-validation exceeding 0.6 for predicted structures (Pham et al., 1 Aug 2024).

6. Limitations, Challenges, and Future Directions

Despite their broad impact, current PLMs face notable challenges:

• Data and supervision bias: Sequence redundancy, overrepresented or poorly annotated families, and limitations of synthetic/metagenomic data can skew learned representations or overfit specific characteristics. High-quality structural annotations remain limiting for some tasks (Cheng et al., 4 Nov 2024).
• Interpretability and causality: Although attention maps and activation analyses suggest biophysical grammar emergence, full causal attribution to sequence/structure determinants is elusive. Extracting actionable folding or function rules remains an open field (Ferruz et al., 2022, Wang et al., 8 Feb 2025).
• Handling of long, multi-domain, or multimeric proteins: Quadratic attention, positional encoding limitations, and fixed tokenization restrict scalability; SSMs and efficient attention variants are partial solutions, but full coverage of proteome-length sequences remains unresolved (Wang et al., 29 Oct 2024).
• Resource constraints: Training PLMs at trillion-token and multi-billion parameter scale is costly, with compute and environmental budget implications. LoRA, model distillation, sparse attention, and small-model backbones (e.g., Phi-3-mini) are promising mitigations (Shah et al., 8 Nov 2024).
• Structural integration and annotation granularity: Explicit structure-aware training now improves predictive tasks where function is tightly coupled to fold or local geometry, but can have negligible or negative impact in tasks driven by sequence motifs or disordered regions (Zhang et al., 7 Feb 2024, Chen et al., 22 May 2025).
• Extending modalities: Joint sequence–structure–function–interaction modeling, wet-lab feedback integration, and leveraging experimental omics data are open frontiers. Methods such as DPLM-2 and SaESM2 represent early progress.

Table: Selected PLM Architectures and Functional Highlights

Model	Architecture	Objective	Notable Capability
ESM-2	Enc-only	MLM	Fast folding, structure-function link
ProtBERT	Enc-only	MLM	Versatile embeddings, function prediction
ProGen2	Dec-only	AR	Tag-controllable generation, zero-shot ranking
Prot42	Dec-only	AR	8k context, target-aware binder design
PST	Enc-only+GNN	MLM+struct	Explicit local structure bias, parameter efficiency
DPLM-2	Multimodal	Diffusion	Co-generation of sequence and 3D structure
LC-PLM (BiMamba-S)	SSM-based	MLM	O(L) scaling, PPI graph context
SaESM2	Enc-only	MLM+contrast	InfoNCE-aligned structure knowledge, token pred.

7. Conclusion

Protein LLMs have transformed computational biology, shifting the paradigm from alignment-based statistical models and energy-based folding to unified, scalable, self-supervised neural architectures. The integration of explicit structural and evolutionary signals, advancement in scalable and efficient architectures, and the ability to condition, interpret, and control protein generation establish PLMs as fundamental computational tools for modern biochemistry, molecular engineering, and synthetic biology. Continued progress will depend on innovations in long-context modeling, computational sustainability, integration of multimodal biological data, and interpretability aligned with the underlying rules of molecular life (Wang et al., 8 Feb 2025, Cheng et al., 4 Nov 2024, Chen et al., 26 Jan 2024).