xTrimoPGLM: Unified 100B-Scale Protein Modeling

• xTrimoPGLM is a unified 100B-scale protein language model that combines autoencoding for understanding and autoregressive blank infilling for generation in a single backbone.
• It employs a GLM-style Transformer with bidirectional prefix encoding and mixed training objectives, achieving faster convergence and state-of-the-art performance across various protein tasks.
• Pretrained on nearly 1 trillion tokens, the model is leveraged for sequence representation, de novo generation, rapid structure prediction, and as a frozen backbone for applications like allergen prediction.

Searching arXiv for papers on xTrimoPGLM and related analyses. xTrimoPGLM is a unified 100B-scale pre-trained transformer for protein sequence modeling introduced by Chen et al. to combine autoencoding-style protein understanding and autoregressive protein generation within a single backbone. The model adopts the GLM paradigm for autoregressive blank infilling with a bidirectional prefix and augments it with a BERT-style masked language modeling objective, with the stated aim of handling understanding tasks, generation tasks, and downstream structure prediction in one protein foundation model. At approximately 101B parameters and approximately 1 trillion training tokens, xTrimoPGLM occupies the extreme high end of protein LLM scale and is presented as a system for sequence representation, out-of-distribution generalization, de novo generation, and fast single-sequence structure prediction via xT-Fold (Chen et al., 2024).

1. Conceptual position and modeling objective

xTrimoPGLM is defined against a longstanding split in protein language modeling. Encoder-only protein LLMs such as ESM-like systems are optimized for bidirectional contextual recovery and therefore support residue-level and protein-level understanding tasks, while decoder-only systems such as GPT-like protein models are optimized for sequence continuation and therefore support generation. Chen et al. frame xTrimoPGLM as an attempt to bridge this divide by jointly optimizing a masked language modeling loss for understanding and a GLM blank-infilling loss for generation on the same parameters (Chen et al., 2024).

In this formulation, the model is not an encoder-decoder composite with separate components. It is a single Transformer backbone shared across both objectives. The underlying claim is not merely architectural consolidation, but compatibility and synergistic joint optimization between the two training regimes. The paper reports that alternating 50/50 batches of MLM and GLM, with attention-mask changes only, keeps validation losses stable for both objectives. It also reports cross-objective transfer during pretraining: MLM-adapted GLM converges approximately $2.2\times$ faster, matching from-scratch GLM loss in about 110B tokens, while GLM-adapted MLM converges approximately $2\times$ faster, reaching a comparable point in about 100B tokens (Chen et al., 2024).

This placement is central to how xTrimoPGLM is interpreted in subsequent work. It is simultaneously treated as a generative model, a representation model, and a high-capacity frozen backbone. A plausible implication is that its significance lies less in any single benchmark and more in establishing a unified pretraining regime for protein foundation models.

2. Architecture and pretraining mechanics

The backbone is a GLM-style Transformer decoder with bidirectional prefix encoding and autoregressive blank infilling. The 100B model is reported as having approximately 101B parameters, 72 Transformer layers, 80 attention heads, hidden size 10,240, and feed-forward size 31,744 with GeGLU activation; normalization is Post-LN stabilized with DeepNorm, and positional encoding uses two-dimensional rotary positional embeddings (2D RoPE) (Chen et al., 2024).

Tokenization is residue-level over protein alphabets with canonical amino acids represented by single-letter codes, plus objective-specific and sequence-boundary tokens. The special tokens explicitly listed are $[MASK]$, $[sMASK]$, $[gMASK]$, and the sequence markers $\langle sop \rangle$, $\langle eop \rangle$, and $\langle eos \rangle$. Ambiguous residues such as $X$, $B$, and $2\times$0 are handled as their own single-letter tokens. Pretraining uses a fixed maximum sequence length of 2,048 and a mix-length strategy that samples context windows of 256, 512, 1,024, and 2,048 in the ratio $2\times$1, concatenating sequences to reduce padding overhead (Chen et al., 2024).

The masked language modeling component is written as

$2\times$2

where approximately $2\times$3 of tokens are masked and predicted from bidirectional context. The GLM component reconstructs masked spans autoregressively after replacing each span with either $2\times$4 or $2\times$5. Short spans use $2\times$6 with a Poisson distribution of $2\times$7, while long end-of-sequence spans use $2\times$8 with at least $2\times$9 of tokens masked. The unified objective is

$[MASK]$0

Training proceeds in two phases: Stage 1 uses pure MLM for approximately 400B tokens with $[MASK]$1, and Stage 2 uses unified training for approximately 600B tokens with $[MASK]$2, corresponding to roughly $[MASK]$3 MLM and $[MASK]$4 GLM (Chen et al., 2024).

Several stabilization mechanisms are reported. To avoid catastrophic divergence when GLM is introduced at 100B scale, the GLM ratio is increased linearly from 0 to the target ratio over $[MASK]$5 steps while the learning rate is kept low at approximately $[MASK]$6, after which the learning rate is returned to the main schedule. Embedding Gradient Shrink is applied with $[MASK]$7. Optimization uses AdamW with $[MASK]$8, $[MASK]$9, and weight decay $[sMASK]$0, gradient clipping at 1.0, dropout 0.1, and mixed precision with Apex O2. The learning rate warms from $[sMASK]$1 to $[sMASK]$2 over $[sMASK]$3 of samples and then follows cosine decay by a factor of 10 to $[sMASK]$4 (Chen et al., 2024).

For autoregressive evaluation, perplexity is defined as

$[sMASK]$5

The paper also notes pseudo-perplexity for masked language modeling, computed by masking each position in turn and evaluating the same form with bidirectional context and $[sMASK]$6 (Chen et al., 2024).

3. Scale, data curation, and systems infrastructure

xTrimoPGLM was pretrained on approximately 1 trillion training tokens, corresponding to a final corpus of approximately 940M unique sequences and approximately 200B residues assembled from UniRef90 and ColabFoldDB-derived sources. The initial data contributions are given as approximately 153M UniRef90 entries and approximately 950M ColabFoldDB entries. After deduplication, short-length filtering, composition screening, and cross-deduplication with UniRef90, the final corpus is constructed and then sampled with bias toward UniRef90, with ColabFoldDB sampling at approximately $[sMASK]$7, effectively tripling or quadrupling UniRef90’s contribution (Chen et al., 2024).

The data sources named for ColabFoldDB include BFD, MGnify, SMAG/MetaEuk/TOPAZ for eukaryotes, MGV for DNA viruses, GPD for bacteriophages, and MetaClust updates, built upon UniProtKB. Validation data are drawn from sequences uploaded to UniProt between January and March 2023 and filtered to remove overlaps greater than $[sMASK]$8 or greater than $[sMASK]$9 sequence identity with the training sets. This combination of metagenomic breadth and explicit overlap filtering is one of the paper’s central attempts to measure out-of-distribution behavior rather than mere memorization (Chen et al., 2024).

The systems configuration is correspondingly large. Training used 96 DGX-A100 servers, each with $[gMASK]$0A100 40GB GPUs, over January to June 2023. Parallelism combines 4-way tensor parallelism, 8-way pipeline parallelism, and 24-way data parallelism through DeepSpeed and Megatron-LM strategies. Forward and backward passes use FP16, while master weights and optimizer states remain in FP32. The global batch is warmed from 240 to 4,224, sequence length is fixed at 2,048, and the total training compute is estimated at approximately $[gMASK]$1 FLOPs (Chen et al., 2024).

These figures place xTrimoPGLM among the largest reported protein LLMs of its time. A plausible implication is that the model’s scientific significance is inseparable from its systems engineering: the architecture, corpus construction, curriculum, and distributed training strategy are coupled rather than independent design choices.

4. Empirical performance across understanding, structure, and generation

On protein understanding, xTrimoPGLM is evaluated on 18 benchmarks across four categories using linear probing and LoRA-based parameter-efficient fine-tuning. The paper reports state of the art in 15 of 18 tasks. In protein structure benchmarks, the reported values are 93.32 Top $[gMASK]$2 contact-map accuracy on trRosetta, compared with 92.19 for ESM2-15B, and 75.61 fold-classification accuracy on SCOP 1.75, compared with 61.10 for Ankh-large. On CASP12/14 secondary structure, however, xTrimoPGLM records 75.33 Q3 accuracy while Ankh-large records 80.70, so the gains are not uniform across all tasks (Chen et al., 2024).

In protein developability, the reported results include solubility accuracy 79.45 versus 76.49 for ESM2-15B; protease stability SRCC 84.21 versus 80.75; temperature stability at $[gMASK]$3 MCC 94.22 versus 83.80 for TemStaPro; optimal temperature SRCC 73.96 versus 62.40 for DeepET; optimal pH AUC 64.99 versus 62.48 for ESM2-15B; cloning failure AUC 84.82 versus 77.09 for ESM2-3B; and material production failure AUC 86.48 versus 79.17 for ESM2-15B. In protein interactions, the reported scores are metal-binding site presence accuracy 82.78, enzyme catalytic efficiency $[gMASK]$4 PCC 74.79, peptide–HLA binding AUC 96.68, and TCR–pMHC binding AUC 95.10. In protein function, the model reaches 98.38 accuracy on antibiotic resistance, 66.00 SRCC on fluorescence intensity, 96.10 SRCC on the GB1 fitness landscape, and 81.60 accuracy on localization, while Ankh-large is higher on localization at 83.20 (Chen et al., 2024).

The paper emphasizes out-of-distribution perplexity as a cross-task indicator. On UniProt sequences uploaded after January 2023 and filtered at sequence identity thresholds 0.5 and 0.9, xTrimoPGLM-100B records perplexities of 10.81 and 6.70, respectively, compared with 10.98 and 7.25 for ESM2-15B and 14.30 and 9.75 for ProGen2-xlarge. On CAMEO and CASP15 sequences used for structure prediction, the reported PLM perplexities are 4.01 and 4.45, compared with 5.21 and 6.18 for the ESMFold PLM based on ESM2-3B (Chen et al., 2024).

These representations are then used in xT-Fold, a single-sequence structure predictor that integrates xTrimoPGLM-100B embeddings with a folding stack similar in spirit to ESMFold. xT-Fold trains on about 380k experimental PDB structures plus about 4M AlphaFoldDB structures with pLDDT $[gMASK]$5, sampled in a 75\% predicted / 25\% real ratio, and uses short recycling of at most 3 passes. The reported TM-scores are 0.86 on CAMEO and 0.70 on CASP15, compared with 0.85 and 0.65 for ESMFold and 0.80 and 0.60 for OmegaFold. The paper further states that PLM-based methods such as xT-Fold, ESMFold, and OmegaFold are approximately $[gMASK]$6–$[gMASK]$7 faster than MSA/template-based pipelines, and that xT-Fold can run on a single A100 GPU, aided by 4-bit quantization and FlashAttention (Chen et al., 2024).

Generation is treated as a first-class capability rather than an auxiliary property. Using $[gMASK]$8-prompted blank infilling with temperature 0.8 and top-$[gMASK]$9 0.8, the authors generated 2,861 sequences from xTrimoPGLM-100B and 2,759 from ProGen2-xlarge under identical sampling. The generated sequences are evaluated with xT-Fold, where xTrimoPGLM yields median pLDDT 85.4 versus 80.8 for ProGen2-xlarge, median Foldseek TM-score 0.658 versus 0.560, and median sequence identity 0.117 versus 0.079. For programmable generation, the paper reports that LoRA-based supervised fine-tuning on curated sets from Fold class, Temperature stability, Localization, Fluorescence, and Fitness outperforms non-finetuned baselines and surpasses ProGen2 and ProtGPT2 under identical SFT settings; a one-step Reinforced Self-Training procedure further improves alignment to desired properties (Chen et al., 2024).

5. Use as a frozen backbone in allergen prediction

Later work uses xTrimoPGLM as a frozen feature extractor rather than as a fine-tuned end-to-end model. In Applm, a framework for allergen prediction, the 100B model is loaded as a pretrained Transformer-style PGLM backbone and its final-layer hidden states are used as per-residue embeddings. For a protein of length $\langle sop \rangle$0, xTrimoPGLM-100B produces an $\langle sop \rangle$1 embedding matrix, which is average-pooled over sequence length to yield a fixed 10,240-dimensional vector per protein. That pooled vector is then passed to a Random Forest classifier implemented with scikit-learn with $\langle sop \rangle$2; the pLM itself is kept frozen in the main method (Wong et al., 14 Aug 2025).

The Applm evaluation is explicitly designed around difficult generalization settings rather than random splits. It includes “By Date” tests on proteins added after 2020, “Homologs” tests in mixed-allergenicity protein families such as Arginine Kinase, Cysteine Protease, Serine Protease, and Tropomyosin, and “Mutations” tests involving single or few amino-acid changes to known allergens. Internal cross-validation enforces inter-split similarity control $\langle sop \rangle$3 and inter-class similarity control $\langle sop \rangle$4, with sequence identity computed by Smith–Waterman local alignment with coverage control through ssearch36. The pipeline is described as producing clean partitions with zero threshold violations (Wong et al., 14 Aug 2025).

Across six curated external test sets, Applm reports average AUROC 0.872 and average AUPRC 0.700, exceeding competitors by 0.065–0.370 in AUROC and 0.104–0.500 in AUPRC; it ranks first in 14/18 AUROC settings and 18/18 AUPRC settings, with differences reported as significant by Friedman and Nemenyi tests. Among the Applm backbones, xTrimoPGLM-100B achieves the highest overall performance, although differences among pLMs are stated to be not statistically significant. In similarity-aware internal cross-validation, pLM embeddings including xTrimoPGLM-100B achieve overall average AUROC 0.844–0.864, compared with 0.742 for One-Hot Encoding and 0.684 for BLOSUM62, and the authors report that LoRA fine-tuning does not improve over the frozen-embedding plus Random Forest configuration (Wong et al., 14 Aug 2025).

This downstream use is important because it shows a distinct operational mode for xTrimoPGLM. Rather than exploiting its generative pathway or large-scale end-to-end adaptation, Applm treats the model as a static representation engine whose final-layer embeddings support subtle homolog discrimination and mutation-impact assessment. The paper interprets this as evidence that xTrimoPGLM captures fine-grained sequence differences relevant to allergenicity, even when no residue-level attribution method is applied (Wong et al., 14 Aug 2025).

6. Compute-optimality, contextual critiques, and limitations

A separate line of analysis places xTrimoPGLM in the scaling-law discussion for protein LLMs. In “Are Protein LLMs Compute Optimal?”, xTrimoPGLM is used as a large-scale comparison point in a single-pass masked-language-modeling setting. That study reports a perplexity of 10.81 for xTrimoPGLM-100B and argues that a 35M encoder-only model trained on a 20M-sequence subset of UniRef50 in a single pass, corresponding to approximately 6.5 billion tokens and roughly 1 hour on a single NVIDIA H100 GPU, reaches perplexity 11.43 on an 8K-sequence holdout. The same section compares ESM-2 15B at 10.98 and describes 11.43 as “comparable” given the compute disparity (Serrano et al., 2024).

The same study reports a common lower bound in training loss around 2.45 across model sizes from 5M to 650M and across token budgets, with and without learning-rate decay, and notes that similar plateaus have been observed in xTrimoPGLM, RITA, and ProteinBERT. Its fitted compute-optimal scaling laws are

$\langle sop \rangle$5

indicating sublinear growth of optimal model size and superlinear growth of optimal tokens with compute in the studied single-pass regime. The paper therefore concludes that widely used pLMs, including xTrimoPGLM in this framing, are likely not compute-optimal under current training practices when the target is single-pass masked-LM loss minimization (Serrano et al., 2024).

That critique is explicitly bounded by several caveats. The scaling-law study uses encoder-only ESM-2 variants with a masked language modeling objective, a 29-token vocabulary, UniRef50-derived subsets, and an 8K holdout with sequence identity at most 0.5 to the training set; it also notes that xTrimoPGLM’s training corpora, filtering, tokenization, and pretraining objectives may differ substantially, and that reported perplexities may come from different test distributions or protocols. A plausible implication is that the compute-optimality argument should not be read as a universal negation of xTrimoPGLM’s downstream value, but as a regime-specific statement about the efficiency of current scaling practice for single-pass masked-LM training (Serrano et al., 2024).

The original xTrimoPGLM paper also identifies substantial limitations. Training and adaptation at 100B scale are resource-intensive, and even LoRA fine-tuning without quantization can require at least $\langle sop \rangle$6A100 80GB. Single-sequence PLM-based folding still lags MSA-based methods on harder out-of-distribution structure sets such as CASP15. Generative hallucinations remain an issue: approximately 21.7\% of generated sequences had low-confidence folds with pLDDT $\langle sop \rangle$7, and low temperature or low top-$\langle sop \rangle$8 can induce repetitive low-complexity fragments. The paper further notes possible biases from metagenomic composition and uneven length distributions, especially for membrane proteins, disordered regions, and extreme lengths (Chen et al., 2024).

Taken together, these results position xTrimoPGLM as both a milestone and a point of debate. It is a concrete demonstration that autoencoding and autoregressive protein language modeling can be unified at 100B scale in one Transformer and that the resulting model can support understanding, generation, fast structure prediction, and frozen-feature transfer. At the same time, later scaling-law analysis suggests that such scale does not automatically imply compute optimality under all objectives or training regimes, and downstream studies such as Applm indicate that the model’s practical impact may depend as much on representation quality and evaluation design as on raw parameter count (Chen et al., 2024)