ProGen2: Scalable Protein Language Models

• ProGen2 is a suite of autoregressive protein language models, scaling from 151M to 6.4B parameters and trained on over a billion diverse protein sequences.
• It extends the original ProGen approach by enhancing sequence diversity, generation quality, and zero-shot fitness prediction through advanced transformer architectures.
• ProGen2 serves as a foundational model for downstream controllable and multimodal systems, catalyzing innovations such as ProCALM and InstructPro for conditional protein design.

ProGen2 is a suite of autoregressive transformer decoder protein LLMs trained with next-token prediction and scaled from 151M to 6.4B parameters on sequence datasets drawn from over a billion proteins from genomic, metagenomic, and immune repertoire databases (Nijkamp et al., 2022). In later taxonomies, it is placed under “LLM Methods for Protein Engineering, Generation and Translation” as a “Protein Generation Model,” with the task framing of protein generation, de novo protein design, and broad exploration of protein sequence space (Xiao et al., 21 Feb 2025). Within the development of protein language modeling, ProGen2 extends the earlier ProGen paradigm of causal sequence generation, but shifts the emphasis toward scale, data distribution, generation quality, and zero-shot fitness prediction, while also serving as a backbone or initialization point for later controllable and multimodal systems (Madani et al., 2020).

1. Lineage within autoregressive protein language modeling

ProGen2 inherits its basic generative formulation from ProGen, which posed protein engineering as an unsupervised sequence generation problem and trained a 1.2B-parameter LLM on approximately 280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component (Madani et al., 2020). In ProGen, conditioning tags were prepended to the amino-acid sequence, the model learned the joint distribution over tags and sequence, and the system demonstrated controllable generation using metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy. That earlier work established the autoregressive protein LM formulation, large-scale conditioning on protein metadata, and the idea that generative sequence models could support protein completion, de novo generation, and zero-shot selection.

Against that background, the 2025 survey describes ProGen2 as “an extended version of ProGen, featuring a larger model size and a more extensive training dataset to enhance sequence diversity,” and further highlights that ProGen2 can predict protein fitness without requiring additional fine-tuning (Xiao et al., 21 Feb 2025). This places ProGen2 in continuity with ProGen while also marking a shift in emphasis: the original ProGen paper foregrounded controllable generation through explicit tags, whereas the survey-grounded description of ProGen2 presents it primarily as a scaled autoregressive generator whose property awareness emerges from pretraining rather than from a rich conditioning interface. A plausible implication is that ProGen2’s importance lies not only in its direct generative performance, but also in clarifying which capabilities arise from large-scale causal modeling alone.

2. Model family, architecture, and objective

ProGen2 is a family of autoregressive transformer decoder models trained with next-token prediction (Nijkamp et al., 2022). The reported family spans five named scales:

Model	Parameters
ProGen2-small	151M
ProGen2-medium	764M
ProGen2-base	764M
ProGen2-large	2.7B
ProGen2-xlarge	6.4B

The models use standard left-to-right causal masking, rotary positional encodings, and a GPT-J-like parallelization of attention and MLP blocks. The paper writes the parallel form as

$$x_{t+1} = x_t + {\rm attn}({\rm ln}(x_t)) + {\rm mlp}({\rm ln}(x_t)),$$

in contrast to a sequential form in which the MLP is applied after the attention-updated hidden state. The stated purpose of this parallel form is improved efficiency. ProGen2 is trained as a causal LLM with maximum likelihood:

$p(x) = \prod_{i=1}^n p(x_i \mid x_{<i}),$

and is evaluated with perplexity, including the explicit token-level formulation

$$ppl(x) = \exp\left(-\frac{1}{n} \sum_{i=1}^n \ln(\text{softmax}(\text{logits}(x)[i]))[x_i]\right).$$

The model release includes the ProGen2 models and code at https://github.com/salesforce/progen (Nijkamp et al., 2022). In the survey literature, ProGen2 is consistently treated as a decoder-style sequence model that generates amino-acid tokens sequentially, rather than as a masked encoder or a multimodal architecture (Xiao et al., 21 Feb 2025). This point matters for interpretation: its strengths and limitations are anchored in causal density modeling of protein sequences.

3. Pretraining corpora and the centrality of data distribution

A defining feature of ProGen2 is its emphasis on training data distribution rather than on parameter count alone (Nijkamp et al., 2022). The standard ProGen2 models are trained on mixtures involving UniRef90 and BFD30, where BFD30 is described as a large database mostly from metagenomic sources and clustered at 30% sequence identity. The paper also defines a BFD90 variant consisting of UniRef90 mixed with representative sequences from clustered UniProtKB, Metaclust, SRC, and MERC at 90% identity; this variant is approximately twice the size of UniRef90 and is described as majority metagenomic. Across these settings, the training corpus comprises over a billion protein sequences from genomic, metagenomic, and immune repertoire databases.

For antibody modeling, the paper introduces ProGen2-OAS, trained on the Observed Antibody Space. The original OAS collection contains 1.5B antibody sequences from 80 immune repertoire sequencing studies and includes heavy and light chains from six species; after clustering at 85% identity, 554M sequences are used for training (Nijkamp et al., 2022). These antibody-specific experiments are important because they decouple domain-specific sequence abundance from broader universal protein pretraining.

The survey literature compresses this picture into a simpler summary, listing UniRef50 as the training dataset for ProGen2 in the generation taxonomy (Xiao et al., 21 Feb 2025). The original paper, however, provides the more detailed account of mixed genomic, metagenomic, and immune-repertoire sources. This suggests that ProGen2’s empirical behavior should be read through the lens of corpus composition and bias, not merely through a single dataset label.

4. Sequence-distribution modeling and generation behavior

On held-out test sets, ProGen2 reports monotonically decreasing perplexity with scale, from 12.9 / 15.0 for ProGen2-small to 9.9 / 13.9 for ProGen2-xlarge on Test-max90 and Test-max50, respectively (Nijkamp et al., 2022). The paper interprets this as evidence that larger models better capture the distribution of observed evolutionary sequences. In the generation setting, the universal ProGen2-xlarge model was used to sample 5,000 sequences, after which structures were predicted with AlphaFold2. The reported median TMscore is 0.89 and the median pLDDT is 73.7, with many generated sequences showing very low sequence identity to natural proteins but substantial structural similarity. The examples include a solenoid fold with 11.2% sequence identity to the closest PDB hit, a multi-domain $\alpha + \beta$ fold with no close structural analogue, and an RTX-like $\beta$-roll fold described as an “idealized” version of the natural counterpart.

The paper also studies finetuned fold-specific generation by finetuning ProGen2-large on 1M proteins from Gene3D/CATH with a two-layer sandwich architecture (Nijkamp et al., 2022). In that setting, more finetuning makes generated sequences more like natural sequences, higher sampling temperature or nucleus probability yields more novel sequences, and lower temperature or more restrictive sampling yields structures closer to known proteins. The result is not a single optimum but a controllable trade-off between novelty and naturalness.

Antibody generation provides a further domain-specific view. ProGen2-OAS generates both unprompted antibodies and prompted sequences beginning with the motif EVQ; prompted generation yields more complete full-length antibodies, looser sampling improves developability through lower aggregation propensity and higher solubility, and ranking generated antibodies with the universal ProGen2-base model improves solubility while aggregation changes only marginally (Nijkamp et al., 2022). The broader significance is that ProGen2’s generative prior is not confined to unconditional sampling; it can also function as a useful scoring model for downstream library filtering.

5. Zero-shot fitness prediction and the mismatch between likelihood and fitness

A central contribution of ProGen2 is the demonstration that sequence likelihoods can support fitness prediction without additional finetuning (Nijkamp et al., 2022). On narrow fitness landscapes, average Spearman correlations are reported as 0.456 for ProGen2-small, 0.505 for ProGen2-base, 0.485 for ProGen2-large, 0.476 for ProGen2-xlarge, and 0.518 for a ProGen2 ensemble. Comparative baselines in the same summary include RITA-XL at 0.443, EVE at 0.511, Tranception without retrieval at 0.447, Tranception with retrieval at 0.503, MSA Transformer at 0.476, and ESM-1v (single) at 0.475. The important empirical point is that the best single-model ProGen2 performance on narrow landscapes occurs at 764M parameters rather than at the largest scale.

For wider or more out-of-distribution landscapes, the pattern changes. Reported metrics include AAV AUC improving from 0.59 to 0.68 across scale, GFP AUC from 0.51 to 0.84, CM AUC from 0.68 to 0.64, and GB1 top100avg from 0.01 to 0.85 (Nijkamp et al., 2022). The GB1 result is singled out as especially notable and is discussed as possible emergent behavior in a difficult, low-homology, epistatic setting. In antibody landscapes, universal protein pretraining often outperforms the antibody-specific OAS model: average binding $\rho$ values range from 0.40 to 0.44 for the universal models versus 0.37 for ProGen2-OAS, and average general-property $\rho$ values reach 0.74 for ProGen2-xlarge versus 0.66 for ProGen2-OAS.

The paper’s interpretation is explicitly distributional. It distinguishes the empirical training distribution $p_0$ from the evolutionary stationary distribution $p^\infty$, notes that $\log p^\infty \propto \log f$, and also states that generally $p(x) = \prod_{i=1}^n p(x_i \mid x_{<i}),$0 (Nijkamp et al., 2022). A recurrent misconception is that lower perplexity or larger scale should uniformly imply better biological fitness prediction. ProGen2 argues against that conclusion: better modeling of $p(x) = \prod_{i=1}^n p(x_i \mid x_{<i}),$1 does not necessarily produce a better proxy for fitness-relevant biology, and misspecification can make an intermediate-sized model more useful than the largest one on narrow landscapes.

6. Controllability, adapters, and multimodal descendants

In the survey taxonomy, ProGen2 is presented primarily as a sequence-only autoregressive generator rather than as a highly controllable multimodal model (Xiao et al., 21 Feb 2025). Later work therefore uses ProGen2 as a foundation to which control mechanisms are added. ProCALM is a direct example: it finetunes ProGen2-base as the autoregressive backbone for conditional protein generation, but injects conditions through continuous conditional adapters rather than through prompting alone (Yang et al., 2024). The ProCALM paper argues that prompting-style methods such as CTRL/ZymCTRL or prefix-tuning are limited to tokenizable, relatively simple conditions and have not demonstrated robust generalization to out-of-distribution functions. By contrast, the adapter route is described as allowing conditioning to live in a learned latent space, so the model can represent richer function descriptions, combine multiple conditions, and interpolate toward rare or unseen conditions.

Architecturally, ProCALM uses ProGen2-base as a decoder-only autoregressive model with 27 layers, 1536 embedding dimension, and 764 million parameters; in the parameter-efficient setting, the original ProGen2 weights are frozen and only the conditioning encoder plus the adapter layers are trained (Yang et al., 2024). The paper explores one-hot encoding of the EC hierarchy, DRFP fingerprints, and CREEP embeddings as function representations, simple one-hot taxonomy over bacteria, eukaryota, archaea, and viruses, and joint EC-taxonomy conditioning through parallel adapters. It reports that ProCALM matches the state of the art ZymCTRL on common EC conditioning, preserves better diversity in some settings, supports taxonomy and joint EC-taxonomy conditioning, and shows evidence of out-of-distribution behavior on rare and held-out ECs. At the same time, it explicitly notes that exact generation for held-out ECs remains difficult, and that better evaluation oracles are needed.

InstructPro extends ProGen2 in a different direction, using ProGen2-base to initialize InstructPro-1B and ProGen2-large to initialize InstructPro-3B (Song et al., 11 Jun 2025). Here ProGen2 serves as decoder initialization inside a system conditioned on natural-language functional instructions and ligand SMILES. Compared with the ProGen2 baseline, which is used with a prefix of the first 32 tokens, InstructPro reports substantially stronger ligand-binding design metrics on its benchmark. On seen ligands, for example, ProGen2 reports DockSR Moderate 66.09%, RMSD 7.773, $p(x) = \prod_{i=1}^n p(x_i \mid x_{<i}),$2 RMSD 56.40%, and pLDDT 72.361, whereas InstructPro-1B reports 81.52%, 4.026, 72.86%, and 88.046, respectively (Song et al., 11 Jun 2025). The comparative lesson is not that ProGen2 already possessed these multimodal capabilities, but that its autoregressive protein prior is a useful substrate for later instruction-following and ligand-aware design.

7. Compute-optimal re-evaluation and enduring limitations

Subsequent scaling-law work treats PROGEN2-xlarge as a major causal language modeling baseline but argues that its original training allocation is not compute-optimal under a fitted efficient frontier (Cheng et al., 2024). Using a 939 million-sequence, 194 billion-token dataset called UniMeta200B, and training over 300 models from 3.5M to 10.7B parameters, that study fits power laws for CLM and MLM and concludes that the compute budget used by PROGEN2-xlarge would have been better spent on a 7.2B CLM model trained on 265B tokens rather than on a 6.4B model trained on 350B tokens. In the direct comparison, PROGEN2-xlarge is listed with 16 heads, dimension 4096, 32 layers, and FLOPs of $p(x) = \prod_{i=1}^n p(x_i \mid x_{<i}),$3, while the alternative 7.2B CLM uses 32 heads, the same dimension, 36 layers, and FLOPs of $p(x) = \prod_{i=1}^n p(x_i \mid x_{<i}),$4.

On protein generation evaluation, the compute-optimal 7.2B CLM is reported to outperform PROGEN2-xlarge on several metrics (Cheng et al., 2024). After generating 40,000 sequences per model and filtering duplicates and high-perplexity sequences, the study retains 8,263 sequences for the 7.2B CLM and 8,466 for PROGEN2-xlarge. The compute-optimal model achieves higher average ESMFold pLDDT, 78.69 versus 74.33; better Foldseek similarity to PDB, with TM-score 0.655 versus 0.522 and SeqID 0.194 versus 0.165; and more clusters, 7,097 versus 4,818. The paper’s conclusion is not that ProGen2 is weak, but that it is a strong generation-focused CLM whose training-token allocation is suboptimal relative to the fitted compute frontier.

These later assessments sharpen the main limitations associated with ProGen2. The survey literature presents it as a sequence-only generator without explicit structure-aware or multimodal control (Xiao et al., 21 Feb 2025). ProCALM shows that controllability can be grafted onto ProGen2 through adapters, but also that out-of-distribution functional discovery remains open rather than solved (Yang et al., 2024). The compute-optimal study shows that naive scaling is not equivalent to efficient scaling, while also noting a caveat: its laws are learned in a regime where most models train for at most one epoch and protein data repetition is limited, so the exact optimum may shift under different data or training regimes (Cheng et al., 2024). Taken together, these works position ProGen2 as a foundational causal protein generator whose enduring significance lies as much in the questions it exposed—about data distribution, controllability, and compute allocation—as in the capabilities it directly demonstrated.