ProteinGym: Protein Fitness Prediction

• ProteinGym Fitness Prediction is a comprehensive framework that benchmarks computational models against diverse deep mutational scanning and variant effect datasets.
• It integrates methods from protein language models, structure-based predictors, meta-learning, and ensemble approaches to enhance predictive accuracy.
• It guides practical applications in protein engineering by leveraging quantitative metrics like Spearman’s correlation and ROC to improve predictions in low-data and complex mutation regimes.

ProteinGym Fitness Prediction refers to computational methods, benchmarks, and model architectures developed for the systematic evaluation and prediction of protein variant effects—specifically, fitness consequences of amino acid substitutions, insertions, or deletions—using the ProteinGym benchmark as a standard testbed. ProteinGym aggregates deep mutational scanning (DMS) and multiplexed assay of variant effect (MAVE) datasets covering diverse protein families, functions, and mutational spectra. Methods evaluated on ProteinGym encompass a broad spectrum of model paradigms, ranging from protein LLMs (PLMs; both masked and autoregressive), structure-based approaches, supervision with DMS data, meta-learning, and ensemble frameworks, providing a comprehensive platform for measuring the generalization, robustness, and biological fidelity of fitness predictors across the protein universe.

1. ProteinGym Benchmark: Scope and Evaluation

ProteinGym is an extensive collection of variant effect assays, designed to enable rigorous, cross-protein comparison of fitness prediction models. Its principal features are:

• Assay Types: Comprises 217 deep mutational scanning (DMS) and multiplexed variant effect assays, totaling ~2.5 million mutants across five phenotype categories: enzymatic activity, protein-protein or ligand binding, expression, organismal fitness (e.g., viral replication), and stability (Notin et al., 2022).
• Mutation Coverage: Includes both single amino acid substitutions and multiple mutants; explicitly incorporates indel (insertion/deletion) assays (69 DMS).
• Function and Taxa Diversity: Proteins span human, eukaryotic, prokaryotic, and viral families with extensive representation of difficult (shallow alignment) cases.
• Evaluation Metrics: Performance is primarily assessed per protein as Spearman’s rank correlation (ρ) between model-predicted scores and experimentally measured fitness, with additional metrics including area under the ROC curve (AUC), top-10% recall, and normalized discounted cumulative gain (NDCG) (Sharma et al., 23 Apr 2025).

2. Model Architectures and Methodological Advances

ProteinGym has catalyzed development across several methodological axes:

• Protein LLMs (PLMs): Both masked LLMs (MLMs, e.g., ESM2, SaProt) and causal autoregressive transformers (e.g., Proust, Tranception) trained on massive unlabeled protein corpora, scoring mutants via pseudo-perplexity or log-likelihood ratios (Bushuiev et al., 2024, Eris, 2 Feb 2026, Notin et al., 2022).
• Structure-Based Predictors: ML models (e.g., ESM-IF1, S3F, SaProt) leveraging 3D structure (experimental or AlphaFold2-predicted), either alone or fused with sequence embeddings. Integrating explicit surface geometric or dynamic features (e.g., S3F, TriFit) further improves accuracy, especially for stability and binding functions (Sharma et al., 23 Apr 2025, Zhang et al., 2024, Cho, 13 Apr 2026).
• Meta-Learning and Few-Shot Generalization: Metalic and FSFP utilize meta-learning and parameter-efficient fine-tuning to achieve rapid adaptation in extreme low-data regimes, leveraging auxiliary assays and test-time adaptation pipelines (Beck et al., 2024, Zhou et al., 2024).
• Test-Time Training (TTT) Frameworks: ProteinTTT customizes pre-trained MLMs to single-sequence targets via rapid self-supervised fine-tuning on the test protein itself, yielding significant gains, especially for proteins lacking deep evolutionary alignments (Bushuiev et al., 2024).
• Multimodal and Ensemble Models: Mechanisms for combining sequence, structure, MSA, evolutionary profiles, or dynamics include mixture-of-experts routers (TriFit), joint transformers (S3F, SaProt), and simple log-probability summation (StructSeq, VenusRAR) (Sharma et al., 23 Apr 2025, Zhang et al., 2024, Tan et al., 30 Jan 2026, Cho, 13 Apr 2026).

3. Mathematical Formulations and Prediction Strategies

Most protein fitness predictors rely on the following zero-shot or few-shot formulations:

• Log-Odds or Pseudo-Perplexity Scoring:
  • For MLMs and hybrid approaches: $$\text{Score(mutant)} = \sum_{i \in T} [\log p(x_i^{\mathrm{mut}} | x_{\{i\}}) - \log p(x_i | x_{\{i\}})]$$
  • where $T$ is the mutated position set (Bushuiev et al., 2024, Zhang et al., 2024).
  • For autoregressive models: $$F_x = \log [P(x^{\mathrm{mut}}) / P(x^{\mathrm{wt}})]$$
  • leveraging full-sequence likelihoods (Notin et al., 2022, Eris, 2 Feb 2026).
• Pseudo-Perplexity: For MLMs, the average negative log-likelihood (masking one position at a time) serves as a robust fitness proxy (Kantroo et al., 2024, Bushuiev et al., 2024).
• Ensemble or Retrieval-Augmented Fusion: Multimodal scores combine predictions from sequence, structure, and alignment-based components via learned or context-aware weights (Sharma et al., 23 Apr 2025, Tan et al., 30 Jan 2026, Cho, 13 Apr 2026).
• Preference or Ranking Losses: Preference-based losses and listwise ranking objectives (e.g., ListMLE, preference-based sigmoid cross-entropy) are commonly used during few-shot fine-tuning or meta-learning to align model outputs with experimental fitness ordering (Beck et al., 2024, Zhou et al., 2024, Teufel et al., 2 Dec 2025).

4. Benchmarked Model Results and Comparative Analysis

Systematic benchmarking on ProteinGym reveals the following performance hierarchies and properties:

• Sequence-Only Zero-Shot Baselines: ESM2-650M achieves Spearman ρ ~0.414; log-odds fitness from Tranception (with inference-time retrieval) and state-of-the-art ensemble approaches reach ρ ~0.455–0.496 (Bushuiev et al., 2024, Notin et al., 2022, Sharma et al., 23 Apr 2025).
• Structure and Surface Fusion: S3F (sequence-structure-surface fusion) improves to ρ = 0.470, and simple ensembling with EVE variants can yield ρ ≈ 0.496 (Zhang et al., 2024, Sharma et al., 23 Apr 2025).
• Multimodal and Agentic Ensembles: Context-aware and chain-of-thought audited ensembles (VenusRAR) set the current record for zero-shot ProteinGym correlation with ρ = 0.551, highlighting the effectiveness of calibrated multimodal aggregation and biophysical reasoning panels (Tan et al., 30 Jan 2026).
• Test-Time or Inference-Time Adaptation:
  • ProteinTTT enables sequence-specific adaptation and achieves state-of-the-art on “hard” targets, especially those lacking deep MSAs (up to +0.05 ρ) (Bushuiev et al., 2024).
  • Inference-only dropout, meta-learning (Metalic), and few-shot transformers (PRIMO, FSFP) close performance gaps in low-data scenarios (up to 0.67 with ~128 labels), substantially elevating few-shot prediction accuracy (Ravuri et al., 31 May 2025, Beck et al., 2024, Teufel et al., 2 Dec 2025, Zhou et al., 2024).
• Indel Scoring: Proust (causal) and ESM2-OFS pseudo-perplexity set the standard for indel benchmarks (ρ = 0.521, 0.574), markedly exceeding previous AR or retrieval-augmented models (Eris, 2 Feb 2026, Kantroo et al., 2024).
• Performance by Task/Difficulty: Largest model gains are seen in proteins with low evolutionary depth, stability and binding assays, and high-mutation-depth or epistatic landscapes. Explicit structure/dynamics embeddings and ensemble designs confer more benefit in these regimes (Sharma et al., 23 Apr 2025, Zhang et al., 2024, Cho, 13 Apr 2026).

5. Methodological Extensions and Practical Recommendations

Practical advances and recommendations distilled from ProteinGym studies:

• Test-time adaptation yields robust, per-protein gain: Self-supervised fine-tuning on the target sequence (ProteinTTT) or inference-time dropout consistently lower pseudo-perplexity/calibrate outputs, particularly in outlier and OOD regimes (Bushuiev et al., 2024, Ravuri et al., 31 May 2025).
• Structure and dynamics are critical for stability and allostery: Additional descriptors (surface topology, B-factors, mode shapes, cross-correlations) should be included for accurate functional impact prediction in especially stability- or binding-focused tasks (Zhang et al., 2024, Cho, 13 Apr 2026).
• Graph-based and epistasis-aware methods: Modeling the sequence landscape as a hypercube graph (EHCube4P) and applying smoothing/denoising is valuable for small, sparse, rugged or highly epistatic assays, though scaling to larger k remains challenging (Daud et al., 20 Jun 2025).
• Few-shot and meta-learned adaptation should combine dataset similarity, LoRA/adapter-based updates, and ranking losses to maximize small-N generalization. (Teufel et al., 2 Dec 2025, Beck et al., 2024, Zhou et al., 2024).
• Benchmarking and task stratification with landscape features: Tools like GraphFLA provide 20 quantitative features (ruggedness, epistasis, neutrality, navigability) that explain 50–67% of inter-task error variance and should be computed for model selection and pipeline design (Huang et al., 28 Oct 2025).

6. Limitations, Open Questions, and Future Research Directions

Key challenges and ongoing areas of development highlighted by ProteinGym-centered work:

• Disordered Region Blind Spots: All current models, including structure/surface-augmented, show sharp drops in predictive power on intrinsically disordered regions; explicit modeling or masking of low-confidence structure is required (Sharma et al., 23 Apr 2025).
• Transfer Beyond Sequence Likelihoods: While masked LM likelihoods and log-odds are effective fitness proxies, higher-order epistasis and non-additive effects remain incompletely modeled. Integration of explicit evolutionary and biophysical priors, beyond local context, is an area for future model innovation (Fan et al., 8 Oct 2025, Zhang et al., 2024).
• Minimum Data Requirements and Adaptation Cost: Even best-in-class meta-learning (Metalic), PRIMO, and FSFP approaches require multiple support mutations for maximal performance on challenging proteins. Further improvements are likely to come from scaling the diversity and size of pre-training DMS libraries, richer representations, and joint multitask/active training strategies (Teufel et al., 2 Dec 2025, Beck et al., 2024).
• Indel-Ready and Generative Workflows: Recent causal models (Proust) and pseudo-perplexity-based encoders (OFS) allow unified handling of substitutions and indels while retaining sequence generative capacity, essential for the next generation of protein design pipelines (Eris, 2 Feb 2026, Kantroo et al., 2024).
• Ensemble Reasoning and Chain-of-Thought Auditing: Agentic frameworks (VenusRAR) leveraging both statistical models and explicit structural reasoning outperform all single-modality or static ensemble predictors in both high-throughput and low-N wet-lab validation (Tan et al., 30 Jan 2026).
• GraphFLA Task Topography: Incorporating explicit quantitative landscape features into modeling and model selection pipelines can substantially close the loop between model performance and biological inference, enabling better task-aware deployment of computational fitness predictors (Huang et al., 28 Oct 2025).

7. Impact and Significance in Computational Protein Engineering

ProteinGym fitness prediction stands as the reference paradigm for quantitative evaluation and systematic comparison of variant effect predictors in protein engineering, functional genomics, and synthetic biology. Advances driven by this benchmark—test-time adaptation, cross-modal fusion, meta-learning, OOD generalization, and rigorous calibration—have substantially expanded the capacity, fidelity, and practical applicability of machine learning models for protein fitness landscapes. The iterative cycle between improved benchmarks and model architectures embodied in the ProteinGym ecosystem is shaping the standards for robust, interpretable, and generalizable protein variant scoring tools and is poised to support the next generation of data-driven protein design strategies (Bushuiev et al., 2024, Cho, 13 Apr 2026, Sharma et al., 23 Apr 2025, Huang et al., 28 Oct 2025, Zhang et al., 2024).