Function-Guided Conditional Generation Using Protein Language Models with Adapters (2410.03634v2)

Abstract: The conditional generation of proteins with desired functions is a key goal for generative models. Existing methods based on prompting of protein LLMs (PLMs) can generate proteins conditioned on a target functionality, such as a desired enzyme family. However, these methods are limited to simple, tokenized conditioning and have not been shown to generalize to unseen functions. In this study, we propose ProCALM (Protein Conditionally Adapted LLM), an approach for the conditional generation of proteins using adapters to PLMs. While previous methods have used adapters for structure-conditioned generation from PLMs, our implementation of ProCALM involves finetuning ProGen2 to condition generation based on versatile representations of protein function-e.g. enzyme family, taxonomy, or natural language descriptions. ProCALM matches or exceeds the performance of existing methods at conditional sequence generation from target functions. Impressively, it can also generalize to rare and unseen functions. Overall, ProCALM is a flexible and computationally efficient approach, and we expect that it can be extended to a wide range of generative LLMs.

• The paper introduces ProCALM, a parameter-efficient method integrating adapters with PLMs to generate enzyme sequences conditioned on taxonomy and functionality.
• It demonstrates flexible, out-of-distribution generalization by successfully generating sequences for rare or unseen enzyme families.
• The approach reduces computational costs and overfitting risks, paving the way for scalable protein design and advanced enzyme engineering applications.

Conditional Enzyme Generation Using Protein LLMs with Adapters

The research paper introduces ProCALM, a novel approach for conditional protein generation utilizing protein LLMs (PLMs) enhanced with adapters. The paper primarily addresses the limitations of existing prompt-based methods for generating protein sequences conditioned on specific functionalities, such as enzymatic activity constrained by enzyme family taxonomy. These conventional methods often struggle with generalizing to unseen functions and rely heavily on tokenized conditions, which restrict flexibility.

Key Contributions

ProCALM leverages parameter-efficient fine-tuning, employing adapters to existing PLM frameworks, specifically ProGen2. This strategy allows for the seamless incorporation of complex conditioning, such as enzyme class combined with taxonomy, and showcases several advantages:

1. Parameter Efficiency: The ProCALM approach significantly reduces computational cost compared to full-model tuning, requiring only a fraction of the GPU hours used by models like ZymCTRL.
2. Flexibility and Generalization: The architecture supports multi-type conditioning, enabling sequence generation conditioned not only by enzyme functionality but also by taxonomy. Impressively, the model can generate sequences representative of rare or unseen enzyme classes, highlighting its capacity for out-of-distribution generalization.
3. Training and Quality of Sequences: ProCALM performs conditional generation with quality comparable to existing models while maintaining diversity among generated sequences. The generated sequences are robust against overfitting, evidenced by consistent perplexity across held-out datasets.

Implications and Opportunities

The capability to generate protein sequences conditioned on rare or unseen enzyme families presents significant opportunities in protein engineering. This advancement could enhance the design of enzymes for industrial applications, where unannotated or novel functionalities are required.

The use of adapters introduces a scalable and efficient mechanism to customize LLMs for specialized tasks, potentially extending beyond proteins to other biological sequence types.

Theoretical and Practical Implications

Theoretically, ProCALM suggests a viable path for improving the generalization capabilities of PLMs by adopting continuous space conditioning over token-based paradigms. Practically, it paves the way for developing highly specialized proteins with tailored functions, minimizing the reliance on exhaustive experimental datasets.

Future Directions

Future research may focus on improving the representation of conditioning information. Exploration of multi-modal conditioning, integrating structural data, textual descriptions, and sequence relationships, could further refine the specificity and utility of generated proteins. Additionally, expanding training datasets to include metagenomic data may enhance the model's capacity to explore the latent space of protein functions.

ProCALM exemplifies a strategic advancement in the field of computational protein design, offering a flexible, efficient framework poised to transform protein engineering across domains.