Large scale paired antibody language models (2403.17889v1)

Abstract: Antibodies are proteins produced by the immune system that can identify and neutralise a wide variety of antigens with high specificity and affinity, and constitute the most successful class of biotherapeutics. With the advent of next-generation sequencing, billions of antibody sequences have been collected in recent years, though their application in the design of better therapeutics has been constrained by the sheer volume and complexity of the data. To address this challenge, we present IgBert and IgT5, the best performing antibody-specific LLMs developed to date which can consistently handle both paired and unpaired variable region sequences as input. These models are trained comprehensively using the more than two billion unpaired sequences and two million paired sequences of light and heavy chains present in the Observed Antibody Space dataset. We show that our models outperform existing antibody and protein LLMs on a diverse range of design and regression tasks relevant to antibody engineering. This advancement marks a significant leap forward in leveraging machine learning, large scale data sets and high-performance computing for enhancing antibody design for therapeutic development.

• The paper presents IgBert and IgT5, advanced models that leverage extensive antibody sequence data to enhance design accuracy.
• It employs transformer architectures with a masked language model strategy, significantly improving sequence recovery and property prediction.
• The models offer practical benefits for antibody engineering, accelerating therapeutic development by elucidating sequence-structure-function relationships.

Advancements in Antibody Design Through Large-Scale Paired Antibody LLMs

Introduction

Antibodies, essential components of the adaptive immune system, hold significant promise in therapeutic development due to their ability to precisely target antigens. The advent of next-generation sequencing (NGS) has provided a comprehensive view of the antibody repertoire, unveiling the immense heterogeneity and specificity inherent in antibody sequences. This plethora of data, while invaluable, poses significant challenges in data handling and interpretation, necessitating sophisticated computational approaches for effective utilization. In this context, the development and application of protein LLMs, inspired by advances in natural language processing, have emerged as powerful tools in deciphering the complex "language" of protein sequences.

Large Scale Paired Antibody LLMs

The paper presents two novel antibody-specific LLMs, IgBert and IgT5, which stand out due to their ability to adeptly handle both paired and unpaired variable region sequences. Trained on the Observed Antibody Space (OAS) dataset—comprising over two billion unpaired sequences and two million paired sequences—these models represent a significant leap in leveraging large-scale datasets for enhancing antibody design.

Data Preparation and Model Training Strategy

A comprehensive strategy involving initial pre-training on general protein sequences, further pre-training on unpaired antibody sequences, and fine-tuning on paired sequences ensures that the models capture both the general grammar of protein sequences and the specific nuances of antibody sequences. The models employ transformer architectures and are trained using a masked LLM (MLM) objective, promoting a deep understanding of the intricate relationships between amino acid residues.

Performance Evaluation on Key Tasks

IgBert and IgT5 models demonstrate superior performance across various design and regression tasks relevant to antibody engineering compared to existing models. Notably, they show remarkable accuracy in sequence recovery tasks, especially in predicting the hypervariable regions crucial for antigen recognition. Furthermore, when applied to downstream tasks such as predicting binding affinities and expression levels, the models outperform state-of-the-art protein and antibody LLMs, underscoring their potential in facilitating the design of more effective therapeutics.

Practical Implications and Future Directions

The ability of IgBert and IgT5 models to accurately predict antibody sequences and properties has profound implications for antibody engineering. By enhancing our understanding of sequence-structure-function relationships, these models can significantly accelerate the development of antibodies with desirable characteristics, such as higher affinity, specificity, and better expression levels. Looking ahead, the integration of structural data and the application of these models in generative tasks hold promise for the de novo design of novel antibody candidates, further extending their utility in therapeutic development.

Conclusion

The development of IgBert and IgT5 represents a pivotal advancement in the application of LLMs to the field of antibody design. By effectively harnessing the vast datasets of antibody sequences, these models provide powerful tools for unlocking the potential of antibodies as therapeutics. As we continue to refine these models and explore new applications, the prospects for antibody-based interventions in treating a wide array of diseases appear ever more promising.