Lyra: An Efficient and Expressive Subquadratic Architecture for Modeling Biological Sequences

Abstract: Deep learning architectures such as convolutional neural networks and Transformers have revolutionized biological sequence modeling, with recent advances driven by scaling up foundation and task-specific models. The computational resources and large datasets required, however, limit their applicability in biological contexts. We introduce Lyra, a subquadratic architecture for sequence modeling, grounded in the biological framework of epistasis for understanding sequence-to-function relationships. Mathematically, we demonstrate that state space models efficiently capture global epistatic interactions and combine them with projected gated convolutions for modeling local relationships. We demonstrate that Lyra is performant across over 100 wide-ranging biological tasks, achieving state-of-the-art (SOTA) performance in many key areas, including protein fitness landscape prediction, biophysical property prediction (e.g. disordered protein region functions) peptide engineering applications (e.g. antibody binding, cell-penetrating peptide prediction), RNA structure analysis, RNA function prediction, and CRISPR guide design. It achieves this with orders-of-magnitude improvements in inference speed and reduction in parameters (up to 120,000-fold in our tests) compared to recent biology foundation models. Using Lyra, we were able to train and run every task in this study on two or fewer GPUs in under two hours, democratizing access to biological sequence modeling at SOTA performance, with potential applications to many fields.

Overview of Lyra: Efficient Modeling of Biological Sequences

The paper presents Lyra, a novel architecture designed for modeling biological sequences with improved efficiency and expressivity, offering subquadratic scaling. Biological sequence modeling is pivotal for understanding molecular functions, and while deep learning models such as CNNs and Transformers have made significant advancements, they often require substantial computational resources. Lyra aims to address these constraints by introducing a biologically-grounded framework through the concept of epistasis, harnessing state space models (SSMs) to capture global sequence interactions and projected gated convolutions (PGCs) for local feature extraction.

Lyra's design integrates SSMs and PGCs, achieving state-of-the-art performance in numerous biological tasks. Noteworthy areas of application include protein fitness landscape prediction, various biophysical property predictions, peptide engineering applications, RNA structure analysis, and CRISPR guide design, among others. The architecture facilitates order-of-magnitude improvements in inference speed and model parameter reduction, with empirical results indicating up to a 120,000-fold reduction in parameters compared to other leading models.

Lyra Architecture and Mathematical Foundation

Deeper exploration into Lyra's architecture reveals that it effectively combines local and global sequence modeling through the use of PGCs and S4D layers, a variant of SSMs. S4D layers approximate polynomial interactions efficiently, offering insight into epistatic interactions — complex phenomena where mutations influence each other within a sequence. The architecture's subquadratic scaling is achieved through mathematical alignment to the Fourier domain via FFT convolutions, which provides computational efficiency benefits over conventional Transformer models with quadratic complexities.

Numerical Results and Performance Metrics

The architecture excels in various biological tasks with significant performance improvement. Specific results include 64x average speedup in inference time and substantial reductions in computational requirements, allowing task execution on minimal hardware—two GPU units within two hours. The impressive parameter reduction of up to 120,000-fold illustrates Lyra’s efficacy and broad accessibility.

Implications of Lyra in Computational Biology

The implications of Lyra's advancements are notable for both theoretical exploration and practical applications. It demonstrates a paradigm shift from reliance on immense computational infrastructures towards efficient, mathematically principled, domain-specific architectures. Potential applications of Lyra include therapeutic development, targeted gene therapy, rapid pathogen surveillance, and improved biomanufacturing processes like enzyme optimization.

Lyra's integration of mathematical structures with biological phenomena challenges prevailing trends in the field, suggesting that further exploration into domain-specific architectures could yield highly efficient models beyond the field of biological sequences. The project's potential scalability and adaptability herald a promising direction for future developments in AI and computational biology.

In summary, this paper contributes to the advancement of biological sequence modeling by successfully aligning mathematical principles with domain specificity, offering substantial reductions in computational requirements and paving the way for democratized access to high-performance biocomputational models.