RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks (2403.00043v2)

Abstract: While RNA has recently been recognized as an interesting small-molecule drug target, many challenges remain to be addressed before we take full advantage of it. This emphasizes the necessity to improve our understanding of its structures and functions. Over the years, sequencing technologies have produced an enormous amount of unlabeled RNA data, which hides a huge potential. Motivated by the successes of protein LLMs, we introduce RiboNucleic Acid LLM (RiNALMo) to unveil the hidden code of RNA. RiNALMo is the largest RNA LLM to date, with 650M parameters pre-trained on 36M non-coding RNA sequences from several databases. It can extract hidden knowledge and capture the underlying structure information implicitly embedded within the RNA sequences. RiNALMo achieves state-of-the-art results on several downstream tasks. Notably, we show that its generalization capabilities overcome the inability of other deep learning methods for secondary structure prediction to generalize on unseen RNA families.

• The paper introduces RiNALMo, a 650M-parameter RNA language model trained on 36M sequences to achieve state-of-the-art structure prediction.
• The model employs techniques like RoPE, SwiGLU, and FlashAttention-2 to enhance efficiency and enable robust cross-family generalization.
• RiNALMo also excels in functional tasks such as splice-site and ribosome loading predictions, broadening its impact in computational biology.

An Expert Overview of RiNALMo: A General-Purpose RNA LLM

The paper "RiNALMo: General-Purpose RNA LLMs Can Generalize Well on Structure Prediction Tasks" introduces a novel approach in the domain of computational biology by applying a large-scale LLM to RNA sequences. The proposed model, RiNALMo, a RiboNucleic Acid LLM, represents a significant advancement at the intersection of machine learning and bioinformatics, focusing on the structural prediction capabilities of RNA.

Model and Dataset

RiNALMo is distinguished by its substantial size, featuring 650 million parameters, which is a significant scale, especially within the context of RNA LLMs. It was pre-trained using 36 million non-coding RNA sequences sourced from several publicly available databases, including RNAcentral, Rfam, and others. The architecture leverages advanced techniques such as a BERT-style Transformer encoder, incorporating architectural innovations like rotary positional embedding (RoPE), SwiGLU activation, and FlashAttention-2 for efficient training.

Structural Prediction Capabilities

The central thesis of the paper posits that RiNALMo can implicitly capture structural information embedded within RNA sequences, outperforming existing methods in secondary structure prediction tasks. In quantitative assessments, RiNALMo achieved state-of-the-art results on various benchmarks, including intra-family and inter-family RNA secondary structure prediction tasks. Notably, it showcases remarkable generalization to RNA families not encountered during training, a significant leap over conventional deep learning models that struggle with cross-family generalization for such tasks.

Functional Tasks and Evaluation

Beyond structural prediction, RiNALMo was evaluated on tasks associated with RNA functions. It demonstrates robust performance in multi-species splice-site prediction, outperforming existing specialized models like SpliceBERT and Spliceator. Furthermore, in predicting mean ribosome loading (MRL), RiNALMo exhibited superior generalization capability to human UTRs, despite being fine-tuned exclusively on randomly derived sequences.

Architectural and Training Considerations

The paper underscores the architectural choices and training regimen that contribute to the model's performance. Techniques such as RoPE and the SwiGLU activation function are highlighted for their roles in enhancing model capacity and representation. Pre-training, conducted using masked LLMing (MLM) with a carefully curated dataset, provides RiNALMo a robust foundation, facilitating its effective application in both structural and functional RNA tasks.

Implications and Future Work

The introduction of RiNALMo holds multiple implications for computational biology. The ability to predict RNA structures and functions with enhanced generalization opens up new possibilities in RNA-related studies and drug discovery. The success of RiNALMo suggests that LLMs, traditionally used for natural language processing, can be repurposed to decode the complex biological information contained in RNA sequences.

Looking forward, the authors suggest exploring the model's application in tertiary structure prediction tasks, potentially creating a unified framework for RNA structural and functional prediction. The findings might inspire further research into other biomolecular LLMs, extending similar methodologies to proteins, DNAs, and other complex biological molecules.

Conclusion

RiNALMo represents a substantial stride in RNA LLMing, offering a more generalized approach to RNA structure prediction tasks. Its ability to effectively generalize across RNA families unseen during training challenges the current limitations of deep learning approaches. The model's applicability to a wide range of tasks, coupled with its robust architecture, positions it as a promising tool for advancing RNA research and applications, ultimately signaling a broader shift towards data-driven methods in understanding biological complexity.