Scaling Bio-xLSTM beyond the billion-parameter regime

Assess the performance of Bio-xLSTM models when scaled to parameter regimes exceeding one billion parameters, including their training feasibility and downstream effectiveness across DNA, protein, and small-molecule sequence tasks.

Background

The paper evaluates Bio-xLSTM at moderate scales (hundreds of thousands to hundreds of millions of parameters) and demonstrates strong results across DNA, protein, and chemical sequence tasks. However, training and evaluation at billion-parameter scale introduces additional challenges related to compute, optimization stability, and generalization across data domains.

The authors explicitly state that understanding Bio-xLSTM’s behavior and performance beyond the billion-parameter regime remains an open question, indicating a need for future work on scaling laws, training protocols, and comprehensive evaluation at larger scales.

References

Finally, assessing Bio-xLSTM's performance in parameter regimes beyond the billion scale remains an open question.

— Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences (2411.04165 - Schmidinger et al., 6 Nov 2024) in Section 6, Discussion