Decode what and how RNA language models encode information about RNA families

Determine what information about messenger RNA (mRNA) and non-coding RNA (ncRNA) families is internally encoded by RNA language models and characterize how these models represent this information within their hidden states.

Background

The paper motivates SAE-RNA by noting that, despite recent progress in RNA LLMs such as RiNALMo, there remains limited understanding of what these models encode internally about RNA families. While prior work in interpretability has focused on input attributions, the internal mechanisms and representations that capture family-level distinctions are not well characterized.

SAE-RNA is proposed to probe hidden states via sparse autoencoders, mapping discovered features to known biological categories. The stated uncertainty highlights a broader need to identify and explain the internal features linked to mRNA and ncRNA family distinctions.