Can sparse autoencoders make sense of latent representations? (2410.11468v2)

Abstract: Sparse autoencoders (SAEs) have lately been used to uncover interpretable latent features in LLMs. Here, we explore their potential for decomposing latent representations in complex and high-dimensional biological data, where the underlying variables are often unknown. Using simulated data, we find that latent representations can encode observable and directly connected upstream hidden variables in superposition. The degree to which they are learned depends on the type of variable and the model architecture, favoring shallow and wide networks. Superpositions, however, are not identifiable if the generative variables are unknown. SAEs can recover these variables and their structure with respect to the observables. Applied to single-cell multi-omics data, we show that SAEs can uncover key biological processes. We further present an automated method for linking SAE features to biological concepts to enable large-scale analysis of single-cell expression models.