Integrated Bayesian non-parametric spatial modeling for cross-sample identification of spatially variable genes (2504.09654v2)

Abstract: Spatial transcriptomics has revolutionized tissue analysis by simultaneously mapping gene expression, spatial topography, and histological context across consecutive tissue sections, enabling systematic investigation of spatial heterogeneity. The detection of spatially variable (SV) genes, which are molecular signatures with position-dependent expression, provides critical insights into disease mechanisms spanning oncology, neurology, and cardiovascular research. Current methodologies, however, confront dual constraints: predominant reliance on predefined spatial pattern templates restricts detection of novel complex spatial architectures, and inconsistent sample selection strategies compromise analytical stability and biological interpretability. To overcome these challenges, we propose a novel Bayesian hierarchical framework incorporating non-parametric spatial modeling and across-sample integration. It takes advantage of the non-parametric technique and develops an adaptive spatial process accommodating complex pattern discovery while maintaining biological interpretability. A novel cross-sample bi-level shrinkage prior is further introduced for robust multi-sample SV gene detection, facilitating more effective information fusion. An efficient variational inference is developed for posterior inference ensuring computational scalability. Comprehensive simulations demonstrate the improved performance of our proposed method over existing analytical frameworks, and its application to DLPFC data reveals interpretable SV genes whose spatial patterns delineate neuroanatomically relevant clusters and gradients, advancing brain transcriptomics.