Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Single Cells Are Spatial Tokens: Transformers for Spatial Transcriptomic Data Imputation (2302.03038v2)

Published 6 Feb 2023 in q-bio.GN, cs.AI, and cs.LG

Abstract: Spatially resolved transcriptomics brings exciting breakthroughs to single-cell analysis by providing physical locations along with gene expression. However, as a cost of the extremely high spatial resolution, the cellular level spatial transcriptomic data suffer significantly from missing values. While a standard solution is to perform imputation on the missing values, most existing methods either overlook spatial information or only incorporate localized spatial context without the ability to capture long-range spatial information. Using multi-head self-attention mechanisms and positional encoding, transformer models can readily grasp the relationship between tokens and encode location information. In this paper, by treating single cells as spatial tokens, we study how to leverage transformers to facilitate spatial tanscriptomics imputation. In particular, investigate the following two key questions: (1) $\textit{how to encode spatial information of cells in transformers}$, and (2) $\textit{ how to train a transformer for transcriptomic imputation}$. By answering these two questions, we present a transformer-based imputation framework, SpaFormer, for cellular-level spatial transcriptomic data. Extensive experiments demonstrate that SpaFormer outperforms existing state-of-the-art imputation algorithms on three large-scale datasets while maintaining superior computational efficiency.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. SpaGE: spatial gene enhancement using scRNA-seq. Nucleic acids research 48, 18 (2020), e107–e107.
  2. Missing data imputation in the electronic health record using deeply learned autoencoders. In Pacific symposium on biocomputing 2017. World Scientific, 207–218.
  3. Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram. Nature methods 18, 11 (2021), 1352–1362.
  4. Missing data in traffic estimation: A variational autoencoder imputation method. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2882–2886.
  5. NicheNet: modeling intercellular communication by linking ligands to target genes. Nature methods 17, 2 (2020), 159–162.
  6. End-to-end object detection with transformers. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. Springer, 213–229.
  7. Rethinking Attention with Performers. In International Conference on Learning Representations.
  8. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882 (2021).
  9. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  10. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
  11. Graph neural networks with learnable structural and positional representations. arXiv preprint arXiv:2110.07875 (2021).
  12. Single-cell RNA-seq denoising using a deep count autoencoder. Nature communications 10, 1 (2019), 390.
  13. Lovedeep Gondara and Ke Wang. 2017. Multiple imputation using deep denoising autoencoders. arXiv preprint arXiv:1705.02737 280 (2017).
  14. Lovedeep Gondara and Ke Wang. 2018. Mida: Multiple imputation using denoising autoencoders. In Pacific-Asia conference on knowledge discovery and data mining. Springer, 260–272.
  15. DrImpute: imputing dropout events in single cell RNA sequencing data. BMC bioinformatics 19, 1 (2018), 1–10.
  16. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16000–16009.
  17. High-plex Multiomic Analysis in FFPE at Subcellular Level by Spatial Molecular Imaging. bioRxiv (2022). https://doi.org/10.1101/2021.11.03.467020
  18. SAVER: gene expression recovery for single-cell RNA sequencing. Nature methods 15, 7 (2018), 539–542.
  19. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
  20. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems 34 (2021), 21618–21629.
  21. Cell clustering for spatial transcriptomics data with graph neural networks. Nature Computational Science 2, 6 (2022), 399–408.
  22. Wei Vivian Li and Jingyi Jessica Li. 2018. An accurate and robust imputation method scImpute for single-cell RNA-seq data. Nature communications 9, 1 (2018), 1–9.
  23. Sign and basis invariant networks for spectral graph representation learning. arXiv preprint arXiv:2202.13013 (2022).
  24. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision. 10012–10022.
  25. A joint model of unpaired data from scRNA-seq and spatial transcriptomics for imputing missing gene expression measurements. arXiv preprint arXiv:1905.02269 (2019).
  26. Deep generative modeling for single-cell transcriptomics. Nature methods 15, 12 (2018), 1053–1058.
  27. Single-cell in situ RNA profiling by sequential hybridization. Nature methods 11, 4 (2014), 360–361.
  28. Stable, fast and accurate: Kernelized attention with relative positional encoding. Advances in Neural Information Processing Systems 34 (2021), 22795–22807.
  29. Deep Learning in Single-Cell Analysis. arXiv preprint arXiv:2210.12385 (2022).
  30. Reviewing autoencoders for missing data imputation: Technical trends, applications and outcomes. Journal of Artificial Intelligence Research 69 (2020), 1255–1285.
  31. Imputing single-cell RNA-seq data by combining graph convolution and autoencoder neural networks. Iscience 24, 5 (2021), 102393.
  32. Jonathan Ronen and Altuna Akalin. 2018. netSmooth: Network-smoothing based imputation for single cell RNA-seq. F1000Research 7 (2018).
  33. Alexander Y Rudensky. 2011. Regulatory T cells and Foxp3. Immunological reviews 241, 1 (2011), 260–268.
  34. Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 464–468.
  35. Cellpose: a generalist algorithm for cellular segmentation. Nature methods 18, 1 (2021), 100–106.
  36. MAGIC: A diffusion-based imputation method reveals gene-gene interactions in single-cell RNA-sequencing data. BioRxiv (2017), 111591.
  37. Attention is all you need. Advances in neural information processing systems 30 (2017).
  38. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).
  39. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning. 1096–1103.
  40. scGNN is a novel graph neural network framework for single-cell RNA-Seq analyses. Nature communications 12, 1 (2021), 1–11.
  41. Sprod for de-noising spatially resolved transcriptomics data based on position and image information. Nature methods 19, 8 (2022), 950–958.
  42. Zelun Wang and Jyh-Charn Liu. 2021. Translating math formula images to LaTeX sequences using deep neural networks with sequence-level training. International Journal on Document Analysis and Recognition (IJDAR) 24, 1-2 (2021), 63–75.
  43. Bi-channel Masked Graph Autoencoders for Spatially Resolved Single-cell Transcriptomics Data Imputation. In NeurIPS 2022 AI for Science: Progress and Promises.
  44. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10033–10041.
  45. Unsupervised spatially embedded deep representation of spatial transcriptomics. Genome Medicine 16, 1 (2024), 12.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now