---
title: 'SPATIA: Multimodal Spatial Transcriptomics Model'
url: https://www.emergentmind.com/topics/spatia
type: topic
---

# SPATIA: Multimodal Spatial Transcriptomics Model

SPATIA is a multi-scale, multimodal model for image-based spatial transcriptomics that learns unified, spatially aware representations of single cells, local neighborhoods, and whole tissues by jointly modeling cell-resolved morphology, gene expression, and spatial context. In the reported formulation, it supports both predictive tasks—such as cell annotation, clustering, gene expression prediction, cross-modal prediction, and biomarker status prediction—and generative tasks that synthesize high-resolution cell images conditioned on gene expression. The model is trained on a multi-scale dataset consisting of 17 million cell-gene pairs, 1 million niche-gene pairs, and 10,000 tissue-gene pairs across 49 donors, 17 tissue types, and 12 disease states [2507.04704].

## 1. Biological setting and multi-scale data model

SPATIA is designed for settings in which cellular phenotype is governed jointly by intrinsic morphology, molecular state, and spatial organization. The underlying data are image-based spatial transcriptomics measurements, with each cell represented by a cropped cell image, a gene expression vector, and a spatial coordinate. Formally, the cell-level dataset is
\[
\mathcal{D}_{\text{cell}} = \{(C_i, g_i, s_i)\}_{i=1}^M,
\]
where \(C_i \in \mathbb{R}^{H \times W \times 3}\) is a high-resolution RGB cropped image, \(g_i \in \mathbb{R}^G\) is a gene expression vector, and \(s_i = (x_i, y_i) \in \mathbb{R}^2\) is a 2D spatial coordinate [2507.04704].

The MIST dataset organizes these measurements at three hierarchical scales. At the cell level, MIST-C contains approximately 17 million cell-gene pairs. At the niche level, MIST-N contains approximately 1 million niche-gene pairs obtained by tiling slides into non-overlapping \(256 \times 256\) pixel niches and aggregating expression within each tile. At the tissue level, MIST-T contains 10,000 tissue-gene entries summarizing whole-slide or large-region organization [2507.04704].

| Scale | Representation | Reported size |
|---|---|---|
| Cell | Cell image, cell gene vector, cell coordinate | ~17 million cell-gene pairs |
| Niche | \(256 \times 256\) image patch, pooled gene profile, niche coordinates | ~1 million niche-gene pairs |
| Tissue | Collection of niche embeddings with positional metadata and aggregated expression | 10,000 tissue-gene pairs |

A niche \(j\) is defined by a fixed-size square region, an image patch \(N_j\), a cell set \(\mathcal{C}_j\), and an aggregated gene expression vector
\[
g^n_j = \sum_{i \in \mathcal{C}_j} g_i.
\]
This construction makes local microenvironmental context an explicit modeling object rather than an implicit by-product of cell-level features. At the tissue level, SPATIA uses collections of niche representations and their positions within the slide to capture large-scale architecture [2507.04704].

## 2. Hierarchical architecture and multimodal fusion

SPATIA combines modality-specific encoders, cross-attention fusion, and hierarchical spatial transformers. At the cell level, cell images are processed by a ViT-based image encoder \(E_{\text{cell}}\), yielding a sequence of morphological tokens
\[
X_i^c = E_{\text{cell}}(C_i) \in \mathbb{R}^{N_{\text{patch}} \times D}.
\]
The reported implementation uses ViTMAEForPreTraining, with encoder hidden size 768, 12 layers, MLP intermediate size 3072, decoder hidden size 512, and 8 decoder layers [2507.04704].

Gene expression is encoded by a gene encoder \(E_{\text{gene}}\) based on scPRINT:
\[
x_i^g = E_{\text{gene}}(g_i) \in \mathbb{R}^D.
\]
The reported gene encoder uses embedding dimension 256, vocabulary size 23,122 genes, dropout 0.1, and a FlashTransformer with 8 blocks, hidden size 256, and intermediate size 1024 [2507.04704].

Cell-level multimodal fusion is implemented with cross-attention, using image tokens as queries and the gene embedding as keys and values:
\[
z^c_i = \mathrm{CrossAttn}(Q = X_i^c,\, K = x_i^g,\, V = x_i^g).
\]
This yields a unified single-cell embedding that aligns fine-grained morphology with molecular state. The niche level then aggregates cell embeddings inside each niche,
\[
\bar{z}_j = \frac{1}{|\mathcal{C}_j|}\sum_{i \in \mathcal{C}_j} z^c_i,
\]
and fuses them with niche image tokens:
\[
z^n_j = \mathrm{CrossAttn}(Q = \bar{z}_j,\, K = X_j^n,\, V = X_j^n).
\]
At the tissue level, SPATIA processes niche embeddings with added 2D positional encodings and produces tissue-contextualized representations through another cross-attention stage,
\[
z^t_j = \mathrm{CrossAttn}(Q = z^n_j,\, K = X_k^t,\, V = X_k^t).
\]
The final unified cell embedding combines cell, niche, and tissue information:
\[
z_i = \mathcal{F}_{\mathrm{fusion}}\bigl(z^c_i, z^n_j, z^t_k\bigr).
\]
The reported fusion layers use image fusion dimension 768, expression fusion dimension 256, and dropout 0.1 [2507.04704].

This hierarchy makes spatial context explicit at multiple biological scales. A plausible implication is that SPATIA treats spatial organization not merely as coordinates appended to a feature vector, but as a structured dependency spanning single-cell morphology, local microenvironment, and tissue architecture.

## 3. Generative decoder, token merging, and optimization

SPATIA includes a conditional diffusion model for gene-conditioned cell image generation. Given a cell-gene pair \((C_i, g_i, s_i)\) and a modified gene vector \(g'_i\), the model defines a conditioning embedding
\[
z_{i,\mathrm{cond}} =
\begin{cases}
F_{\mathrm{fusion}}(C_i, g_i, s_i), & \text{training} \\
F_{\mathrm{fusion}}(C_i, g'_i, s_i), & \text{inference}.
\end{cases}
\]
A frozen VAE maps images into latent space, and a lightweight U-Net of approximately 40M parameters denoises those latents under cross-attention conditioning on \(z_{i,\mathrm{cond}}\) [2507.04704].

The diffusion objective is the standard latent denoising loss:
\[
\mathcal{L}_{\mathrm{diff}} = \mathbb{E}_{i,t,\epsilon}\bigl[\|\epsilon - \hat{\epsilon}\|_{2}^{2}\bigr],
\]
where \(\hat{\epsilon} = \epsilon_{\omega}(x_{i,t}, t, z_i)\). To steer generation toward the condition while preserving diversity, SPATIA uses classifier-free guidance:
\[
\tilde{\epsilon}_\theta(\mathbf{x}_t, t)
=
\epsilon_\theta(\mathbf{x}_t, t)
+
w \cdot
\bigl(
\epsilon_\theta(\mathbf{x}_t \mid z_{i,\mathrm{cond}}, t)
-
\epsilon_\theta(\mathbf{x}_t, t)
\bigr).
\]
Token importance is then defined through the absolute difference between conditional and unconditional noise predictions:
\[
I = \left| \epsilon_\theta(\mathbf{x}_t \mid y, t) - \epsilon_\theta(\mathbf{x}_t, t) \right|.
\]
These scores drive a training-free token-merging procedure during inference [2507.04704].

The token-merging mechanism preserves the top \(M\) most important tokens as anchors and merges less important tokens according to cosine similarity in query-key space,
\[
s_{ij}= \frac{q(x_{i})^{\top} k(x_{j})}{\|q(x_{i})\| \|k(x_{j})\|},
\]
followed by gated similarity,
\[
\tilde{s}_{ij}=
\begin{cases}
s_{ij}, & \text{if } s_{ij}\ge \kappa, \\
0, & \text{otherwise}.
\end{cases}
\]
Unlike exclusive many-to-one merging, the reported method allows overlaps, so a token can be associated with multiple anchor tokens. The reported result is approximately 37% faster generation without loss of image quality, as measured by PSNR and SSIM [2507.04704].

Pretraining combines self-supervised reconstruction with diffusion. Image reconstruction uses
\[
\mathcal{L}_{\text{img\_recon}} = \frac{1}{N} \sum_{i=1}^N \| \hat{C}_i - C_i \|_2^2,
\]
and the total reconstruction objective is
\[
\mathcal{L}_{\text{total}}
=
\lambda_{\text{cell}}\, \mathcal{L}_{\text{img\_recon}}
+
\lambda_{\text{gene}}\, \mathcal{L}_{\text{gene\_recon}}.
\]
The hierarchical modules are trained concurrently, with AdamW at learning rate \(1\mathrm{e}{-3}\) for the main model; the diffusion U-Net uses AdamW with learning rate \(1\mathrm{e}{-4}\), cosine decay, 500 warm-up steps, batch size 256, and 25k steps on 3M cell-gene pairs sampled from MIST-C [2507.04704].

## 4. Benchmarks, reported performance, and ablations

SPATIA is benchmarked against 13 existing models across 12 individual tasks spanning cell annotation, cell clustering, gene imputation, cross-modal prediction, and image generation. On cell annotation in the Multiple Sclerosis dataset, SPATIA reports F1 \(= 0.725\) and precision \(= 0.734\), compared with scGPT at \(0.703 / 0.729\), CellPLM at \(0.709 / 0.702\), scBERT at \(0.599 / 0.604\), and CellTypist at \(0.667 / 0.693\). On cell clustering, it reports ARI \(= 0.870\) and NMI \(= 0.831\), compared with CellPLM at \(0.867 / 0.823\), scGPT at \(0.856 / 0.828\), PCA at \(0.843 / 0.812\), and Geneformer at \(0.461 / 0.586\) [2507.04704].

In biomarker status prediction on the BCNB invasive breast cancer cohort, SPATIA reports the highest AUC and balanced accuracy for ER, PR, and HER2. The reported values are \(0.902 / 0.785\) for ER, \(0.825 / 0.731\) for PR, and \(0.744 / 0.643\) for HER2. The strongest comparison model in the reported table, UNI, gives \(0.891 / 0.775\), \(0.820 / 0.712\), and \(0.732 / 0.641\), respectively [2507.04704].

For cross-modal prediction, the reported Pearson and Spearman correlations are 0.43 and 0.41. For cross-modal generation, the reported image quality metrics are PSNR \(= 24.80\) and SSIM \(= 0.65\). Baselines such as UNI and CONCH are listed as not directly supporting image generation in that comparison [2507.04704].

The ablation study isolates the contributions of reconstruction, hierarchy, and multimodal fusion. A cell-level-only model reports loss 0.405 and accuracy 0.93; adding the MAE loss gives 0.396 and 0.94; adding multi-level hierarchy gives 0.369 and 0.97; and the full model with fusion gives 0.361 and 0.98. The reported pattern assigns the largest gain to the multi-level hierarchy, with additional improvement from cross-modal fusion [2507.04704].

Qualitatively, generated cell images are reported as realistic at single-cell resolution and responsive to transcriptomic perturbations. This suggests that the learned representation captures a usable correspondence between gene-expression variation and morphological variation, rather than only supporting discriminative downstream tasks.

## 5. Relation to adjacent spatially aware modeling

SPATIA belongs to a wider methodological trend in which spatial structure is built directly into learning systems rather than treated as a secondary feature. In spatial omics, the statistical framework presented in “pasta” emphasizes that imaging-based assays can be represented as point patterns or irregular lattices and sequencing-based assays as regular lattices, with explicit attention to homogeneity, edge effects, neighbor definitions, and scale [2412.01561]. SPATIA is complementary to that framework: it is a transformer-based predictive and generative model, whereas “pasta” systematizes the use of spatial statistics such as K/L functions, Moran’s I, and local indicators of spatial association.

Comparable design choices appear in other spatial domains. SPATE-GAN introduces a spatio-temporal association metric as an embedding for generative modeling of turbulent flows, log-Gaussian Cox processes, and weather fields [2109.15044]. SPATIAL uses unidirectional temporal LSTMs within sensors and bidirectional spatial LSTMs across sensors for environmental forecasting [2108.11875]. SpaRRTa evaluates whether frozen visual foundation models encode abstract, relational 3D structure between objects [2601.11729]. SPUR injects First-Order Ambisonics features into large audio-language models to support direction, elevation, distance, and overlap reasoning [2511.06606]. Spatia for video generation preserves a 3D scene point cloud as persistent spatial memory and updates it through visual SLAM [2512.15716]. Spadas unifies dataset-level and point-level search over spatial repositories [2412.04805]. The spherical spatial autoregressive model extends SAR-style dependence to responses on unit spheres in finite or infinite-dimensional Hilbert spaces [2601.16385]. TPiet-QL extends SOLAP to temporal queries over spatial objects that split, merge, or otherwise change discretely over time [1108.0831].

This suggests that “spatial awareness” now denotes several distinct but related operations: multiscale cellular context in omics, space-time dependence in environmental fields, relational viewpoint reasoning in vision, listener-centric geometry in audio, persistent scene memory in video, and unified indexing or autoregression in spatial statistics and databases. SPATIA’s distinctive contribution within that landscape is the joint modeling of morphology, transcriptomics, and biological spatial context at cell, niche, and tissue scales [2507.04704].

## 6. Limitations and prospective extensions

The reported limitations are largely tied to data regime, geometry, and interpretability. SPATIA relies on very large paired image-gene datasets, is presently formulated for 2D single-section data, defines niches as fixed non-overlapping \(256 \times 256\) pixel tiles, and is trained on Xenium-like image-based spatial transcriptomics, so transfer to MERFISH, CosMx, or Visium HD may require adaptation or fine-tuning. The paper also notes that more systematic interpretability is needed to relate specific genes and morphological features to model outputs [2507.04704].

The reported future directions include extending the framework to additional spatial omics modalities such as proteins and chromatin accessibility, incorporating temporal dynamics, scaling to larger cohorts and clinical applications, improving niche and tissue modeling with explicit graphs or 3D context, and using the generative module for virtual perturbation screens and in silico experiments [2507.04704].

A plausible implication is that SPATIA is best understood not as a closed architecture for a single benchmark, but as a template for biologically grounded spatial representation learning: modality-specific encoders, cross-attention fusion, hierarchical context aggregation, and conditional generation are all treated as components that can be replaced or extended as spatial omics technologies evolve.

Source: https://www.emergentmind.com/topics/spatia