Img2ST-Net: Deep Learning for ST Prediction

• The paper introduces Img2ST-Net, a fully convolutional framework that reformulates histology-to-transcriptomics as an image-to-image prediction problem, addressing signal sparsity and computational bottlenecks.
• It employs a UNet-style architecture with region-wise inference and contrastive alignment to produce dense, multi-channel gene expression maps with robust spatial coherence.
• Quantitative metrics and visual assessments demonstrate superior performance over baselines, making the method a promising tool for advancing high-resolution spatial omics research.

Img2ST-Net is a fully convolutional deep learning framework for high-resolution spatial transcriptomics (ST) prediction from whole-slide histology images. It is designed to address scalability, computational inefficiency, and signal sparsity limitations arising in modern ST platforms such as Visium HD, which offer single-cell or subcellular spatial resolution. Img2ST-Net reformulates histology-to-transcriptomics inference as an image-to-image prediction problem, producing dense multi-channel gene expression maps in a manner that is both efficient and spatially coherent. Notable architectural and algorithmic components include super-pixel map modeling, region-wise inference, and region-level contrastive alignment between image features and frozen gene embeddings. The introduction of a structural-similarity-based evaluation metric, SSIM-ST, further enables robust quality assessment in sparse high-resolution scenarios (Zhu et al., 20 Aug 2025).

1. Network Architecture

Img2ST-Net leverages a UNet-style, fully convolutional architecture optimized for parallel, high-resolution prediction. The model processes an input histology region $X \in \mathbb{R}^{3 \times H \times W}$ (e.g., RGB channels of a $448 \times 448$ pixel patch) and outputs a predicted gene expression tensor $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$, where $C$ is the number of genes and $H' \times W'$ the grid of ST bins in the region. The architecture comprises:

• Encoder: $L$ stages of downsampling, each with two $3 \times 3$ convolution + ReLU layers and $2 \times 2$ max-pooling.

$$F_0 = X; \quad F_\ell = f^{\text{enc}}_\ell(F_{\ell-1}), \quad \ell = 1 \ldots L$$

• Decoder: $L$ upsampling stages, each with bilinear upsampling, skip connections, and two $448 \times 448$0 convolution + ReLU layers.

$448 \times 448$1

• Output head: $448 \times 448$2 convolution projecting decoder output to $448 \times 448$3 gene channels.

$448 \times 448$4

Novel elements include region-wise inference (processing $448 \times 448$5200 HD bins per forward pass) and a contrastive alignment head that aligns decoder features with a frozen gene embedding space. The architecture allows for parallelized, high-throughput prediction, avoiding the computational bottlenecks of spot-wise regression.

2. Super-Pixel Formulation

Rather than treating individual Visium HD bins as isolated targets, Img2ST-Net models the ST output as a "super-content image" with $448 \times 448$6 channels. For each input region $448 \times 448$7,

• Input: $448 \times 448$8
• Output: $448 \times 448$9

Each pixel $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$0 gives the predicted expression of gene $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$1 at spatial bin $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$2. Whole-slide inference is achieved by tessellating the WSI into patches and stitching together the outputs, effectively reconstructing a dense, high-dimensional transcriptome map over the tissue section. This enables the preservation of spatial structure and facilitates efficient computation in contrast to sequential spot regression.

3. Learning Objectives and Evaluation Metrics

Img2ST-Net employs a composite loss combining pixel-level regression and regional contrastive alignment:

• Pixel-wise regression (MSE):

$\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$3

• Regional contrastive alignment: Each ground-truth vector $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$4 is embedded by a frozen MLP $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$5 into $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$6. Decoder features $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$7 are projected by a learnable linear head $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$8 to $\hat{Y} \in \mathbb{R}^{C \times H' \times W'}$9. The InfoNCE loss:

$C$0

where $C$1 denotes cosine similarity, and $C$2 is the temperature.

• Total loss: $C$3 with $C$4.
• SSIM-ST metric: To address instability of Pearson correlation under sparsity, SSIM-ST measures local structural similarity on $C$5 or $C$6 windows for each gene. The SSIM is averaged over all gene channels and windows for a spatially-aware map coherence assessment.

4. Datasets, Preprocessing, and Implementation

Training and evaluation employ Visium HD datasets:

• Breast Cancer (Nagendran et al. 2023): 4 FFPE sections, resolutions 2, 8, 16 µm.
• Colorectal Cancer (Oliveira et al. 2025): 5 sections (3 tumor, 2 normal), same resolutions.
• Experiments focus on 8 and 16 µm due to extreme sparsity at 2 µm.

Preprocessing includes log$C$7 normalization and retention of the top 250 highly expressed genes. Regions of $C$8 pixels ($C$9196 bins at 8 µm) are the effective input unit. Training employs SGD (momentum$H' \times W'$0, weight decay$H' \times W'$1–4), initial learning rate $H' \times W'$2–4 with cosine annealing, batch size 64, and 4$H' \times W'$3NVIDIA RTX A6000 GPUs, converging after 50–100 epochs.

5. Quantitative and Computational Results

Evaluation uses MSE, MAE, and SSIM-ST, with baselines including ST-Net, HisToGene, His2ST, and BLEEP. Key metrics for Img2ST-Net are:

Dataset (Resolution)	MSE	MAE	SSIM-ST
Breast Cancer (8 µm)	0.0647	0.1737	0.0756
CRC (8 µm)	0.2587	0.2711	0.0144
Breast Cancer (16 µm)	0.1657	0.2506	0.0937
CRC (16 µm)	0.7981	0.5208	0.0081

Img2ST-Net achieves the lowest MSE/MAE and highest SSIM-ST in the majority of cases, and demonstrates robust performance across resolutions. Computationally, region-wise inference provides a $H' \times W'$4 speedup at 8 µm versus spot-wise: 0.43 min/epoch vs 12.10 min/epoch, with near-linear scaling as bin count increases.

6. Qualitative and Biological Assessment

Visualizations indicate that Img2ST-Net accurately recapitulates gene expression "hotspots" at tumor margins and around vascular structures, aligning with biological ground truth and preserving microenvironmental boundaries more faithfully than baselines. Gene-wise mean-expression trajectories show close adherence to ground-truth trends. Downstream cell-type deconvolution (e.g., CD8A/CD68 immunocyte localization) shows spatial distributions concordant with known histological patterns. These qualitative and biological validations underscore the coherence and reliability of the predicted ST maps.

7. Ablations, Limitations, and Prospects

Ablation studies on Breast Cancer (8/16 µm) indicate that omitting the contrastive loss increases MSE and lowers SSIM-ST (e.g., MSE $H' \times W'$5 at 8 µm), but region-wise image-to-image inference matches the accuracy of spot-wise baselines while providing a $H' \times W'$6 speed advantage. Noted limitations include:

• Degradation of accuracy and SSIM-ST in highly sparse 2 µm data.
• Current restriction to prediction of the top 250 genes; expanding to genome-wide prediction may require model adaptations such as dimensionality reduction or transformer-based modules.
• Full-scale biological validation on independent samples and integration with cell segmentation pipelines remain open directions.
• There is potential for augmenting the framework with multi-modal input (e.g., immunohistochemistry) or self-supervised pretraining for improved robustness.

Img2ST-Net defines a scalable, resolution-aware paradigm for histology-to-ST prediction, with enhanced computational performance, improved spatial map coherence, and practical viability for high-resolution omics inference (Zhu et al., 20 Aug 2025).