- The paper introduces SEAL, a two-stage framework that combines transcriptomic pretraining with LoRA-based contrastive and reconstruction learning to align histology images with spatial gene expression across 723,250 paired samples.
- Spatial transcriptomics-guided fine-tuning improves molecular slide-level tasks by an average of 1.5%, raises patch-level gene-expression correlation by up to 17.3%, and reduces scanner-dependent clustering in multi-scanner testing.
- SEAL enables robust cross-modal retrieval, achieving 3.1× higher image-to-gene correlation than OmiCLIP while highlighting limitations from Visium resolution, distribution shifts, and modest task-specific clinical gains.
Motivation and problem statement
Pathology foundation models (FMs) trained on large histopathology image collections, either vision-only or vision-language, have become the standard feature extractors for whole-slide image (WSI) analysis. However, existing multimodal extensions rely on bulk RNA-seq or language captions, neither of which provides the spatial resolution needed to link fine-grained morphological patterns to their molecular correlates. Spatial transcriptomics (ST) offers spatially resolved gene expression profiles that can be co-registered to histology, but prior work on morphology–ST integration has focused almost exclusively on patch-level gene expression prediction, without establishing whether such alignment improves general-purpose downstream clinical tasks.
The paper introduces SEAL (Spatially Enhanced Alignment Learning), a vision–omics self-supervised finetuning framework designed to infuse localized molecular information into pretrained pathology vision encoders rather than training new encoders from scratch. The central hypothesis is that cohesive vision–omics alignment on diverse paired morphology–ST data can enhance image-based clinical tasks while enabling new cross-modal capabilities such as gene-to-image retrieval.
Method
SEAL operates in two stages. In Stage I, a variational autoencoder (SEAL-omics) is trained exclusively on a transcriptomic reconstruction task over an aligned gene panel of 2,000 highly variable genes supplemented with 584 COSMIC Tier 1 genes. To handle the high sparsity and heterogeneous variance of ST data, the VAE is augmented with planar normalizing flows for a more expressive posterior, and the reconstruction objective combines a scale-invariant Barlow Twins-style correlation loss with a magnitude-aware MSE term. This design choice is validated by ablation: models trained only with Huber, L1, or MSE losses show lower and less stable Spearman and Pearson correlations during pretraining.
In Stage II, the pretrained vision encoder—equipped with low-rank adaptation (LoRA) matrices at rank 8—is jointly optimized with three objectives: an InfoNCE contrastive loss aligning image and gene embeddings, an image-to-gene reconstruction loss ensuring embeddings are directly predictive of expression, and a gradient-reversal domain-adaptation penalty encouraging invariance to study-of-origin batch effects. LoRA serves both parameter efficiency and mitigation of catastrophic forgetting; ablations show full finetuning yields marginally better patch-level prediction (+0.5%) but degrades slide-level task performance (−1.2%), confirming the trade-off between molecular adaptation and preservation of general-purpose morphological features.
Training uses MAPLE (Morphology Aligned Dataset with Paired Local Expression), comprising 723,250 paired H&E patches (224×224 px) and Visium ST spots from 353 patients across 14 organs, drawn from HEST-1k, public studies, and internal Mass General Brigham data. Splits are stratified by patient to prevent leakage.
SEAL was applied to five FMs spanning ViT-Base to ViT-Huge and multiple pretraining recipes: CONCH, H-Optimus-mini, Phikon-v2, UNI-v2-h, and Virchow-v2. Evaluation covers 38 slide-level tasks across 14 datasets: 18 molecular status tasks, 14 pathway expression tasks, and marker/treatment response tasks, aggregated via both attention-based MIL (ABMIL) and mean pooling.
All five encoders benefit from SEAL. On molecular subtyping, SEAL achieves an average improvement of 1.5% across encoders and tasks under ABMIL, with Virchow-v2-SEAL showing the largest relative gain (+3.1%) and UNI-v2-SEAL the smallest (+0.6%). Mean pooling shows a similar trend (+1.3% average). Virchow-v2-SEAL also outperforms Virchow-v2 finetuned with the PathOmCLIP recipe on slide-level tasks, indicating that even with ST guidance, the finetuning recipe must be carefully optimized to transfer gains across spatial scales. Gains on purely morphological tasks (grading, subtyping) are smaller, which the authors attribute to saturated performance from large-scale DINOv2 pretraining on >100M patches—a plausible but untested explanation.
A notable secondary finding concerns robustness. Using the ICAIRD dataset of 100 renal cell carcinoma slides digitized on five scanners, SEAL embeddings form weaker scanner-dependent clusters than baseline embeddings: average Adjusted Rand Index drops from 0.67 to 0.44 and Mutual Information from 1.25 to 0.9 (significant for four of five encoders, excluding Phikon-v2). This suggests multimodal molecular finetuning suppresses non-biological acquisition artifacts, consistent with prior observations that multimodal FMs are more robust than unimodal ones.
Patch-level gene expression prediction
On MAPLE-Test-70k (six organs), linear probes on frozen SEAL embeddings significantly outperform vision-only baselines across all five encoders (Wilcoxon signed-rank, p<0.001). Virchow-v2 gains the most (+17.3% average PCC; e.g., prostate PCC rises from 0.483 to 0.576) and H-Optimus-mini the least (+7.3%). MSE reductions follow the same pattern, up to 11% for Virchow-v2. On the out-of-distribution HEST-Bench (nine organs, including Xenium-based tasks despite no Xenium training data), improvements are smaller (Phikon-v2 largest at +1.33% average PCC), reflecting distributional shift from Visium training data—an honest limitation of the evaluation.
Against dedicated ST prediction baselines (STNet, BLEEP, OmiCLIP, PathOmCLIP, all using Virchow-v2 where applicable), SEAL leads on all six test organs with an average PCC gain of 2.4% over the next-best model (lung +16.1%, kidney +0.4%). Two results are particularly instructive. First, predictive approaches (SEAL, STNet, PathOmCLIP) consistently outperform retrieval-based approaches (BLEEP, OmiCLIP), implying that explicitly learning a morphology-to-expression mapping generalizes better than nearest-neighbor retrieval—even when retrieval references include samples from the same cohorts as training data. Second, STNet—a simple ridge probe on frozen FM embeddings—outperforms more complex finetuning frameworks, underscoring that strong frozen representations plus simple heads remain a formidable baseline.
Ablations and design considerations
Four ablations support the recipe's components. Removing Stage I transcriptomic pretraining costs 1.48% patch-level and 1.57% slide-level performance, confirming the value of structured unimodal embeddings before cross-modal alignment. Replacing LoRA with full finetuning trades slide-level performance for marginal patch-level gains, as noted above. Data-scaling experiments (1%, 10%, 100% of MAPLE-Train, matched iteration counts) show monotonic improvement with data volume for Virchow-v2, though some per-task results are noisy—for example, CPTAC COAD TP53 drops −8.1% at 1% data before recovering +8.0% at 100%. Finally, the scale-invariant reconstruction objective outperforms pure regression losses throughout pretraining.
Cross-modal retrieval
On two internal prostate cancer cohorts processed at distinct sequencing cores (94,142 query spots vs. 31,192 support spots, introducing staining batch effects), SEAL image-to-gene retrieval achieves on average 3.1× higher PCC than OmiCLIP across gene panels, and dramatically outperforms BLEEP despite sharing the same Virchow-v2 backbone (PCC 0.146 vs. 0.003 for all genes). The SEAL decoder further outperforms SEAL retrieval in zero-shot settings, reinforcing the conclusion that retrieval-based ST prediction is limited by embedding-space quality alone. These results indicate that the combined contrastive-plus-reconstruction objective produces a joint embedding space more robust to processing-induced batch effects than contrastive-only alternatives.
Gene-to-image retrieval is evaluated qualitatively: pathway-derived transcriptomic queries (e.g., androgen signaling, myogenesis) activate tumor and stromal regions respectively in prostate cancer slides, consistent with known biology, whereas BLEEP and OmiCLIP activate nearly identical regions for functionally opposite pathways. Because there is no well-defined quantitative metric for gene-to-image retrieval, this evidence remains interpretive rather than statistically validated.
Limitations and open questions
The paper concedes several constraints. Evaluation covers five FMs but not all publicly available models. Training relies on Visium spot-resolution data (55 µm spots aggregating multiple cells), not single-cell resolution; transferability to Xenium or Visium HD is suggested only indirectly by moderate HEST-Bench Xenium gains and remains unverified at scale. The omics encoder is a lightweight MLP-based VAE rather than a heavier single-cell transcriptomics FM, a choice justified by dataset size but potentially limiting expressiveness. Slide-level gains, while consistent, are modest (average ~1.5% on molecular tasks), and individual tasks show variability—including occasional significant regressions—so the practical benefit for any specific clinical endpoint requires case-by-case validation. Whether the emergent batch-effect robustness persists under larger or more heterogeneous scanner distributions is likewise untested beyond the five-scanner ICAIRD setup.
Conclusion
SEAL demonstrates that localized molecular supervision from paired ST–histology data can be infused into existing pathology foundation models through a parameter-efficient, architecture-agnostic finetuning recipe, yielding consistent improvements across 38 slide-level and 15 patch-level tasks, improved scanner robustness, and a geometrically meaningful joint embedding space supporting cross-modal retrieval. The consistency of gains across five backbones of different sizes and pretraining methods suggests ST-guided finetuning is a general capability rather than a dataset-specific effect, positioning molecular alignment as a viable post-training stage for computational pathology.