CellSymphony: Single-Cell Multimodal Analysis

• CellSymphony is a multimodal framework that fuses Xenium spatial gene expression with cell-level histology for true single-cell pathomics.
• It leverages pretrained encoders (scGPT for RNA and UNI2 for morphology) to generate robust, high-dimensional embeddings for precise cell characterization.
• Transformer-based fusion and contrastive learning within CellSymphony enhance cell type annotation and niche discovery across breast, lung, and prostate cancers.

CellSymphony is a modular multimodal framework for integrating spatial gene expression data from 10x Genomics Xenium with cell-level morphological information extracted directly from paired high-resolution H&E histology images, with the goal of achieving single-cell pathomics at true single-cell spatial and molecular resolution. It leverages foundation model-derived embeddings from Xenium transcriptomic profiles and histology images, learns joint representations that fuse spatial gene expression with morphological context, and is designed for accurate cell type annotation and discovery of tissue microenvironmental niches across breast, lung, and prostate cancers (Acosta et al., 13 Aug 2025).

1. Definition and analytical scope

CellSymphony is positioned around a specific technical problem: Xenium provides subcellular-resolution profiling of complex tumor tissues, but extracting robust cell-level features from histology and integrating them with spatial transcriptomics remains difficult. The framework addresses this by coupling molecular and morphological measurements at the level of individual cells rather than relying on global or tile-based image features (Acosta et al., 13 Aug 2025).

Within the terminology of the source paper, the central objective is “single-cell pathomics,” meaning the joint analysis of cell-resolved transcriptomic state, histomorphology, and spatial context. The framework is explicitly described as flexible and modular. A plausible implication is that CellSymphony is intended not merely as a task-specific classifier, but as a general multimodal representation-learning stack for tissue ecosystems in which cell identity and tissue organization must be read simultaneously from molecular and image-derived signals.

2. Modality-specific embeddings at true single-cell resolution

The framework begins with paired embeddings for each cell, one transcriptomic and one morphological. For transcriptomic embedding, each cell’s gene count vector from Xenium, comprising up to approximately 5,000 genes, is embedded using scGPT, yielding a fixed-length transcriptomic feature vector. For morphology, the same cell is represented by a 224 $\times$ 224 px H&E patch centered at the nucleus at 0.5 $\mu$m/px, processed by UNI2 (Acosta et al., 13 Aug 2025).

A defining implementation detail is that CellSymphony does not use global or tile-based histology features. Instead, it extracts intermediate spatial tokens from UNI2 and aggregates those spatially closest to the nucleus. The resulting morphology representation is a 1536-dimensional embedding per cell. This design ties fine-grained morphological context directly to the indexed cell, rather than to a broader field of view.

The use of scGPT for RNA and UNI2 for H&E places CellSymphony within the broader foundation-model paradigm in computational biology and digital pathology. In the paper’s formulation, these pretrained encoders are used as transfer-learning components that provide generalizable, high-quality cell representations. This suggests that the framework’s performance depends in part on the prior biological structure already captured by the pretrained models, with CellSymphony contributing the cell-level pairing and fusion strategy.

3. Transformer architectures for multimodal cell type annotation

Cell type annotation in CellSymphony is organized around several transformer variants that fuse gene, morphology, and spatial information in different combinations (Acosta et al., 13 Aug 2025).

Architecture	Inputs	Distinguishing feature
Unimodal Transformer	Gene (GEB) or morphology (MEB) embeddings	Single-modality processing
Spatial Transformer	Gene or morphology with spatial coordinates	Sinusoidally encoded spatial coordinate tokens
Dual-Modality Transformer	Gene and morphology embeddings	Modality-specific tokens for both inputs
Multi-Input Transformer	Gene, morphology, and spatial tokens	Relative positional attention for cross-modality interactions

Across these variants, each input is projected into a shared embedding space and processed by a 6-layer transformer encoder. Outputs are mean pooled and passed through a linear head for classification. Training uses AdamW for 20 epochs with batch size 64, learning rate $3 \times 10^{-5}$, and cross-entropy loss with class balancing.

The paper describes the basic transformer fusion conceptually through fused tokens of the form $$[\mathrm{CLS}; \mathbf{z}_{\text{GEB}}; \mathbf{z}_{\text{MEB}}; \mathbf{z}_{\text{Spatial}}]$$, followed by transformer fusion, mean pooling, and a linear head. In this formulation, the Multi-Input Transformer is the most expressive variant because it jointly exposes the encoder to molecular state, morphology, and explicit spatial context. The reported results indicate that this added structure is not uniformly necessary across all tissues, but becomes particularly informative in tissue architectures with strong regional organization.

4. Cross-modal alignment through contrastive learning

Beyond supervised cell type classification, CellSymphony uses unsupervised contrastive learning to align transcriptomic and morphological representations in the absence of explicit cell-type labels (Acosta et al., 13 Aug 2025). Each modality is projected through its own non-linear projection head into a 128-dimensional latent space. The objective is an InfoNCE loss that attracts gene and morphology embeddings from the same cell and repels embeddings from different cells in the same batch:

$$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\mathrm{sim}(\mathbf{u}_i, \mathbf{v}_i)/\tau)}{\sum_{j=1}^{N} \exp(\mathrm{sim}(\mathbf{u}_i, \mathbf{v}_j)/\tau)}$$

Here, $\mathbf{u}_i$ and $\mathbf{v}_i$ are the gene and morphology projections for cell $i$, $\mathrm{sim}$ is typically cosine similarity, and $\tau$ is a temperature parameter. The loss is computed bidirectionally, from morphology to transcriptomics and vice versa, to promote a shared aligned space.

After training, UMAP and clustering are used to assess separation of cell types and spatial niches. The paper reports that this contrastive alignment produces morphology-derived projections that reveal refined microenvironmental structure. In lung, lymphocyte-rich clusters with high B- and T-cell enrichment become sharply spatially localized. In breast, macrophage/fibroblast compartments are better resolved and epithelial subtypes stratify into distinct clusters. In prostate, gradients of epithelial differentiation and clearer separation of glandular structures are observed, matching known histology.

A notable clarification follows from these results: the advances in niche discovery do not require gene expression at inference. Once trained, the morphology model alone identifies meaningful biological structure. This directly addresses a common misconception that multimodal training necessarily implies multimodal deployment.

5. Empirical evaluation across breast, lung, and prostate cancers

CellSymphony was evaluated on breast cancer with $\mu$0 cells, lung cancer with $\mu$1, and prostate cancer with $\mu$2 (Acosta et al., 13 Aug 2025). Two forms of ground truth were used: AI-sTIL, described as morphology-based segmentation/classification, and SingleR, described as reference-based transcriptomic annotation.

The reported results indicate that gene-only models, including unimodal and spatial transformers, show decent but variable performance. The source gives concrete examples: prostate lymphocyte classification under SingleR can be as low as $\mu$3, whereas lung tumoral cells can reach $\mu$4. This variability establishes the baseline difficulty of cell typing from a single modality.

The Dual-Modality Transformer, which combines gene and morphology embeddings, yields large gains. In breast tissue using AI-sTIL labels, the reported $\mu$5 scores for fibroblasts, lymphocyte, and tumor rise to $\mu$6. The Multi-Input Transformer, which additionally incorporates spatial tokens, improves performance further, especially in lung tissue. Under SingleR labels in lung, the paper reports epithelial $\mu$7, T cell $\mu$8, and macrophage $\mu$9.

These results support two specific interpretations given in the paper. First, multimodal fusion improves annotation relative to unimodal alternatives. Second, spatial context mattered most in lung, likely due to structurally distinct immune/tumor regions. The latter is framed in the source as an interpretation rather than a direct mechanistic proof, and it situates the strongest gains in tissues where regional organization is especially pronounced.

6. Significance, modularity, and future directions

The paper characterizes CellSymphony as enabling robust, accurate single-cell-level annotation by leveraging the synergy between gene expression and high-resolution morphology, thereby overcoming noise and ambiguity inherent in either modality alone (Acosta et al., 13 Aug 2025). It is further described as outperforming unimodal and patch-based approaches, especially in complex tissue architectures such as tumor–immune regions and cellular gradients.

The framework is also presented as generalizing across different cancer types without manual tuning, a claim grounded in the reported breast, lung, and prostate evaluations. Because its representations are foundation model-derived and its fusion mechanism is modular, the paper describes it as highly modular and extendable to additional modalities, including spatial proteomics, multiplexed imaging, and graphs based on spatial adjacency, as well as to integration of prior biological knowledge such as gene ontology and regulatory networks.

From an encyclopedic perspective, the significance of CellSymphony lies in the convergence of three design commitments: true single-cell pairing between Xenium and histology, transformer-based multimodal fusion for supervised annotation, and contrastive cross-modal alignment for unsupervised niche discovery. The source paper presents these together as a framework for deciphering the physiological and phenotypic orchestration of cells within complex tissue ecosystems. A plausible implication is that CellSymphony occupies an intermediate position between spatial omics integration, computational pathology, and representation learning: it is neither purely an annotation pipeline nor purely a foundation-model application, but a multimodal single-cell analysis framework built around the correspondence between morphology, transcriptomic state, and tissue context.