scMPT: Multimodal Fusion for Single-Cell Analysis

• scMPT is a multimodal fusion model that integrates a single-cell foundation model (scGPT) with text embeddings from ranked gene names.
• It employs frozen encoders and separate dense layers to concat transcriptomic and textual representations for supervised cell-type prediction.
• Empirical results demonstrate that scMPT leverages complementary strengths to improve classification across diverse single-cell datasets.

Searching arXiv for scMPT and closely related single-cell foundation model papers to ground the article in current literature. scMPT is a multimodal fusion model for single-cell analysis that combines a single-cell foundation model, scGPT, with large-language-model-derived text embeddings computed from “cell sentences,” i.e., gene names sorted by decreasing expression with zero-count genes omitted. It was introduced in the context of studying whether LLMs should be used as alternatives to, or complements for, single-cell foundation models, and is positioned explicitly as a complementary approach. The model uses frozen encoders for both modalities, applies separate dense layers to the two embeddings, concatenates the resulting representations, and performs supervised cell-type prediction through a final softmax layer. The motivating claim is that scGPT captures richer transcriptomic structure from large-scale gene expression data, whereas a text embedding model such as Ember-V1 can encode biological knowledge present in language, especially marker genes and some simple expression-pattern regularities (Palayew et al., 14 Jul 2025).

1. Conceptual position and motivation

scMPT was introduced to address a specific asymmetry in contemporary single-cell modeling. Single-cell foundation models such as scGPT have demonstrated strong performance on downstream biological tasks, but they do not directly exploit the large amount of biological knowledge encoded in text. Conversely, recent work using LLMs on single-cell data showed that text models can be competitive on some tasks, but these approaches were often framed as replacements for domain-specific single-cell models rather than as complementary systems (Palayew et al., 14 Jul 2025).

The central motivation for scMPT is therefore representational complementarity. The paper argues that scGPT and LLM-based encoders exhibit different strengths on different datasets, and that these differences can be exploited through fusion rather than model substitution. In this formulation, scGPT contributes transcriptomic structure learned from large-scale gene-expression corpora, while the LLM branch contributes priors derived from textual biology, especially marker-gene associations and regularities recoverable from ranked gene-name sequences (Palayew et al., 14 Jul 2025).

This positioning distinguishes scMPT from methods that treat natural-LLMs as standalone cell-state encoders. A plausible implication is that scMPT belongs to a broader class of multimodal systems in which heterogeneous pretrained encoders are preserved and coupled only at the representation level, thereby minimizing interference between modalities while testing whether the modalities are genuinely complementary.

2. Input representation and architecture

The input to the LLM branch is a “cell sentence,” defined as gene names sorted by decreasing expression, with zero-count genes omitted. An example format given in the paper is: “A cell with genes ranked by expression: RAB3B MT-CO1 CHN1 …” These sentences are encoded by a text embedding model, chosen as Ember-V1 after a comparative benchmark of text encoders on the same task family (Palayew et al., 14 Jul 2025).

The scMPT architecture is deliberately simple. It consists of five stages:

1. A frozen Ember-V1 encoder produces a text-based cell embedding.
2. A frozen scGPT encoder produces a single-cell embedding.
3. Each embedding is passed through its own dense layer.
4. The dense-layer outputs are concatenated.
5. The fused representation is passed to a final output layer for cell-type prediction (Palayew et al., 14 Jul 2025).

The dense layers have output dimension 4096 and use ReLU activation, while the final layer uses softmax (Palayew et al., 14 Jul 2025). Both encoders are left frozen, explicitly to reduce compute and preserve learned domain knowledge. The paper further states that the classification head is intentionally simple so that observed gains are more likely to reflect modality synergy rather than additional capacity in the fusion module (Palayew et al., 14 Jul 2025).

Mathematically, the architecture is summarized as learned projections of the two embeddings,

$$h_{\text{scGPT}} = f_{\text{scGPT}}(z_{\text{scGPT}}), \qquad h_{\text{LLM}} = f_{\text{LLM}}(z_{\text{LLM}}),$$

followed by concatenation,

$h = [h_{\text{scGPT}}; h_{\text{LLM}}],$

and a softmax classifier,

$\hat{y} = \mathrm{softmax}(W h + b).$

The training objective is the usual cross-entropy loss for multi-class classification,

$\mathcal{L} = - \sum_{c} y_c \log(\hat{y}_c).$

The paper notes that this is an implied objective for the softmax classifier rather than a novel custom fusion loss (Palayew et al., 14 Jul 2025).

3. Choice of Ember-V1 and the role of text-derived biological priors

Before constructing scMPT, the study benchmarked several text embedding models on cell sentences in a GenePT-style setup. The compared encoders included ada-002, text-embedding-3-large, all-MiniLM-L12-v2, gte-large-en-v1.5, bge-large-en-v1.5, mxbai-embed-large-v1, GIST-small-embedding-v0, stella-en-400M-v5, and Ember-V1. Ember-V1 was reported as the best-performing open encoder for the authors’ purposes, which motivated its use as the LLM branch in scMPT (Palayew et al., 14 Jul 2025).

The broader significance of this encoder choice lies in the paper’s argument that LLM-derived embeddings can encode biologically meaningful structure even when fed only ranked gene names. The paper attributes Ember-V1’s usefulness to at least two sources. First, it appears to capture marker-gene biology through language-pretrained associations. Second, it exploits simple expression-pattern and lexical regularities that survive the conversion of a cell into a text sequence (Palayew et al., 14 Jul 2025).

This suggests that the text encoder is not serving merely as a generic semantic compressor. Rather, within the limits of the cell-sentence representation, it acts as a repository of textual biological priors that can complement the more directly transcriptomic inductive biases of scGPT. In that sense, scMPT operationalizes a division of labor between biological text priors and expression-model priors.

4. Training protocol, datasets, and evaluation tasks

scMPT is trained as a supervised classifier using AdamW and an exponential decay learning rate schedule. Hyperparameters, specifically learning rate, epochs, batch size, and decay rate, are selected by grid search, and results are reported over 5 random seeds using mean and standard deviation (Palayew et al., 14 Jul 2025).

The study evaluates the model on datasets previously used in GenePT, CELLama, and scGPT evaluations. The datasets listed are Aorta, Pancreas, Bones, Artery, Myeloid, MS, and a Tabula Sapiens subsample (Palayew et al., 14 Jul 2025). The principal downstream tasks are cell-type classification and clustering, with clustering performed by k-means and evaluated using ARI and AMI. An additional disease phenotype prediction task is conducted on Aorta, specifically aneurysm versus control (Palayew et al., 14 Jul 2025).

For text encoders, the evaluation protocol includes zero-shot classification using a 10-nearest-neighbor classifier in embedding space, and clustering by k-means with the number of clusters fixed to the known number of cell types. For scMPT, the procedure is to train on the provided train split, evaluate on the test split, and report accuracy, macro-weighted precision, recall, and F1 (Palayew et al., 14 Jul 2025).

The paper also includes experiments with generative LLMs, although these are distinct from scMPT proper. In those experiments, only 100 randomly selected test cells per dataset are classified due to cost, structured outputs are used, and a top-3 reranking pipeline allows the LLM to choose among the top 3 cell types suggested by scGPT (Palayew et al., 14 Jul 2025). These experiments support the broader complementarity thesis, but the core scMPT formulation remains the frozen-encoder fusion model.

5. Biological interpretation and ablation findings

A central contribution of the scMPT study is its attempt to identify what biological information the LLM branch is contributing. Using integrated gradients and LIME on an Ember-V1 + MLP classifier, the authors report that top-attribution genes often correspond to known marker genes for the predicted cell type. For the Pancreas dataset, the examples given are Alpha: GCG, TTR, NEUROD1, SLC30A8; Beta: INS, IAPP, ADCYAP1; Ductal: KRT19, MMP7; Endothelial: PLVAP, RGCC; and Acinar: PRSS1, PRSS3, REG1A (Palayew et al., 14 Jul 2025).

The paper strengthens this interpretation by comparing cosine similarity among marker-gene embeddings. For Ember-V1, intra-similarity exceeds inter-similarity: in Pancreas, 0.644 versus 0.623; in Aorta, 0.667 versus 0.653 (Palayew et al., 14 Jul 2025). The study interprets this as evidence that the encoder has internalized biologically meaningful marker structure.

The ablation results indicate that performance depends not only on semantic biology but also on sequence-level regularities. Replacing gene names with hashes causes only a moderate drop in performance, but replacing each gene-name instance with a unique random hash causes performance to collapse. Shuffling all genes destroys performance. Shuffling only genes inside the encoder’s context window reduces performance, but not catastrophically. Shuffling only the top 10% of in-context genes causes a disproportionately large drop, while using only the top 10% of in-context genes still gives strong performance (Palayew et al., 14 Jul 2025).

These results support four specific factors identified by the paper: marker-gene knowledge, the top-ranked genes in the cell sentence, gene-order information, and lexical similarity even after gene-name ablation (Palayew et al., 14 Jul 2025). Biologically, the conclusion is that the LLM branch captures canonical cell markers, coarse expression programs, and cell-type-specific rank structure. This explains why Ember-V1 can perform strongly on datasets where marker genes are highly diagnostic, while still being weaker than scGPT on tasks that demand richer expression modeling.

6. Empirical performance and comparison with unimodal baselines

The empirical evaluation shows that scMPT usually matched or exceeded the better of its two component models, although the degree of improvement varied by dataset (Palayew et al., 14 Jul 2025). The pattern is not uniform, and the paper emphasizes complementarity rather than blanket dominance.

The main reported results are as follows:

Dataset/task	Model	Reported result
Pancreas	scMPT	accuracy 0.962, F1 0.745
Pancreas	scGPT full fine-tune (reported)	accuracy 0.968, F1 0.718
Pancreas	Ember-V1 + MLP	accuracy 0.974, F1 0.684
Aorta	scMPT	accuracy 0.971, F1 0.960
Aorta	scGPT + MLP	accuracy 0.968, F1 0.954
Aorta	Ember-V1 + MLP	accuracy 0.940, F1 0.889
Artery	scMPT	accuracy 0.962, F1 0.931
Artery	scGPT + MLP	accuracy 0.961, F1 0.929
Artery	Ember-V1 + MLP	accuracy 0.949, F1 0.903
Bones	scMPT	accuracy 0.684, F1 0.554
Bones	Ember-V1 + MLP	accuracy 0.674, F1 0.541
Bones	scGPT + MLP	accuracy 0.630, F1 0.498
Tabula Sapiens	scMPT	accuracy 0.764, F1 0.290
Tabula Sapiens	scGPT + MLP	accuracy 0.736, F1 0.261
Tabula Sapiens	Ember-V1 + MLP	accuracy 0.748, F1 0.236
Myeloid	scMPT	accuracy 0.664, F1 0.369
Myeloid	scGPT fine-tuned	accuracy 0.642, F1 0.346
Myeloid	scGPT from scratch	accuracy 0.606, F1 0.309
Myeloid	Ember-V1 + MLP	accuracy 0.601, F1 0.325
MS	scMPT	accuracy 0.837, F1 0.704
MS	scGPT fine-tuned	accuracy 0.856, F1 0.703
MS	scGPT + MLP	accuracy 0.845, F1 0.726
MS	Ember-V1 + MLP	accuracy 0.687, F1 0.568
Aorta disease phenotype	scMPT	accuracy 0.894, F1 0.831
Aorta disease phenotype	Ember-V1 + MLP	accuracy 0.863, F1 0.799
Aorta disease phenotype	scGPT + MLP	accuracy 0.862, F1 0.797

On Pancreas, scMPT was reported as competitive with full fine-tuning of scGPT and improved over the unimodal heads in F1 (Palayew et al., 14 Jul 2025). On Aorta and Artery, the gains over scGPT + MLP are small but positive. On Bones and Tabula Sapiens, the fusion yields clearer improvements. On Myeloid, scMPT exceeds the reported fine-tuned scGPT and Ember-V1 + MLP baselines. On MS, scGPT remains strongest on some metrics, and the paper explicitly notes that scMPT is competitive rather than uniformly best (Palayew et al., 14 Jul 2025).

The study also provides zero-shot text-encoder results that contextualize the LLM branch. On Aorta, Ember-V1 achieved accuracy 0.906 and F1 0.841, compared with ada-002 at accuracy 0.872 and F1 0.716, all-MiniLM-L12-v2 at accuracy 0.855 and F1 0.644, and scGPT at accuracy 0.960 and F1 0.949. Across datasets, Ember-V1 was especially strong on Pancreas, with accuracy 0.984 and F1 0.778, and on Bones, with accuracy 0.427 and F1 0.322, although scGPT remained better on some datasets, especially MS (Palayew et al., 14 Jul 2025).

Taken together, these results support the paper’s claim that the two modalities have nonidentical error profiles. This suggests that the benefit of scMPT is not merely averaging but combining representations whose strengths vary with dataset composition, marker-gene salience, and the complexity of the expression structure.

7. Relation to alternative fusion strategies, limitations, and significance

The paper also evaluates a different kind of complementarity: a generative LLM reranking pipeline in which a reasoning model selects among the top 3 cell types predicted by scGPT. On 100-cell subsets, o3-mini performed best overall among generative models, with accuracies of 0.980 on Pancreas, 0.393 on Myeloid, and 0.473 on MS. In the fusion setup, o3-mini + scGPT achieved 0.930 on Pancreas, 0.530 on Myeloid, and 0.763 on MS, outperforming scGPT alone, o3-mini alone on Myeloid and MS, and standard LLM fusion approaches (Palayew et al., 14 Jul 2025). The paper concludes that reasoning models complement scGPT better than standard generative LLMs, though it also states that this route is less effective than representation-level fusion.

Within this framework, scMPT represents the paper’s principal answer to how LLMs should be integrated into single-cell analysis: not as substitutes for single-cell foundation models, but as complementary sources of prior knowledge fused at the representation level (Palayew et al., 14 Jul 2025). That conclusion is explicitly presented as one of the study’s main claims.

The paper also notes several limitations. Evaluation of fusion methods is limited to a small set of datasets and tasks; only cell sentences were studied as text representations; more advanced multimodal fusion architectures were not explored; interpretability methods are mostly local, though partially aggregated; and generative LLM fusion was tested only on small subsets because of cost (Palayew et al., 14 Jul 2025). These limitations are important for constraining interpretation. In particular, they indicate that the present evidence supports the complementarity hypothesis in a specific experimental regime rather than establishing a general theorem about multimodal single-cell learning.

In the context of related literature, scMPT is best understood as a fusion-based single-cell classification framework grounded in the coexistence of two pretrained representation families: transcriptomic foundation models and biology-informed text encoders. Its significance lies less in architectural novelty than in an empirical and interpretive claim: language-pretrained models can contribute real biological knowledge, especially marker-gene knowledge, and that contribution can measurably improve single-cell analysis when fused with scGPT (Palayew et al., 14 Jul 2025).