Mix-Geneformer: Unified scRNA-seq Modeling

• The paper introduces Mix-Geneformer, a unified Transformer-based model that jointly pretrains on human and mouse scRNA-seq data to capture both shared and species-specific gene patterns.
• The model employs a hybrid self-supervised objective combining masked language modeling and SimCSE-style contrastive learning to improve cell-type classification and zero-shot transfer.
• Despite strong performance, the approach faces challenges in computational cost, reproducibility, and missing methodological details such as masking ratio and gene vocabulary alignment.

Searching arXiv for the primary paper and closely related work mentioned in the provided material. Using the arXiv search tool to retrieve the Mix-Geneformer paper and related entries. Mix-Geneformer is a Transformer-based foundation model for unified representation learning across human and mouse single-cell RNA sequencing (scRNA-seq) data. It was introduced to address the species-specific pretraining regime of Geneformer and Mouse-Geneformer, which limits cross-species generalization and translational use. Mix-Geneformer jointly pretrains on a mixed corpus of approximately 50 million cells and combines masked language modeling (MLM) with SimCSE-style contrastive learning to learn cell representations that capture both shared and species-specific gene patterns (Nishio et al., 10 Jul 2025). In the reported experiments, the model supports cell-type classification, zero-shot transfer, and in silico perturbation, while also exhibiting limitations in computational cost, zero-shot variability, and reproducibility due to unspecified implementation details (Nishio et al., 10 Jul 2025).

1. Conceptual framing and motivation

Mix-Geneformer was proposed in response to a practical limitation of species-specific foundation models for scRNA-seq. Geneformer is pretrained on human data and Mouse-Geneformer on mouse data; this species-matched setup yields strong within-species performance but constrains cross-species transfer and translational applications. The central motivation is that conserved biological programs co-exist with species-specific regulation, so separate models cannot simultaneously represent shared and divergent structure and may be brittle under domain shift between species (Nishio et al., 10 Jul 2025).

The model’s stated objective is to learn a single unified representation across human and mouse scRNA-seq such that one pretrained model can support cell-type classification in either species, in silico perturbation for hypothesis generation and cross-species disease modeling, and zero-shot transfer across species and tissues without task-specific fine-tuning (Nishio et al., 10 Jul 2025). This suggests a shift from species-isolated pretraining toward comparative transcriptomic modeling in which shared latent structure is learned implicitly through mixed-corpus self-supervision rather than through explicit ortholog constraints.

The paper positions Mix-Geneformer against both species-specific Transformer models and other scRNA-seq representation learners. Relative to Geneformer and Mouse-Geneformer, its distinctive features are joint human-plus-mouse pretraining and the addition of a SimCSE contrastive objective. Relative to scDeepSort and scVAE, it is presented as a Transformer-based alternative operating on rank-ordered gene tokens (Nishio et al., 10 Jul 2025).

2. Architecture and input representation

Mix-Geneformer adopts a BERT-style encoder with 6 Transformer encoder layers, 4 self-attention heads, hidden dimension 256, SiLU activation, and a maximum input sequence length of 2,048 tokens per cell (Nishio et al., 10 Jul 2025). The paper does not report the total parameter count, classifier head design, or the use of special tokens such as $[CLS]$ (Nishio et al., 10 Jul 2025).

The model follows the “genes as tokens, cells as sequences” paradigm. Each cell is converted to a fixed-length sequence of up to 2,048 informative genes using a rank-value encoding procedure. The ordering is determined by within-cell rank, from highest to lower values, thereby encoding relative expression structure rather than raw count magnitude alone (Nishio et al., 10 Jul 2025). Sparse zeros are handled implicitly through preprocessing and feature selection; genes with zero expression are unlikely to appear among the top 2,048 selected genes (Nishio et al., 10 Jul 2025).

A notable point is that the paper does not specify the gene vocabulary size, an explicit human-mouse ortholog mapping, or a unified gene identifier namespace. It also does not describe species embeddings, species-specific adapters, or domain-adversarial components (Nishio et al., 10 Jul 2025). Consequently, the cross-species unification mechanism is procedural rather than architectural: a shared encoder is exposed to mixed-species corpora under common self-supervised objectives.

The core preprocessing scheme is rank-value encoding. For a cell with raw counts $e_i$ for gene $i$, normalization and scaling are defined as

$\tilde{e}_i = \frac{e_i}{\sum_j e_j}$

and

$$s_i = \tilde{e}_i \cdot \mathrm{median}\{\tilde{e}_k : \tilde{e}_k > 0\} \cdot 10{,}000.$$

The top $L=2{,}048$ genes with the largest $s_i$ are selected, and within-cell ranks are assigned by descending order:

$r_i = \mathrm{rank}_{\text{desc}(s_i)}.$

The paper emphasizes that this encoding focuses on high-variance, informative genes that define cell state, but it does not provide an explicit global variance weighting formula beyond the within-cell selection induced by $s_i$ (Nishio et al., 10 Jul 2025).

3. Self-supervised objectives and cross-species unification

Mix-Geneformer is trained with a hybrid self-supervised objective combining MLM and SimCSE-style contrastive learning. The total loss is reported as

$$L_{\mathrm{total}} = L_{\mathrm{MLM}} + L_{\mathrm{SimCSE}}.$$

No weighting coefficients are reported; the formulation therefore implies equal weighting of the two components (Nishio et al., 10 Jul 2025).

For MLM, a subset $e_i$0 of input tokens is masked and the model predicts the original tokens from surrounding context:

$e_i$1

The mask ratio and masking strategy are not reported (Nishio et al., 10 Jul 2025). This omission is significant because masking design can materially affect inductive bias in gene-token models.

The contrastive component uses SimCSE-style learning. For each cell, two slightly perturbed views form a positive pair, while other views in the batch serve as negatives. Cosine similarity is defined as

$e_i$2

and the reported loss is

$e_i$3

The augmentation mechanism used to create the two views and the temperature $e_i$4 are not specified (Nishio et al., 10 Jul 2025).

Cross-species unification is attributed to joint training on a mixed human-mouse corpus together with hybrid self-supervision that encourages context-aware token modeling and stable cell-level embeddings. The paper does not describe explicit ortholog alignment, cross-species positive pairs, species embeddings, or domain-adversarial training (Nishio et al., 10 Jul 2025). A plausible implication is that species alignment emerges indirectly from shared training statistics and the representational pressure of a common encoder, rather than from an explicit biological correspondence objective.

4. Data, preprocessing, and training protocol

The pretraining corpus, termed Mix-Genecorpus-50M, contains approximately 50,000,000 cells pooled from Genecorpus-30M for human and Mouse-Genecorpus-20M for mouse (Nishio et al., 10 Jul 2025). Quality control removes doublets, ambient RNA contamination, and low-viability cells (Nishio et al., 10 Jul 2025). After QC, rank-value encoding is applied cell by cell.

For downstream evaluation, the reported human organs are spleen, brain, immune, kidney, large intestine, liver, lung, pancreas, and placenta. The mouse organs are brain, heart, kidney, large intestine, limb muscle, mammary gland, spleen, thymus, and tongue. Each organ is split 80/20 into train and test sets (Nishio et al., 10 Jul 2025).

The pretraining hyperparameters reported in the paper are limited but specific. The model uses 6 layers, 4 heads, $e_i$5, SiLU activation, maximum sequence length 2,048, batch size 8 per GPU, maximum learning rate $e_i$6 with a cosine scheduler, and 10 pretraining epochs (Nishio et al., 10 Jul 2025). The optimizer, number and type of accelerators, wall-clock pretraining time, and memory footprint are not reported (Nishio et al., 10 Jul 2025).

Fine-tuning for cell-type classification uses maximum length 2,048, 10 epochs, AdamW, peak learning rate $e_i$7, batch size 8, and 500 warmup steps (Nishio et al., 10 Jul 2025). The reported hardware is eight A100 GPUs, and training took approximately 3 days (Nishio et al., 10 Jul 2025). No additional regularization or curriculum is described.

The absence of several implementation details is an important part of the model’s documented profile. The paper does not report the MLM masking ratio, the SimCSE temperature, the augmentation design, the vocabulary alignment strategy, or released code and checkpoints (Nishio et al., 10 Jul 2025). These omissions limit exact reproducibility.

5. Empirical performance

The principal evaluation domains are cell-type classification, zero-shot transfer, UMAP structure preservation, and in silico perturbation. In mouse cell-type classification, Mix-Geneformer matches or slightly exceeds Mouse-Geneformer in several fine-tuned settings. Reported examples include brain at 97.3% for Mix-Geneformer with contrastive learning versus 96.9% for Mouse-Geneformer, kidney at 95.6% versus 94.9%, and thymus at 97.5% versus 97.0% (Nishio et al., 10 Jul 2025).

In zero-shot mouse evaluations, Mix-Geneformer without contrastive learning often performs best among the compared variants. Examples include limb muscle at 97.2% for Mix-Geneformer without contrastive learning versus 96.3% for Mouse-Geneformer, spleen at 95.8% versus 94.2%, and thymus at 89.3% versus 84.4% (Nishio et al., 10 Jul 2025). This indicates that the effect of the contrastive term is not uniformly beneficial across deployment regimes.

In human cell-type classification, Human-Geneformer slightly leads overall in fine-tuned settings, but the gaps are described as minor. Reported examples include spleen at 98.9% for Human-Geneformer versus 98.3% for Mix-Geneformer with contrastive learning, and brain at 96.8% versus 95.9% (Nishio et al., 10 Jul 2025). In zero-shot human results, Human-Geneformer generally leads, but Mix-Geneformer with contrastive learning is reported as best on placenta at 80.7% versus 75.2% (Nishio et al., 10 Jul 2025).

The paper also compares Mix-Geneformer with scDeepSort and scVAE on mouse organs. For fine-tuned classification, the reported brain accuracies are 97.3 for Mix-Geneformer, 96.9 for Mouse-Geneformer, 58.5 for scDeepSort, and 76.2 for scVAE; for kidney the corresponding values are 95.6, 94.9, 57.6, and 56.2 (Nishio et al., 10 Jul 2025). These results place Mix-Geneformer at or near the best-performing method among the reported baselines on the evaluated mouse datasets.

UMAP analyses are used as qualitative evidence for manifold preservation. For mouse spleen, Mix-Geneformer preserves global topology and local density relative to Mouse-Geneformer, with cluster centroids aligning within approximately 1 UMAP unit in both axes and with comparable compactness for key populations such as NK/T cells and the erythroid lineage (Nishio et al., 10 Jul 2025). For human spleen, Mix-Geneformer and Human-Geneformer similarly show cluster centroids differing by approximately 1 UMAP unit while maintaining relative spatial relationships among major cell populations (Nishio et al., 10 Jul 2025).

6. In silico perturbation and interpretability

Mix-Geneformer is also evaluated as an in silico perturbation model. The procedure described in the paper perturbs a target gene by decreasing its within-cell rank or value to simulate deletion, or increasing it to simulate activation, then recomputes the cell embedding and measures the cosine similarity shift toward a predefined disease-state embedding (Nishio et al., 10 Jul 2025). The cosine similarity used in this analysis is

$e_i$8

If the perturbed embedding becomes more similar to the disease reference, the gene is inferred to drive that transition (Nishio et al., 10 Jul 2025).

The reported perturbation results include several genes with in vivo support. In mouse diabetic kidney disease activation experiments, Spp1, Sgk1, Cfb, and Rock2 yield positive cosine shifts; a reported example is Sgk1 with rank 19, cosine_shift $e_i$9, and $i$0 (Nishio et al., 10 Jul 2025). In mouse ADPKD, deletion of Umod shifts embeddings toward ADPKD, with rank 19, cosine_shift $i$1, and $i$2 (Nishio et al., 10 Jul 2025). In human Alzheimer’s activation experiments, HSPB1, S100B, and HSP90AA1 are recovered; one example is HSPB1 with rank 11, cosine_shift $i$3, and $i$4 (Nishio et al., 10 Jul 2025).

The paper reports p-values for these perturbation findings but does not specify the statistical test used (Nishio et al., 10 Jul 2025). Interpretability is therefore demonstrated primarily through perturbation-response ranking and UMAP consistency rather than through attention analysis, attribution methods, or gradient-based gene importance. The absence of mechanistic interpretability tools is explicit in the description; any use of saliency or attention-based explanation would be an extension beyond what is reported (Nishio et al., 10 Jul 2025).

7. Positioning, limitations, and broader context

Mix-Geneformer differs from Geneformer and Mouse-Geneformer in three stated respects: training jointly on human and mouse rather than on a single species, adding a SimCSE contrastive objective, and using the same rank-value preprocessing to construct sequences of length 2,048 (Nishio et al., 10 Jul 2025). Empirically, it matches or exceeds species-specific models in several fine-tuned mouse settings, remains competitive in some zero-shot settings, and supports in silico perturbation across both species (Nishio et al., 10 Jul 2025).

At the same time, the paper identifies several limitations. Pretraining on 50 million cells is computationally intensive, but the hardware and cost are not disclosed (Nishio et al., 10 Jul 2025). Zero-shot performance varies substantially across tissues and species; the paper notes that including SimCSE improves some cases, such as placenta, but degrades others, including some mouse zero-shot settings (Nishio et al., 10 Jul 2025). Reproducibility is further impeded by unspecified details such as MLM mask ratio, masking strategy, SimCSE augmentations, contrastive temperature, and gene vocabulary or ortholog mapping strategy (Nishio et al., 10 Jul 2025). Code, checkpoints, data-access links, licensing, and curation scripts are not provided in the paper (Nishio et al., 10 Jul 2025).

The broader significance of Mix-Geneformer lies in its demonstration that mixed-species pretraining plus hybrid self-supervision can produce a single representation space useful for comparative transcriptomics. This suggests a foundation-model regime in which translational hypotheses, perturbational analyses, and cross-species classification are handled by one encoder rather than by separate species-specific models. The paper itself points to future work in additional organisms and diseases, and to mechanisms for balancing shared versus species-specific features, including ortholog-aware objectives, species embeddings, and domain-adversarial training (Nishio et al., 10 Jul 2025).

A related but distinct use of the term “Mix-Geneformer” appears in microbial genomics, where MicroFuse is described as a protein-to-genome mixture-of-experts fusion model for operon reasoning and is explicitly characterized as “precisely the kind of multimodal, expert-routed ‘Mix-Geneformer’” that integrates gene or protein identity with genome-context organization (Cho, 9 May 2026). In that usage, “Mix-Geneformer” refers more broadly to a mixture-of-experts fusion layer operating over gene- and genome-scale foundation model representations (Cho, 9 May 2026). This suggests a broader conceptual family in which unified biological representation learning may proceed either through mixed-corpus self-supervision, as in the human-mouse scRNA-seq model, or through expert-routed multimodal fusion, as in operon reasoning.