---
title: 'SCxGEN-32M: Single-Cell Language Pretraining'
url: https://www.emergentmind.com/topics/scxgen-32m
type: topic
---

# SCxGEN-32M: Single-Cell Language Pretraining

Searching arXiv for SCxGEN-32M and related primary sources.
arXiv search query: SCxGEN-32M OKR-CELL single-cell foundation model cell-language pretraining
SCxGEN-32M is a large-scale single-cell–language pretraining resource introduced in connection with the OKR-CELL framework, where it denotes a dataset of **32 million cell-text pairs** constructed from **CELLxGENE** single-cell data and text derived from metadata plus retrieval-augmented large-language-model enrichment [2601.05648]. In this setting, “32M” refers to the number of paired examples rather than a model parameter count. SCxGEN-32M functions as the main pretraining corpus for **OKR-CELL**, an **Open-world Language Knowledge-Aided Robust Single-Cell Foundation Model** designed for cross-modal alignment between scRNA-seq profiles and biomedical text [2601.05648].

## 1. Identity, provenance, and scope

SCxGEN-32M is described as a corpus built from **32 million single-cell data from the CellxGene platform**, including **scRNA-seq matrices** and **corresponding metadata annotations from diverse human tissues and organs** [2601.05648]. The paper states that the dataset consists of **32 million cell-text pairs**, and also states that each cell is associated with **9 key metadata items**, with explicitly mentioned fields including **cell type**, **tissue**, **disease**, **sex / gender**, and **developmental stage** [2601.05648].

The resource is not itself the model. The model in the paper is **OKR-CELL**, while **SCxGEN-32M** is the pretraining dataset/resource on which OKR-CELL is trained [2601.05648]. The paper frames this pairing as a response to two limitations of earlier cell-language pretraining systems: **weak textual supervision** based on sparse metadata and **noise sensitivity** in multimodal cell-text pairs [2601.05648].

A central distinguishing feature is that SCxGEN-32M is organized as a **cross-modal** resource. Each training example is a pair of the form
$$
(\text{cell transcriptome}_i,\ \text{text description}_i)
$$
for \(i = 1,\dots,32{,}000{,}000\) [2601.05648]. This makes the dataset suitable not only for cell-only representation learning, but also for multimodal objectives such as cell-text alignment and bidirectional retrieval.

## 2. Corpus construction and text generation pipeline

The construction pipeline begins with collection of cell profiles and metadata from **CELLxGENE**, followed by metadata standardization using ontology resources, generation of an original text description from metadata, optional enrichment through an **LLM + RAG** workflow, semantic reliability screening, and finally retention of the resulting cell-text pair [2601.05648].

The paper states that metadata terms such as **disease**, **gender**, **tissue**, and **cell type** are mapped to standardized textual descriptions by referencing the **Cell Ontology database**, and also mentions **OBO Foundry** as a standardization resource [2601.05648]. The text side therefore begins as metadata-derived structured biomedical description rather than free-form annotation.

The enrichment stage uses a retrieval-augmented generation pipeline. The knowledge base is assembled from domain resources including **PubMed**; specifically, the paper says that it extracts **only abstracts** and then cleans and splits the corpus into **500-token chunks** [2601.05648]. These chunks are embedded with **BioBERT** and stored in a vector database for semantic retrieval [2601.05648]. At generation time, structured metadata fields such as **Cell Type**, **Tissue**, **Sex**, and **Development Stage** are converted into a query, encoded by **BioBERT**, and used to retrieve top-\(K\) snippets; the retrieved snippets and standardized metadata text are then passed to **DeepSeek-V3** to generate an enriched description [2601.05648]. The exact value of \(K\) is not reported.

The generated text is not accepted unconditionally. Reliability screening is performed with **Clinical-Longformer** as a feature extractor \(f_{RS}(\cdot)\). For original text \(T_i^{ori}\) and augmented text \(T_i^{aug}\), the paper computes
$$
s(T_i^{ori}, T_i^{aug})= \frac{f_{RS}(T_i^{ori})^\top f_{RS}(T_i^{aug})} {\|f_{RS}(T_i^{ori})\|\,\|f_{RS}(T_i^{aug})\|}
$$
and discards the generated text if \(s < \alpha\), retaining it if \(s \ge \alpha\) [2601.05648]. The threshold \(\alpha\) is mentioned but its numerical value is not reported.

The text side of SCxGEN-32M is therefore more elaborate than ontology labels alone. The supplementary prompt described in the paper requests a **single academic essay** of about **550–600 English words**, designed to incorporate cell type definition, tissue definition, disease definition, sex-specific context, developmental-stage context, canonical mechanisms and pathways, and interactions among these factors [2601.05648]. This suggests that SCxGEN-32M is intended to provide semantically dense supervision rather than short metadata strings.

## 3. RNA representation and model context in OKR-CELL

SCxGEN-32M is used to pretrain **OKR-CELL**, whose architecture contains three trainable components: a **cell encoder**, a **text encoder**, and a **cross-modal projector** [2601.05648]. The cell encoder is a **transformer-based scGPT-derived model** with **embedding dimension 128**, **6 transformer blocks**, **4 attention heads per block**, and **feedforward hidden dimension 128** [2601.05648]. The text encoder is **Clinical-Longformer**, based on **Longformer**, pretrained on about **2 million clinical notes** from **MIMIC-III**; the backbone supports **4096** tokens, while OKR-CELL truncates text input to **1024** tokens, producing **768-dimensional** text features [2601.05648]. The projector maps the **128-dimensional** cell embedding into the **768-dimensional** text feature space [2601.05648].

On the cell side, the paper represents the cell-gene matrix as
$$
\mathbf{X} \in \mathbb{R}^{N \times G}
$$
where \(N\) is the number of cells and \(G\) is the number of genes [2601.05648]. It states that **HVG selection** is performed, but also specifies in implementation details that the model feeds only genes with **non-zero expression levels**, following scGPT, with a **maximum input length of 1200**; if more than 1200 nonzero genes exist, it **randomly samples 1200 genes** at each iteration [2601.05648].

Gene identities are tokenized as
$$
t_g^{(i)} = [id(g_1^{(i)}), id(g_2^{(i)}), \ldots, id(g_M^{(i)})]
$$
with special tokens \(<cls>\) and \(<pad>\) [2601.05648]. Expression preprocessing applies **log1p**, performs HVG selection, bins nonzero values into \(O\) equal-interval bins, and keeps zeros as zero:
$$
x_{j}^{(i)} = \begin{cases} k, & \text{if } X_{i,j} > 0 \text{ and } X_{i,j} \in [o_k, o_{k+1}] \\ 0, & \text{if } X_{i,j} = 0 \end{cases}
$$
[2601.05648].

The input embedding is defined as
$$
h^{(i)} = emb_g(t_g^{(i)}) + emb_e(x^{(i)})
$$
and the transformer stack is written
$$
h_0^{(i)} = h^{(i)}, \qquad h_l^{(i)} = \mathrm{transformer\_block}(h_{l-1}^{(i)}), \quad l=1,\dots,n
$$
with the final cell embedding taken from the \(<cls>\) token [2601.05648].

The pretraining schedule is two-stage. In the first stage, the text encoder is frozen and optimization is applied to the **cell encoder** and **cell-to-text projector**; in the second stage, the text encoder is partially unfrozen, specifically only the **last self-attention layer** of Clinical-Longformer [2601.05648]. This makes SCxGEN-32M not only a corpus but also the substrate for a specific cell-language pretraining regime.

## 4. Objective functions and robust cross-modal alignment

The overall OKR-CELL training objective on SCxGEN-32M is
$$
\mathcal{L} = \mathcal{L}_{GEP} + \mathcal{L}_{GEPC} + \mathcal{L}_{CRA}
$$
[2601.05648]. The first two terms are intra-cell objectives and the third is the cross-modal robust alignment objective.

For **Gene Expression Prediction (GEP)**, masked genes are predicted by a 2-layer MLP:
$$
\tilde{x}^{(i)} = MLP\left(h_n^{(i)}\right)
$$
with loss
$$
\mathcal{L}_{GEP} = \frac{1}{|\mathcal{M}_{mask}|} \sum_{j \in \mathcal{M}_{mask}} ce\!\left(\tilde{x}_{j}^{(i)}, x_{j}^{(i)}\right)
$$
and the mask ratio randomly chosen from
$$
\{0.25,\ 0.50,\ 0.75\}
$$
[2601.05648].

For **Gene Expression Prediction for Cell Modelling (GEPC)**, the paper defines
$$
q_j = MLP\left(emb_g\left(t_g^{(i)}\right)\right)
$$
$$
\tilde{x}_j^{(i)} = q_j \cdot W \cdot h_c^{(i)}
$$
$$
\mathcal{L}_{GEPC} = \frac{1}{|\mathcal{M}_{mask}|} \sum_{j \in \mathcal{M}_{mask}} ce\!\left(\tilde{x}_j^{(i)}, x_j^{(i)}\right)
$$
where \(W \in \mathbb{R}^{d \times d}\) is a learnable bridge matrix [2601.05648].

The distinctive component is **Cross-modal Robust Alignment (CRA)**, which combines **PSW** (Progressive Sample Weighting), **CMMB** (Coupled Momentum-updated Memory Bank), and **CCA** (Cross-modal Complementary Alignment loss) [2601.05648]. The system maintains two memory banks, \(\mathbf{B}_c\) for cell features and \(\mathbf{B}_t\) for text features, as momentum-updated queues; the queue size is denoted by \(B\), but the numerical value is not reported [2601.05648].

Positive-pair reliability is estimated from both **symmetry** and **temporal stability**. The symmetry-based reliability is
$$
Rel^{sym}_{i,i} = \frac{1}{2}\left( \frac{\exp(S_{i,i}/\sigma)}{\sum_{k=1}^N \exp(S_{i,k}^{C \to T}/\sigma)} + \frac{\exp(S_{i,i}/\sigma)}{\sum_{k=1}^N \exp(S_{i,k}^{T \to C}/\sigma)} \right)
$$
[2601.05648]. Temporal stability uses a memory container \(\mathbf{Q} \in \mathbb{R}^{M \times P}\), with
$$
\mu_{i,i}(p) = \frac{1}{P-1} \sum_{p=1}^{P-1} Rel^{sym}_{i,i}(p)
$$
$$
q_{i,i}(p) = \sqrt{ \frac{ \sum_{p=1}^{P} \left(Rel^{sym}_{i,i}(p) - \mu_{i,i}(p)\right)^2 }{P} }
$$
and global statistics
$$
\mu_p = \frac{1}{M} \sum_{i=1}^M q_{i,i}(p), \qquad \sigma_p = \sqrt{ \frac{\sum_{i=1}^M \left(q_{i,i}(p)-\mu_p\right)^2}{M} }
$$
with noisy-positive threshold
$$
c_{np} = \mu_p + \alpha_{np}\sigma_p, \qquad \alpha_{np} = 3
$$
and stability term
$$
Rel^{stab}_{i,i} = \exp\left( -\frac{(q_{i,i}(p)-\mu_p)^2}{2\sigma_p^2} \right)
$$
[2601.05648].

The combined positive weight is
$$
W_{i,i}^{pos} = \begin{cases} Rel^{sym}_{i,i}, & \text{if } Rel^{sym}_{i,i} \le c_{np} \lor p \le 3 \\ Rel^{sym}_{i,i} \cdot Rel^{stab}_{i,i}, & \text{if } Rel^{sym}_{i,i} > c_{np} \end{cases}
$$
and the positive-pair loss is
$$
\mathcal{L}_{pos}(C_i, T_i) = - W^{pos}_{i,i} \left( \log \frac{\exp(S_{i,i}/\sigma)}{\sum_{k=1}^N \exp(S_{i,k}^{C\to T}/\sigma)} + \log \frac{\exp(S_{i,i}/\sigma)}{\sum_{k=1}^N \exp(S_{i,k}^{T\to C}/\sigma)} \right)
$$
[2601.05648].

The negative term is
$$
\mathcal{L}_{neg}(C_i, T_i) = \frac{\sum_{j \neq i}^{N+B} \tan(S^{C \to T}_{i,j})} {\left(\sum_{k=1}^{N+B} \tan(S^{C \to T}_{i,k})\right)^\omega} + \frac{\sum_{j \neq i}^{N+B} \tan(S^{T \to C}_{i,j})} {\left(\sum_{k=1}^{N+B} \tan(S^{T \to C}_{i,k})\right)^\omega}
$$
with \(\omega \in [0,1]\) [2601.05648]. The curriculum-based **PSW** schedule is written in the paper with \(\gamma_{\text{start}} = 2\), \(\gamma_{\text{end}} = 10\), and \(\alpha = 0.5\), and defines pair weights that emphasize easy negatives early and harder negatives later [2601.05648]. The final combined terms are
$$
\mathcal{L}_{CCA}(C_i, T_i) = \mathcal{L}_{pos}(C_i, T_i) + \lambda \cdot \mathcal{L}_{neg}(C_i, T_i)
$$
$$
\mathcal{L}_{CRA}(C_i, T_i) = \mathcal{L}_{pos}(C_i, T_i) + \lambda \cdot w^{PSW}(C_i, T_i)\cdot \mathcal{L}_{neg}(C_i, T_i)
$$
with
$$
\lambda = 5
$$
[2601.05648].

A plausible implication is that SCxGEN-32M is defined not only by scale, but by a training design that explicitly treats both **metadata-derived noise** and **LLM-generated text noise** as first-class optimization problems.

## 5. Empirical role in pretraining and downstream evaluation

After pretraining on SCxGEN-32M, OKR-CELL is reported to obtain cutting-edge results across **6 evaluation tasks**: **cell clustering**, **cell-type annotation**, **batch-effect correction**, **few-shot annotation**, **zero-shot cell-type annotation**, and **bidirectional cell-text retrieval** [2601.05648].

On **cell clustering**, the reported datasets are **Blood**, **Kidney**, **hPancreas**, and **hPBMC**, with metrics **ARI**, **AMI**, and **AvgBIO** [2601.05648]. The paper reports, for example, **Blood** results of **ARI 0.387** and **AMI 0.584** for OKR-CELL, compared with second-best **scGPT** at **ARI 0.175** and **AMI 0.374**; on **Kidney**, it reports **AMI 0.732** versus **0.371** for scGPT; on **hPancreas**, it reports **ARI 0.374** and **AMI 0.535** [2601.05648].

On **batch-effect correction**, the datasets are **hPancreas** and **hPBMC**, with metrics including **AvgBIO** and **AvgBatch** [2601.05648]. For **hPancreas**, the paper reports **AvgBIO = 0.708**, **scGPT = 0.687**, and **AvgBatch = 0.790**, and states that OKR-CELL performs best on nearly all detailed indicators except **PCR index** [2601.05648].

On **few-shot annotation**, the datasets are **Zheng68k** and **Great Apes**, using \(K \in \{1,5,9\}\) shots per class with a frozen backbone and linear head [2601.05648]. On **Zheng68k, 9-shot**, OKR-CELL reaches **accuracy 0.787**, compared with **0.554** for **scCLIP-GPT** and **0.190** for **LangCell**; on **Great Apes**, it reports **0.412** for OKR-CELL, **0.361** for LangCell, and **0.264** for scCLIP-GPT [2601.05648].

On **zero-shot cell-type annotation**, the datasets are **Eye**, **Prostate Gland**, and **Great Apes** [2601.05648]. For **Prostate Gland**, the paper reports **accuracy 0.511** for OKR-CELL and **0.299** for runner-up **scCLIP-GPT**, and describes this as a **+70.9%** relative advantage over LangCell on that dataset [2601.05648].

For **bidirectional cell-text retrieval**, the paper introduces **SCxGEN-CT5K**, a benchmark of **5K cell-text pairs** from **36 tissues**, all **unseen in training**, split into **5 folds**, evaluated by average **R@1, R@5, R@10** [2601.05648]. In **cell-to-text retrieval**, OKR-CELL reports **R@1 = 3.32**, **R@5 = 19.7**, and **R@10 = 42.9**; the paper states that this beats LangCell by **2.1%** at R@1 and surpasses the second-best **scCLIP-GPT** by **+11.9%** absolute at R@5 and **+16.0%** absolute at R@10 [2601.05648].

The paper also includes ablation evidence for the specific value of enriched text. On **hPBMC**, **original text + InfoNCE** gives **Accuracy 0.958** and **F1 0.934**, while **LLM-enriched text + InfoNCE** gives **Accuracy 0.964** and **F1 0.951**; on **spleen**, the corresponding numbers are **0.747 / 0.726** and **0.762 / 0.735** [2601.05648]. This indicates that the SCxGEN-32M text-construction strategy contributes beyond raw metadata alignment.

## 6. Robustness, implementation details, and caveats

A major theme of SCxGEN-32M and OKR-CELL is robustness to noise. The paper discusses noise from **sequencing variability**, **gene dropout**, **batch effects**, **incomplete or subjective descriptions**, **mismatched or partially matched cell-text pairs**, and **LLM hallucinations** in generated text [2601.05648].

In corrupted-test experiments with **gene dropout rates 10%–50%**, the paper reports that on **small intestine**, OKR-CELL accuracy drops from **91.9% to 87.2%**, whereas **scCLIP-GPT** drops from **89.1% to 84.1%** [2601.05648]. On **hPBMC**, with dropout increasing from **10% to 40%**, OKR-CELL remains **above 97%**, while scCLIP-GPT drops from **95.2% to 92.4%** [2601.05648]. Under noisy multimodal training with **30%** shuffling/perturbation of gene expression values and **30%** shuffling of cell-text relationships, the paper states that OKR-CELL remains robust and often superior; on **spleen**, **scCLIP-GPT** noisy F1 drops by **6%** from **73.3 to 68.9**, and on **colon**, **OKR-CELL(noisy)** exceeds **scCLIP-GPT(noisy)** by **15.7% F1** [2601.05648]. In zero-shot annotation on the **eye** dataset, **OKR-CELL(noisy)** reaches **F1 = 54.06%**, reported as **28.37%** higher than **scCLIP-GPT(noisy)** [2601.05648].

Training details are concrete in several respects. The optimizer is **Adam**, the learning rate is **1e-4**, the batch size is **14 per GPU**, training runs for **6 epochs**, and the paper states **weight decay: 0.9 applied after each epoch**, while also noting that this wording is ambiguous [2601.05648]. Training took about **3 days** on **8 nodes**, each with **4 NVIDIA A100 40GB GPUs**, for a total of **32 A100 40GB GPUs** [2601.05648]. For the 32M pretraining corpus, the paper reports **99.6% training**, **0.2% validation**, and **5000 samples** used to build **SCxGEN-CT5K**, while also noting that these quantities do not sum neatly and that the split description is somewhat inconsistent [2601.05648].

Several limitations are explicit or strongly implied. The paper does not report the **total parameter count** of OKR-CELL, the exact **projector architecture**, the full list of all **9 metadata fields**, the top-\(K\) retrieval value in RAG, acceptance rates after filtering, deduplication details, or release information for code, checkpoints, or the dataset [2601.05648]. It also notes future work toward **multi-omics**, **gene-level textual annotations**, and a **chat-based multimodal LLM**, which implies that the present system is primarily **scRNA-seq only**, uses **cell-level text**, and is **not yet a conversational model** [2601.05648].

## 7. Relation to other “32M” resources and naming ambiguities

SCxGEN-32M belongs to a broader class of resources whose names include “32M,” but the paper evidence distinguishes it clearly from several unrelated systems. In **“Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training”**, SCxGEN-32M is explicitly the **32 million cell-text pair** resource for OKR-CELL [2601.05648].

By contrast, **“Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation”** introduces **MRES-32M**, a vision-language grounding dataset with **over 32.2M high-quality masks and captions** on **1M images**, and explicitly notes that this is a different resource; it provides no evidence that **SCxGEN-32M** is an alias of **MRES-32M** [2312.08007]. Similarly, **“SemChunk-C: Semantic Segmentation for C Code”** defines a **SemChunk-C 32M** model as a **32M-parameter Ettin-based encoder-only** chunker for C-family source code, and the string **“SCxGEN-32M” does not appear anywhere in the paper** [2606.23697]. The retrieval paper **“Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report”** likewise describes **mxbai-edge-colbert-v0-32m**, a **32M-parameter** late-interaction retriever, not SCxGEN-32M [2510.14880].

This naming comparison matters because “32M” is overloaded across domains. In SCxGEN-32M, it denotes **dataset scale measured in paired cell-text examples**; in SemChunk-C 32M and mxbai-edge-colbert-v0-32m, it denotes **model parameter count**; and in MRES-32M, it denotes a **32M-scale multimodal annotation corpus** in vision-language segmentation [2601.05648; 2606.23697; 2510.14880; 2312.08007]. A plausible implication is that confusion arises from shared scale notation rather than shared methodology or lineage.

Source: https://www.emergentmind.com/topics/scxgen-32m