SCxGEN-32M: Single-Cell Language Pretraining

• SCxGEN-32M is a large-scale dataset comprising 32M cell-text pairs from scRNA-seq profiles and richly annotated CELLxGENE metadata.
• It utilizes a retrieval-augmented pipeline to generate enriched text descriptions that mitigate noise and improve semantic supervision.
• The dataset underpins OKR-CELL, a cross-modal foundation model achieving superior performance in clustering, annotation, batch correction, and cell-text retrieval.

Searching arXiv for SCxGEN-32M and related primary sources. arXiv search query: SCxGEN-32M OKR-CELL single-cell foundation model cell-language pretraining SCxGEN-32M is a large-scale single-cell–language pretraining resource introduced in connection with the OKR-CELL framework, where it denotes a dataset of 32 million cell-text pairs constructed from CELLxGENE single-cell data and text derived from metadata plus retrieval-augmented large-language-model enrichment (Wang et al., 9 Jan 2026). In this setting, “32M” refers to the number of paired examples rather than a model parameter count. SCxGEN-32M functions as the main pretraining corpus for OKR-CELL, an Open-world Language Knowledge-Aided Robust Single-Cell Foundation Model designed for cross-modal alignment between scRNA-seq profiles and biomedical text (Wang et al., 9 Jan 2026).

1. Identity, provenance, and scope

SCxGEN-32M is described as a corpus built from 32 million single-cell data from the CellxGene platform, including scRNA-seq matrices and corresponding metadata annotations from diverse human tissues and organs (Wang et al., 9 Jan 2026). The paper states that the dataset consists of 32 million cell-text pairs, and also states that each cell is associated with 9 key metadata items, with explicitly mentioned fields including cell type, tissue, disease, sex / gender, and developmental stage (Wang et al., 9 Jan 2026).

The resource is not itself the model. The model in the paper is OKR-CELL, while SCxGEN-32M is the pretraining dataset/resource on which OKR-CELL is trained (Wang et al., 9 Jan 2026). The paper frames this pairing as a response to two limitations of earlier cell-language pretraining systems: weak textual supervision based on sparse metadata and noise sensitivity in multimodal cell-text pairs (Wang et al., 9 Jan 2026).

A central distinguishing feature is that SCxGEN-32M is organized as a cross-modal resource. Each training example is a pair of the form

$$(\text{cell transcriptome}_i,\ \text{text description}_i)$$

for $i = 1,\dots,32{,}000{,}000$ (Wang et al., 9 Jan 2026). This makes the dataset suitable not only for cell-only representation learning, but also for multimodal objectives such as cell-text alignment and bidirectional retrieval.

2. Corpus construction and text generation pipeline

The construction pipeline begins with collection of cell profiles and metadata from CELLxGENE, followed by metadata standardization using ontology resources, generation of an original text description from metadata, optional enrichment through an LLM + RAG workflow, semantic reliability screening, and finally retention of the resulting cell-text pair (Wang et al., 9 Jan 2026).

The paper states that metadata terms such as disease, gender, tissue, and cell type are mapped to standardized textual descriptions by referencing the Cell Ontology database, and also mentions OBO Foundry as a standardization resource (Wang et al., 9 Jan 2026). The text side therefore begins as metadata-derived structured biomedical description rather than free-form annotation.

The enrichment stage uses a retrieval-augmented generation pipeline. The knowledge base is assembled from domain resources including PubMed; specifically, the paper says that it extracts only abstracts and then cleans and splits the corpus into 500-token chunks (Wang et al., 9 Jan 2026). These chunks are embedded with BioBERT and stored in a vector database for semantic retrieval (Wang et al., 9 Jan 2026). At generation time, structured metadata fields such as Cell Type, Tissue, Sex, and Development Stage are converted into a query, encoded by BioBERT, and used to retrieve top-$K$ snippets; the retrieved snippets and standardized metadata text are then passed to DeepSeek-V3 to generate an enriched description (Wang et al., 9 Jan 2026). The exact value of $K$ is not reported.

The generated text is not accepted unconditionally. Reliability screening is performed with Clinical-Longformer as a feature extractor $f_{RS}(\cdot)$. For original text $T_i^{ori}$ and augmented text $T_i^{aug}$, the paper computes

$$s(T_i^{ori}, T_i^{aug})= \frac{f_{RS}(T_i^{ori})^\top f_{RS}(T_i^{aug})} {\|f_{RS}(T_i^{ori})\|\,\|f_{RS}(T_i^{aug})\|}$$

and discards the generated text if $s < \alpha$, retaining it if $s \ge \alpha$ (Wang et al., 9 Jan 2026). The threshold $i = 1,\dots,32{,}000{,}000$0 is mentioned but its numerical value is not reported.

The text side of SCxGEN-32M is therefore more elaborate than ontology labels alone. The supplementary prompt described in the paper requests a single academic essay of about 550–600 English words, designed to incorporate cell type definition, tissue definition, disease definition, sex-specific context, developmental-stage context, canonical mechanisms and pathways, and interactions among these factors (Wang et al., 9 Jan 2026). This suggests that SCxGEN-32M is intended to provide semantically dense supervision rather than short metadata strings.

3. RNA representation and model context in OKR-CELL

SCxGEN-32M is used to pretrain OKR-CELL, whose architecture contains three trainable components: a cell encoder, a text encoder, and a cross-modal projector (Wang et al., 9 Jan 2026). The cell encoder is a transformer-based scGPT-derived model with embedding dimension 128, 6 transformer blocks, 4 attention heads per block, and feedforward hidden dimension 128 (Wang et al., 9 Jan 2026). The text encoder is Clinical-Longformer, based on Longformer, pretrained on about 2 million clinical notes from MIMIC-III; the backbone supports 4096 tokens, while OKR-CELL truncates text input to 1024 tokens, producing 768-dimensional text features (Wang et al., 9 Jan 2026). The projector maps the 128-dimensional cell embedding into the 768-dimensional text feature space (Wang et al., 9 Jan 2026).

On the cell side, the paper represents the cell-gene matrix as

$i = 1,\dots,32{,}000{,}000$1

where $i = 1,\dots,32{,}000{,}000$2 is the number of cells and $i = 1,\dots,32{,}000{,}000$3 is the number of genes (Wang et al., 9 Jan 2026). It states that HVG selection is performed, but also specifies in implementation details that the model feeds only genes with non-zero expression levels, following scGPT, with a maximum input length of 1200; if more than 1200 nonzero genes exist, it randomly samples 1200 genes at each iteration (Wang et al., 9 Jan 2026).

Gene identities are tokenized as

$i = 1,\dots,32{,}000{,}000$4

with special tokens $i = 1,\dots,32{,}000{,}000$5 and $i = 1,\dots,32{,}000{,}000$6 (Wang et al., 9 Jan 2026). Expression preprocessing applies log1p, performs HVG selection, bins nonzero values into $i = 1,\dots,32{,}000{,}000$7 equal-interval bins, and keeps zeros as zero:

$i = 1,\dots,32{,}000{,}000$8

(Wang et al., 9 Jan 2026).

The input embedding is defined as

$i = 1,\dots,32{,}000{,}000$9

and the transformer stack is written

$K$0

with the final cell embedding taken from the $K$1 token (Wang et al., 9 Jan 2026).

The pretraining schedule is two-stage. In the first stage, the text encoder is frozen and optimization is applied to the cell encoder and cell-to-text projector; in the second stage, the text encoder is partially unfrozen, specifically only the last self-attention layer of Clinical-Longformer (Wang et al., 9 Jan 2026). This makes SCxGEN-32M not only a corpus but also the substrate for a specific cell-language pretraining regime.

4. Objective functions and robust cross-modal alignment

The overall OKR-CELL training objective on SCxGEN-32M is

$K$2

(Wang et al., 9 Jan 2026). The first two terms are intra-cell objectives and the third is the cross-modal robust alignment objective.

For Gene Expression Prediction (GEP), masked genes are predicted by a 2-layer MLP:

$K$3

with loss

$K$4

and the mask ratio randomly chosen from

$K$5

(Wang et al., 9 Jan 2026).

For Gene Expression Prediction for Cell Modelling (GEPC), the paper defines

$K$6

$K$7

$K$8

where $K$9 is a learnable bridge matrix (Wang et al., 9 Jan 2026).

The distinctive component is Cross-modal Robust Alignment (CRA), which combines PSW (Progressive Sample Weighting), CMMB (Coupled Momentum-updated Memory Bank), and CCA (Cross-modal Complementary Alignment loss) (Wang et al., 9 Jan 2026). The system maintains two memory banks, $K$0 for cell features and $K$1 for text features, as momentum-updated queues; the queue size is denoted by $K$2, but the numerical value is not reported (Wang et al., 9 Jan 2026).

Positive-pair reliability is estimated from both symmetry and temporal stability. The symmetry-based reliability is

$K$3

(Wang et al., 9 Jan 2026). Temporal stability uses a memory container $K$4, with

$K$5

$K$6

and global statistics

$K$7

with noisy-positive threshold

$K$8

and stability term

$K$9

(Wang et al., 9 Jan 2026).

The combined positive weight is

$f_{RS}(\cdot)$0

and the positive-pair loss is

$f_{RS}(\cdot)$1

(Wang et al., 9 Jan 2026).

The negative term is

$f_{RS}(\cdot)$2

with $f_{RS}(\cdot)$3 (Wang et al., 9 Jan 2026). The curriculum-based PSW schedule is written in the paper with $f_{RS}(\cdot)$4, $f_{RS}(\cdot)$5, and $f_{RS}(\cdot)$6, and defines pair weights that emphasize easy negatives early and harder negatives later (Wang et al., 9 Jan 2026). The final combined terms are

$f_{RS}(\cdot)$7

$f_{RS}(\cdot)$8

with

$f_{RS}(\cdot)$9

(Wang et al., 9 Jan 2026).

A plausible implication is that SCxGEN-32M is defined not only by scale, but by a training design that explicitly treats both metadata-derived noise and LLM-generated text noise as first-class optimization problems.

5. Empirical role in pretraining and downstream evaluation

After pretraining on SCxGEN-32M, OKR-CELL is reported to obtain cutting-edge results across 6 evaluation tasks: cell clustering, cell-type annotation, batch-effect correction, few-shot annotation, zero-shot cell-type annotation, and bidirectional cell-text retrieval (Wang et al., 9 Jan 2026).

On cell clustering, the reported datasets are Blood, Kidney, hPancreas, and hPBMC, with metrics ARI, AMI, and AvgBIO (Wang et al., 9 Jan 2026). The paper reports, for example, Blood results of ARI 0.387 and AMI 0.584 for OKR-CELL, compared with second-best scGPT at ARI 0.175 and AMI 0.374; on Kidney, it reports AMI 0.732 versus 0.371 for scGPT; on hPancreas, it reports ARI 0.374 and AMI 0.535 (Wang et al., 9 Jan 2026).

On batch-effect correction, the datasets are hPancreas and hPBMC, with metrics including AvgBIO and AvgBatch (Wang et al., 9 Jan 2026). For hPancreas, the paper reports AvgBIO = 0.708, scGPT = 0.687, and AvgBatch = 0.790, and states that OKR-CELL performs best on nearly all detailed indicators except PCR index (Wang et al., 9 Jan 2026).

On few-shot annotation, the datasets are Zheng68k and Great Apes, using $T_i^{ori}$0 shots per class with a frozen backbone and linear head (Wang et al., 9 Jan 2026). On Zheng68k, 9-shot, OKR-CELL reaches accuracy 0.787, compared with 0.554 for scCLIP-GPT and 0.190 for LangCell; on Great Apes, it reports 0.412 for OKR-CELL, 0.361 for LangCell, and 0.264 for scCLIP-GPT (Wang et al., 9 Jan 2026).

On zero-shot cell-type annotation, the datasets are Eye, Prostate Gland, and Great Apes (Wang et al., 9 Jan 2026). For Prostate Gland, the paper reports accuracy 0.511 for OKR-CELL and 0.299 for runner-up scCLIP-GPT, and describes this as a +70.9% relative advantage over LangCell on that dataset (Wang et al., 9 Jan 2026).

For bidirectional cell-text retrieval, the paper introduces SCxGEN-CT5K, a benchmark of 5K cell-text pairs from 36 tissues, all unseen in training, split into 5 folds, evaluated by average R@1, R@5, R@10 (Wang et al., 9 Jan 2026). In cell-to-text retrieval, OKR-CELL reports R@1 = 3.32, R@5 = 19.7, and R@10 = 42.9; the paper states that this beats LangCell by 2.1% at R@1 and surpasses the second-best scCLIP-GPT by +11.9% absolute at R@5 and +16.0% absolute at R@10 (Wang et al., 9 Jan 2026).

The paper also includes ablation evidence for the specific value of enriched text. On hPBMC, original text + InfoNCE gives Accuracy 0.958 and F1 0.934, while LLM-enriched text + InfoNCE gives Accuracy 0.964 and F1 0.951; on spleen, the corresponding numbers are 0.747 / 0.726 and 0.762 / 0.735 (Wang et al., 9 Jan 2026). This indicates that the SCxGEN-32M text-construction strategy contributes beyond raw metadata alignment.

6. Robustness, implementation details, and caveats

A major theme of SCxGEN-32M and OKR-CELL is robustness to noise. The paper discusses noise from sequencing variability, gene dropout, batch effects, incomplete or subjective descriptions, mismatched or partially matched cell-text pairs, and LLM hallucinations in generated text (Wang et al., 9 Jan 2026).

In corrupted-test experiments with gene dropout rates 10%–50%, the paper reports that on small intestine, OKR-CELL accuracy drops from 91.9% to 87.2%, whereas scCLIP-GPT drops from 89.1% to 84.1% (Wang et al., 9 Jan 2026). On hPBMC, with dropout increasing from 10% to 40%, OKR-CELL remains above 97%, while scCLIP-GPT drops from 95.2% to 92.4% (Wang et al., 9 Jan 2026). Under noisy multimodal training with 30% shuffling/perturbation of gene expression values and 30% shuffling of cell-text relationships, the paper states that OKR-CELL remains robust and often superior; on spleen, scCLIP-GPT noisy F1 drops by 6% from 73.3 to 68.9, and on colon, OKR-CELL(noisy) exceeds scCLIP-GPT(noisy) by 15.7% F1 (Wang et al., 9 Jan 2026). In zero-shot annotation on the eye dataset, OKR-CELL(noisy) reaches F1 = 54.06%, reported as 28.37% higher than scCLIP-GPT(noisy) (Wang et al., 9 Jan 2026).

Training details are concrete in several respects. The optimizer is Adam, the learning rate is 1e-4, the batch size is 14 per GPU, training runs for 6 epochs, and the paper states weight decay: 0.9 applied after each epoch, while also noting that this wording is ambiguous (Wang et al., 9 Jan 2026). Training took about 3 days on 8 nodes, each with 4 NVIDIA A100 40GB GPUs, for a total of 32 A100 40GB GPUs (Wang et al., 9 Jan 2026). For the 32M pretraining corpus, the paper reports 99.6% training, 0.2% validation, and 5000 samples used to build SCxGEN-CT5K, while also noting that these quantities do not sum neatly and that the split description is somewhat inconsistent (Wang et al., 9 Jan 2026).

Several limitations are explicit or strongly implied. The paper does not report the total parameter count of OKR-CELL, the exact projector architecture, the full list of all 9 metadata fields, the top-$T_i^{ori}$1 retrieval value in RAG, acceptance rates after filtering, deduplication details, or release information for code, checkpoints, or the dataset (Wang et al., 9 Jan 2026). It also notes future work toward multi-omics, gene-level textual annotations, and a chat-based multimodal LLM, which implies that the present system is primarily scRNA-seq only, uses cell-level text, and is not yet a conversational model (Wang et al., 9 Jan 2026).

7. Relation to other “32M” resources and naming ambiguities

SCxGEN-32M belongs to a broader class of resources whose names include “32M,” but the paper evidence distinguishes it clearly from several unrelated systems. In “Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training”, SCxGEN-32M is explicitly the 32 million cell-text pair resource for OKR-CELL (Wang et al., 9 Jan 2026).

By contrast, “Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation” introduces MRES-32M, a vision-language grounding dataset with over 32.2M high-quality masks and captions on 1M images, and explicitly notes that this is a different resource; it provides no evidence that SCxGEN-32M is an alias of MRES-32M (Wang et al., 2023). Similarly, “SemChunk-C: Semantic Segmentation for C Code” defines a SemChunk-C 32M model as a 32M-parameter Ettin-based encoder-only chunker for C-family source code, and the string “SCxGEN-32M” does not appear anywhere in the paper (Nazarov et al., 12 May 2026). The retrieval paper “Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report” likewise describes mxbai-edge-colbert-v0-32m, a 32M-parameter late-interaction retriever, not SCxGEN-32M (Takehi et al., 16 Oct 2025).

This naming comparison matters because “32M” is overloaded across domains. In SCxGEN-32M, it denotes dataset scale measured in paired cell-text examples; in SemChunk-C 32M and mxbai-edge-colbert-v0-32m, it denotes model parameter count; and in MRES-32M, it denotes a 32M-scale multimodal annotation corpus in vision-language segmentation (Wang et al., 9 Jan 2026, Nazarov et al., 12 May 2026, Takehi et al., 16 Oct 2025, Wang et al., 2023). A plausible implication is that confusion arises from shared scale notation rather than shared methodology or lineage.