OKR-CELL: Open-world Cell-Language Model

• The paper introduces OKR-CELL, a cross-modal cell-language foundation model that enriches sparse metadata with open-world biomedical knowledge.
• It combines a scGPT-based cell encoder and a Clinical-Longformer text encoder with a cross-modal projector using retrieval-augmented generation and robust contrastive alignment.
• Empirical evaluations show superior performance in cell clustering, batch-effect correction, and cell-type annotation under noisy multimodal conditions.

Searching arXiv for OKR-CELL and closely related cell-language foundation models. OKR-CELL, short for Open-world Language Knowledge-Aided Robust Single-Cell Foundation Model, is a single-cell foundation model for cross-modal cell-language pre-training that is designed to improve representation learning for single-cell RNA-seq and, more broadly, future single-cell multi-omics settings by leveraging natural language as a complementary knowledge source (Wang et al., 9 Jan 2026). The model addresses two limitations attributed to earlier cell-language pre-training approaches: textual supervision that is restricted to sparse metadata and cross-modal contrastive learning that is fragile under multimodal noise. Its framework combines open-world knowledge enrichment of cell-level text through an LLM-based workflow with retrieval-augmented generation (RAG) and a Cross-modal Robust Alignment (CRA) objective that incorporates sample reliability assessment, curriculum learning, and coupled momentum contrastive learning (Wang et al., 9 Jan 2026).

1. Conceptual scope and problem setting

OKR-CELL is positioned within the broader development of transformer-style single-cell foundation models such as scBERT, Geneformer, scFoundation, and scGPT, which treat genes as tokens and expression patterns as a biological language (Wang et al., 9 Jan 2026). In that setting, the central motivation is that transcriptomic models can learn strong embeddings from gene expression matrices but do not necessarily ground cellular states in rich biological semantics expressed in human language.

The model specifically targets two shortcomings. First, prior cell-language methods such as LangCell and scMULAN are described as relying on cell-level text that is often limited to sparse metadata, including fields such as cell type, tissue, sex, disease, and developmental stage, rather than richer contextual descriptions (Wang et al., 9 Jan 2026). Second, standard CLIP/InfoNCE-style objectives are treated as insufficiently robust for biological multimodal supervision because cell-text pairs may be incomplete, weakly matched, partially incorrect, or affected by technical noise and batch effects (Wang et al., 9 Jan 2026). In this formulation, OKR-CELL aims simultaneously to improve the semantic depth of the text modality and to improve robustness of cell-text alignment.

A plausible implication is that the model treats language not merely as an annotation channel but as a structured biological prior. The paper’s framing suggests that the language side is intended to encode contextual mechanisms, tissue specialization, disease context, and broader open-world knowledge that are not captured by metadata concatenation alone (Wang et al., 9 Jan 2026).

2. Architecture and pre-training pipeline

OKR-CELL is a cross-modal Cell-Language pre-training framework with three trainable components: a cell encoder, a text encoder, and a cross-modal projector (Wang et al., 9 Jan 2026). The cell encoder is based on scGPT, while the text branch uses Clinical-Longformer, a Longformer-based model pretrained on about 2 million clinical notes from MIMIC-III (Wang et al., 9 Jan 2026).

For the cell branch, the input cell-gene matrix is written as

$\mathbf{X} \in \mathbb{R}^{N \times G},$

where $N$ is the number of cells and $G$ the number of genes (Wang et al., 9 Jan 2026). Genes are tokenized as

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$

and expression values are binned as

$$x_{j}^{(i)} = \begin{cases} k, & \text{if } X_{i,j} > 0 \text{ and } X_{i,j} \in [o_k, o_{k+1}] \ 0, & \text{if } X_{i,j} = 0. \end{cases}$$

Gene identity embeddings and expression-bin embeddings are fused by

$h^{(i)} = emb_g(t_{g}^{(i)}) + emb_e(x^{(i)}),$

then passed through transformer blocks

$$h_0^{(i)} = h^{(i)}, \quad h_l^{(i)} = \text{transformer\_block}(h_{l-1}^{(i)}) \quad \forall l \in [1, n],$$

with a special $<cls>$ token whose final hidden state $h_{cls}^{(i)} \in \mathbb{R}^{D}$ serves as the cell embedding (Wang et al., 9 Jan 2026).

The implementation uses a lightweight scGPT-style configuration with embedding dimension $d=128$, 6 transformer blocks, 4 attention heads, and hidden dimension 128 (Wang et al., 9 Jan 2026). The text encoder produces 768-dimensional embeddings, has a maximum input sequence length of 4096, and uses an actual truncation length of 1024 during training (Wang et al., 9 Jan 2026). Because the cell and text encoders operate in different dimensions, a cell-to-text projector maps cell representations into the text embedding space for alignment (Wang et al., 9 Jan 2026).

Training is organized in two stages. In Stage 1, the text encoder is frozen while the cell encoder and projector are trained to establish basic cross-modal alignment. In Stage 2, only the last self-attention layer of Clinical-Longformer is unfrozen for continued joint training (Wang et al., 9 Jan 2026). Both stages optimize intra-modal cellular objectives together with the cross-modal alignment objective.

3. Open-world knowledge aided text enrichment

One of the two central innovations of OKR-CELL is its enrichment of cell-level text using open-world biomedical knowledge (Wang et al., 9 Jan 2026). The goal is to replace sparse metadata strings with richer descriptions that encode canonical cell-type biology, tissue context, disease context, sex effects, developmental or aging stage, and broader biological priors (Wang et al., 9 Jan 2026).

The workflow has four stages. Step 1 standardizes metadata fields such as disease, gender, tissue, and cell type using the Cell Ontology / OBO Foundry (Wang et al., 9 Jan 2026). Step 2 constructs an external biomedical knowledge base from PubMed abstracts, which are cleaned, chunked into semantically coherent snippets such as 500-token chunks, encoded using BioBERT, and stored in a vector database for semantic retrieval (Wang et al., 9 Jan 2026). Step 3 performs RAG-enhanced text generation: metadata components such as Cell Type, Tissue, Sex, and Development Stage are combined into a templated query, the query is encoded by BioBERT, and the top-$N$0 most semantically similar snippets are retrieved; the generation model is DeepSeek-V3 (Wang et al., 9 Jan 2026). Step 4 applies Reliability Screening (RS) to suppress hallucinations (Wang et al., 9 Jan 2026).

The semantic screening stage compares the original text $N$1 and the augmented text $N$2 through cosine similarity:

$N$3

If the similarity is below a threshold $N$4, the augmented text is discarded; otherwise it is retained (Wang et al., 9 Jan 2026).

The prompt design in the supplementary material asks the LLM to generate a single continuous academic essay of 550 to 600 words, with high-density biomedical terminology, systematic interaction analysis, physiological contextualization, and no fabricated specific quantitative data or unique experimental claims (Wang et al., 9 Jan 2026). An example given for a Pvalb+ GABAergic interneuron in Dorsolateral Prefrontal Cortex (DLPFC) with Alzheimer’s Disease, Female, and Late-onset (80+ years) illustrates the intended shift from metadata concatenation to context-rich mechanistic description (Wang et al., 9 Jan 2026).

This suggests that OKR-CELL treats text augmentation as a knowledge distillation layer between biomedical literature and cell-level supervision. The model is therefore not limited to ontology labels; it is trained on language that embeds mechanistic and contextual priors.

4. Training objectives and the CRA formulation

The second central innovation is the Cross-modal Robust Alignment (CRA) objective (Wang et al., 9 Jan 2026). OKR-CELL is trained with both intra-modal generative pre-training on cells and cross-modal robust alignment between cells and text (Wang et al., 9 Jan 2026).

The intra-modal part follows scGPT-style objectives. For Gene Expression Prediction (GEP),

$N$5

and

$N$6

For Gene Expression Prediction for Cell Modelling (GEPC),

$N$7

$N$8

and

$N$9

These are described as capturing local gene-gene dependencies and global cell-level consistency, respectively (Wang et al., 9 Jan 2026).

CRA contains three components: Progressive Sample Weighting (PSW), Coupled Momentum-updated Memory Bank (CMMB), and Cross-modal Complementary Alignment (CCA) loss (Wang et al., 9 Jan 2026). CMMB maintains two dynamic memory banks, $G$0 for cell embeddings and $G$1 for text embeddings, using momentum encoders to enlarge the negative set beyond the current batch (Wang et al., 9 Jan 2026).

PSW introduces a curriculum schedule through

$G$2

with $G$3, $G$4, and $G$5 (Wang et al., 9 Jan 2026). The sample-pair weight is defined piecewise so that easy negatives are emphasized early and hard negatives later (Wang et al., 9 Jan 2026). The motivation is that aggressive hard-negative emphasis in early training can destabilize learning under noisy supervision.

Positive-pair reliability is estimated using symmetry and stability. The symmetry-based score is

$G$6

and stability is computed from a historical lookup table

$G$7

using epochwise fluctuations in $G$8 (Wang et al., 9 Jan 2026). A noisy-pair threshold is then defined as

$G$9

with $$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$0, and temporal stability reliability is

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$1

The combined positive weight becomes

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$2

This means early epochs use symmetry only, while later epochs additionally penalize unstable positive pairs (Wang et al., 9 Jan 2026).

The positive loss is

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$3

while the negative term is

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$4

The CCA loss is

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$5

and the final CRA loss is

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$6

with

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$7

The overall training objective is

$$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$8

The paper identifies CRA as the principal mechanism by which the model becomes resistant to noisy multimodal supervision (Wang et al., 9 Jan 2026).

5. Data scale, implementation, and evaluation tasks

OKR-CELL is pretrained on SCxGEN-32M, a corpus of 32 million cell-text pairs collected from CELLxGENE (Wang et al., 9 Jan 2026). Each pair contains scRNA-seq information together with metadata-derived text, and the corpus is described as a whole-human pretraining set spanning diverse tissues, organs, developmental stages, and cell type categories (Wang et al., 9 Jan 2026). The whole-human pretraining split is reported as 99.6% training, 0.2% validation, and 5000 samples reserved for the retrieval benchmark SCxGEN-CT5K; the paper notes these printed percentages do not sum to 100%, which suggests a likely typographical inconsistency in the source description (Wang et al., 9 Jan 2026).

The scRNA-seq preprocessing follows scGPT conventions: only genes with non-zero expression are fed into the model; the maximum input length is 1200; if a cell contains more than 1200 non-zero genes, 1200 genes are randomly sampled per iteration; the gene generation ratio is selected uniformly from $$t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],$$9; preprocessing includes log1p transformation and HVG selection; and expression values are binned to reduce scale inconsistency across batches (Wang et al., 9 Jan 2026).

Training uses Adam, batch size 14 per GPU, initial learning rate $$x_{j}^{(i)} = \begin{cases} k, & \text{if } X_{i,j} > 0 \text{ and } X_{i,j} \in [o_k, o_{k+1}] \ 0, & \text{if } X_{i,j} = 0. \end{cases}$$0, weight decay 0.9 applied after each epoch, 6 training epochs, and hardware comprising 8 nodes × 4 NVIDIA A100 40GB GPUs per node, for a total of 32 A100 GPUs; reported training time is approximately 3 days (Wang et al., 9 Jan 2026). The temperature $$x_{j}^{(i)} = \begin{cases} k, & \text{if } X_{i,j} > 0 \text{ and } X_{i,j} \in [o_k, o_{k+1}] \ 0, & \text{if } X_{i,j} = 0. \end{cases}$$1, robustness parameter $$x_{j}^{(i)} = \begin{cases} k, & \text{if } X_{i,j} > 0 \text{ and } X_{i,j} \in [o_k, o_{k+1}] \ 0, & \text{if } X_{i,j} = 0. \end{cases}$$2, and memory-bank size $$x_{j}^{(i)} = \begin{cases} k, & \text{if } X_{i,j} > 0 \text{ and } X_{i,j} \in [o_k, o_{k+1}] \ 0, & \text{if } X_{i,j} = 0. \end{cases}$$3 appear in the formulations but are not numerically specified in the visible text (Wang et al., 9 Jan 2026).

The model is evaluated on 6 tasks: cell clustering, batch-effect correction, traditional cell-type annotation, few-shot cell-type annotation, zero-shot cell-type annotation, and bidirectional cell-text retrieval (Wang et al., 9 Jan 2026). Metrics include ARI, AMI/NMI, and AvgBIO for clustering; AvgBIO, AvgBatch, and eight detailed indicators for batch correction; Accuracy and Macro-F1 for annotation; and R@1, R@5, R@10 for retrieval (Wang et al., 9 Jan 2026). Baselines include Geneformer, scBERT, scFoundation, scGPT, LangCell, and scCLIP-GPT, where the latter uses the same cell and text encoders as OKR-CELL but replaces CRA with standard CLIP InfoNCE and uses only original text (Wang et al., 9 Jan 2026).

6. Empirical performance, ablations, and interpretive issues

The paper reports that OKR-CELL obtains cutting-edge results across all six evaluation tasks (Wang et al., 9 Jan 2026). In cell clustering, it achieves the best performance across the main datasets, including Blood, Kidney, hPancreas, and hPBMC (Wang et al., 9 Jan 2026). Reported values include Blood ARI 0.387, AMI 0.584, compared with second-best scGPT at ARI 0.175, AMI 0.374, and Kidney AMI 0.732 versus 0.371 for scGPT (Wang et al., 9 Jan 2026). For hPancreas, the paper reports ARI 0.374 and AMI 0.535 (Wang et al., 9 Jan 2026).

In batch-effect correction, OKR-CELL is reported as best on both biological signal retention and batch mixing for hPancreas and hPBMC (Wang et al., 9 Jan 2026). On hPancreas, it reaches AvgBIO = 0.708 and AvgBatch = 0.790, compared with scGPT at AvgBIO 0.687 (Wang et al., 9 Jan 2026). Across eight specific indicators, it is best on all except PCR index (Wang et al., 9 Jan 2026).

For traditional cell-type annotation, the paper states that OKR-CELL consistently outperforms all baselines, including scCLIP-GPT, in both accuracy and F1-score across datasets including small-intestine, spleen, hPBMC, and additional analyses on eye and kidney (Wang et al., 9 Jan 2026). The full per-dataset table is not reproduced in the supplied material, but the text reports strong linear separability and biologically meaningful rather than random confusions (Wang et al., 9 Jan 2026).

For few-shot annotation, performance is highlighted on Zheng68k and Great Apes with $$x_{j}^{(i)} = \begin{cases} k, & \text{if } X_{i,j} > 0 \text{ and } X_{i,j} \in [o_k, o_{k+1}] \ 0, & \text{if } X_{i,j} = 0. \end{cases}$$4 shots per class and a frozen backbone (Wang et al., 9 Jan 2026). Reported values include Zheng68k 9-shot Accuracy 0.787, compared with 0.554 for scCLIP-GPT and 0.190 for LangCell, and Great Apes Accuracy 0.412, compared with 0.361 for LangCell and 0.264 for scCLIP-GPT (Wang et al., 9 Jan 2026).

For zero-shot annotation, the paper emphasizes results on Eye, Prostate Gland, and Great Apes (Wang et al., 9 Jan 2026). On Prostate Gland, OKR-CELL achieves Accuracy 0.511 versus 0.299 for scCLIP-GPT (Wang et al., 9 Jan 2026). The paper interprets these results as particularly important because zero-shot annotation depends directly on semantic cell-text alignment.

For bidirectional cell-text retrieval, the paper introduces SCxGEN-CT5K, consisting of 5000 cell-text pairs from 36 tissues that are all unseen during training (Wang et al., 9 Jan 2026). In cell-to-text retrieval, OKR-CELL reaches R@1 = 3.32, R@5 = 19.7, and R@10 = 42.9 (Wang et al., 9 Jan 2026). The evaluation is strict: only strictly one-to-one annotated pairs count as positives, while even same-cell-type or same-parent-type pairs are treated as negatives (Wang et al., 9 Jan 2026).

The robustness experiments are central to the paper’s claims. Under simulated gene dropout from 10% to 50%, OKR-CELL shows smaller degradation than scCLIP-GPT (Wang et al., 9 Jan 2026). In small intestine, OKR-CELL accuracy drops from 91.9% to 87.2%, a 5.1% decrease, while scCLIP-GPT drops from 89.1% to 84.1%, a 5.6% decrease (Wang et al., 9 Jan 2026). In hPBMC, as dropout rises from 10% to 40%, OKR-CELL remains above 97%, while scCLIP-GPT drops from 95.2% to 92.4% (Wang et al., 9 Jan 2026). Under noisy multimodal pretraining with 30% gene expression perturbation and 30% shuffled cell-text mismatches, OKR-CELL degrades little and in some cases improves, whereas scCLIP-GPT(noisy) degrades more substantially (Wang et al., 9 Jan 2026).

The ablation study isolates the contributions of open-world text enrichment and CRA (Wang et al., 9 Jan 2026). On hPBMC, Original text + InfoNCE gives Accuracy 0.958, F1 0.934, while LLM-enriched text + InfoNCE improves to Accuracy 0.964, F1 0.951 (Wang et al., 9 Jan 2026). Further replacing InfoNCE with CCA and then adding PSW, CMMB, and full CRA yields the best results, culminating in full CRA: 0.976 / 0.959 on hPBMC (Wang et al., 9 Jan 2026). On spleen, the sequence 0.762 / 0.735 with enriched text + InfoNCE, 0.770 / 0.751 with CCA, 0.773 / 0.750 with +PSW, 0.780 / 0.763 with +CMMB, and 0.776 / 0.765 with full CRA indicates that the robust objective is beneficial even though one intermediate configuration yields the highest reported accuracy (Wang et al., 9 Jan 2026).

A plausible interpretation is that the model’s reported gains arise from two separable sources: richer language supervision and more robust alignment under noisy cell-text correspondence. That interpretation is directly supported by the ablation structure, which first changes the text source and then changes the contrastive formulation (Wang et al., 9 Jan 2026).

7. Relation to prior work, scope, and limitations

OKR-CELL differs from earlier single-cell foundation models in three ways explicitly emphasized in the source material: it treats language as cell-level biological supervision rather than merely labels; it incorporates open-world biological knowledge through ontology normalization, PubMed retrieval, BioBERT semantic search, DeepSeek-V3 generation, and semantic filtering; and it explicitly models noisy cross-modal supervision through CRA instead of using plain CLIP-style alignment (Wang et al., 9 Jan 2026).

The model’s strongest reported use cases are cell type annotation when labels are scarce, few-shot adaptation, zero-shot annotation, batch-effect resistant representation learning, cell-text retrieval, and future multi-omics integration in which text can serve as a common interface (Wang et al., 9 Jan 2026). The paper also proposes future directions including gene-level textual annotations, extension to multi-omics integration, and chat-based multimodal LLM systems for single-cell data interrogation (Wang et al., 9 Jan 2026).

Several limitations are either stated directly or clearly indicated in the provided material. Training appears focused primarily on scRNA-seq, despite a broader multi-omics motivation (Wang et al., 9 Jan 2026). The text-enrichment pipeline depends on both LLM generation quality and retrieval quality (Wang et al., 9 Jan 2026). Several implementation specifics are underreported, including the exact momentum coefficient, memory bank size, and temperature values (Wang et al., 9 Jan 2026). The retrieval benchmark is relatively small compared with the pretraining scale, and enriched texts may still inherit literature bias or ontology bias even after reliability screening (Wang et al., 9 Jan 2026).

A plausible source of confusion is the similarity between the acronym “OKR” in OKR-CELL and its use in agent-systems literature. A separate work, “Agents meet OKR: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation” (Zheng et al., 2023), uses “OKR” in the organizational sense of Objects and Key Results for hierarchical agent orchestration. By contrast, OKR-CELL uses “OKR” to denote Open-world Language Knowledge-Aided Robust in a biological foundation model (Wang et al., 9 Jan 2026). The two works are therefore associated by acronym rather than by domain or method.

Within the single-cell multimodal literature, OKR-CELL is best understood as a cross-modal foundation model that couples literature-grounded text augmentation with a reliability-aware contrastive objective. Its technical distinctiveness lies in the CRA formulation and its biological significance lies in showing that cell-language alignment becomes more effective when the text side is enriched from open-world biomedical knowledge rather than constrained to sparse metadata (Wang et al., 9 Jan 2026).