---
title: 'OKR-CELL: Open-world Cell-Language Model'
url: https://www.emergentmind.com/topics/okr-cell
type: topic
---

# OKR-CELL: Open-world Cell-Language Model

Searching arXiv for OKR-CELL and closely related cell-language foundation models.
OKR-CELL, short for **Open-world Language Knowledge-Aided Robust Single-Cell Foundation Model**, is a single-cell foundation model for **cross-modal cell-language pre-training** that is designed to improve representation learning for **single-cell RNA-seq** and, more broadly, future **single-cell multi-omics** settings by leveraging natural language as a complementary knowledge source [2601.05648]. The model addresses two limitations attributed to earlier cell-language pre-training approaches: textual supervision that is restricted to sparse metadata and cross-modal contrastive learning that is fragile under multimodal noise. Its framework combines open-world knowledge enrichment of cell-level text through an **LLM-based workflow with retrieval-augmented generation (RAG)** and a **Cross-modal Robust Alignment (CRA)** objective that incorporates sample reliability assessment, curriculum learning, and coupled momentum contrastive learning [2601.05648].

## 1. Conceptual scope and problem setting

OKR-CELL is positioned within the broader development of transformer-style single-cell foundation models such as **scBERT**, **Geneformer**, **scFoundation**, and **scGPT**, which treat genes as tokens and expression patterns as a biological language [2601.05648]. In that setting, the central motivation is that transcriptomic models can learn strong embeddings from gene expression matrices but do not necessarily ground cellular states in rich biological semantics expressed in human language.

The model specifically targets two shortcomings. First, prior cell-language methods such as **LangCell** and **scMULAN** are described as relying on cell-level text that is often limited to sparse metadata, including fields such as cell type, tissue, sex, disease, and developmental stage, rather than richer contextual descriptions [2601.05648]. Second, standard **CLIP/InfoNCE-style** objectives are treated as insufficiently robust for biological multimodal supervision because cell-text pairs may be incomplete, weakly matched, partially incorrect, or affected by technical noise and batch effects [2601.05648]. In this formulation, OKR-CELL aims simultaneously to improve the semantic depth of the text modality and to improve robustness of cell-text alignment.

A plausible implication is that the model treats language not merely as an annotation channel but as a structured biological prior. The paper’s framing suggests that the language side is intended to encode contextual mechanisms, tissue specialization, disease context, and broader open-world knowledge that are not captured by metadata concatenation alone [2601.05648].

## 2. Architecture and pre-training pipeline

OKR-CELL is a **cross-modal Cell-Language pre-training framework** with three trainable components: a **cell encoder**, a **text encoder**, and a **cross-modal projector** [2601.05648]. The cell encoder is based on **scGPT**, while the text branch uses **Clinical-Longformer**, a Longformer-based model pretrained on about **2 million clinical notes** from **MIMIC-III** [2601.05648].

For the cell branch, the input cell-gene matrix is written as
$$
\mathbf{X} \in \mathbb{R}^{N \times G},
$$
where $N$ is the number of cells and $G$ the number of genes [2601.05648]. Genes are tokenized as
$$
t_{g}^{(i)} = \left[id(g_{1}^{(i)}), id(g_{2}^{(i)}), \ldots, id(g_{M}^{(i)})\right],
$$
and expression values are binned as
$$
x_{j}^{(i)} = \begin{cases} k, & \text{if } X_{i,j} > 0 \text{ and } X_{i,j} \in [o_k, o_{k+1}] \\ 0, & \text{if } X_{i,j} = 0. \end{cases}
$$
Gene identity embeddings and expression-bin embeddings are fused by
$$
h^{(i)} = emb_g(t_{g}^{(i)}) + emb_e(x^{(i)}),
$$
then passed through transformer blocks
$$
h_0^{(i)} = h^{(i)}, \quad h_l^{(i)} = \text{transformer\_block}(h_{l-1}^{(i)}) \quad \forall l \in [1, n],
$$
with a special $<cls>$ token whose final hidden state $h_{cls}^{(i)} \in \mathbb{R}^{D}$ serves as the cell embedding [2601.05648].

The implementation uses a lightweight scGPT-style configuration with **embedding dimension $d=128$**, **6 transformer blocks**, **4 attention heads**, and **hidden dimension 128** [2601.05648]. The text encoder produces **768-dimensional** embeddings, has a maximum input sequence length of **4096**, and uses an actual truncation length of **1024** during training [2601.05648]. Because the cell and text encoders operate in different dimensions, a **cell-to-text projector** maps cell representations into the text embedding space for alignment [2601.05648].

Training is organized in **two stages**. In **Stage 1**, the text encoder is frozen while the cell encoder and projector are trained to establish basic cross-modal alignment. In **Stage 2**, only the **last self-attention layer** of Clinical-Longformer is unfrozen for continued joint training [2601.05648]. Both stages optimize intra-modal cellular objectives together with the cross-modal alignment objective.

## 3. Open-world knowledge aided text enrichment

One of the two central innovations of OKR-CELL is its enrichment of cell-level text using open-world biomedical knowledge [2601.05648]. The goal is to replace sparse metadata strings with richer descriptions that encode canonical cell-type biology, tissue context, disease context, sex effects, developmental or aging stage, and broader biological priors [2601.05648].

The workflow has four stages. **Step 1** standardizes metadata fields such as disease, gender, tissue, and cell type using the **Cell Ontology / OBO Foundry** [2601.05648]. **Step 2** constructs an external biomedical knowledge base from **PubMed** abstracts, which are cleaned, chunked into semantically coherent snippets such as **500-token chunks**, encoded using **BioBERT**, and stored in a vector database for semantic retrieval [2601.05648]. **Step 3** performs **RAG-enhanced text generation**: metadata components such as Cell Type, Tissue, Sex, and Development Stage are combined into a templated query, the query is encoded by BioBERT, and the **top-$K$** most semantically similar snippets are retrieved; the generation model is **DeepSeek-V3** [2601.05648]. **Step 4** applies **Reliability Screening (RS)** to suppress hallucinations [2601.05648].

The semantic screening stage compares the original text $T_i^{ori}$ and the augmented text $T_i^{aug}$ through cosine similarity:
$$
s(T_{i}^{ori},T_{i}^{aug})= \frac {f_{RS}(T_{i}^{ori})^{\top }\cdot f_{st}(T_{i}^{aug})} {\left\| f_{st}(T_{i}^{ori})\right\| \left\| f_{st}(T_{i}^{aug}\right) \| }.
$$
If the similarity is below a threshold $\alpha$, the augmented text is discarded; otherwise it is retained [2601.05648].

The prompt design in the supplementary material asks the LLM to generate a **single continuous academic essay** of **550 to 600 words**, with high-density biomedical terminology, systematic interaction analysis, physiological contextualization, and no fabricated specific quantitative data or unique experimental claims [2601.05648]. An example given for a **Pvalb+ GABAergic interneuron** in **Dorsolateral Prefrontal Cortex (DLPFC)** with **Alzheimer’s Disease**, **Female**, and **Late-onset (80+ years)** illustrates the intended shift from metadata concatenation to context-rich mechanistic description [2601.05648].

This suggests that OKR-CELL treats text augmentation as a knowledge distillation layer between biomedical literature and cell-level supervision. The model is therefore not limited to ontology labels; it is trained on language that embeds mechanistic and contextual priors.

## 4. Training objectives and the CRA formulation

The second central innovation is the **Cross-modal Robust Alignment (CRA)** objective [2601.05648]. OKR-CELL is trained with both **intra-modal generative pre-training** on cells and **cross-modal robust alignment** between cells and text [2601.05648].

The intra-modal part follows scGPT-style objectives. For **Gene Expression Prediction (GEP)**,
$$
\tilde{x}^{(i)} = MLP\left(h_{n}^{(i)}\right),
$$
and
$$
\mathcal{L}_{GEP} = \frac{1}{|\mathcal{M}_{mask}|} \sum_{j \in \mathcal{M}_{mask}} ce\left(\tilde{x}_{j}^{(i)}, x_{j}^{(i)}\right).
$$
For **Gene Expression Prediction for Cell Modelling (GEPC)**,
$$
q_j = MLP\left(emb_g\left(t_g^{(i)}\right)\right),
$$
$$
\tilde{x}_j^{(i)} = q_j \cdot W \cdot h_c^{(i)},
$$
and
$$
\mathcal{L}_{GEPC} = \frac{1}{|\mathcal{M}_{mask}|} \sum_{j \in \mathcal{M}_{mask}} ce\left(\tilde{x}_j^{(i)}, x_j^{(i)}\right).
$$
These are described as capturing local gene-gene dependencies and global cell-level consistency, respectively [2601.05648].

CRA contains three components: **Progressive Sample Weighting (PSW)**, **Coupled Momentum-updated Memory Bank (CMMB)**, and **Cross-modal Complementary Alignment (CCA) loss** [2601.05648]. CMMB maintains two dynamic memory banks, $\mathbf{B}_c$ for cell embeddings and $\mathbf{B}_t$ for text embeddings, using momentum encoders to enlarge the negative set beyond the current batch [2601.05648].

PSW introduces a curriculum schedule through
$$
\gamma(e) = \gamma_{\text{start}} + (\gamma_{\text{end}} - \gamma_{\text{start}})\cdot e^{\alpha}, \quad e = \frac{t}{T} \in [0,1],
$$
with $\gamma_{\text{start}}=2$, $\gamma_{\text{end}}=10$, and $\alpha=0.5$ [2601.05648]. The sample-pair weight is defined piecewise so that easy negatives are emphasized early and hard negatives later [2601.05648]. The motivation is that aggressive hard-negative emphasis in early training can destabilize learning under noisy supervision.

Positive-pair reliability is estimated using **symmetry** and **stability**. The symmetry-based score is
$$
Rel^{sym}_{i,i} = \frac{1}{2}\left( \frac{\exp(S_{i,i}/\sigma)}{ \sum_{k=1}^N \exp(S_{i,k}^{C \to T}/\sigma)} + \frac{\exp(S_{i,i}/\sigma)}{ \sum_{k=1}^N \exp(S_{i,k}^{T \to C}/\sigma)} \right),
$$
and stability is computed from a historical lookup table
$$
\mathbf{Q} \in \mathbb{R}^{M \times P}
$$
using epochwise fluctuations in $Rel^{sym}$ [2601.05648]. A noisy-pair threshold is then defined as
$$
c_{np}= \mu_p + \alpha_{np}\sigma_p,
$$
with $\alpha_{np}=3$, and temporal stability reliability is
$$
Rel^{stab}_{i,i} = \exp\left( -\frac{(q_{i,i}(p) - \mu_p)^2}{2\sigma_p^2} \right).
$$
The combined positive weight becomes
$$
W_{i,i}^{pos} = \begin{cases} Rel^{sym}_{i,i}, & \text{if } Rel^{sym}_{i,i} \leq c_{np} \lor p \leq 3 \\ Rel^{sym}_{i,i} \cdot Rel^{stab}_{i,i}, & \text{if } Rel^{sym}_{i,i} > c_{np}. \end{cases}
$$
This means early epochs use symmetry only, while later epochs additionally penalize unstable positive pairs [2601.05648].

The positive loss is
$$
\mathcal{L}_{pos}(C_i, T_i) = -W^{pos}_{i,i} \left( \log \left( \frac{\exp(S_{i,i}/\sigma)}{ \sum_{k=1}^N \exp(S_{i,k}^{C \to T}/\sigma)} \right) + \log \left( \frac{\exp(S_{i,i}/\sigma)}{ \sum_{k=1}^N \exp(S_{i,k}^{T \to C}/\sigma)} \right) \right),
$$
while the negative term is
$$
\mathcal{L}_{neg}(C_i, T_i) = \frac{\sum_{j \neq i}^{N+B} \tan(S^{C \to T}_{i,j})}{\left(\sum_{k=1}^{N+B} \tan(S^{C \to T}_{i,k})\right)^\omega} + \frac{\sum_{j \neq i}^{N+B} \tan(S^{T \to C}_{i,j})}{\left(\sum_{k=1}^{N+B} \tan(S^{T \to C}_{i,k})\right)^\omega}.
$$
The CCA loss is
$$
\mathcal{L}_{CCA}(C_i, T_i) = \mathcal{L}_{pos}(C_i, T_i) + \lambda \cdot \mathcal{L}_{neg}(C_i, T_i),
$$
and the final CRA loss is
$$
\mathcal{L}_{CRA}(C_i, T_i) = \mathcal{L}_{pos}(C_i, T_i) + \lambda \cdot w^{PSW}(C_i, T_i) \cdot \mathcal{L}_{neg}(C_i, T_i),
$$
with
$$
\lambda = 5.
$$
The overall training objective is
$$
\mathcal{L} = \mathcal{L}_{GEP} + \mathcal{L}_{GEPC} + \mathcal{L}_{CRA}.
$$
The paper identifies CRA as the principal mechanism by which the model becomes resistant to noisy multimodal supervision [2601.05648].

## 5. Data scale, implementation, and evaluation tasks

OKR-CELL is pretrained on **SCxGEN-32M**, a corpus of **32 million cell-text pairs** collected from **CELLxGENE** [2601.05648]. Each pair contains scRNA-seq information together with metadata-derived text, and the corpus is described as a whole-human pretraining set spanning diverse tissues, organs, developmental stages, and cell type categories [2601.05648]. The whole-human pretraining split is reported as **99.6% training**, **0.2% validation**, and **5000 samples** reserved for the retrieval benchmark **SCxGEN-CT5K**; the paper notes these printed percentages do not sum to 100%, which suggests a likely typographical inconsistency in the source description [2601.05648].

The scRNA-seq preprocessing follows scGPT conventions: only genes with **non-zero expression** are fed into the model; the maximum input length is **1200**; if a cell contains more than 1200 non-zero genes, **1200 genes are randomly sampled** per iteration; the gene generation ratio is selected uniformly from $\{0.25, 0.50, 0.75\}$; preprocessing includes **log1p transformation** and **HVG selection**; and expression values are binned to reduce scale inconsistency across batches [2601.05648].

Training uses **Adam**, **batch size 14 per GPU**, **initial learning rate $1\times10^{-4}$**, **weight decay 0.9 applied after each epoch**, **6 training epochs**, and hardware comprising **8 nodes × 4 NVIDIA A100 40GB GPUs per node**, for a total of **32 A100 GPUs**; reported training time is approximately **3 days** [2601.05648]. The temperature $\sigma$, robustness parameter $\omega$, and memory-bank size $B$ appear in the formulations but are not numerically specified in the visible text [2601.05648].

The model is evaluated on **6 tasks**: **cell clustering**, **batch-effect correction**, **traditional cell-type annotation**, **few-shot cell-type annotation**, **zero-shot cell-type annotation**, and **bidirectional cell-text retrieval** [2601.05648]. Metrics include **ARI**, **AMI/NMI**, and **AvgBIO** for clustering; **AvgBIO**, **AvgBatch**, and eight detailed indicators for batch correction; **Accuracy** and **Macro-F1** for annotation; and **R@1, R@5, R@10** for retrieval [2601.05648]. Baselines include **Geneformer**, **scBERT**, **scFoundation**, **scGPT**, **LangCell**, and **scCLIP-GPT**, where the latter uses the same cell and text encoders as OKR-CELL but replaces CRA with standard **CLIP InfoNCE** and uses only original text [2601.05648].

## 6. Empirical performance, ablations, and interpretive issues

The paper reports that OKR-CELL obtains cutting-edge results across all six evaluation tasks [2601.05648]. In **cell clustering**, it achieves the best performance across the main datasets, including **Blood**, **Kidney**, **hPancreas**, and **hPBMC** [2601.05648]. Reported values include **Blood ARI 0.387, AMI 0.584**, compared with second-best scGPT at **ARI 0.175, AMI 0.374**, and **Kidney AMI 0.732** versus **0.371** for scGPT [2601.05648]. For **hPancreas**, the paper reports **ARI 0.374** and **AMI 0.535** [2601.05648].

In **batch-effect correction**, OKR-CELL is reported as best on both biological signal retention and batch mixing for **hPancreas** and **hPBMC** [2601.05648]. On **hPancreas**, it reaches **AvgBIO = 0.708** and **AvgBatch = 0.790**, compared with scGPT at **AvgBIO 0.687** [2601.05648]. Across eight specific indicators, it is best on all except **PCR index** [2601.05648].

For **traditional cell-type annotation**, the paper states that OKR-CELL consistently outperforms all baselines, including **scCLIP-GPT**, in both accuracy and F1-score across datasets including **small-intestine**, **spleen**, **hPBMC**, and additional analyses on **eye** and **kidney** [2601.05648]. The full per-dataset table is not reproduced in the supplied material, but the text reports strong linear separability and biologically meaningful rather than random confusions [2601.05648].

For **few-shot annotation**, performance is highlighted on **Zheng68k** and **Great Apes** with $K \in \{1,5,9\}$ shots per class and a frozen backbone [2601.05648]. Reported values include **Zheng68k 9-shot Accuracy 0.787**, compared with **0.554** for scCLIP-GPT and **0.190** for LangCell, and **Great Apes Accuracy 0.412**, compared with **0.361** for LangCell and **0.264** for scCLIP-GPT [2601.05648].

For **zero-shot annotation**, the paper emphasizes results on **Eye**, **Prostate Gland**, and **Great Apes** [2601.05648]. On **Prostate Gland**, OKR-CELL achieves **Accuracy 0.511** versus **0.299** for scCLIP-GPT [2601.05648]. The paper interprets these results as particularly important because zero-shot annotation depends directly on semantic cell-text alignment.

For **bidirectional cell-text retrieval**, the paper introduces **SCxGEN-CT5K**, consisting of **5000 cell-text pairs** from **36 tissues** that are all unseen during training [2601.05648]. In cell-to-text retrieval, OKR-CELL reaches **R@1 = 3.32**, **R@5 = 19.7**, and **R@10 = 42.9** [2601.05648]. The evaluation is strict: only strictly one-to-one annotated pairs count as positives, while even same-cell-type or same-parent-type pairs are treated as negatives [2601.05648].

The robustness experiments are central to the paper’s claims. Under simulated gene dropout from **10% to 50%**, OKR-CELL shows smaller degradation than scCLIP-GPT [2601.05648]. In **small intestine**, OKR-CELL accuracy drops from **91.9% to 87.2%**, a **5.1%** decrease, while scCLIP-GPT drops from **89.1% to 84.1%**, a **5.6%** decrease [2601.05648]. In **hPBMC**, as dropout rises from **10% to 40%**, OKR-CELL remains **above 97%**, while scCLIP-GPT drops from **95.2% to 92.4%** [2601.05648]. Under noisy multimodal pretraining with **30% gene expression perturbation** and **30% shuffled cell-text mismatches**, OKR-CELL degrades little and in some cases improves, whereas **scCLIP-GPT(noisy)** degrades more substantially [2601.05648].

The ablation study isolates the contributions of open-world text enrichment and CRA [2601.05648]. On **hPBMC**, **Original text + InfoNCE** gives **Accuracy 0.958, F1 0.934**, while **LLM-enriched text + InfoNCE** improves to **Accuracy 0.964, F1 0.951** [2601.05648]. Further replacing InfoNCE with CCA and then adding PSW, CMMB, and full CRA yields the best results, culminating in **full CRA: 0.976 / 0.959** on hPBMC [2601.05648]. On **spleen**, the sequence **0.762 / 0.735** with enriched text + InfoNCE, **0.770 / 0.751** with CCA, **0.773 / 0.750** with +PSW, **0.780 / 0.763** with +CMMB, and **0.776 / 0.765** with full CRA indicates that the robust objective is beneficial even though one intermediate configuration yields the highest reported accuracy [2601.05648].

A plausible interpretation is that the model’s reported gains arise from two separable sources: richer language supervision and more robust alignment under noisy cell-text correspondence. That interpretation is directly supported by the ablation structure, which first changes the text source and then changes the contrastive formulation [2601.05648].

## 7. Relation to prior work, scope, and limitations

OKR-CELL differs from earlier single-cell foundation models in three ways explicitly emphasized in the source material: it treats language as **cell-level biological supervision** rather than merely labels; it incorporates **open-world biological knowledge** through ontology normalization, PubMed retrieval, BioBERT semantic search, DeepSeek-V3 generation, and semantic filtering; and it explicitly models **noisy cross-modal supervision** through CRA instead of using plain CLIP-style alignment [2601.05648].

The model’s strongest reported use cases are **cell type annotation when labels are scarce**, **few-shot adaptation**, **zero-shot annotation**, **batch-effect resistant representation learning**, **cell-text retrieval**, and future **multi-omics integration** in which text can serve as a common interface [2601.05648]. The paper also proposes future directions including **gene-level textual annotations**, extension to **multi-omics integration**, and **chat-based multimodal LLM systems** for single-cell data interrogation [2601.05648].

Several limitations are either stated directly or clearly indicated in the provided material. Training appears focused primarily on **scRNA-seq**, despite a broader multi-omics motivation [2601.05648]. The text-enrichment pipeline depends on both LLM generation quality and retrieval quality [2601.05648]. Several implementation specifics are underreported, including the exact momentum coefficient, memory bank size, and temperature values [2601.05648]. The retrieval benchmark is relatively small compared with the pretraining scale, and enriched texts may still inherit literature bias or ontology bias even after reliability screening [2601.05648].

A plausible source of confusion is the similarity between the acronym “OKR” in OKR-CELL and its use in agent-systems literature. A separate work, **“Agents meet OKR: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation”** [2311.16542], uses “OKR” in the organizational sense of **Objects and Key Results** for hierarchical agent orchestration. By contrast, OKR-CELL uses “OKR” to denote **Open-world Language Knowledge-Aided Robust** in a biological foundation model [2601.05648]. The two works are therefore associated by acronym rather than by domain or method.

Within the single-cell multimodal literature, OKR-CELL is best understood as a cross-modal foundation model that couples literature-grounded text augmentation with a reliability-aware contrastive objective. Its technical distinctiveness lies in the CRA formulation and its biological significance lies in showing that cell-language alignment becomes more effective when the text side is enriched from open-world biomedical knowledge rather than constrained to sparse metadata [2601.05648].

Source: https://www.emergentmind.com/topics/okr-cell