Cell2Sentence: Transformer-Based Single-Cell Modeling

• Cell2Sentence (C2S) is a framework that converts single-cell gene expression data into ordered gene-token sentences for transformer-based analysis.
• The method leverages a decoder-only transformer with ranking-based tokenization and sparse transcoder approximation to extract interpretable decision circuits.
• It integrates cross-modal semantic alignment by fusing gene text embeddings with structural clustering, outperforming traditional scRNA-seq methods.

Cell2Sentence (C2S) is a single-cell modeling framework that treats each cell as a “sentence” whose “words” are genes, allowing transformer-based language-model machinery to operate on ordered gene lists rather than only on conventional expression matrices. In the reported literature, C2S appears both as a decoder-only single-cell foundation model for cell-type annotation and as a semantic embedding mechanism that injects NCBI-derived gene knowledge into clustering pipelines. A subsequent interpretability study applies transcoders to C2S in order to extract sparse internal “decision circuits,” with the stated aim of tracing final predictions back to interpretable features and ultimately to gene tokens (Hosokawa et al., 18 Sep 2025, Xu et al., 11 Jun 2026).

1. Sentence formulation and biological semantics

The central premise of C2S is to serialize each cell’s highly expressed genes into a token sequence. In the decoder-only formulation, each assayed cell’s top-expressed genes are turned into a 128-token “sentence,” and downstream tasks are predicted via the final token’s representation (Hosokawa et al., 18 Sep 2025). In the clustering-oriented formulation used in scLLM-DSC, C2S views each cell as a “sentence” whose “words” are gene tokens drawn from a curated vocabulary of gene symbols and summaries, and each cell is represented by its top-$K$ genes with $K=2048$ (Xu et al., 11 Jun 2026).

In the scLLM-DSC setting, vocabulary construction begins with the set $G=\{g_1,\ldots,g_M\}$ of all protein-coding genes in the organism. For each gene $g_j$, NCBI provides its official symbol, a one-line functional summary, known pathways, and related fields; these are concatenated into a structured prompt,

$$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$

The gene-level vocabulary $V$ consists of all gene symbols plus a small set of common “function” tokens such as “kinase” and “transcription_factor,” derived from the summaries (Xu et al., 11 Jun 2026).

Sentence formation is explicitly rank-based. For each cell $i$, one takes its raw or log-normalized expression $x_i\in\mathbb{R}^D$, selects the top-$K$ genes by descending expression magnitude, sorts them so that $$x_i[\mathrm{idx}_i[1]]\ge\cdots\ge x_i[\mathrm{idx}_i[K]]$$, and forms

$K=2048$0

Only gene symbols, and not numeric counts, are used as tokens so that the LLM attends to functional relationships rather than to explicit abundance values (Xu et al., 11 Jun 2026).

This published usage indicates that C2S is not reducible to a single fixed network specification. A plausible implication is that the unifying object is the cell-to-sequence transformation itself: expression profiles are converted into ordered gene-token strings so that pretrained or adapted transformer components can exploit gene co-occurrence, ordering, and external biological knowledge.

2. Decoder-only C2S as a single-cell foundation model

One reported implementation adapts the decoder-only “Pythia” transformer to the single-cell domain (Hosokawa et al., 18 Sep 2025). At its core, this C2S model consists of a learned token embedding for each gene, with vocabulary size $K=2048$1, plus standard positional embeddings for up to 128 ranks. The transformer stack contains $K=2048$2 decoder layers. Each layer comprises a multi-head self-attention block with $K=2048$3 heads, hidden dimension $K=2048$4, key/query/value projections of size $K=2048$5, and an output projection back to $K=2048$6; it also contains a two-layer feed-forward network with expansion factor $K=2048$7, that is, $K=2048$8, with each linear layer followed by a GELU nonlinearity (Hosokawa et al., 18 Sep 2025).

For supervised cell-type annotation, the model uses a lightweight classification head: one learned linear projection followed by a softmax applied to the final “:” token hidden state of size $K=2048$9 to predict one of $G=\{g_1,\ldots,g_M\}$0 cell-type labels. Fine-tuning minimizes the standard cross-entropy loss

$G=\{g_1,\ldots,g_M\}$1

where $G=\{g_1,\ldots,g_M\}$2 is the last-token hidden state and $G=\{g_1,\ldots,g_M\}$3 is the one-hot cell-type label (Hosokawa et al., 18 Sep 2025).

Pre-training the tokenizer and decoder stack on 57 million cells and biological abstracts is described as instilling both gene-expression and literature-derived co-occurrence statistics (Hosokawa et al., 18 Sep 2025). In that formulation, C2S is explicitly positioned as a state-of-the-art single-cell foundation model, and the later interpretability work is motivated by the observation that its decision-making processes are less interpretable than traditional methods such as differential gene expression analysis (Hosokawa et al., 18 Sep 2025).

A key technical motivation for the interpretability pipeline is that the MLP sublayers within each transformer block are characterized as highly polysemantic and densely connected. Directly tracing how individual gene tokens drive a final prediction is therefore presented as intractable in the raw model, which motivates replacing each MLP with a sparse, separately trained transcoder (Hosokawa et al., 18 Sep 2025).

3. Transcoder approximation of MLP sublayers

Following Dunefsky et al. (NeurIPS ’24), the interpretability study trains a separate transcoder for each MLP block of C2S (Hosokawa et al., 18 Sep 2025). For a single MLP block $G=\{g_1,\ldots,g_M\}$4, with input $G=\{g_1,\ldots,g_M\}$5 and output $G=\{g_1,\ldots,g_M\}$6, the corresponding transcoder is a wide, sparsely activating autoencoder that approximates the original two-layer feed-forward function while exposing interpretable “features.”

The encoder is

$G=\{g_1,\ldots,g_M\}$7

with $G=\{g_1,\ldots,g_M\}$8, corresponding to expansion factor $G=\{g_1,\ldots,g_M\}$9, and producing a nonnegative activation vector $g_j$0. The decoder is

$g_j$1

with $g_j$2, recovering the original output dimension (Hosokawa et al., 18 Sep 2025).

Each transcoder minimizes

$g_j$3

The first term enforces faithful function approximation, and the second enforces sparsity so that only a small subset of the 8192 features activate per input (Hosokawa et al., 18 Sep 2025).

Training is reported at the granularity of 60 million input tokens per transcoder, with peak learning rate $g_j$4, batches of 2048 tokens, and the $g_j$5 regularization coefficient above (Hosokawa et al., 18 Sep 2025). Validation on the Heart Cell Atlas v2 yielded the following comparison:

• Original MLP: validation loss 2.48
• Transcoder-replaced MLP: 4.63
• MLPs removed entirely: 12.67
• KL divergence: $g_j$6 versus $g_j$7

These values are presented as confirming that transcoders capture the vast majority of MLP functionality in a sparse, interpretable substrate (Hosokawa et al., 18 Sep 2025).

4. Decision-circuit extraction and attribution formalism

Once the transcoders are trained, circuit extraction proceeds by tracing contributions from final predictions back through sparse transcoder features and attention-mediated token interactions (Hosokawa et al., 18 Sep 2025). For a target prediction such as the final token’s classification logits, the method first identifies the top transcoder feature or features in the last layer that contributed most strongly, measured by the product of the feature’s activation $g_j$8 and its decoder-row norm.

The recursion then follows two pathways. The first is the MLP approximation pathway, in which connections from feature $g_j$9 in layer $$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$0 to feature $$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$1 in layer $$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$2 are captured by fixed decoder–encoder dot products,

$$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$3

which are input-invariant weights. Combined with the input-dependent activations $$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$4, these provide quantitative attribution (Hosokawa et al., 18 Sep 2025).

The second pathway runs through attention heads. Information flows across tokens via each head’s OV matrices and attention weights $$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$5. By linearizing each head’s update, the method obtains attributions from source token $$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$6 to transcoder feature $$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$7 in the next layer (Hosokawa et al., 18 Sep 2025). This produces a hybrid graph in which sparse MLP features and prompt tokens are both explicit objects of analysis.

To manage graph size, the procedure prunes to retain only the top-$$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$8 strongest paths at each step and then merges paths into a sparse computational subgraph termed the “decision circuit.” All retained edges are collected into an adjacency matrix $$T_j = \text{“Gene: [symbol}_j\text{]. Summary: [text}_j\text{]. Pathways: […].”}$$9 whose nodes are transcoder features $V$0 and prompt tokens $V$1. A simplified logical-gate view is written as

$V$2

where $V$3 thresholds sparse feature activations, $V$4 is a small activation cutoff, and $V$5 is the classifier nonlinearity (Hosokawa et al., 18 Sep 2025).

The significance of this construction is methodological rather than merely visual. It provides a principled attribution route from a final label back to layer-specific sparse features and, through attention-mediated transport, to the gene tokens that seeded the prediction.

5. Biological correspondence and interpretability evidence

A case study on an artery endothelial cell illustrates the intended biological reading of a C2S decision circuit (Hosokawa et al., 18 Sep 2025). The extracted circuit targeted the most strongly activated last-layer feature, identified as feature ID 3353. A distilled subcircuit highlighted three gene tokens driving the prediction of “endothelial cell of artery”:

• VWF: von Willebrand factor, described as the canonical Weibel–Palade body resident marker.
• PTPRB: VE-PTP, described as a junctional tyrosine phosphatase critical for vascular integrity.
• SPARCL1: hevin, described as an IBD-associated matricellular protein that stabilizes quiescent endothelium.

The study states that the presence of these genes, each with well-characterized roles in endothelial biology, among the top contributors to the predicted label demonstrates that transcoder-based circuit tracing recovers biologically plausible regulatory modules (Hosokawa et al., 18 Sep 2025). This is the central empirical claim linking internal C2S features to recognizable biology.

The work also reports a broader interpretability statistic based on human evaluation of layer 12 features. Specifically, 35% of the “live” features—defined as those with $V$6, where $V$7 is the per-token activation probability—correspond to gene-level tokens in a semantically coherent way (Hosokawa et al., 18 Sep 2025). This is contrasted with the original model’s opaque neuron activations.

At the same time, limitations are stated directly. Current circuits can still be large and require manual feature-to-biology mapping (Hosokawa et al., 18 Sep 2025). A plausible implication is that C2S interpretability, in this form, is mechanistic but not fully automated: sparse structure makes analysis possible, yet domain expertise remains necessary to assign biological meaning to individual features and subgraphs.

6. Cross-modal semantic alignment and clustering

In scLLM-DSC, C2S is used as a semantic component within a cross-modal deep structural clustering framework rather than as a decoder-only predictor (Xu et al., 11 Jun 2026). The framework establishes a semantically grounded representation by synergizing two views: a Knowledge-Driven Semantic View derived from NCBI gene priors and contextualized Cell2Sentence embeddings, and a Structure-Aware Topological View extracted via a graph-guided encoder. The stated motivation is that traditional scRNA-seq clustering collapses each gene to a mere index and is therefore semantically agnostic.

The semantic branch uses a frozen transformer-based text encoder $V$8, for example OpenAI text-embedding-3-small, mapping token sequences or prompts into $V$9. Gene prompts are embedded as

$i$0

and cell sentences are embedded as

$i$1

Internally, $i$2 is described as a multi-layer Transformer with self-attention, layer normalization, and a final pooling head, with weights kept frozen (Xu et al., 11 Jun 2026).

In parallel, the method constructs an abundance-weighted semantic embedding

$i$3

so that for cell $i$4,

$i$5

The two semantic views are fused as

$i$6

This fused semantic representation is then aligned to a structural embedding $i$7 extracted by the scCDCG autoencoder+graph-cut backbone (Xu et al., 11 Jun 2026).

Alignment is implemented with two projection heads $i$8 and $i$9, each a Linear→ReLU→Linear MLP, which map the semantic and structural views into a shared $x_i\in\mathbb{R}^D$0-dimensional space. After $x_i\in\mathbb{R}^D$1 normalization of each row, the cross-modal similarity matrix is

$x_i\in\mathbb{R}^D$2

The model minimizes a symmetric InfoNCE loss over a minibatch of $x_i\in\mathbb{R}^D$3 cells together with a variance regularization term, producing

$x_i\in\mathbb{R}^D$4

In the reported configuration, the structural encoder uses scCDCG with layer sizes $x_i\in\mathbb{R}^D$5, so $x_i\in\mathbb{R}^D$6; the projection heads output $x_i\in\mathbb{R}^D$7; the contrastive temperature is $x_i\in\mathbb{R}^D$8; the variance weight is $x_i\in\mathbb{R}^D$9; and the fusion weight is $K$0 (Xu et al., 11 Jun 2026).

The evaluation uses six public scRNA-seq benchmarks from scCluBench: Mauro Pancreas, Sonya Liver, Sapiens Liver, Muris Brain, Muris Liver, and Muris Limb Muscle. Preprocessing uses no additional filtering or batch correction, and always selects the 2048 highest-expressed genes per cell. Training follows a two-stage procedure: pre-train the autoencoder with $K$1 for 200 epochs, then jointly fine-tune all modules with the full objective

$K$2

with $K$3, for another 200 epochs, using Adam with learning rate $K$4, five seeds, and an NVIDIA A800 GPU (Xu et al., 11 Jun 2026).

The reported outcome is that scLLM-DSC significantly outperforms eleven state-of-the-art baselines in clustering accuracy (Xu et al., 11 Jun 2026). In this context, C2S functions as the mechanism that converts priority-sorted gene lists into natural-language-like sequences so that a frozen LLM can infuse external biological knowledge into the latent space. Taken together with the transcoder study, the published record presents C2S as both a representational interface between transcriptomes and LLMs and a substrate for mechanistic interrogation of single-cell foundation models (Hosokawa et al., 18 Sep 2025, Xu et al., 11 Jun 2026).