GenVarFormer: Mutation-Centric Transformer

• The paper introduces a mutation-centric transformer that models long-range genomic effects from sparse somatic mutations, achieving 26-fold higher prediction correlation.
• It employs a novel mutation-only representation with gene context and custom RoPE, efficiently processing data over a 16 Mbp window without redundant sequence information.
• Patient-level embeddings derived from the model significantly improve prognostic assessments and subtype clustering compared to traditional gene expression measures.

Searching arXiv for the specified GenVarFormer paper and closely related cited work. GenVarFormer (GVF) is a transformer-based architecture for predicting gene expression from sparse, long-range somatic mutations in cancer. It is designed to model mutation effects up to 8 million base pairs away from a gene by only considering mutations and their local DNA context, while omitting the vast intermediate sequence. Using data from 864 breast cancer samples from The Cancer Genome Atlas, the model is reported to predict gene expression with 26-fold higher correlation across samples than current models, to generalize to unseen genes and samples simultaneously, and to yield patient embeddings that are more informative than ground-truth gene expression for predicting overall patient survival in luminal A breast cancer (Laub et al., 29 Sep 2025).

1. Problem setting and design objective

GenVarFormer is motivated by a specific difficulty in cancer genomics: most cancer somatic mutations occur in the non-coding genome and are rare or unique, whereas one of the primary mechanisms by which non-coding driver mutations contribute to cancer is by affecting gene expression, potentially from millions of nucleotides away. The model is presented as addressing four constraints simultaneously: modeling long-range effects, handling the extreme sparsity of somatic mutations, generalizing to unseen genes, and maintaining efficiency under highly variable sparse inputs (Laub et al., 29 Sep 2025).

The architecture is explicitly framed against limitations of existing predictors of gene expression from mutations. Prior approaches are described as being unable to simultaneously handle interactions spanning millions of base pairs, the extreme sparsity of somatic mutations, and generalization to unseen genes. GenVarFormer therefore adopts a mutation-centric representation rather than a contiguous-sequence formulation: it processes only mutated sites and their local context, and it omits long intervals of unmutated reference sequence. A plausible implication is that the model is tailored to settings in which regulatory effects are distal while the observed mutational signal remains sparse and irregular.

The reported mutation density is approximately 6 mutations per megabase in cancer, and the mutation counts per instance are described as Zipf-distributed. These properties are central to the model’s formulation: the task is not merely long-context modeling, but long-context modeling under severe sparsity and sample-specific variation in token count.

2. Mutation representation and transformer architecture

GenVarFormer considers mutations in a 16 Mbp window centered on a gene’s transcription start site, which is described as over 340x longer than previous models. Each mutation is represented through multiple feature channels that are embedded into vectors in $\mathbb{R}^d$ with $d=64$ (Laub et al., 29 Sep 2025).

Component	Representation
ALT	Alternative DNA sequence, encoded with a single-layer transformer and mean pooling
ILEN	Indel length, signed and symmetric log-transformed
VAF	Variant allele fraction
Flanking DNA	32 bp sequences from both 5′ and 3′ ends, processed by a shallow ConvNova CNN
POS	Relative to the gene's start, rounded to the nearest 100 bp

Gene context is added to this mutation representation. The gene embeddings are derived from Borzoi using gene promoter and/or coding sequences, then projected via a ConvNova CNN, mean pooled, and added to each mutation’s vector embedding, scaled by a small learnable parameter. The paper states that, because these features are derived solely from DNA sequence, the model is not tied to a fixed gene set and can generate predictions for entirely novel genes provided their sequences are available.

After mutation embeddings are augmented with gene context, they are treated as a sequence. Rotary positional embeddings (RoPE) encode mutation positions and are implemented for irregular and sparse positions. The sequence is then processed by a 4-layer Transformer with 4 heads per layer, mean pooled, and projected to a scalar predicting gene expression. The summary equations given for this pipeline are

$$\{ \mathbf{m}_0, ..., \mathbf{m}_n \} \in \mathbb{R}^d$$

and

$$\hat{y} = \mathrm{proj}\left( \mathrm{meanpool}\left(\mathrm{Transformer}(\mathbf{m}_i, \mathrm{RoPE}(POS_i))\right) \right).$$

The representation strategy is explicitly local at the mutation level and global at the aggregation level. This suggests a division of labor in which local sequence perturbation is encoded per mutation, while long-range regulatory interaction is learned through the transformer over sparse mutation tokens rather than over contiguous genomic sequence.

3. Objective function and systems-level implementation

The training objective is a Gradient Aligned Regression Loss, described as being inspired by recent work showing advantages for regression of gene expression and as minimizing pairwise distances among the predictions and their true targets (Laub et al., 29 Sep 2025, Zhu et al., 2024). The model is optimized with AdamW, and the reported hyperparameters include $d_{\text{model}}=64$, 4 layers, 4 heads, no dropout, and learning rate $10^{-4}$.

A substantial part of the reported contribution lies in systems design for sparse, variable-length genomic inputs. Nested tensors in PyTorch are used to accommodate highly variable numbers of mutations per sample and to eliminate wasteful padding. A bin packing sampler is used so that no batch exceeds a set mutation count, which the paper describes as preventing out-of-memory errors common with naive fixed-size batches. Custom RoPE and FlashAttention-2 are used for arbitrary mutation positions and for memory and throughput efficiency.

The paper reports that this implementation is over 1,170x faster and more memory-efficient than state-of-the-art sequence models on equivalent window sizes. That comparison is tied to matched window size at 524 kbp. Because the model skips unmutated reference sequence, the efficiency claim is not merely an engineering optimization over the same tokenization scheme; it depends on the mutation-only formulation itself.

4. Data, preprocessing, and generalization protocol

The empirical study uses 864 breast cancer samples from TCGA with both whole-genome sequencing and RNA-seq. Bulk tumor RNA-seq is “purified” using InstaPrism to estimate cancer cell-specific expression and remove stromal, immune, and normal components (Laub et al., 29 Sep 2025).

The evaluation design is structured around generalization. The paper defines unseen samples (US), unseen genes (UG), and unseen genes and samples simultaneously (USG) splits. Genes are split by chromosome for gene-based splits, and cross-validation is used for hyperparameter selection. Normalization includes linear regression on the top 10 gene expression principal components to remove confounders, followed by z-scoring per split.

This protocol matters because the paper positions GenVarFormer as the first model of its kind to generalize to unseen genes and samples simultaneously. That claim is tied not only to model architecture but also to the strict holdout design. In this setting, gene-level generalization is not treated as interpolation over previously observed gene identities; it depends on the use of DNA-sequence-derived gene features.

5. Reported empirical performance and downstream utility

On the most challenging USG test set, GenVarFormer achieved an average sample-wise Pearson correlation of $r = 0.219$. The reported baselines are $r = 0.0081$ for Lasso/Hotspot models, $r = 0.0043$ for Borzoi, and $r = 0.0749$ for mean subtype gene expression (Laub et al., 29 Sep 2025). The abstract summarizes this as 26-fold higher correlation across samples than current models.

The long-range design is directly supported by ablation. Increasing the window from 524 kbp to 16 Mbp is reported to double the model’s prediction performance. The paper presents this as evidence that modeling very long-range regulatory effects is important for mutation-to-expression prediction in cancer.

The work also introduces patient-level representations derived from the model. Mean-pooled gene-level embeddings are concatenated into patient-level vectors, and UMAP visualizations are reported to show clustering by breast cancer subtype (PAM50) as well as finer substructure than available from gene expression or hotspot features. In the luminal A breast cancer subtype, with $d=64$0, Cox proportional hazard models using GVF embeddings achieved a concordance index of $d=64$1, compared with $d=64$2 for ground-truth gene expression. For progression-free survival and for other subtypes, the embeddings were generally the best or second best mutation-based feature set. The embeddings also performed well for PAM50 subtype and early/late stage prediction, even when using only mutation features.

These results are used in the paper to argue for two biomedical utilities. First, by predicting the effect of distal non-coding mutations on gene expression, the model can help flag functionally significant mutations regardless of frequency or prior annotation. Second, the learned embeddings can serve as prognostic biomarkers. This suggests that the architecture is intended not only as a predictor of molecular phenotype but also as a representation learner over sparse somatic variation.

6. Scope, interpretation, and terminological ambiguity around “GVF”

Within cancer genomics, “GVF” denotes GenVarFormer. The acronym is, however, overloaded across arXiv literature. In robotics, GVF can denote “Generative Visual Foresight” in GVF-TAPE (Zhang et al., 30 Aug 2025). In UAV guidance, GVF refers to “Guiding Vector Field” (Ntouros et al., 19 Jan 2026). In reinforcement learning, GVF commonly denotes “General Value Function” (Ring, 2021). In arithmetic geometry, GVF can denote “Globally Valued Field” (Destic et al., 2024). In bibliographic and search contexts, disambiguation is therefore necessary.

A common misconception would be to treat GenVarFormer as a conventional long-context genomic sequence model. The reported formulation is narrower and more specialized: it encodes only mutated sites and their local DNA context, adds sequence-derived gene context, and omits the intervening reference sequence. Another possible misconception is that the model is gene-specific. The paper explicitly argues the opposite: by using sequence-based gene embeddings, GVF is not tied to a fixed gene set and can generate predictions for entirely novel genes provided their sequences are available (Laub et al., 29 Sep 2025).

The paper also states extension potential beyond the reported benchmark. It could, in principle, accommodate germline variation, rare inherited diseases, or coding mutation effect modeling via extension. As stated, this is prospective rather than experimentally established within the reported study. The central established claim remains narrower: GenVarFormer is a mutation-centric transformer for predicting gene expression from sparse, long-range somatic mutations in breast cancer, with strict generalization to unseen genes and samples, and with downstream patient embeddings that are informative for survival and subtype-related analyses.