Gengram: Retrieval-Augmented Genomic Memory

• Gengram is a retrieval-augmented conditional memory module that explicitly captures multi-base motifs in genomic sequences via a specialized hashing scheme.
• It integrates with Transformer-based genomic models by aggregating k-mer motifs from a causal window, thereby shifting motif retrieval from implicit to explicit representation.
• Empirical results demonstrate up to 14% improvement on functional genomics tasks, validating its efficiency and biological relevance in motif-dominated predictions.

Gengram is a retrieval-augmented conditional memory module for genomic foundation models (GFMs) that introduces an explicit and highly efficient lookup primitive for multi-base motifs through a genomic-specific hashing scheme, with the stated aim of establishing genomic “syntax” (Xu et al., 29 Jan 2026). It is designed to address a central limitation of current GFMs: standard Transformer-based models ingest single-nucleotide inputs and rely on dense neural computation to implicitly reconstruct biologically consequential multi-nucleotide motifs such as splice signals, promoter elements, codons, UTR motifs, and repetitive elements. Gengram instead provides explicit motif memory, retrieving stored embeddings for $k$-mers with $k \in \{1,2,3,4,5,6\}$ from a local causal window, aggregating these multi-scale motif representations, and injecting them into the backbone through a gate-controlled residual branch. In the reported experiments, integration into state-of-the-art GFMs yields substantial gains of up to $14\%$ across several functional genomics tasks while also producing mechanistic signals that align with biological knowledge (Xu et al., 29 Jan 2026).

1. Conceptual framing and motivation

The introduction of Gengram is framed by two linked observations. First, GFMs have become strong but expensive and indirect: they learn from single-nucleotide inputs and rely on large-scale pretraining plus dense attention to reconstruct biological motifs. Second, conditional computation suggests an alternative path. Inspired by conditional memory methods such as Engram in LLMs, Gengram is presented as a genomic analogue: a hash-based memory system for $k$-mers that can be selectively retrieved and injected only where useful (Xu et al., 29 Jan 2026).

Within this framing, Gengram is not described as replacing the Transformer backbone. Rather, it supplies an additional modeling primitive in which recurring sequence fragments become explicit reusable symbols in a learned memory system. The paper refers to this as genomic “syntax,” defined as the structured composition of short biological patterns into recognizable functional elements. The relevant claim is not merely that motifs exist, but that a GFM can benefit from treating them as directly retrievable units rather than as patterns that must always be approximated through dense attention over single bases.

A plausible implication is that this reformulation shifts part of the representational burden from implicit pattern induction to explicit motif access. The paper’s emphasis on splice signals, promoter elements, codons, UTR motifs, and repetitive elements suggests that Gengram is especially targeted at tasks in which short, compositionally meaningful motifs dominate predictive structure.

2. Retrieval-augmented motif memory

At its core, Gengram maintains multi-scale motif memory tables. For each $N \in \{1,2,3,4,5,6\}$, it defines a learnable table

$$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$

where $\Sigma = \{A,T,C,G,N\}$ is the genomic alphabet, $d_m$ is the memory embedding dimension, and each possible $N$-gram has its own lookup slot. The mapping is specified by

$$h_N(\cdot): \Sigma^N \rightarrow \{0,1,\ldots,|\Sigma|^N - 1\}.$$

Because $k \in \{1,2,3,4,5,6\}$0 is small, the paper states that this hash can be implemented as a deterministic base-$k \in \{1,2,3,4,5,6\}$1 encoding and is therefore collision-free. This deterministic, motif-specific indexing is the genomic-specific hashing mechanism that differentiates Gengram from generic retrieval modules (Xu et al., 29 Jan 2026).

For a position $k \in \{1,2,3,4,5,6\}$2, the module searches a causal local window

$k \in \{1,2,3,4,5,6\}$3

enumerates all contiguous $k \in \{1,2,3,4,5,6\}$4-grams within that window,

$k \in \{1,2,3,4,5,6\}$5

and then deduplicates them,

$k \in \{1,2,3,4,5,6\}$6

The deduplication step is explicit: the design is intended to focus on distinct motif types rather than raw frequency alone.

Each unique motif is then retrieved by direct table lookup,

$k \in \{1,2,3,4,5,6\}$7

and the retrieved vectors are mean-pooled within each $k \in \{1,2,3,4,5,6\}$8-gram scale,

$k \in \{1,2,3,4,5,6\}$9

The stated reason for mean pooling is scale stabilization across positions and sequences, which makes the memory branch easier to control during training.

The per-scale summaries are concatenated,

$14\%$0

and then projected as

$14\%$1

Here $14\%$2 is used for gating and $14\%$3 carries the content to be injected. Gating is computed from the normalized current token state and normalized projected memory:

$14\%$4

The memory contribution is then transformed and added residually:

$14\%$5

This architecture makes the memory conditional rather than unconditional: motif information is not always injected, but is written only when the current hidden state is compatible with the retrieved memory content.

3. Backbone integration and computational profile

Gengram is integrated into state-of-the-art GFMs as a lightweight module inserted into selected Transformer layers. It is applied before attention and connected through a residual path, so the backbone architecture remains standard. The paper evaluates this design in Dense and MoE settings and across three attention variants—MHA, GQA, and MLA—arguing from these experiments that the module is architecture-agnostic (Xu et al., 29 Jan 2026).

Layer placement is treated as an empirical design question. For the $14\%$6B MoE model, the best configuration is reported as insertion at layers $14\%$7, a shallow/middle/deep trio that ranked first in four of five benchmark categories. The selected local window is $14\%$8. The paper connects this choice to the approximately $14\%$9–$k$0 bp helical repeat of B-form DNA, suggesting that $k$1 bp captures roughly two helical turns and therefore useful phasing structure. This suggests that the window is intended not only as a computational truncation but also as a biologically informed inductive bias.

The efficiency argument is explicit. Because the alphabet is tiny, lookup is $k$2 per $k$3-mer, and retrieval complexity for fixed window size and fixed $k$4-set is given as

$k$5

which becomes

$k$6

when $k$7 and $k$8 are fixed. The paper further states that the module uses negligible extra parameters—around $k$9M additional parameters—while preserving efficiency. The broader claim is therefore that Gengram provides explicit motif retrieval without changing asymptotic sequence scaling.

4. Pretraining regime and evaluation protocol

The reported pretraining corpus is constructed from human and non-human primate genomes from HPRC release 2 and NCBI RefSeq, with a balanced $N \in \{1,2,3,4,5,6\}$0 human/non-human ratio (Xu et al., 29 Jan 2026). Three datasets are specified: a $N \in \{1,2,3,4,5,6\}$1B-token $N \in \{1,2,3,4,5,6\}$2k dataset for ablations, a $N \in \{1,2,3,4,5,6\}$3B-token $N \in \{1,2,3,4,5,6\}$4k dataset for formal $N \in \{1,2,3,4,5,6\}$5B pretraining, and a $N \in \{1,2,3,4,5,6\}$6B-token $N \in \{1,2,3,4,5,6\}$7k dataset for continued pretraining.

The main model is named Gengram-10B-L3_6_10-W21-8k. Its architecture consists of $N \in \{1,2,3,4,5,6\}$8 Transformer layers, hidden size $N \in \{1,2,3,4,5,6\}$9, $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$0 attention heads, RMSNorm, RoPE, SwiGLU, GQA with $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$1 query groups, $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$2 experts in MoE, top-2 routing, auxiliary loss load balancing, BF16, and flash attention. Gengram itself is inserted at layers $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$3, $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$4, and $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$5; uses $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$6-mer sizes $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$7 to $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$8; uses embedding dimension $$\mathbf{M}^{(N)} \in \mathbb{R}^{|\Sigma|^{N} \times d_m},$$9 per $\Sigma = \{A,T,C,G,N\}$0-mer; uses local window $\Sigma = \{A,T,C,G,N\}$1; and uses short convolution kernel size $\Sigma = \{A,T,C,G,N\}$2.

Evaluation is carried out on $\Sigma = \{A,T,C,G,N\}$3 datasets across five categories: Genomic Structure Understanding, Gene Regulation Prediction, Epigenetic Profiling, Variant Effect and Clinical Impact, and Evolutionary Analysis. The benchmark suites include Genomic Benchmarks (GB), Nucleotide Transformer Benchmarks (NTB), Long-Range Benchmarks (LRB), and Genos Benchmarks (GeB). The protocol freezes the pretrained model as the backbone and performs downstream prediction with MLP or XGBoost, including zero-shot feature extraction evaluation.

Implementation details reported for training include Megatron-LM, $\Sigma = \{A,T,C,G,N\}$4 GPUs per node, pipeline parallelism $\Sigma = \{A,T,C,G,N\}$5, BF16, flash attention, group query attention, MoE with grouped GEMM, and auxiliary-loss load balancing. The code is released on GitHub, and the checkpoint is released on Hugging Face.

5. Empirical results, ablations, and architectural generalization

The headline empirical result is that Gengram yields up to $\Sigma = \{A,T,C,G,N\}$6 improvement over state-of-the-art GFMs on downstream tasks, with especially strong gains on motif-dominated tasks (Xu et al., 29 Jan 2026). Relative to the baseline $\Sigma = \{A,T,C,G,N\}$7B model, Gengram-10B-L3_6_10-W21-8k is reported to improve almost all metrics. The paper highlights strong performance on splice site tasks, exon classification, species classification, and long-range regulatory tasks.

Benchmark result	Score
Multi-species exon classification	0.9832
Splice site identification	0.9009
Primate-mammal species classification at 8k	0.9611
Primate-mammal species classification at 32k	0.9630

The reported multi-species exon classification score of $\Sigma = \{A,T,C,G,N\}$8 is identified as best in the corresponding table. The comparative message extends beyond within-model gains. Gengram-10B is trained on only $\Sigma = \{A,T,C,G,N\}$9B tokens, whereas Genos-10B is listed at $d_m$0T tokens, Evo2-40B at $d_m$1T tokens, and NT-v3 at $d_m$2T tokens. The paper states that Gengram is competitive or better on many tasks despite this smaller pretraining budget.

A notable result concerns context length. Gengram-10B trained at $d_m$3k is reported to match or exceed a baseline trained at $d_m$4k. The paper interprets this as evidence that the local window mechanism provides an intrinsic long-context advantage, partially compensating for a shorter nominal context length. This suggests that explicit local motif retrieval may substitute for some functions otherwise delegated to extended sequence context.

The ablation studies support several design choices. Comparing Gengram-10B-L3_6_10-W21-8k with the window-free Gengram-10B-L3_6_10-8k, the paper reports that the local window substantially improves performance, especially on splice site detection, histone mark prediction, and promoter/enhancer-related tasks. Continued pretraining to $d_m$5k benefits window-free Gengram substantially, whereas windowed Gengram gains only marginally from the same continued pretraining; the stated interpretation is that the window itself already encodes a kind of long-context inductive bias. Layer insertion ablations indicate that deeper insertion generally lowers validation loss, while the best multi-layer placement remains $d_m$6. Window-size selection shows validation loss minimized around window size $d_m$7.

The same section of results also supports the architecture-agnostic claim. Across all six tested combinations of Dense or MoE with MHA, GQA, or MLA, adding Gengram consistently lowers validation loss. On this evidence, the module is presented not as a backbone-specific adjustment but as a reusable component for future GFMs.

6. Interpretability, MoE behavior, and limitations

A substantial part of the paper is devoted to mechanistic interpretability. One reported finding is that windowed retrieval produces smoother and less spiky residual dynamics: when the $d_m$8 norm of hidden states is compared with and without windowing under frozen weights, the windowed version exhibits smoother block-output norms (Xu et al., 29 Jan 2026). The interpretation given is that the local window imposes structural stabilization.

A second diagnostic uses a LogitLens-style measure of prediction readiness, computing the KL divergence between intermediate-layer and final-layer predictions,

$d_m$9

With Gengram, early layers align with the final prediction more quickly. The paper interprets this as front-loading predictive computation, making useful structure available earlier in depth and easing optimization.

A third line of evidence examines residual write strength,

$N$0

where $N$1 is the Gengram residual contribution. These writes are reported to be sparse and high-contrast, with peaks aligned to motif-like regions, especially around TATA-like elements, poly-T tracts, and promoter-proximal boundaries. The paper presents this as evidence that Gengram writes selectively at biologically meaningful sites rather than indiscriminately memorizing local patterns.

The learned memory space is also probed for reverse-complement structure. In the $N$2-mer memory space, some layers show significant within-vs-between separation for reverse-complement pairs, suggesting biologically meaningful strand symmetry. At the same time, the paper reports that the memory does not trivially collapse synonymous codons into a single cluster. This is presented as evidence against the misconception that Gengram functions merely as a codon-to-amino-acid dictionary; instead, the memory appears to encode richer context-dependent motif structure.

For MoE backbones, Gengram is additionally claimed to stabilize sparse training. The paper argues that genomic data contain many repetitive, low-complexity sequences that can destabilize expert routing, and that Gengram’s gating filters noise before it re-enters the backbone. In Top-2 MoE experiments with $N$3, $N$4, and $N$5 experts, load-balancing loss is reported to be more stable with Gengram across sparsity settings.

The limitations are explicit. Scaling remains underexplored: larger corpora, longer contexts, and longer training runs require study. The current design uses fixed $N$6-mer sizes and fixed windows, and adaptive or learnable choices are identified as possible extensions. The evaluation is benchmark-centric, with real biological applications and experimentally validated downstream tasks left for future work. The paper also notes that effects on subtle genetic variation and direct biological assays remain to be tested. Taken together, these caveats delimit the scope of the current evidence while leaving open the broader possibility that explicit motif memory plus conditional retrieval could become a standard primitive in genomics-oriented foundation modeling.