Genomics Language Models

• Genomics language models are self-supervised neural sequence models that learn the statistical and functional structure of DNA.
• They employ diverse architectures—transformers, convolutional hybrids, state-space, and recurrent models—to capture both local motifs and long-range dependencies.
• Applications include variant effect prediction, sequence design, and transfer learning, with extensions to multi-modal and cross-omics analysis.

A Genomics LLM (gLM) is a self-supervised neural sequence model designed to learn the intrinsic statistical and functional structure of genomic DNA sequences, analogously to LLMs in natural language processing. By treating the genome as a language—with a vocabulary of nucleotides (A, C, G, T) or larger “words” (k-mers, BPE tokens)—these models aim to capture the underlying biological grammar governing transcription factor binding, chromatin organization, and functional element interactions across a wide range of scales (Benegas et al., 2024).

1. Model Families, Tokenization, and Training Objectives

1.1 Architectural Families

gLMs span four principal architectural classes:

• Convolutional hybrids: Models such as GPN deploy dilated convolutional neural networks in lieu of attention to efficiently capture local motifs.
• Transformer-based encoders: Vanilla, sparse, or hierarchical-attention Transformers (e.g., DNABERT, GENA-LM, Nucleotide Transformer) are predominant in tasks requiring long-range dependency modeling. Typical parameter scales range from 110 M (BERT-base style) to several billion.
• State-Space Models (SSMs): HyenaDNA (Hyena operator), Caduceus (Mamba SSM), and JanusDNA (hybrid attention-Mamba-MoE) achieve linear or subquadratic complexity in sequence length, processing up to megabase-scale contexts (Duan et al., 22 May 2025). JanusDNA’s design uniquely enables global bidirectional context and processes ∼1 Mb DNA per step on a single 80 GB GPU.
• Recurrent/other architectures: Some models (e.g., LookingGlass) use RNNs for fragment-level modeling but are generally outperformed by newer architectures in terms of long-range context handling (Benegas et al., 2024).

1.2 Tokenization Approaches

gLMs rely on varying granularity for input representation:

• Single-nucleotide tokens ({A,C,G,T}): Maximal resolution, natural for variant-aware modeling.
• k-mer tokens: Overlapping or non-overlapping substrings of length k (e.g., DNABERT uses overlapping 6-mers; v ≈ 4⁶ = 4096). This approach balances sequence length and motif capacity.
• Byte-Pair Encoding (BPE) or Unigram Subwords: BPE trained on DNA discovers frequent substrings, increasing vocabulary richness and reducing sequence length (as adopted by DNABERT-2, GENA-LM, VQDNA-HRQ) (Li et al., 2024).
• Learnable vocabulary via vector quantization: VQDNA adaptively tokenizes DNA into pattern-aware embeddings, discovering a high-resolution lexicon jointly with LLM training. Hierarchical Residual Quantization (HRQ) enhances this with multi-scale codebooks to disentangle coarse motifs from fine mutations (Li et al., 2024).

1.3 Training Objectives

• Autoregressive (Causal) Language Modeling (CLM):

$$L_\mathrm{CLM}(\theta) = -\sum_{i=1}^N \log P_\theta(x_i \mid x_{1:i-1})$$

• Masked Language Modeling (MLM):

$$L_\mathrm{MLM}(\theta) = -\mathbb{E}_\mathcal{K}\; \sum_{i\in \mathcal{K}} \log P_\theta(x_i \mid X_{-\mathcal{K}})$$

• Genomic-specialized variants:
  • Masked-span/progressive unmasking for regulatory grammars.
  • Contrastive species/region embeddings, e.g., separating regulatory logic across taxa.
  • MSA-based MLM (e.g., GPN-MSA leverages multispecies alignments to encode conservation).
  • Multimodal joint objectives (e.g., prediction of chromatin state tokens, epigenetic marks, RNA structure features) (Benegas et al., 2024).

2. Data, Scalability, and Pretraining Practices

2.1 Genomic Corpora

Training data scale spans:

• Human reference genomes: Whole-genome (e.g., GRCh38, ~3 Gb); variant-augmented sets (e.g., 1000 Genomes Project).
• Multi-species assemblages: From tens to hundreds of eukaryotic and prokaryotic genomes, plus model organisms in both plants and animals (Benegas et al., 2024).
• MSAs and specialized datasets: Cross-vertebrate alignments, regulatory element–focused windows, cellular or population-specific cohorts.
• VQDNA and HRQ: Train on billions of bases across combined human and multi-species sets, learning maximal pattern diversity (Li et al., 2024).

2.2 Data Processing and Partitioning

• Windowing: Sequences are split into fixed-length windows (1–131 kb). Randomized starts mitigate frame artifacts, especially for k-mer and 6-mer tokenization (Wu et al., 11 Feb 2025).
• Repetitive element management: Downweighting or subsampling of repeats is common (>50% of repeat regions in some sets).
• Species tokens: Special tokens pre-pended to encode species identity, necessary for cross-species models (Benegas et al., 2024).
• Cross-species and MSA approaches: Crucial for robust feature learning and phylogenetic transfer, but must handle rapid evolution of noncoding grammar.

3. Core Applications and Model Evaluation

3.1 Variant Effect and Fitness Prediction

• Unsupervised functional constraint is measured via the per-site log–likelihood ratio (LLR) between alleles:

$$\mathrm{LLR} = \log P(X_i=a \mid X_{-i}) - \log P(X_i=b \mid X_{-i})$$

Lower LLR predicts rarer variants in populations and correlates with pathogenicity or effect size in experimental screens (Benegas et al., 2024).

3.2 Sequence Design and Generation

• De novo regulatory/coding sequence generation: CLM-based models can synthesize sequences conditioned on control tags, e.g., regLM (HyenaDNA) generates regulatory elements that are validated in silico for motif density and activity. MegaDNA designs full-length phage genomes, and EVO constructs novel CRISPR-Cas loci (Benegas et al., 2024).
• Prompt-based design and optimization: Prompt-responsive generation conditions models to produce sequences with desired activity profiles; e.g., the Generator model separates high/low-activity promoters effectively (Wu et al., 11 Feb 2025).

3.3 Transfer Learning and Embedding Utility

• gLM embeddings enable:
  • Coding vs. noncoding genomic window clustering (e.g., GPN).
  • Feature extraction for downstream models (annotation, accessibility, variant prioritization).
  • Fine-tuning for low-data tasks (e.g., annotation in held-out cell types or species) often yields performance gains over scratch training (Benegas et al., 2024).

3.4 Evaluation Metrics

• Perplexity: Standard metric on held-out data,

$$\operatorname{perplexity} = \exp \left(-\tfrac{1}{N} \sum_{i=1}^N \log P(x_i\mid x_{<i})\right)$$

• Functional performance: Rank correlation (Spearman’s ρ) with mutational scans, AUROC on pathogenic/benign variant labels, motif enrichment.
• Design evaluation: Polygraph features (composition, k-mer, motif, activity), spatial organization (for genome-length synthesis).
• Transfer generalization: Performance is measured on held-out cell types, tissues, or phylogenetic distances against gold standard labels (Benegas et al., 2024, Wu et al., 11 Feb 2025).

4. Limitations, Privacy Risks, and Interpretability

4.1 Computational and Scalability Challenges

• Quadratic complexity of attention (O(n²) time/memory) restricts standard Transformer contexts to ≤12 kb. SSM, hierarchical/sparse attention, and vision-inspired (OpticalDNA) approaches achieve subquadratic or linear scaling for megabase contexts (Duan et al., 22 May 2025, Xiang et al., 2 Feb 2026).
• Data sparsity in functional annotation (∼3% under strong constraint in human) and dominance of nonfunctional/repetitive sequences may bias model learning.

4.2 Memorization and Privacy

gLMs trained or fine-tuned on sensitive human genomes demonstrate measurable memorization risk:

• Three complementary attack vectors:
  • Perplexity gap detection,
  • Canary extraction via prefix conditioning and beam search,
  • Membership inference (LiRA AUC-based score).
• Findings: No single metric captures all leakage. Both full and parameter-efficient fine-tuning (e.g., LoRA) can induce memorization. Repetition in training set increases extractable sequences in a monotonic fashion, even in SSMs and convolutional architectures (Nemecek et al., 9 Mar 2026).
• Best practices: Multi-vector auditing, deduplication, canary testing, and differential privacy integration are recommended for privacy-sensitive deployments.

4.3 Interpretability

• Attention maps may localize functional interactions (e.g., SpliceBERT identifies donor/acceptor communication).
• Mask-reconstructions can recover established motifs (e.g., TF-binding sites).
• Latent spaces cluster according to element class, suggesting internalization of sequence roles (Benegas et al., 2024).

5. Innovations, Multimodal Extensions, and Future Directions

5.1 Multi-modal and Hybrid Models

• Multi-modal gLMs (e.g., Geneverse pipeline (Liu et al., 2024)) integrate DNA sequence, protein structure, and spatial transcriptomics within a unified parameter-efficient (e.g., LoRA) framework for cross-omics tasks.
• Adapters and fusion modules (e.g., CodonMoE (Du et al., 6 Aug 2025)) enable DNA foundation models to transfer knowledge to RNA analysis without explicit RNA pretraining, via lightweight codon-level mixture-of-expert layers.

5.2 Open Research Questions

• Multi-scale modeling: Determining the optimal context window (from motifs to megabase-scale loops and genome topology) per downstream task.
• Population genetics integration: Infusion of mutation rate and demographic signals into pretraining.
• Modeling structural variation (insertions, deletions, inversions), and capacity scaling with respect to genome complexity (high-GC, repeat-rich, polyploid).
• Scaling hypothesis: Systematic benchmarking to validate whether performance grows logarithmically with data/model/compute.
• Interpreting emergent phenomena: Recent evidence shows in-context learning arises naturally in next-nucleotide gLMs, matching scaling laws observed in NLP (Breslow et al., 16 Nov 2025).

5.3 Model Deployment and Clinical Applications

• Foundation models like JanusDNA (Duan et al., 22 May 2025), GENERator (Wu et al., 11 Feb 2025), VQDNA (Li et al., 2024), and cross-modal frameworks (Liu et al., 2024) are being evaluated for disease variant effect prediction, synthetic sequence design, genome annotation, and experimental design pipelines.
• Parameter-efficient tuning (LoRA, adaptive rank sampling) enables efficient fine-tuning on specific genome annotation tasks, an avenue showing strong performance with <2% of parameters updated (Zhan et al., 2024).

6. Summary Table: Core Model Dimensions

Model Family	Attention Complexity	Typical Seq Length	Tokenization	Representative Models
Transformer	$O(n^2)$	≤12 kb	k-mers, BPE	DNABERT, NT, GENA-LM
Sparse/Hierarchical	$O(n\sqrt{n})$–$O(n\log n)$	up to 1 Mb	k-mers, BPE	MegaDNA, HyenaDNA, JanusDNA
SSM (State-Space)	$O(n)$–$O(n\log n)$	up to 1 Mb	Nucleotides, k-mers	HyenaDNA, Caduceus
VQ/HRQ + Transformer	$O(n)$–$$L_\mathrm{MLM}(\theta) = -\mathbb{E}_\mathcal{K}\; \sum_{i\in \mathcal{K}} \log P_\theta(x_i \mid X_{-\mathcal{K}})$$0	up to 32 kb	VQ-learned tokens	VQDNA, VQDNA-HRQ

7. Outlook

Genomics LLMs represent a rapidly advancing paradigm at the confluence of statistical sequence modeling and biological insight. By leveraging architectural advances from NLP LLMs, adapting for the peculiarities of genomic data, and integrating emerging strategies in multi-modality and privacy, gLMs are poised to drive forward both algorithmic and biological discovery. Key open directions include maximizing biological fidelity in representation, benchmarking against functional ground truth, and scaling models to exploit the full throughput of genomic and multi-omic sequencing (Benegas et al., 2024, Nemecek et al., 9 Mar 2026, Duan et al., 22 May 2025).