Phylogenetic Language Modeling in Genomics

• Phylogenetic language modeling is the integration of evolutionary trees and substitution models into neural frameworks to analyze genomic sequences.
• It employs methodologies such as multispecies alignments, transformer attention, and variational inference to capture evolutionary signals in nucleotide and protein data.
• These approaches improve variant effect prediction, gene annotation, and phylogenetic inference by achieving superior metrics like AUROC and ELBO.

Phylogenetic language modeling in genomics refers to the integration of phylogenetic principles—specifically the modeling of nucleotide or amino acid evolution across trees—into the design, training, and inference objectives of neural LLMs for biological sequence data. These frameworks explicitly encode evolutionary relationships among sequences, either by utilizing multispecies alignments, directly modeling tree-based mutation processes, or by extracting evolutionary signals through learned sequence embeddings. The resulting models have shown increased accuracy and robustness in key genomics applications, such as variant effect prediction, functional genome annotation, and phylogenetic inference.

1. Fundamentals of Phylogenetic Language Modeling

Phylogenetic language modeling formalizes the use of evolutionary histories in the statistical learning of genome and protein sequence features. The central concept is the explicit incorporation of phylogenetic trees and substitution processes—such as Felsenstein’s F81 model—within the loss function of deep neural sequence models. Given a multispecies alignment and the corresponding minimal spanning tree over species at each genomic position, models are trained to maximize the likelihood under an explicit substitution model, parameterized by the neural network output, e.g., $P_{F81}( y^{(i)}\,|\,\theta,\,T^{(i)} )$ for nucleotide data (Albors et al., 4 Mar 2025).

A contrasting paradigm leverages multiple sequence alignments (MSAs) as model input, either using transformers or other architectures, and implicitly encodes phylogenetic structure via learned attention patterns or embedding relationships (Lupo et al., 2022). Additionally, variational inference frameworks now use LLMs as feature extractors, feeding representations into downstream modules designed to generate and optimize phylogenetic trees and branch lengths jointly (Duan et al., 2024).

2. Architectures and Phylogenetic Loss Formulations

Three modeling strategies exemplify the integration of phylogenetic context:

• Explicit Phylogenetic Loss (PhyloGPN): In PhyloGPN, a convolutional neural network maps a 481-bp human DNA window to parameters $\theta$ for an F81 substitution model; these parameters define a multinomial equilibrium distribution $\pi = \mathrm{softmax}(\theta)$. For each training instance, the likelihood of the observed aligned nucleotides (up to 447 species) under the inferred tree $T^{(i)}$ is computed using Felsenstein's pruning algorithm, and the neural network minimizes the negative log-likelihood plus a conditioning term to prevent reference base "cheating". For numerical stability, a sigmoid-based upper bound replaces the standard likelihood (Albors et al., 4 Mar 2025).
• MSA Transformer and Attention-driven Phylogeny: The MSA Transformer distinguishes between row-attention (co-evolutionary signal) and column-attention (phylogenetic relationships). The latter's attention matrices, aggregated across all positions, are highly predictive of inter-sequence (row) Hamming distances, exhibiting coefficient-of-determination $R^2$ up to 0.99 within protein families and $\sim0.6$ for universal models (Lupo et al., 2022). This explicit correlation demonstrates transformer layers internalize fine-grained tree-structured dependencies.
• End-to-end Structure Generation (PhyloGen): PhyloGen leverages a pre-trained genomic LLM (DNABERT2) to encode sequences, constructs pairwise "distance" matrices from embeddings, and applies neighbor-joining to yield initial tree topologies. It then uses variational autoencoding (VAE) with specific encoders/decoders for both topology and branch-length learning, integrating a scoring function for stable optimization (Duan et al., 2024).

3. Training Protocols and Inference Mechanisms

Training procedures in this field exploit large-scale multispecies alignments, typically spanning hundreds of vertebrate genomes. Data are sampled according to genome-wide distributions to address coverage biases across chromosomes and sex chromosomes (autosomes, X, Y). Batching is performed over non-overlapping genomic blocks to manage memory and ensure species diversity (Albors et al., 4 Mar 2025).

Optimization leverages AdamW and similar stochastic optimizers, with careful avoidance of data augmentation or masking, as phylogenetic objectives inherently control for overfitting via evolutionary trajectories. At inference, state-of-the-art models such as PhyloGPN require only a single DNA sequence as input—alignment-free—while retaining phylogenetically informed predictions by projecting new sequences into the learned $\theta$ space (Albors et al., 4 Mar 2025). In PhyloGen, no alignment or explicit substitution model is required for inference: raw sequences are embedded, and phylogenetic trees are generated and scored directly (Duan et al., 2024).

4. Applications and Empirical Performance

Phylogenetic LLMs provide several high-impact applications and benchmarks:

• Variant Effect Prediction: PhyloGPN computes a log-likelihood ratio (LLR) $\log[\pi_{alt} / \pi_{ref}]$ for SNVs in human genomes, achieving superior AUROC on ClinVar and regulatory OMIM datasets compared to prior gLMs, and leading in deep mutational scanning correlative accuracy ($\rho$) across 22/25 protein experiments. GPN-MSA, which requires alignment at inference, marginally surpasses zero-shot regulatory variant prediction (Albors et al., 4 Mar 2025).
• Genomic Embedding and Transfer Learning: Embeddings from phylogenetic models enable downstream classifiers for gene-finding, enhancer annotation, and functional genomics tasks. PhyloGPN embeddings attain state-of-the-art AUROC in chromatin accessibility, histone modification, and disease variant prediction, while large effective contexts (PhyloGPN-X) benefit gene-finding (Albors et al., 4 Mar 2025).
• Phylogenetic Tree Inference: PhyloGen outperforms established MCMC and variational-inference methods (MrBayes, SBN, VBPI, ARTree, GeoPhy) in marginal log-likelihood (MLL) and ELBO across eight benchmarks (27–64 taxa; 378–2520 sites), achieving faster convergence and high topological diversity (0.89 vs 0.36) (Duan et al., 2024). Embedding-driven inference is robust to sequence length, species composition, and requires neither alignment nor explicit substitution modeling.
• Encoding and Disentangling Phylogenetic Signals: MSA-based models differentiate between coevolutionary constraints and phylogenetic correlations, showing that contact prediction is degraded two- to three-fold less under simulated phylogenetic noise in MSA Transformer versus Potts models—demonstrating inherent robustness to overcounted phylogenetic signal (Lupo et al., 2022).

5. Interpretability, Limitations, and Future Directions

Analyses of transformer attention reveal that early column-attention heads encode pairwise sequence similarity corresponding to phylogenetic relationships, with regression weights showing universality across protein families (Lupo et al., 2022). PhyloGen provides interpretable visualizations: aquatic vs. terrestrial species cluster accurately; posterior node supports are clustered; and splits in tree topology are traceable to specific k-mer signals (Duan et al., 2024).

Several limitations remain. PhyloGPN is built on the F81 substitution model, which lacks the expressiveness of GTR-class models or context-specific substitution schemes, potentially limiting its generalization in highly variable evolutionary scenarios (Albors et al., 4 Mar 2025). Extending model receptive fields beyond 6 kbp via attention/state-space models, incorporating gene trees or local genealogies, and modeling indels or epigenetic marks represent open technical directions.

A plausible implication is that as richer pre-trained genomic LLMs are built on ever-expanding alignment data, they will not only improve the accuracy of functional annotation but also provide new tools for nonparametric phylogenetic inference—free from the constraints of predefined substitution models or manual alignment procedures.

6. Comparative Table of Core Approaches

Model/Framework	Phylogenetic Input	Loss/Objectives	Applications
PhyloGPN	Explicit: MSA + trees	Phylogenetic likelihood; F81	Variant effect prediction, transfer learning
MSA Transformer	Implicit: MSA	Masked LM (row/col attention)	Structure, function, phylogenetic encoding
PhyloGen	None; LM embeddings	Variational ELBO on trees, branches	Phylogenetic inference, structure generation

This summary illustrates the spectrum of strategies, from explicit evolutionary modeling within modern neural architectures, to embedding-based approaches enabling new forms of end-to-end phylogenetic inference. Each approach extends the capacity of genomic LLMs to learn, encode, and exploit evolutionary structure for diverse tasks in genomics and evolutionary biology (Albors et al., 4 Mar 2025, Lupo et al., 2022, Duan et al., 2024).