ViroBench: Viral NFM Benchmark

• ViroBench is a comprehensive benchmark for viral nucleotide foundation models, assessing taxonomy classification, host prediction, and generative tasks.
• It employs a curated corpus of 58,314 high-quality viral samples with detailed annotations to simulate realistic temporal, phylogenetic, and host-driven shifts.
• The benchmark uniquely links next-token likelihood with biological validity to diagnose latent biosecurity risk in viral sequence generation.

ViroBench is a benchmark for nucleotide foundation models (NFMs) in viral genomics, introduced as the first comprehensive and large-scale benchmark specifically designed for NFMs in viral settings. It evaluates models along two axes—biological understanding and latent biosecurity risk—across 18 scenarios spanning classification and generation, and is built from a curated corpus of 58,314 high-quality viral samples with taxonomic, chronological, host, and nucleic-acid annotations (Ye et al., 25 May 2026).

1. Definition, scope, and naming

ViroBench was proposed to address a specific gap: prior NFM evaluation in virology had been fragmented, often based on inconsistent datasets and biologically unrealistic splits, especially random splits that allow closely related sequences to appear in both training and test sets. The benchmark is explicitly motivated by viral genomics properties that make such evaluation inadequate: extreme diversity, DNA/RNA modality differences, long-tailed label distributions, rapid evolution, phylogenetic distribution shift, and temporal drift. It also extends beyond discriminative performance to assess whether sequence generators exhibit behaviors relevant to latent biosecurity risk (Ye et al., 25 May 2026).

The benchmark is framed around two distinct but coupled questions. The first is whether a model encodes viral biological structure well enough to support taxonomy classification and host prediction under realistic extrapolation regimes. The second is whether generative models that appear statistically competent by language-model criteria also preserve biological constraints when asked to model viral genomes or coding sequences. This separation is central to ViroBench’s design, because the paper argues that generative quality in viral settings cannot be reduced to likelihood alone (Ye et al., 25 May 2026).

The name should be distinguished from similarly named viral benchmarks. In particular, VirBench is a different resource focused on deterministic viral sequence retrieval from NCBI Virus and explicitly uses the name VirBench, not ViroBench (Nasri et al., 4 Jun 2026). ViroBench, by contrast, is a viral nucleotide foundation model benchmark.

2. Corpus construction and task structure

ViroBench is constructed from 273,974 virus-associated TaxIDs collected from NCBI RefSeq and GenBank. For each entry, the benchmark integrates metadata along three axes: taxonomy, chronology, and host. After removing incomplete taxonomy or missing timestamps, retaining verified high-quality assemblies, and resolving species-level redundancies, the pipeline produces a final curated corpus of 58,314 labeled viral samples. Appendix details report an intermediate reduction from 204,603 TaxIDs after initial QC to 67,749 TaxIDs with valid assemblies and comprehensive annotation, before the final corpus is formed (Ye et al., 25 May 2026).

Host metadata required extensive normalization. The raw metadata contained more than 8,000 inconsistent host strings, which were mapped into coarse categories using Qwen3-235B. The final categories are: A bacterial host, B fungi/oomycetes/plant pathogens, C plant host, D1 humans and non-human primates, D2 livestock / companion animals, D3 wild vertebrates, E arthropod / invertebrate host, and F other or uncertain. On a validation subset with 100 instances per class, Qwen achieved 96.25% host-label accuracy; GLM-5 and Kimi-K2.5 achieved 94.25% and 95.63% (Ye et al., 25 May 2026).

The benchmark is organized into two dimensions and four task types.

Dimension	Task types	Scenario construction
Biological understanding	Taxonomy classification; Host prediction	ALL, DNA, RNA × genus-disjoint and temporal splits = 12 scenarios
Latent biosecurity risk	Genome modeling; CDS generation	Short, Medium, Long length buckets = 6 scenarios

For classification, ViroBench evaluates both taxonomy classification and host prediction on three scopes—ALL viruses, DNA viruses, and RNA viruses—under two split regimes: genus-disjoint and temporal. This yields 12 classification scenarios. Taxonomy prediction spans five hierarchical labels (from Kingdom to Family). For the ALL-virus classification setting, the hierarchy label counts are 29 / 55 / 214 / 390 / 319, the genus-disjoint split ratio is 8:1:1, the time span is 1982.06 → 2025.07, and the temporal cutoffs are 2017.10 / 2020.02. For DNA viruses, the corresponding counts are 9 / 14 / 20 / 51 / 160 with time span 1982.06 → 2024.09 and temporal cutoffs 2022.07 / 2023.08. For RNA viruses, the counts are 4 / 12 / 30 / 58 / 139 with time span 1982.06 → 2025.07 and temporal cutoffs 2017.03 / 2017.11 (Ye et al., 25 May 2026).

For generation, ViroBench evaluates genome modeling and CDS generation across Short, Medium, and Long buckets defined by the 33rd and 66th percentiles of sequence lengths. In genome modeling, the benchmark uses all contigs, with buckets 855–1,440 bp (43,040 examples), 1,441–2,192 bp (43,280), and 2,193–1,385,869 bp (56,772). In CDS generation, the benchmark uses host-balanced, diversity-aware subsampling with up to 500 non-redundant sequences per host, yielding 3,575 short CDSs (153–330 bp), 5,495 medium CDSs (333–765 bp), and 9,179 long CDSs (768–26,784 bp) (Ye et al., 25 May 2026).

3. Evaluation protocol and metrics

For classification, ViroBench adopts a frozen-backbone protocol. Pretrained NFMs are kept fixed, embeddings are extracted, and a lightweight standardized classifier is trained on top. Pooling follows model-specific conventions: mean pooling for many BERT-style models, final token for Evo, HyenaDNA, and NT v1/v2, and CLS-token pooling for DNABERT. Genomes are segmented into fixed-length non-overlapping windows, plus an extra tail window. The training configurations use window sizes 512, 1024, 2048 with sampled windows per sequence 8, 4, 2. The classification head is trained with learning rates $10^{-2}, 10^{-3}, 10^{-4}$, weight decay 0.01, maximum 300 epochs, balanced class weights, and early stopping patience 30. Results are reported as mean (standard deviation) across hyperparameter settings (Ye et al., 25 May 2026).

The classification metrics are AUPRC, Recall, Precision, and Macro-F1, reflecting the benchmark’s emphasis on imbalanced viral label distributions. The appendix defines

$$P_i = \frac{TP_i}{TP_i + FP_i}, \qquad R_i = \frac{TP_i}{TP_i + FN_i},$$

and

$$\text{Macro-F1} = \frac{1}{C} \sum_{i=1}^{C} \frac{2 \cdot P_i \cdot R_i}{P_i + R_i}.$$

AUPRC is reported as

$\text{AUPRC} = \sum_{n} (R_n - R_{n-1}) P_n.$

These choices make the benchmark diagnostic rather than accuracy-centric, since viral class imbalance and split difficulty can otherwise obscure failure modes (Ye et al., 25 May 2026).

For generation, the benchmark uses a 128-bp prompt in genome modeling and a 129-bp prompt in CDS generation. Genome modeling is scored by BPB (bits per base), where lower values are better. CDS generation is evaluated using both surface-level sequence agreement and biological plausibility: Edit Distance, Exact Match Accuracy, CDS Success Rate, k-mer JSD, and k-mer KS. The appendix gives

$$\mathrm{ED}(y, \hat{y}) = \frac{\mathrm{LD}(y, \hat{y})}{|y|},$$

and

$$\mathrm{EMA}(y,\hat{y}) = \frac{1}{|y|}\sum_{i=1}^{|y|}\mathbb{I}\!\left[\hat{y}_i = y_i\right].$$

For distributional comparison, the benchmark uses an adaptive $k$ for $k$-mer JSD,

$$k = \mathrm{clamp}(\mathrm{round}(0.7\log_{4} L_{\mathrm{eff}}), 1, 13),$$

followed by

$$\mathrm{kmer\text{-}JSD}(p,q) = \frac{1}{2}\sum_i p_i \log_2\frac{p_i}{m_i} + \frac{1}{2}\sum_i q_i \log_2\frac{q_i}{m_i},$$

with $$P_i = \frac{TP_i}{TP_i + FP_i}, \qquad R_i = \frac{TP_i}{TP_i + FN_i},$$0, and

$$P_i = \frac{TP_i}{TP_i + FP_i}, \qquad R_i = \frac{TP_i}{TP_i + FN_i},$$1

This metric stack is meant to expose divergence between statistical plausibility and biological validity (Ye et al., 25 May 2026).

4. Model coverage and empirical findings on biological understanding

ViroBench evaluates 66 NFMs and 4 conventional baselines—BLAST, Kraken2, BiLSTM, and CNN—across a broad architectural range: BERT encoders, Transformer decoders, GPT, Hyena, StripedHyena, StripedHyena2, Bi-Mamba, MoE Transformers, and diffusion/U-Net systems. The model set includes families such as AIDO.DNA, AIDO.RNA, BiRNA-BERT, Caduceus, DNABERT, DNABERT-2, DNABERT-S, Evo, Evo1.5, Evo2, GENA-LM, GENERator v2, Genos, GenomeOcean, Grover, HyenaDNA, LucaOne, LucaVirus, Nucleotide Transformer, NT v3, OmniReg-GPT, RNA-FM, RiNALMo, RNABERT, and the in-house ViroHyena, ViroDNABERT2, and ViroCaduceus lines. The benchmark groups pretraining corpora into four lineages: D (diverse viral coverage), P (phage-specific coverage), R (RNA-specific coverage), and N (non-viral coverage) (Ye et al., 25 May 2026).

The first major empirical conclusion is that NFMs show performance degradation in biological understanding under phylogenetic and temporal shifts. The benchmark demonstrates this directly with macro-F1 drops between genus-disjoint and temporal settings. On ALL-virus taxonomy, LucaVirus-default-step3.8M drops from 75.88 to 64.91, AIDO.DNA-7B from 69.87 to 55.27, GenomeOcean-4B from 71.53 to 52.28, and LucaOne-default-step36M from 69.79 to 57.45. On DNA taxonomy, LucaVirus-default-step3.8M drops from 82.20 to 69.17; on RNA taxonomy, the same model drops from 85.83 to 73.28 (Ye et al., 25 May 2026).

The degradation is even more severe for host prediction. The paper highlights Genos-10B in the ALL-host setting, where macro-F1 falls from 56.49 under genus-disjoint evaluation to 5.54 under temporal split. Comparable all-host declines appear in DNABERT-S (80.17 → 47.41), GenomeOcean-4B (81.67 → 48.75), AIDO.DNA-7B (80.65 → 47.47), and LucaVirus-default-step3.8M (84.56 → 54.84). This pattern supports the paper’s claim that host prediction is less robust than taxonomy classification and that viral NFMs remain weak at extrapolation under realistic shift (Ye et al., 25 May 2026).

The error structure is not random. Family-level confusion matrices and phylogenetic overlays show that mistakes cluster among phylogenetically neighboring clades. The paper uses AIDO.DNA-7B to illustrate confusion concentrated around related lineages such as Autoscriptoviridae and Autotranscriptaviridae, and the appendix generalizes this pattern across additional models. This indicates that current NFMs often learn coarse evolutionary organization while struggling with finer lineage boundaries (Ye et al., 25 May 2026).

The third major conclusion concerns pretraining data composition. ViroBench’s controlled ablations show that taxonomic diversity in pretraining data outweighs parameter scale. Using the in-house ViroBland corpus (216 Mb, drawn from human reference genome, multi-species genomes, and viral sequences), the authors trained lightweight ViroHyena variants and compared them with HyenaDNA-Large-1M. Mean macro-F1 across classification tasks increases from 23.48 in the baseline to 39.32 in ViroHyena-436K, an absolute gain of 15.84 and a relative gain of 67.5%. Larger in-domain variants continue to improve—ViroHyena-1M reaches 36.77, ViroHyena-6M reaches 44.16, and ViroHyena-253M reaches 44.67—but the paper emphasizes that gains saturate, suggesting that viral-domain diversity, not scale alone, is the dominant factor (Ye et al., 25 May 2026).

5. Generation tasks and the latent biosecurity axis

ViroBench’s generative side is designed to measure latent biosecurity risk, not to optimize virus design. It therefore treats genome modeling and CDS generation as diagnostic tasks. Genome modeling asks whether models can assign high probability to authentic viral continuations. CDS generation asks whether continuation remains biologically coherent—preserving open reading frame, avoiding premature internal stops, and maintaining coding-like sequence statistics. The benchmark’s core finding is that these notions can diverge sharply (Ye et al., 25 May 2026).

On genome modeling, the strongest BPB values come from Evo2-40B and Evo1.5. Evo2-40B reaches 1.9010 on short genomes, 1.8651 on medium genomes, and 1.8660 on long genomes; Evo1.5 reaches 1.9230, 1.9035, and 1.8772 on the same buckets. However, ViroBench shows that good BPB does not guarantee biologically valid generation. The paper notes that models such as GenomeOcean-4B and GENERator-v2-3B can achieve non-worst BPB while still exhibiting substantially elevated JSD, showing a decoupling between next-token likelihood and global biological plausibility (Ye et al., 25 May 2026).

The CDS results make this decoupling explicit. Even the strongest models have very low CDS Success Rate, especially for longer sequences. For Evo2-40B, CSR is 1.4270% on short CDSs, 0.6005% on medium CDSs, and 0.0545% on long CDSs. For Evo1.5, the corresponding values are 0.5315%, 0.2548%, and 0.0109%. HyenaDNA-Large-1M reaches 0.8392% on short CDSs but only 0.0182% on medium CDSs and 0.0000% on long CDSs. GenomeOcean-4B yields 0.3077%, 0.0728%, and 0.0218% (Ye et al., 25 May 2026).

The error analysis explains why. In the CDS-Short setting, only 0.98% of generated continuations satisfy CDS validity, only 5.01% end with a canonical terminal stop codon at the expected position, 76.84% contain premature internal stop codons, and 72.81% both lack the expected terminal stop and contain at least one internal premature stop. Attribution analysis at the terminal decoding step further shows that successful generations depend more strongly on prompt context, whereas failed generations lose long-range dependence on the CDS prefix. The paper interprets this as evidence that local next-token competence is insufficient for sustained coding validity (Ye et al., 25 May 2026).

ViroBench extends this argument with AlphaFold3-based structural validation on 1,143 paired targets. Only 22/1143 predicted generated proteins achieve TM-like ≥ 0.50, only 13/1143 exceed 0.70, and only 44/1143 have C$$P_i = \frac{TP_i}{TP_i + FP_i}, \qquad R_i = \frac{TP_i}{TP_i + FN_i},$$2-RMSD ≤ 5 Å. The median TM-like is 0.054, median C$$P_i = \frac{TP_i}{TP_i + FP_i}, \qquad R_i = \frac{TP_i}{TP_i + FN_i},$$3-RMSD is 23.74 Å, and generated proteins have lower confidence than true targets, with median pLDDT dropping from 75.14 to 62.88 and median pTM from 0.50 to 0.37. The benchmark therefore argues that statistical likelihood and biological functional validity can decouple, and that this decoupling is itself a measurable component of latent biosecurity risk (Ye et al., 25 May 2026).

6. Position in the viral benchmark landscape and stated limitations

Within the broader ecosystem of viral benchmarks, ViroBench occupies the viral nucleotide foundation model niche. It differs from ViroGym, which is a benchmark for viral proteins and zero-shot protein LLMs, curated around 79 DMS assays, 552,937 mutated amino acid sequences, 21 influenza neutralisation tasks, and a SARS-CoV-2 pandemic-prediction task (Zhou et al., 6 Mar 2026). It also differs from VirBench, which evaluates deterministic viral sequence retrieval and agentic use of NCBI Virus through 120 manually curated queries and a reproducible retrieval layer (Nasri et al., 4 Jun 2026). This suggests a broader benchmark stratification in virology: ViroBench for viral nucleotide models, ViroGym for viral protein models, and VirBench for viral data-access reliability.

The paper also emphasizes diagnostic interpretability. ViroBench provides family-level confusion structure, host-stratified generation analyses, split-wise performance gaps, and architecture/pretraining-lineage comparisons. A plausible implication is that it is intended less as a single leaderboard than as a measurement framework for model failure under biologically realistic stressors. This is reinforced by the public release of datasets and code at https://github.com/QIANJINYDX/ViroBench (Ye et al., 25 May 2026).

The authors identify several limitations. Host prediction is simplified to single-label classification over coarse host categories, even though many viruses are multi-host. Temporal labels are based on NCBI deposit or recorded dates and therefore reflect surveillance and sequencing effort as much as true emergence time. Genus-disjoint evaluation is biologically stronger than random split, but not perfect, because recombination and reassortment can preserve homologous sequence blocks across genera. The in-house ViroHyena models currently operate with a maximum context of about 8k tokens. Finally, the benchmark’s generative tasks are explicitly not presented as a virus-design system; rather, the benchmark is meant to make risk-relevant behavior measurable and visible without providing guidance for constructing or validating viruses (Ye et al., 25 May 2026).

Taken together, ViroBench establishes a viral-genomics evaluation framework in which robustness under phylogenetic and temporal shift, biological validity of generated sequence, and biosecurity-aware measurement are treated as first-class criteria. Its central result is not merely that some models outperform others, but that viral NFM evaluation requires biologically realistic splits, function-aware generative metrics, and explicit attention to how domain-specific pretraining data shape both capability and failure.