---
title: 'ViroBench: Viral NFM Benchmark'
url: https://www.emergentmind.com/topics/virobench
type: topic
---

# ViroBench: Viral NFM Benchmark

ViroBench is a benchmark for **nucleotide foundation models (NFMs)** in viral genomics, introduced as **the first comprehensive and large-scale benchmark specifically designed for NFMs in viral settings**. It evaluates models along two axes—**biological understanding** and **latent biosecurity risk**—across **18 scenarios** spanning classification and generation, and is built from a curated corpus of **58,314 high-quality viral samples** with taxonomic, chronological, host, and nucleic-acid annotations [2605.25388].

## 1. Definition, scope, and naming

ViroBench was proposed to address a specific gap: prior NFM evaluation in virology had been fragmented, often based on inconsistent datasets and biologically unrealistic splits, especially **random splits** that allow closely related sequences to appear in both training and test sets. The benchmark is explicitly motivated by viral genomics properties that make such evaluation inadequate: **extreme diversity**, **DNA/RNA modality differences**, **long-tailed label distributions**, **rapid evolution**, **phylogenetic distribution shift**, and **temporal drift**. It also extends beyond discriminative performance to assess whether sequence generators exhibit behaviors relevant to **latent biosecurity risk** [2605.25388].

The benchmark is framed around two distinct but coupled questions. The first is whether a model encodes viral biological structure well enough to support **taxonomy classification** and **host prediction** under realistic extrapolation regimes. The second is whether generative models that appear statistically competent by language-model criteria also preserve biological constraints when asked to model viral genomes or coding sequences. This separation is central to ViroBench’s design, because the paper argues that generative quality in viral settings cannot be reduced to likelihood alone [2605.25388].

The name should be distinguished from similarly named viral benchmarks. In particular, **VirBench** is a different resource focused on deterministic viral sequence retrieval from NCBI Virus and explicitly uses the name **VirBench**, not ViroBench [2606.06749]. ViroBench, by contrast, is a viral **nucleotide foundation model** benchmark.

## 2. Corpus construction and task structure

ViroBench is constructed from **273,974 virus-associated TaxIDs** collected from NCBI RefSeq and GenBank. For each entry, the benchmark integrates metadata along three axes: **taxonomy**, **chronology**, and **host**. After removing incomplete taxonomy or missing timestamps, retaining verified high-quality assemblies, and resolving species-level redundancies, the pipeline produces a final curated corpus of **58,314** labeled viral samples. Appendix details report an intermediate reduction from **204,603 TaxIDs** after initial QC to **67,749 TaxIDs** with valid assemblies and comprehensive annotation, before the final corpus is formed [2605.25388].

Host metadata required extensive normalization. The raw metadata contained more than **8,000** inconsistent host strings, which were mapped into coarse categories using **Qwen3-235B**. The final categories are: **A** bacterial host, **B** fungi/oomycetes/plant pathogens, **C** plant host, **D1** humans and non-human primates, **D2** livestock / companion animals, **D3** wild vertebrates, **E** arthropod / invertebrate host, and **F** other or uncertain. On a validation subset with **100 instances per class**, Qwen achieved **96.25%** host-label accuracy; **GLM-5** and **Kimi-K2.5** achieved **94.25%** and **95.63%** [2605.25388].

The benchmark is organized into two dimensions and four task types.

| Dimension | Task types | Scenario construction |
|---|---|---|
| Biological understanding | Taxonomy classification; Host prediction | ALL, DNA, RNA × genus-disjoint and temporal splits = 12 scenarios |
| Latent biosecurity risk | Genome modeling; CDS generation | Short, Medium, Long length buckets = 6 scenarios |

For classification, ViroBench evaluates both **taxonomy classification** and **host prediction** on three scopes—**ALL viruses**, **DNA viruses**, and **RNA viruses**—under two split regimes: **genus-disjoint** and **temporal**. This yields **12 classification scenarios**. Taxonomy prediction spans **five hierarchical labels (from Kingdom to Family)**. For the ALL-virus classification setting, the hierarchy label counts are **29 / 55 / 214 / 390 / 319**, the genus-disjoint split ratio is **8:1:1**, the time span is **1982.06 → 2025.07**, and the temporal cutoffs are **2017.10 / 2020.02**. For DNA viruses, the corresponding counts are **9 / 14 / 20 / 51 / 160** with time span **1982.06 → 2024.09** and temporal cutoffs **2022.07 / 2023.08**. For RNA viruses, the counts are **4 / 12 / 30 / 58 / 139** with time span **1982.06 → 2025.07** and temporal cutoffs **2017.03 / 2017.11** [2605.25388].

For generation, ViroBench evaluates **genome modeling** and **CDS generation** across **Short**, **Medium**, and **Long** buckets defined by the 33rd and 66th percentiles of sequence lengths. In genome modeling, the benchmark uses all contigs, with buckets **855–1,440 bp** (**43,040** examples), **1,441–2,192 bp** (**43,280**), and **2,193–1,385,869 bp** (**56,772**). In CDS generation, the benchmark uses host-balanced, diversity-aware subsampling with up to **500 non-redundant sequences per host**, yielding **3,575** short CDSs (**153–330 bp**), **5,495** medium CDSs (**333–765 bp**), and **9,179** long CDSs (**768–26,784 bp**) [2605.25388].

## 3. Evaluation protocol and metrics

For classification, ViroBench adopts a **frozen-backbone** protocol. Pretrained NFMs are kept fixed, embeddings are extracted, and a lightweight standardized classifier is trained on top. Pooling follows model-specific conventions: mean pooling for many BERT-style models, final token for **Evo**, **HyenaDNA**, and **NT v1/v2**, and CLS-token pooling for **DNABERT**. Genomes are segmented into **fixed-length non-overlapping windows**, plus an extra tail window. The training configurations use window sizes **512, 1024, 2048** with sampled windows per sequence **8, 4, 2**. The classification head is trained with learning rates **\(10^{-2}, 10^{-3}, 10^{-4}\)**, weight decay **0.01**, maximum **300** epochs, balanced class weights, and early stopping patience **30**. Results are reported as **mean (standard deviation)** across hyperparameter settings [2605.25388].

The classification metrics are **AUPRC**, **Recall**, **Precision**, and **Macro-F1**, reflecting the benchmark’s emphasis on imbalanced viral label distributions. The appendix defines
\[
P_i = \frac{TP_i}{TP_i + FP_i},
\qquad
R_i = \frac{TP_i}{TP_i + FN_i},
\]
and
\[
\text{Macro-F1} = \frac{1}{C} \sum_{i=1}^{C} \frac{2 \cdot P_i \cdot R_i}{P_i + R_i}.
\]
AUPRC is reported as
\[
\text{AUPRC} = \sum_{n} (R_n - R_{n-1}) P_n.
\]
These choices make the benchmark diagnostic rather than accuracy-centric, since viral class imbalance and split difficulty can otherwise obscure failure modes [2605.25388].

For generation, the benchmark uses a **128-bp prompt** in genome modeling and a **129-bp prompt** in CDS generation. Genome modeling is scored by **BPB** (bits per base), where lower values are better. CDS generation is evaluated using both surface-level sequence agreement and biological plausibility: **Edit Distance**, **Exact Match Accuracy**, **CDS Success Rate**, **k-mer JSD**, and **k-mer KS**. The appendix gives
\[
\mathrm{ED}(y, \hat{y}) = \frac{\mathrm{LD}(y, \hat{y})}{|y|},
\]
and
\[
\mathrm{EMA}(y,\hat{y}) = \frac{1}{|y|}\sum_{i=1}^{|y|}\mathbb{I}\!\left[\hat{y}_i = y_i\right].
\]
For distributional comparison, the benchmark uses an adaptive \(k\) for \(k\)-mer JSD,
\[
k = \mathrm{clamp}(\mathrm{round}(0.7\log_{4} L_{\mathrm{eff}}), 1, 13),
\]
followed by
\[
\mathrm{kmer\text{-}JSD}(p,q) = \frac{1}{2}\sum_i p_i \log_2\frac{p_i}{m_i} + \frac{1}{2}\sum_i q_i \log_2\frac{q_i}{m_i},
\]
with \(m=\tfrac{1}{2}(p+q)\), and
\[
\mathrm{kmer\text{-}KS}(p,q) = \sup_t |P(t)-Q(t)|.
\]
This metric stack is meant to expose divergence between statistical plausibility and biological validity [2605.25388].

## 4. Model coverage and empirical findings on biological understanding

ViroBench evaluates **66 NFMs** and **4 conventional baselines**—**BLAST**, **Kraken2**, **BiLSTM**, and **CNN**—across a broad architectural range: **BERT encoders**, **Transformer decoders**, **GPT**, **Hyena**, **StripedHyena**, **StripedHyena2**, **Bi-Mamba**, **MoE Transformers**, and **diffusion/U-Net** systems. The model set includes families such as **AIDO.DNA**, **AIDO.RNA**, **BiRNA-BERT**, **Caduceus**, **DNABERT**, **DNABERT-2**, **DNABERT-S**, **Evo**, **Evo1.5**, **Evo2**, **GENA-LM**, **GENERator v2**, **Genos**, **GenomeOcean**, **Grover**, **HyenaDNA**, **LucaOne**, **LucaVirus**, **Nucleotide Transformer**, **NT v3**, **OmniReg-GPT**, **RNA-FM**, **RiNALMo**, **RNABERT**, and the in-house **ViroHyena**, **ViroDNABERT2**, and **ViroCaduceus** lines. The benchmark groups pretraining corpora into four lineages: **D** (diverse viral coverage), **P** (phage-specific coverage), **R** (RNA-specific coverage), and **N** (non-viral coverage) [2605.25388].

The first major empirical conclusion is that NFMs show **performance degradation in biological understanding under phylogenetic and temporal shifts**. The benchmark demonstrates this directly with macro-F1 drops between genus-disjoint and temporal settings. On **ALL-virus taxonomy**, **LucaVirus-default-step3.8M** drops from **75.88** to **64.91**, **AIDO.DNA-7B** from **69.87** to **55.27**, **GenomeOcean-4B** from **71.53** to **52.28**, and **LucaOne-default-step36M** from **69.79** to **57.45**. On **DNA taxonomy**, **LucaVirus-default-step3.8M** drops from **82.20** to **69.17**; on **RNA taxonomy**, the same model drops from **85.83** to **73.28** [2605.25388].

The degradation is even more severe for **host prediction**. The paper highlights **Genos-10B** in the ALL-host setting, where macro-F1 falls from **56.49** under genus-disjoint evaluation to **5.54** under temporal split. Comparable all-host declines appear in **DNABERT-S** (**80.17 → 47.41**), **GenomeOcean-4B** (**81.67 → 48.75**), **AIDO.DNA-7B** (**80.65 → 47.47**), and **LucaVirus-default-step3.8M** (**84.56 → 54.84**). This pattern supports the paper’s claim that host prediction is less robust than taxonomy classification and that viral NFMs remain weak at extrapolation under realistic shift [2605.25388].

The error structure is not random. Family-level confusion matrices and phylogenetic overlays show that mistakes cluster among **phylogenetically neighboring clades**. The paper uses **AIDO.DNA-7B** to illustrate confusion concentrated around related lineages such as **Autoscriptoviridae** and **Autotranscriptaviridae**, and the appendix generalizes this pattern across additional models. This indicates that current NFMs often learn coarse evolutionary organization while struggling with finer lineage boundaries [2605.25388].

The third major conclusion concerns pretraining data composition. ViroBench’s controlled ablations show that **taxonomic diversity in pretraining data outweighs parameter scale**. Using the in-house **ViroBland** corpus (**216 Mb**, drawn from human reference genome, multi-species genomes, and viral sequences), the authors trained lightweight **ViroHyena** variants and compared them with **HyenaDNA-Large-1M**. Mean macro-F1 across classification tasks increases from **23.48** in the baseline to **39.32** in **ViroHyena-436K**, an absolute gain of **15.84** and a relative gain of **67.5%**. Larger in-domain variants continue to improve—**ViroHyena-1M** reaches **36.77**, **ViroHyena-6M** reaches **44.16**, and **ViroHyena-253M** reaches **44.67**—but the paper emphasizes that gains saturate, suggesting that viral-domain diversity, not scale alone, is the dominant factor [2605.25388].

## 5. Generation tasks and the latent biosecurity axis

ViroBench’s generative side is designed to measure **latent biosecurity risk**, not to optimize virus design. It therefore treats **genome modeling** and **CDS generation** as diagnostic tasks. Genome modeling asks whether models can assign high probability to authentic viral continuations. CDS generation asks whether continuation remains biologically coherent—preserving open reading frame, avoiding premature internal stops, and maintaining coding-like sequence statistics. The benchmark’s core finding is that these notions can diverge sharply [2605.25388].

On genome modeling, the strongest BPB values come from **Evo2-40B** and **Evo1.5**. **Evo2-40B** reaches **1.9010** on short genomes, **1.8651** on medium genomes, and **1.8660** on long genomes; **Evo1.5** reaches **1.9230**, **1.9035**, and **1.8772** on the same buckets. However, ViroBench shows that good BPB does not guarantee biologically valid generation. The paper notes that models such as **GenomeOcean-4B** and **GENERator-v2-3B** can achieve non-worst BPB while still exhibiting substantially elevated **JSD**, showing a decoupling between next-token likelihood and global biological plausibility [2605.25388].

The CDS results make this decoupling explicit. Even the strongest models have very low **CDS Success Rate**, especially for longer sequences. For **Evo2-40B**, CSR is **1.4270%** on short CDSs, **0.6005%** on medium CDSs, and **0.0545%** on long CDSs. For **Evo1.5**, the corresponding values are **0.5315%**, **0.2548%**, and **0.0109%**. **HyenaDNA-Large-1M** reaches **0.8392%** on short CDSs but only **0.0182%** on medium CDSs and **0.0000%** on long CDSs. **GenomeOcean-4B** yields **0.3077%**, **0.0728%**, and **0.0218%** [2605.25388].

The error analysis explains why. In the **CDS-Short** setting, only **0.98%** of generated continuations satisfy CDS validity, only **5.01%** end with a canonical terminal stop codon at the expected position, **76.84%** contain premature internal stop codons, and **72.81%** both lack the expected terminal stop and contain at least one internal premature stop. Attribution analysis at the terminal decoding step further shows that successful generations depend more strongly on prompt context, whereas failed generations lose long-range dependence on the CDS prefix. The paper interprets this as evidence that local next-token competence is insufficient for sustained coding validity [2605.25388].

ViroBench extends this argument with **AlphaFold3**-based structural validation on **1,143 paired targets**. Only **22/1143** predicted generated proteins achieve **TM-like ≥ 0.50**, only **13/1143** exceed **0.70**, and only **44/1143** have **C\(\alpha\)-RMSD ≤ 5 Å**. The median **TM-like** is **0.054**, median **C\(\alpha\)-RMSD** is **23.74 Å**, and generated proteins have lower confidence than true targets, with median **pLDDT** dropping from **75.14** to **62.88** and median **pTM** from **0.50** to **0.37**. The benchmark therefore argues that **statistical likelihood and biological functional validity can decouple**, and that this decoupling is itself a measurable component of latent biosecurity risk [2605.25388].

## 6. Position in the viral benchmark landscape and stated limitations

Within the broader ecosystem of viral benchmarks, ViroBench occupies the **viral nucleotide foundation model** niche. It differs from **ViroGym**, which is a benchmark for **viral proteins** and zero-shot protein language models, curated around **79 DMS assays**, **552,937 mutated amino acid sequences**, **21 influenza neutralisation tasks**, and a SARS-CoV-2 pandemic-prediction task [2603.06740]. It also differs from **VirBench**, which evaluates deterministic viral sequence retrieval and agentic use of NCBI Virus through **120 manually curated queries** and a reproducible retrieval layer [2606.06749]. This suggests a broader benchmark stratification in virology: ViroBench for viral nucleotide models, ViroGym for viral protein models, and VirBench for viral data-access reliability.

The paper also emphasizes diagnostic interpretability. ViroBench provides family-level confusion structure, host-stratified generation analyses, split-wise performance gaps, and architecture/pretraining-lineage comparisons. A plausible implication is that it is intended less as a single leaderboard than as a measurement framework for model failure under biologically realistic stressors. This is reinforced by the public release of datasets and code at **https://github.com/QIANJINYDX/ViroBench** [2605.25388].

The authors identify several limitations. **Host prediction** is simplified to **single-label** classification over coarse host categories, even though many viruses are multi-host. **Temporal labels** are based on NCBI deposit or recorded dates and therefore reflect surveillance and sequencing effort as much as true emergence time. **Genus-disjoint** evaluation is biologically stronger than random split, but not perfect, because recombination and reassortment can preserve homologous sequence blocks across genera. The in-house **ViroHyena** models currently operate with a maximum context of about **8k tokens**. Finally, the benchmark’s generative tasks are explicitly not presented as a virus-design system; rather, the benchmark is meant to make risk-relevant behavior **measurable and visible** without providing guidance for constructing or validating viruses [2605.25388].

Taken together, ViroBench establishes a viral-genomics evaluation framework in which **robustness under phylogenetic and temporal shift**, **biological validity of generated sequence**, and **biosecurity-aware measurement** are treated as first-class criteria. Its central result is not merely that some models outperform others, but that viral NFM evaluation requires biologically realistic splits, function-aware generative metrics, and explicit attention to how domain-specific pretraining data shape both capability and failure.

Source: https://www.emergentmind.com/topics/virobench