Collaborative estimation and evaluation of SARS-CoV-2 variant nowcasting in the United States

Abstract: The ability to estimate and predict pathogen variant dynamics can inform public health responses, including planning for increased transmission or severity, shifts in population immunity, or changes to vaccine or therapeutic effectiveness. The COVID-19 pandemic demonstrated the importance of monitoring SARS-CoV-2 variant evolution through viral genome sequencing, enabling predictive models to estimate variant frequencies in the recent past, present, and short-term future. Collaborative forecasting Hubs provided a valuable way to centralize predictive modeling of epidemiological indicators such as cases, hospitalizations, and deaths during the pandemic; however, none existed for variant dynamics. Here, we discuss the creation of the United States SARS-CoV-2 Variant Nowcast Hub, designed to solicit estimates of the relative abundance of a specified set of SARS-CoV-2 variants at the U.S. state level. We discuss the design decisions and challenges in building the Hub and its scoring procedures. Using submissions from the Hub's first respiratory virus season (nowcast dates October 9th, 2024 to June 4th, 2025), we evaluate five individual models and a baseline model. We found that the baseline model, which pools sequences across the U.S., performs well overall, with most individual models performing similarly or slightly worse. Locations with lower sequencing volumes exhibited greater variability in model performance. Models submitted for a single location outperformed those submitted for all locations, potentially due to greater timeliness and magnitude of local data. Much remains to be investigated regarding relative model performance across different phases of variant emergence, and we conclude by proposing future directions within and beyond this Hub.

• The paper introduces a centralized infrastructure to nowcast SARS-CoV-2 variant frequencies at the state level using genomic sequence data.
• The methodology employs Bayesian multinomial logistic regression and transformer models evaluated with proper scoring rules under data delays.
• The evaluations reveal that fully pooled baseline models can outperform specialized approaches in low sequence volume scenarios, informing public health responses.

Collaborative Estimation and Evaluation of SARS-CoV-2 Variant Nowcasting in the United States

Introduction and Motivation

This work details the design, implementation, and evaluation of the United States SARS-CoV-2 Variant Nowcast Hub, a collaborative infrastructure for the real-time estimation and short-range forecasting ("nowcasting") of SARS-CoV-2 variant/clade frequencies at state-level spatial granularity. Building on lessons from collaborative Hubs for COVID-19 epidemiological indicators, this project directly addresses the gap in centralized, transparent, and standardized estimation of variant proportions using genomic sequence data. Critical challenges arise in this domain due to rapidly evolving prediction targets (variants defined by phylogenetic nomenclature), non-trivial delays in genomic sequence reporting, sparsity and heterogeneity of sequence counts, and a complex mapping between latent epidemiological quantities (true population variant frequencies) and observed sampled sequence counts. The Hub was designed to efficiently catalog and evaluate nowcast submissions from multiple modeling teams to support public health needs regarding variant-driven surge anticipation, intervention targeting, and vaccine strain selection.

Hub Infrastructure and Data Management

A distinguishing aspect of the Variant Nowcast Hub is the dynamic definition of predictive targets. Each week, using curated Nextstrain clade designations, a set of up to nine high-frequency circulating clades plus an additional "other" category is selected based on recent frequency and abundance cutoffs. Clade-target definitions, the mapping to Pango lineages, and curated sequence metadata are versioned and distributed to participating teams to ensure reproducibility of both nowcasts and post hoc evaluations. The Hub systematically tracks both the sequence data available as of each nowcast date (which can differ substantially from the eventual sequence set due to lags and truncation) and the evolving clade assignment models used to aggregate and evaluate model outputs.

Model Submission Protocols

Modeling teams are required to submit forecasts of the latent clade proportions $\theta_{l,t}$ for specified U.S. jurisdictions and time points: daily estimates spanning 31 days in the past (hindcast), the present, and 10 days ahead (forecasting). Submissions may comprise both point estimates and full probabilistic trajectories (samples from the model's predictive multivariate distribution over clade proportions). Notably, the targets to be forecasted are not directly observed: models are evaluated indirectly through multinomial sampling of the predicted clade proportions using the realized (delayed) sequence counts for each jurisdiction and date.

The Hub does not request direct forecasts of future sequence counts or predictions for as-yet-undefined clades, constraining the scope to the retrospective and prospective prevalence of clades recognized at nowcast time.

Evaluation Methodology

Given indirect observation of true clade frequencies, the evaluation protocol leverages proper scoring rules for both point and probabilistic forecasts under multinomial sampling. The energy score is employed for multivariate probabilistic evaluation, reflecting the calibration and sharpness of full predictive distributions under the multinomial observation model, and the categorical Brier score is used for point forecast evaluation. The evaluation takes care to avoid "data leakage": only periods with no sequence observation at nowcast time are considered for fair assessment. Analyses are stratified by jurisdiction, nowcast date, and prediction horizon to provide detailed insights into calibration and forecast sharpness under variable sequencing intensity and epidemiological dynamics. Relative model skill is summarized using a scaled relative skill score with respect to a Hub baseline.

Model Architectures and Comparative Assessment

During the initial assessment period, five independently developed models (UMass-HMLR, UGA-multicast, LANL-CovTransformer, CADPH-CATaMaran, CADPH-CATaLog) and the Hub baseline were evaluated. Models range from hierarchical Bayesian multinomial logistic regressions to transformer-based deep learning ensembles, differing in data sources (GenBank, GISAID, California COVIDNet) and the jurisdictional scope of training and application.

A key finding is that the Hub baseline—a Bayesian multinomial logistic regression pooling sequence data across all U.S. jurisdictions—performed equivalently or better than most more complex or locally specialized models, particularly in jurisdictions with low sequence volume. For California, two locally specialized models leveraging timelier and denser state-level sequence data marginally outperformed the baseline on Brier score. Patterns of performance variation correlated with both the volume and timeliness of sequence data and the complexity of local variant emergence dynamics.

During periods of variant turnover, such as the emergence of clade 25A, all models, including probabilistic nowcasts, exhibited degraded predictive calibration early in the period but recovered as the number of sequences assigned to the emergent clade increased and its prevalence stabilized.

Methodological Advances and Limitations

This work introduces two significant methodological advances: (1) the principled retrospective assignment of sequences to clades at the nowcast time using time-stamped clade assignment models, enabling fair downstream evaluation even as clade definitions change rapidly, and (2) the use of a Monte Carlo multinomial observation framework to map model-predicted latent proportions to observed sequence counts for use with proper scoring rules.

Limitations include the restricted geographic scope to U.S. states, partial genomic surveillance coverage (only GenBank-sequenced data used for both modeling and evaluation), and the current limited number of probabilistic modeling teams, which may restrict the generalizability of comparative inferences. The scaled relative skill score, used for aggregate comparisons, is not a proper scoring rule and thus should not be interpreted as strictly optimal for incentivizing calibrated predictions.

Implications and Future Directions

The infrastructure and evaluation framework established by the U.S. SARS-CoV-2 Variant Nowcast Hub provides a scalable template for collaborative, reproducible, and real-time evaluation of pathogen variant nowcasts. These advances have implications for deployment in future outbreak scenarios and for extension to other rapidly evolving pathogens (e.g., influenza, norovirus), enabling improved monitoring of variant-driven surges, better planning for public health interventions, and data-driven support for vaccine strain selection and viral surveillance strategies.

As sequencing technology evolves and global data-sharing practices continue to improve, future developments may include expanded cross-national collaborative Hubs, integration of additional data sources (e.g., wastewater surveillance, serological data), and further methodological advances in joint modeling of sequence abundances and lineages, particularly under data scarcity or biased sampling.

Conclusion

The U.S. SARS-CoV-2 Variant Nowcast Hub establishes reproducible infrastructure and robust evaluation methodologies for collaborative estimation of latent viral variant frequencies using sparse and delayed genomic data across jurisdictions. Comparative evaluation indicates that probabilistic and locally specialized models do not consistently outperform fully pooled baselines under conditions of extreme sequence sparsity, underscoring the importance of pooling information and the limitations imposed by available data volumes. These standardized systems enhance the transparency, comparability, and real-time utility of variant nowcasting, with broader utility for infectious disease surveillance architectures in future outbreaks.