EpiBench: Cross-Domain Benchmarking

• EpiBench is a comprehensive framework that standardizes benchmarking across various fields such as epidemic forecasting, multimodal research workflows, and bioinformatics.
• It employs rigorous methodologies like modular evaluation pipelines, retrieval-augmented reasoning, and deterministic grading to provide actionable insights into AI/ML performance.
• The platform facilitates transparent comparisons and community-driven extensions, driving progress through granular error analysis and reproducible experimental protocols.

EpiBench refers to a set of independent, field-specific benchmarking platforms and datasets designed to support rigorous, reproducible evaluation in domains ranging from epidemic forecasting and scientific workflow automation to evidence-grounded clinical reasoning in neurology, epigenomics analysis, antibody-specific epitope prediction, and FPGA preemption. Each EpiBench instantiation targets unique methodological challenges, experimental protocols, task formulations, and evaluation objectives, offering standardized benchmarks or workflows critical for the progression of ML, AI, and scientific research in its domain. This entry presents a systematic overview and comparison of EpiBench variants across these domains, with technical emphasis on design, evaluation methodology, and significant results.

1. EpiBench for Automated Epidemic Forecasting

The original EpiBench platform was conceived to address the lack of systematic, reproducible benchmarking for AI/ML-based epidemic forecasting by enforcing a modular, transparent evaluation protocol and infrastructure (Srivastava et al., 2021). The platform ingests ground-truth time series (primarily JHU CSSE COVID-19 for U.S. states and counties), processes submissions of fully automated forecasting methods, and runs all models on a common temporal schedule. Data preprocessing is fully documented (e.g., smoothing, anomaly correction), and each model’s specific preprocessing choices are tracked as part of its methodology.

Every week, the platform triggers all active methods to forecast 1–4 week-ahead case and death counts, evaluating submissions under a uniform suite of point, probabilistic, and interval metrics:

• Mean Absolute Error:

$$\mathrm{MAE} = \frac{1}{n}\sum_{i=1}^n |y_i - \hat{y}_i|$$

• Symmetric Mean Absolute Percentage Error (SMAPE)
• Log-Score and CRPS for probabilistic forecasts
• Interval coverage and penalized interval score

An automated meta-ensemble powered by a Random Forest regressor combines forecasts from multiple SIkJα-based compartmental models. This ensemble consistently achieved 10–15% lower MAE over 4-week-ahead death forecasting than both the leading human-expert submissions (e.g., COVID-19 Forecast Hub, YYG Paramsearch) and the CDC's official ensemble, particularly outperforming in nonstationary or rapidly changing regimes (e.g., Florida, Texas during surges).

The platform’s open, version-controlled submission and continuous evaluation infrastructure facilitate community-driven extensibility, making it straightforward for teams to contribute new tasks or datasets (e.g., dengue, influenza), and to compare intervention-free AI/ML methods against human-expert baselines under strictly uniform conditions.

2. EpiBench for Multimodal Multi-Turn Research Workflows

A distinct variant of EpiBench functions as a fine-grained, episodic benchmark to assess the process-level capabilities of research agents in navigating, extracting, and integrating scientific evidence across multimodal sources and literature (Dong et al., 7 Apr 2026). This benchmark departs from single-turn QA by challenging agents to:

• Proactively search for relevant papers using citation cues and hints,
• Extract and interpret figures, tables, and textual artifacts,
• Accumulate and correctly reuse evidence over multi-turn workflows,
• Integrate findings from multiple papers to answer cross-document, constraint-style questions, often enforcing a "memory-only" final turn to isolate sustained evidence use.

Benchmark design consists of 102 multi-turn episodes drawn from a set of 485 papers, categorized into easy (≈4 turns) and hard (≈6 turns) splits, with up to 6 figures and 5 papers per episode. Tasks include resourceful literature search, figure extraction, memory maintenance, and cross-paper reasoning.

Evaluation focuses on:

• Episode Success Rate (ESR): fraction of episodes with all turns correct,
• Turn-level accuracy (pre-final vs. final/fusion),
• Evidence Correctness (EC): grounding of answers in required artifacts,
• Minimality Gap (MG): efficiency of tool use relative to minimum path,
• Attribution of errors (retrieval, perception, judgment, memory).

The strongest current agents (e.g., GPT-5.2) achieve ESR ≈ 58.3% (easy) and 29.2% (hard), with a notable drop in performance on final fusion turns, indicating that cross-modal, cross-document memory integration remains a significant bottleneck.

3. EpiBench for Evidence-Grounded Epilepsy Clinical Reasoning

In the neurology/clinical AI domain, EpiBench is established atop the EpiGraph knowledge graph, providing a multi-task suite to systematically benchmark LLM-based agents on evidence-intensive epilepsy reasoning (Dai et al., 10 May 2026). The platform integrates 24,324 entities and 32,009 evidence-triplets from 48,166 peer-reviewed papers and clinical ontologies. Five core tasks are included:

• Clinical Decision Accuracy (CDA): MCQ and open-ended expert questions,
• Clinical Report Generation (CRG): EEG-based impression generation,
• Biomarker-Driven Precision Medicine (BPM): pharmacogenomic ASM selection,
• Treatment Recommendation (TR): patient-specific therapy choice,
• Deep Research Planning (DRP): structured research proposal generation.

A critical methodological innovation is the Graph-RAG retrieval-augmented protocol: relevant EpiGraph subgraphs are dynamically retrieved and prepended to LLM prompts, enabling multi-hop, evidence-grounded reasoning. Across all tasks, Graph-RAG augmentation leads to 11–41 percentage point improvements in Top-1 accuracy, especially in precision medicine where gains reach +41%, indicating that domain-specific knowledge graphs encode actionable facts not recoverable from parametric model weights alone.

Experimental pipelines use both open and closed-source LLMs, ablate retriever settings, and analyze gains in guideline concordance, safety, and structural reasoning at a granular level.

4. EpiBench for Verifiable Epigenomics Workflow Decisions

In bioinformatics, EpiBench functions as a deterministic, verifiable benchmark assessing AI agents’ ability to make scientifically rigorous, assay-specific analysis decisions at key workflow inflection points in epigenomic assays (CUT&Tag/CUT&RUN, ATAC-seq, ChIP-seq, methylation-seq) (Muralidharan et al., 11 Jun 2026).

The formalism specifies each evaluation in terms of a workflow state $S = (F, M, C)$ (files, metadata, context), and agent actions $a \in A$ (inspection commands + structured answer). A deterministic grader $G$ compares produced output against ground truth. Benchmarks span 106 evaluations and ~5K trajectories, with each answer scored pass/fail based on exact or interval-based field-level checks.

The highest endpoint pass rates are 45.0% (GPT-5.5 / Pi harness), with significant performance drop-offs on particularly assay-specific or judgment-intensive tasks (e.g., normalization decisions in CUT&RUN or unit-layer confusion in ATAC-seq). Manual analysis indicates that while agents often complete correct file and tool usage, they frequently misapply domain conventions—failing to override defaults with assay-specific reasoning.

The benchmark provides full-compatible harnesses and grader protocols, enabling extension to new assays, integration of new models, and reproducible, granular analysis of failure modes (execution vs. judgment).

5. EpiBench for Antibody-Specific Epitope Prediction

In the structural biology/protein modeling context, EpiBench refers to a benchmarking pipeline for antibody-specific epitope prediction, based on the AsEP dataset—the largest curated set of antibody–antigen complexes (N=1,723) (Liu et al., 2024). The resource includes rigorous filtering, epitope clustering, and per-complex graph representations using protein LLM embeddings (AntiBERTy/ESM2) and explicit structures.

Evaluation encompasses node classification (epitope vs. non-epitope) and bipartite link-prediction (interface contacts), using metrics such as F1, MCC, AUC-ROC, and AUPR, with stratified train-validation-test splits by epitope ratio and epitope group. Baseline comparisons indicate that traditional protein-binding site predictors (EpiPred, ESMFold, MaSIF-site) perform poorly (MCC < 0.04).

The WALLE architecture, combining GCNs and LM embeddings with bipartite decoding, delivers an MCC of 0.210—a 5–10× improvement—demonstrating substantial gains from integrating geometric and sequential features and enabling per-residue attribution for model interpretability.

6. Extensible FPGA Preemption Benchmark Suite

Lastly, in FPGA system research, EpiBench provides an open-source, preemption-enabled benchmark suite for evaluating scheduling and context-switch performance in multi-tenant hardware environments (Malik et al., 10 Nov 2025). The suite consists of 27 workloads (AI/ML, cryptography, image/video processing, signal processing, general computing) deployed on slot-partitioned, partially reconfigurable Zynq-7000 platforms.

Technical details include:

• Modular slot/partition management with context save/restore (latencies per frame ≈ 0.0311/0.0337 ms),
• Secure state isolation (AES-128/SHA-256),
• Workload diversity spanning from deeply pipelined (e.g., H.264) to scalar soft-core (MIPS, RISC-V),
• Quantitative metrics: context-switch overhead, throughput, turnaround, fairness (Jain's index).

The suite is accompanied by developer APIs, scripts, and validation tools, supporting reproducible evaluations and incremental extension with new accelerators and scheduling policies.

7. Significance and Cross-Domain Impact

Across all domains, the creation of EpiBench reflects the increasing demand—both scientific and operational—for standardized, reproducible, and extensible evaluation platforms tailored to the complexities of modern AI/ML systems. Each instantiation enforces strict, domain-appropriate evaluation protocols (rolling-origin retraining, process-level diagnosis, deterministic grading, or retrieval-augmented reasoning), surfaces critical challenges (e.g., multi-modal memory bottlenecks, domain-specific scientific judgment, nontrivial ensemble boosts, knowledge gaps in LLMs), and exposes the limitations of generic or naïve modeling paradigms in real-world settings. EpiBench platforms enable granular error analysis, algorithmic progress, and facilitate transparent comparison driven by rigorous, community-validated benchmarks.

For current methodological frontiers and implementation resources, refer to each EpiBench platform’s original publication and repository (Srivastava et al., 2021, Dong et al., 7 Apr 2026, Dai et al., 10 May 2026, Muralidharan et al., 11 Jun 2026, Liu et al., 2024, Malik et al., 10 Nov 2025).