SciMLBench: Unified SciML Benchmarking

• SciMLBench is a standardized benchmark initiative that unifies over 100 diverse scientific ML tasks with clear problem specifications and FAIR data practices.
• It improves reproducibility by enforcing containerized environments, fixed train/validation/test splits, and rigorous documentation for cross-domain comparisons.
• The platform employs a hierarchical taxonomy and a six-category rubric to assess benchmarks, facilitating systematic evaluation across fields such as physics, chemistry, and climate science.

SciMLBench is a standardized scientific machine learning (SciML) benchmarking initiative, developed to address fragmentation and lack of reproducibility in evaluating machine learning frameworks across scientific disciplines. Under MLCommons governance, SciMLBench unifies over 100 diverse scientific ML tasks into a coherent, extensible ontology accessible via an open submission workflow, strict rubric-based endorsement, and a centralized web portal that supports programmatic search and metadata integration for continued evolution and community governance (Hawks et al., 6 Nov 2025).

1. Motivation and Foundational Objectives

SciML spans fields including physics, chemistry, materials science, biology, and climate science, featuring data that is heterogeneous in both structure and modality. Prior to SciMLBench, existing benchmarks were siloed by domain (e.g., XAI-BENCH for explainability, PDEBench for PDE surrogates, FastML for low-latency edge ML), impeding reproducibility, degrading portability of methods, and creating obstacles for systematic cross-domain algorithmic comparison (Hawks et al., 6 Nov 2025, Thiyagalingam et al., 2021). SciMLBench formalizes benchmarks as first-class, machine-readable entities, encoding a unified specification—including problem formulation, FAIR-compliant datasets, performance metrics, open reference solutions, reproducible software environments, and comprehensive documentation.

Key objectives include:

• Standardization of description, implementation, and evaluation of scientific ML tasks.
• Improved reproducibility through fixed train/validation/test splits, containerized environments, and open reference code.
• Enabling of cross-domain benchmarking and meta-analysis, allowing stakeholders to identify and assemble tasks by scientific domain, AI-motif (e.g., surrogate modeling, generative tasks), or compute motif (e.g., latency-bound, memory-bound).
• Sustained community governance and scalable extensibility under the MLCommons Science Working Group.

2. Hierarchical Benchmark Taxonomy

SciMLBench organizes all benchmarks within a rigorous three-tier hierarchy:

Level	Scope	Example
Scientific-Level	Abstracted, canonical ML or surrogate tasks	Navier–Stokes PDE emulation (PDEBench)
Application-Level	Complete data-to-insight pipelines, with workflow logic	OCP adsorption energy prediction with GNNs
System-Level	Hardware-centric tests of power, throughput, latency	FastML sub-μs jet-tagging on FPGAs, MLPerf HPC scaling benchmarks

Scientific-level benchmarks focus on canonical ML problems abstracted from deployment specifics. Application-level benchmarks encompass full pipelines, integrating preprocessing, model training, inference, and specialized metrics. System-level benchmarks profile hardware under fixed task constraints, quantifying trade-offs in latency, energy, and throughput (Hawks et al., 6 Nov 2025, Duarte et al., 2022). This stratification enables users to select and compare benchmarks aligned with methodological focus and operational targets.

3. Six-Category Rubric and Endorsement Process

All SciMLBench entries are evaluated against a stringent six-category, 0–5 ordinal rubric:

1. Software Environment: mandates open, documented, containerized, or scripted code without manual patching.
2. Problem Specification & Constraints: requires unambiguous task definition, I/O formats, and explicit system constraints (power, latency, memory).
3. Dataset: insists on full FAIR (Findable, Accessible, Interoperable, Reusable) compliance with canonical train/val/test splits and persistent URLs.
4. Performance Metrics: demands complete and quality metric specifications.
5. Reference Solution: public, metric-evaluated and fully documented implementation.
6. Documentation: comprehensive background, motivation, evaluation protocol, and a peer-reviewed or preprint manuscript.

An average score $S \geq 4.5$ denotes endorsement as an “MLCommons Science Benchmark.” Benchmarks are scored by MLCommons Science Working Group experts. The aggregation method permits custom weighting:

$$S = \frac{1}{6}\sum_{i=1}^{6} s_i ~~~\text{or}~~~ S = \sum_{i=1}^{6} w_i s_i,~ \sum_{i=1}^{6} w_i=1$$

(Hawks et al., 6 Nov 2025).

4. Submission Workflow, Metadata Standards, and Governance

The SciMLBench submission process comprises five stages:

1. Proposal via public GitHub, detailing metadata (domain, AI-motif, problem spec, dataset location, constraints).
2. Automated validation of metadata completeness, dataset accessibility, and container build reproducibility.
3. Expert review and rubric assessment by the MLCommons Science Working Group, including feedback and revision.
4. Scoring, endorsement (if qualifying), and addition to the central registry.
5. Ontology integration: updating the searchable web portal and publishing new entries (Appendix Table A1) (Hawks et al., 6 Nov 2025).

Benchmark metadata uses a controlled JSON/YAML schema, including DOIs, GitHub links, domain tags, motif tags, scores, constraints, and machine-readable metric definitions. A reference container registry (Docker/Singularity) and a RESTful API for programmatic search/filtering are integral. Extensibility is critical: new domains, motifs, and compute tags are incorporated by extending controlled vocabularies. Future integration targets include RDF/OWL semantic definitions and workflow languages (e.g., CWL, Nextflow).

Governance resides with the MLCommons Science Working Group, which maintains the rubric, adjudicates submission edge cases, and periodically revises metadata schemas.

5. Emerging System Workload Patterns and Clustering

SciMLBench surfaces emerging hardware and computational motifs by clustering static and dynamic feature vectors from benchmarks—incorporating rubric scores and system metrics (e.g., GPU utilization, power traces). Compute motif tags classify tasks as latency-bound (e.g., sub-millisecond FPGA inference), memory-bound (large 3D grids), throughput-bound (satellite data streams), or utilization-bound (long-horizon policy optimization).

The profiling workflow:

1. Feature vector collection: combining rubric categories and dynamic runtime metrics per benchmark.
2. Cosine distance computation: $$d_{ij} = 1 - \frac{v_i \cdot v_j}{||v_i||\,||v_j||}$$.
3. Agglomerative hierarchical clustering, with dendrogram thresholding $\tau$ for $K$ clusters (e.g., Low-Power, High-Power, or Mixed).
4. Mapping clusters to user priorities, recommending workloads by operational relevance (Hawks et al., 6 Nov 2025).

This facilitates system architects’ identification of representative tasks and patterns for next-generation hardware and algorithm co-design.

6. Representative Benchmarks and Case Studies

SciMLBench aggregates canonical tasks across domains, verified implementations, and innovations from both ML and domain-specific perspectives. Notable examples include:

• High-Energy Physics: Jet-classification (Duarte et al. 2022) for sub-microsecond FPGA inference; Smart-Pixels (Parpillon et al. 2024) on ASICs (Duarte et al., 2022).
• Chemistry: MOLGEN (SELFIES generative modeling, Fang et al. 2024); OCP DFT regression with GNNs (Chanussot 2021).
• Materials Science: Materials Project (bandgap/formation energy prediction, Jain 2013); SuperCon3D (superconductor design).
• Biology & Medicine: BiasBench (Luo et al. 2025, scRNA-seq annotation + QA); MedQA (Jin 2020, board-exam QA).
• Climate & Earth Science: ClimateLearn (Nguyen 2023, weather forecasting with physics-informed loss); HDR ML Anomaly (Campolongo 2025).
• Computational Science & AI: SciCode (Tian 2024, code generation test suite); MLPerf HPC (Thiyagalingam et al., 2021).
• Mathematics: FrontierMath (Glazer 2024); AIME (structured high-school math problems).

Implementations and rubric scores for these are in the public GitHub registry and Appendix Table A1 (Hawks et al., 6 Nov 2025).

7. Architecture, Extensibility, and Community Impact

The SciMLBench architecture consists of:

• Data/metadata layers: strictly defined schemas, persistent identifiers, and FAIR dataset integration.
• Software layers: reference container images (Docker/Singularity), RESTful APIs.
• Web portal: interactive filtering and task recommendation based on clustering and motif selection.

Extensibility supports the addition of novel scientific domains, AI/ML motifs, and compute categories. Integration of workflow languages and semantic web standards is planned to further enhance interoperability (Hawks et al., 6 Nov 2025).

The unification of previously disparate efforts under SciMLBench—together with rigorous methodology, container-based reproducibility, and open community governance—establishes a scalable, trustworthy foundation for reproducible scientific machine learning benchmarking. Through standardized protocols and endorsement, it promotes high-quality, cross-domain, and system-level evaluation, catalyzing both methodological progress and practical deployment across the full spectrum of scientific computing.