SciBench Benchmark Datasets Overview

• SciBench Benchmark Datasets are standardized, large-scale resources designed to rigorously assess computational models with authentic scientific reasoning and data tasks.
• They encompass SciBench for curricular problem solving, scBench for single-cell RNA-seq analysis, and SDRBench for evaluating lossy compressors in various scientific domains.
• Deterministic protocols and reproducible methodologies are employed to guide model diagnostics and algorithmic improvements in real-world scientific workflows.

SciBench benchmark datasets comprise a class of standardized, large-scale evaluation resources designed for rigorous assessment of computational models and algorithms in scientific domains. These resources span complex scientific reasoning, real-world data analytics, and scientific data reduction, and are characterized by their curated coverage, precise evaluation protocols, and integration with state-of-the-art computational methods. Exemplary datasets in this class include SciBench for collegiate-level problem-solving assessment, scBench for single-cell RNA-seq analysis, and SDRBench for evaluating lossy compressor performance in scientific data reduction.

1. Scope and Purpose

SciBench benchmarks serve as authoritative reference suites targeting advanced scientific reasoning, data analysis fidelity, and data reduction quality. Unlike high-school-level or synthetic benchmarks, these datasets are derived from real-world scientific workflows, standard undergraduate textbooks, and actual domain-specific experiments. Their primary purposes are:

• Rigorous, replicable evaluation of LLMs, AI agents, and compression codecs under representative conditions.
• Systematic measurement of reasoning proficiency, computational robustness, and error-tolerance in real scientific problem domains.
• Guidance for model development, algorithmic diagnostics, and reproducibility practices in academic and industrial research.

2. Dataset Composition and Domain Coverage

SciBench

SciBench (Wang et al., 2023) consists of 986 benchmark items aggregated from collegiate-level textbook questions and university exams. Its composition is as follows:

Subset	Content Type	Count	Domains
Textbook	Open-response	789	Mathematics, Chemistry, Physics
Multimodal	Visual + LaTeX	94	Physics, Chemistry, Math
Closed Exams	MC/TF/Free-response	103	Data Mining, ML, Differential Eq

Physics, chemistry, and mathematics are represented via canonical textbooks (e.g., "Fundamentals of Physics," "Physical Chemistry," "Calculus: Early Transcendentals"). Problems include a range of subtopics: calculus, probability, quantum mechanics, thermodynamics, classical mechanics, and more. Metadata annotates each instance with original source, solution availability, answer units, numerical values, format (visual context, question type), and, for exams, point values.

scBench

scBench (Workman et al., 9 Feb 2026) targets single-cell RNA-seq (scRNA-seq) analysis pipelines. The benchmark comprises 394 verifiable evaluation problems, constructed from real scRNA-seq workflows across six sequencing platforms (e.g., 10× Chromium, BD Rhapsody, MissionBio Tapestri). Each "evaluation" provides:

• A pre-analysis data snapshot (AnnData .h5ad format) stripped of shortcut features.
• Natural-language prompt and machine-readable JSON output schema.
• Deterministic grader for answer validation.

Tasks span QC, normalization, dimensionality reduction, clustering, cell typing, differential expression, and trajectory inference.

SDRBench

SDRBench (Zhao et al., 2021) evaluates lossy compressors on nine real-world scientific data objects representative of large-scale simulation and experiment output (e.g., EXAALT, HACC, CESM-ATM, EXAFEL, QMCPack). Domains include molecular dynamics, climate modeling, turbulence, cosmology, and quantum Monte Carlo, with varying data dimensionality and precision.

3. Evaluation Protocols and Grading Methodologies

Each benchmark suite enforces explicit, deterministic evaluation methodologies.

SciBench

SciBench presents immutable benchmark files, with no train/validation/test splits, to maintain standardized comparability. Problem types include free-response, multiple-choice, and problems with/without visual components. LLM responses are scored according to correctness using numerical answers (rounded to three decimal places), LaTeX benchmarks, and exam-specific grading rubrics (Wang et al., 2023).

scBench

scBench defines five grader families, automating pass/fail judgment on structured agent outputs:

• NumericTolerance: Absolute/relative numeric bounds.
• MultipleChoice: Option matching.
• MarkerGenePrecisionRecall: Computes precision and recall on gene lists, with configurable thresholds.
• LabelSetJaccard: Jaccard index on predicted label sets (default pass if $J \geq 0.90$).
• DistributionComparison: Agent-generated and ground-truth category proportions must agree within a specified margin (e.g., ±5 percentage points).

Performance metrics include aggregate accuracy, precision, recall, $F_1$, and 95% confidence intervals, calculated over triplicate model runs per evaluation as described in (Workman et al., 9 Feb 2026).

SDRBench

SDRBench uses the Z-checker tool for protocol automation. Metrics reported are:

• Compression Ratio (CR)
• Throughput (MB/s)
• Maximum Absolute/Relative Errors
• MSE, PSNR, NRMSE
• SSIM for 1D/2D data
• Error autocorrelation and error histograms

Compressors SZ, ZFP, and SZ(Hybrid) are evaluated under specified error bounds, with runs executed on fixed hardware (e.g., Argonne “Bebop” nodes) for reproducibility (Zhao et al., 2021).

4. Standardized Tasks, Coverage, and Metadata

SciBench and scBench are distinguished by the breadth and rigor of their task and domain coverage:

Benchmark	Task Categories	Platforms/Coverage
SciBench	Problem solving in math, physics, chemistry; multimodal	3 collegiate domains, 10 textbooks, 7 exams
scBench	QC, normalization, DR, clustering, typing, DE, trajectory	6 sequencing platforms
SDRBench	Compression evaluation	9 multidomain scientific datasets

All tasks are meticulously tagged with metadata: problem source, subtopic, solution availability, platform, evaluation type, complexity notes, and field-level attributes as appropriate.

5. Benchmarks in Practice: Findings and Impact

SciBench Results

Representative LLMs show unsatisfactory performance on SciBench, with best scores reaching only 43.22%. No prompting strategy consistently dominates. Error analysis identifies ten distinct problem-solving deficits, and improvements in specific skills often trade off with declines in others. This highlights the challenge of robust, generalizable scientific reasoning in current LLMs (Wang et al., 2023).

scBench Results

Eight advanced AI agents under mini-SWE-agent framework yield aggregate accuracies from 29.2% (Gemini 2.5 Pro) to 52.8% (Claude Opus 4.6) across 394 evaluations. Task difficulty is stratified: normalization and QC (well-documented) reach cross-model means of 70.4% and 55.3%, while differential expression and cell typing remain most challenging. Performance varies as much by sequencing platform as by model; poorly documented platforms can result in >30 percentage-point drops (Workman et al., 9 Feb 2026).

SDRBench Findings

Compression-error distribution, throughput, rate–distortion trade-offs, effects of data dimensionality, and PSNR-vs-SSIM diagnostics are systematically characterized. Hybrid codecs excel at aggressive reduction; plain SZ is suitable at mid–high bit rates; ZFP is optimal for near-lossless. Structural similarity is best measured using 2D SSIM alongside PSNR for perceptual fidelity. Lossy compressors at practical error bounds achieve 20–100× compression, vastly exceeding lossless methods (Zhao et al., 2021).

6. Practical Guidance and Limitations

Recommended usage includes:

• Direct integration of benchmark test harnesses (e.g., mini-SWE-agent for scBench) for plug-in evaluation.
• Isolation and reproducibility of evaluation environments—with removal of shortcut data to enforce genuine computational analysis.
• Routine regression testing and validation through canonical evaluation subsets.

Caveats include the discretization of complex scientific judgment into binary outcomes, lack of full pipeline (multi-step, human-in-the-loop) capture, and partial coverage of emerging task categories or data platforms. Both scBench and SDRBench provide standardized schemas and automated tooling, with extensibility for future methodological advances.

7. Access and Extensibility

All benchmarks provide detailed documentation, data download scripts, and comprehensive metric schemas:

• SciBench: Released as immutable files with LaTeX-encoded problems and solutions.
• scBench: Canonical data snapshots, grader code, and integration harnesses are available; public subset enables setup validation (Workman et al., 9 Feb 2026).
• SDRBench: Full benchmark, Z-checker tool, and code for automated runs accessible via https://sdrbench.github.io and GitHub (Zhao et al., 2021).

Extending the benchmarks is supported via open formats (JSON, H5AD, LaTeX), plugin interfaces, and rigorous data/metric provenance tracking. Expansion to new domains (e.g., additional omic assays, expanded pipeline tasks) is inherent to the benchmark design.