SciEvalKit: Scientific AI Evaluation Toolkit

• SciEvalKit is an open-source evaluation toolkit that benchmarks AI models on complex scientific tasks across physics, chemistry, life science, earth science, astronomy, and materials science.
• It employs a modular, four-layered architecture supporting diverse inputs such as text, image, code, and symbolic data to ensure rigorous and reproducible assessments.
• The toolkit uses expert-curated benchmarks and standardized metrics like accuracy and code pass rate to evaluate scientific competencies, including symbolic reasoning and hypothesis generation.

SciEvalKit is an open-source evaluation toolkit designed to benchmark AI models on scientific tasks requiring sophisticated domain knowledge, multimodal perception, and reasoning across six core scientific domains. It departs from general-purpose evaluation platforms by emphasizing authentic, expert-grade benchmarks and a principled taxonomy of scientific competencies, supporting rigorous, transparent, and reproducible assessment of scientific general intelligence in AI systems (Wang et al., 26 Dec 2025).

1. System Overview and Motivation

SciEvalKit addresses the need for capability-oriented and discipline-diverse evaluation in scientific AI. General-purpose benchmarks—dominated by generic question answering (QA), image understanding, or aggregate leaderboard metrics—fail to capture essential scientific skills such as scientific code generation, symbolic reasoning, or hypothesis generation. SciEvalKit is structured to:

• Provide expert-curated scientific benchmarks reflecting authentic, domain-specific challenges, not synthetic or trivial tasks.
• Operate across a principled taxonomy of scientific competencies, including symbolic reasoning, code generation, multimodal perception/reasoning, hypothesis generation, and knowledge understanding.
• Support a unified, extensible pipeline for handling text, image, code, and symbolic inputs.
• Ensure transparent, reproducible, and comparable results, with modular interfaces for new datasets and models.
• Focus on six primary scientific domains: physics, chemistry, life science, earth science, astronomy, and materials science.

Unlike broad-domain benchmarks such as MMLU or SuperGLUE, SciEvalKit is explicitly designed for the demands and subtleties of AI4Science (Wang et al., 26 Dec 2025).

2. Architecture and Module Structure

SciEvalKit comprises a four-layered, modular architecture, each with strict responsibilities to enable scalability and extensibility:

Layer	Core Responsibilities	Example Interfaces/Files
Dataset Layer	Dataset registry; unified interfaces for text, image, video; prompt construction; normalization, indexing, metadata, and media caching.	data_ingestion.py, TextBaseDataset, ImageBaseDataset
Model Inference Layer	Model loading (local/meta/API); abstract .generate() interface; batching, retries, supports vLLM/PyTorch and cloud API providers.	model_adapters.py
Evaluation Layer	Modular .evaluate() for each dataset; exact match, semantic, code execution, or LLM-as-judge; utilities for judge construction, code testing, metric aggregation.	metrics.py
Report & Storage Layer	Logging of predictions, reasoning traces, and metadata; metrics serialization; helper functions for reproducibility; support for CSV, JSON, XLSX.	reporting.py

This architecture supports batch evaluation, data/model integration, custom metric functions, and structured result reporting, ensuring both flexibility and methodological rigor (Wang et al., 26 Dec 2025).

3. Scientific Competencies and Task Taxonomy

SciEvalKit organizes tasks across seven core scientific competencies (capabilities), each grounded in specific representative datasets and evaluation methodologies:

Competency	Task Nature	Representative Benchmarks
Scientific Multimodal Perception	Entity detection/localization in scientific images	SLAKE (radiology VQA); input: CT/MRI+question, output: single-token
Scientific Multimodal Understanding	Interpretation of imagery and textual context alignment	SFE (scientific figures); input: plot+text, output: free-form or MCQ
Scientific Multimodal Reasoning	Visual-textual chain-of-thought inference	MSEarth (map+question), output: explanation or label
Scientific Symbolic Reasoning	Symbolic derivation, units, equations	CMPhysBench, PHYSICS; input: physics problem, output: numeric/formula
Scientific Code Generation	Text-to-code for research workflows	SciCode (w/unit tests), AstroVisBench (Jupyter)
Science Hypothesis Generation	Open-ended research hypothesis formulation	ResearchBench
Scientific Knowledge Understanding	Factual/mechanistic QA spanning sciences	ChemBench, MaScQA, ProteinLMBench, ClimaQA, etc.

Capability-based aggregation reports per-competency performance (mean of constituent benchmarks), delivering a fine-grained profile of model strengths and weaknesses over mere aggregate scores (Wang et al., 26 Dec 2025).

4. Supported Domains, Benchmarks, and Datasets

Six major scientific domains are represented, each with curated benchmark datasets:

• Life Science: ProteinLMBench, BioProBench, TRQA, Biology-Instructions, Mol-Instructions, PEER.
• Chemistry: ChemBench, ChemBench4K, SMolInstruct.
• Earth Science: ClimaQA, EarthSE, MSEarth.
• Materials Science: MaScQA.
• Physics: CMPhysBench, PHYSICS.
• Astronomy: AstroVisBench.

Datasets span MCQ, complex symbolic tasks, multimodal VQA, code generation with unit tests, and open-ended hypothesis generation. Many datasets are derived from real-world problems and professional scientific practice, supporting both breadth and depth in evaluation (Wang et al., 26 Dec 2025).

5. Evaluation Protocols and Metrics

The evaluation pipeline conforms to reproducible and interpretable methodology:

• Workflow:

1. build_dataset: load samples and prompts 2. build_model_from_config: instantiate model (local or API) 3. infer_data: batch prediction generation with error tolerance 4. evaluate: dataset-specific metric computation 5. aggregate: combine per-benchmark scores into capability dimensions 6. report: serialize metrics and logs

• Core Quantitative Metrics:
  • Accuracy (MCQ): $\frac{\#correct}{\#total}$
  • Precision/Recall/F₁ (binary/multi-label):
  • $Precision = \frac{TP}{TP + FP}$,
  • $Recall = \frac{TP}{TP + FN}$,

  $$F_1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$$ - Code-execution pass rate:

  $$Pass\_Rate = \frac{\#unit\,tests\,passed}{\#total\,tests}$$ - Semantic/LLM-based metrics: BLEU, VQA accuracy, semantic overlap

• Aggregated results enable capability-level radar charts and fine-grained bar plots across domains and modalities (Wang et al., 26 Dec 2025).

6. Extensibility, Integration, and Usage

SciEvalKit exposes interfaces for integrating custom datasets and models:

• Dataset Integration: Inherit from TextBaseDataset/ImageBaseDataset, implement load_data and build_prompt, specify evaluation logic (e.g., exact match, LLM-as-judge).
• Model Integration: Specify config (JSON/YAML), loading via unified adapter regardless of local/remote model type.
• Batch and CLI Usage: Support for YAML configs, concurrent evaluation over multiple datasets and models.
• Reproducibility: Full manifest logging (dataset, model, seeds, timestamp); intermediate files and resume logic.

Example code usage:

from scievalkit import build_dataset, build_model_from_config, run_evaluation dataset = build_dataset('ChemBench', split='test') model = build_model_from_config('my_model.json') results = run_evaluation( model=model, dataset=dataset, batch_size=4, output_dir='results/chembench') print(results.metrics)

A 0.72 accuracy on ChemBench indicates the model answers 72% of chemistry MCQs correctly; capability vectors reveal granular strengths and weaknesses (Wang et al., 26 Dec 2025).

7. Quality Assurance, Reporting, and Community Maintenance

SciEvalKit emphasizes rigorous verification and community-driven maintenance:

• Logging/Versioning: Each run captures dataset/model version, random seeds, and manifest metadata; intermediate output files enable recovery and incremental evaluation.
• Reporting: Tabular (CSV/JSON/XLSX) summaries; visualization scripts for radar/bar plots; sample output includes timestamp and per-competency metrics.
• Open-Source Governance: GitHub Actions enforce CI standards, with issue templates and quarterly releases. After three major contributions, participants can gain co-authorship in reports.
• Community Updatability: Governance ensures incorporation of new benchmarks, models, and methodological advances, supporting rapid evolution and reproducibility.

SciEvalKit thus provides a robust, extensible infrastructure for benchmarking scientific general intelligence in AI, integrating expert benchmarks, rigorous metrics, and tooling for the scientific AI community (Wang et al., 26 Dec 2025).