STEM-Bench: Evaluating Multimodal STEM Models

• STEM-Bench is a broad term for diverse benchmark suites and methodologies used to evaluate STEM capabilities across modalities, languages, and task granularities.
• It covers evaluations from dynamic, code-backed physics problem solving to domain-specific agentic tasks and bioinformatics analysis.
• Evaluations combine accuracy metrics with robustness diagnostics, criterion-weighted grading, and deterministic artifact verification for detailed performance insights.

STEM-Bench is not a uniformly standardized official title in the recent arXiv literature. The term is best understood as a broad label for benchmark suites and benchmark-construction methodologies that evaluate models and agents on science, technology, engineering, and mathematics across multiple modalities, languages, and workflow granularities. Within that landscape, the literature spans a large multimodal K–12 benchmark officially named STEM, a dynamic undergraduate-physics benchmark named SymPyBench, an Arabic benchmark suite named 3LM, a relative capability evaluation framework named STEM, an expert-authored frontier agent benchmark named COMPOSITE-STEM, and a domain-specific bioinformatics benchmark named scBench (Shen et al., 2024, Imani et al., 5 Dec 2025, Boussaha et al., 21 Jul 2025, Hu et al., 16 Aug 2025, Waters et al., 10 Apr 2026, Workman et al., 9 Feb 2026).

1. Terminology and scope

The naming of STEM-Bench is itself a point of clarification. The paper "Measuring Vision-Language STEM Skills of Neural Models" explicitly introduces a benchmark dataset officially named STEM and states that the authors do not use the term “STEM-Bench” (Shen et al., 2024). The paper "STEM: Efficient Relative Capability Evaluation of LLMs through Structured Transition Samples" likewise states that it does not use the term “STEM-Bench,” and that the closest concept in that paper is the Significant Transition Samples pool and the balanced STS subset used in evaluation (Hu et al., 16 Aug 2025).

Other papers use the term more loosely as a category. SymPyBench contrasts itself with “generic STEM-Bench-style suites,” arguing that such suites often prioritize breadth across many disciplines and static items, whereas SymPyBench emphasizes dynamic, executable, parameterized university-level physics problems (Imani et al., 5 Dec 2025). The 3LM paper states that its openly available Arabic-native STEM benchmarks and Arabic code suite make it “a practical candidate to serve as, or inform, a STEM-Bench resource in Arabic” (Boussaha et al., 21 Jul 2025).

This suggests that STEM-Bench functions less as a single canonical benchmark than as an umbrella designation for STEM evaluation resources. Under that usage, the term covers at least five recurring targets: multimodal STEM question answering, dynamic symbolic and numerical reasoning, relative capability estimation, agentic scientific task execution, and domain-specific scientific data analysis.

2. Benchmark families and design axes

Recent STEM-Bench-style resources differ along several axes: static versus dynamic item generation, text-only versus multimodal inputs, multiple-choice versus executable outputs, broad curricular coverage versus narrow domain depth, and item-level grading versus workflow-level verification.

Resource	Official characterization	Key scope
STEM	Multimodal vision-language benchmark	448 skills, 1,073,146 K–8 questions
SymPyBench	Dynamic, code-backed physics benchmark	15,045 university-level physics problems
3LM	Arabic benchmark suite	Native STEM, synthetic STEM, Arabic code
STEM	Structured Transition Evaluation Method	STS pool and balanced STS subset
COMPOSITE-STEM	Expert-written agent benchmark	70 tasks in physics, chemistry, biology, math
scBench	Scientific agent benchmark	394 verifiable scRNA-seq problems

The STEM benchmark is breadth-oriented. It covers science, technology, engineering, and mathematics, is multimodal by construction, and includes text plus at least one image for each item, with 448 distinct STEM skills and 1,073,146 multiple-choice questions spanning pre-K through grade 8 (Shen et al., 2024). By contrast, SymPyBench is narrower in domain but deeper per problem. It contains 15,045 undergraduate-level physics problems, each parameterized, paired with step-by-step reasoning, and backed by executable Python code that produces the ground-truth solution for arbitrary parameter settings (Imani et al., 5 Dec 2025).

3LM emphasizes language-specific evaluation. It is a three-part Arabic benchmark suite comprising 865 native STEM multiple-choice questions, 1,744 synthetic Arabic STEM multiple-choice questions, and 542 Arabic code prompts derived from HumanEval+ and MBPP+ (Boussaha et al., 21 Jul 2025). COMPOSITE-STEM shifts the focus from static answer selection to agentic scientific work. It contains 70 expert-written tasks across physics, chemistry, biology, and mathematics, including multimodal reference files and both exact-match and semantic grading (Waters et al., 10 Apr 2026). scBench pushes further toward real scientific analysis by presenting agents with AnnData files and requiring executable recovery of biologically meaningful results in single-cell RNA-seq workflows (Workman et al., 9 Feb 2026).

A further design axis is whether the benchmark is itself a dataset or a methodology for constructing evaluation subsets. The STEM paper from 2025 falls into the latter category: it is not a single benchmark in the conventional sense, but an evaluation framework and benchmark-construction methodology that extracts high-discrimination Significant Transition Samples from existing benchmarks such as MMLU, MMLU-Pro, GPQA, SuperGPQA, GSM8K, and MATH (Hu et al., 16 Aug 2025).

3. Evaluation protocols and scoring paradigms

STEM-Bench-style evaluation no longer relies on accuracy alone. The literature now combines conventional answer accuracy with robustness diagnostics, criterion-weighted grading, deterministic artifact verification, and relative capability positioning.

SymPyBench retains standard metrics such as Exact Match Accuracy and Partial Accuracy, but adds three dynamic robustness metrics computed over “problem groups” formed by paraphrases, formats, and parameter changes: Consistency Score, Confusion Rate, and Complete Failure Rate (Imani et al., 5 Dec 2025). Their role is diagnostic rather than merely aggregate. Consistency Score measures whether all controlled variants are answered correctly; Confusion Rate flags problem groups with approximately 40–60% accuracy across variants; Complete Failure Rate isolates groups where all variants are incorrect.

The relative-evaluation STEM framework formalizes benchmark discriminability and transition structure. For each benchmark $j$, it defines

$$D_j = \sigma_{S_j} \times \rho_{S_j,\log(P)}, \qquad w_j = \frac{D_j}{\sum_{j=1}^{m} D_j},$$

where $\sigma$ is the standard deviation of scores and $\rho$ is the Pearson correlation between benchmark score and $\log(P)$ across model sizes (Hu et al., 16 Aug 2025). At the sample level, it uses an Inference Result Vector with ternary entries $v_i \in \{-1,0,1\}$ and assigns a Transition Index to samples with a single monotonic $0 \rightarrow 1$ transition across a scale-ordered model family. Evaluation then proceeds on a balanced STS subset, and the target model is positioned between anchor models by identifying where accuracy drops sharply across TI levels.

COMPOSITE-STEM replaces single-answer scoring with a portable “evaluation contract.” Tasks are graded by exact match, by a five-model LLM jury, or by a hybrid of the two. The grading protocol specifies weighted criteria, normalizes weights to a 0–100 scale, and aggregates criterion outcomes into a normalized Harbor reward scalar (Waters et al., 10 Apr 2026). This permits partial credit for scientifically meaningful outputs while retaining explicit per-criterion audit trails in portex_detail.json.

scBench takes a different route: deterministic graders over structured JSON outputs. Its grader families include NumericTolerance, MultipleChoice, MarkerGenePrecisionRecall, LabelSetJaccard, and DistributionComparison. For example,

$$\text{precision@}K = \frac{|P \cap G|}{|P|}, \qquad \text{recall@}K = \frac{|P \cap G|}{|G|}$$

for marker-gene recovery, and

$J(A,B) = \frac{|A \cap B|}{|A \cup B|}$

for label-set overlap (Workman et al., 9 Feb 2026). Replicates are aggregated in two stages: each run yields $s_{i,r}\in\{0,1\}$, per-evaluation means are computed as $$D_j = \sigma_{S_j} \times \rho_{S_j,\log(P)}, \qquad w_j = \frac{D_j}{\sum_{j=1}^{m} D_j},$$0, and overall accuracy is then $$D_j = \sigma_{S_j} \times \rho_{S_j,\log(P)}, \qquad w_j = \frac{D_j}{\sum_{j=1}^{m} D_j},$$1.

The original STEM benchmark also supplements accuracy with a human-referenced “exam score” based on IXL SmartScore, where a score above 90 indicates mastery for science, engineering, and math; the benchmark reports both test-set accuracy and human-comparable exam-style scoring (Shen et al., 2024). The resulting picture is that STEM-Bench evaluation has become multidimensional: correctness, stability, discrimination, reproducibility, and human-comparable mastery are all explicit targets.

4. Construction methodologies and curation practices

A defining feature of current STEM-Bench work is that benchmark construction has itself become a technical contribution. The methodologies vary, but they share a concern with leakage control, reproducibility, and the removal of artifacts that would let models succeed without the intended reasoning.

SymPyBench begins from open, Creative Commons–licensed university-level physics materials. OCR with Tesseract extracts text, heuristic checks remove items dependent on figures, tables, or other questions, and LLaMA-3.2-90B-Vision-Instruct reformats the remaining problems into a standardized schema with reasoning steps, variables, and constants (Imani et al., 5 Dec 2025). The same pipeline then generates parameterized templates and executable Python functions using SymPy and Pint. Validation executes the generated code on the original numerical values and checks numerical tolerance and unit equivalence; approximately 88% of problems yielded validated code, while non-conforming functions were discarded. The paper also reports staged filtered rates of about 5% for dependency or visual issues, about 4.5% structured JSON issues, about 1% template mismatches, less than 1% variation structure errors, and about 12% Python code failures.

3LM combines native collection, Arabic-aware synthetic generation, and human-in-the-loop translation QA. The native STEM component uses web scraping or API queries, regex-based categorization, Pix2Tex for math-aware OCR, and a four-stage pipeline with Qwen3-235B-A22B for extraction, classification, formatting, and manual quality assurance (Boussaha et al., 21 Jul 2025). The synthetic STEM set adapts the YourBench pipeline and enforces difficulty at or above 6 on a 1–10 scale. The code suite translates HumanEval+ and MBPP+ with GPT-4o, uses backtranslation with ROUGE-L F1 thresholds of at least 0.85 for HumanEval and at least 0.8 for MBPP, and escalates sub-threshold prompts to review by native Arabic speakers with Python expertise.

The STEM relative-evaluation framework constructs its pool offline through multi-scale inference on a reference family, IRV filtering, STS extraction, TI assignment, and balanced subset sampling (Hu et al., 16 Aug 2025). Its central curation operation is not question generation but the removal of abnormal, non-monotonic, or invalid-output samples, which the paper interprets as contamination, overfitting, or structural non-informativeness.

COMPOSITE-STEM emphasizes expert authorship and rubric refinement. Its 70 tasks were contributed by doctoral-level researchers, distinguished faculty, postdocs, and practitioners, and the Portex team ran multi-round curation through Portex Datalab with live model audits, rubric refinement, and error inspection (Waters et al., 10 Apr 2026). scBench similarly foregrounds construction rigor: it reproduces published analyses from raw counts, defines a precise JSON schema, selects a grader family, calibrates tolerances across multiple valid methods, strips precomputed embeddings and cached labels, and performs an anti-shortcut review that explicitly attempts to solve each task by reading .obs or .uns, using prior biological knowledge, or exploiting alternative valid methods (Workman et al., 9 Feb 2026).

Across these benchmarks, curation is not merely dataset assembly. It is a formal control layer governing what counts as evidence of capability.

5. Empirical findings on model capability

The empirical record across STEM-Bench-style resources is heterogeneous, but several patterns recur: performance is highly sensitive to modality, output format, and task realism; static multiple-choice scores can substantially overstate broader capability; and workflow execution remains difficult even for frontier systems.

On the multimodal STEM benchmark, zero-shot CLIP variants achieve about 53–55 average accuracy, with the best zero-shot result reported as about 54.9 for RN50x64, while GPT-3.5-Turbo reaches 49.0 and human participants in a small study average 83.0% (Shen et al., 2024). Full fine-tuning of CLIP ViT-L/14@336px improves macro average accuracy to 76.3, but the paper still reports that recent model advances “only help master a very limited number of lower grade-level skills (2.5% in the third grade).”

In Arabic evaluation, 3LM shows a large gap between visible-option and hidden-option settings. Models consistently perform better on MCQ than on completion. Among base models, Qwen2.5-72B leads native MCQ at about 94.45% and synthetic MCQ at about 91.9%, while completion scores are much lower, at about 62.31% on native and about 57.6% on synthetic (Boussaha et al., 21 Jul 2025). The synthetic Arabic STEM benchmark is harder than the native benchmark, and distractor perturbation causes measurable declines across all models.

SymPyBench reaches beyond point estimates by reporting both performance and stability. Sonnet-3.7, Gemini-2.0-Flash, and Llama4-Maverick-17B-128E achieve exact match accuracies above 64%; Sonnet-3.7 attains the highest Consistency Score at 42.42% and the lowest Confusion Rate at 6.06%; Gemini-2.0-Flash achieves the highest Partial Accuracy at 71.43% (Imani et al., 5 Dec 2025). The benchmark also identifies recurring failure modes: arithmetic instability, unit handling errors, hallucination under under-specified inputs, implicit simplification bias, and format-specific brittleness. A notable finding is that accuracy on MC-Symbolic can far exceed accuracy on MC-Numerical, which the paper attributes to arithmetic and unit-conversion errors rather than conceptual misunderstanding.

The STEM transition-based evaluation framework reports strong sample efficiency for relative positioning. Using balanced STS subsets of size 100, it places both LLaMA3-8B and GLM4-9B in the correct reference-family interval in 100% of trials, whereas random sampling places GLM4-9B correctly in 88% of trials and the Bayesian baseline has 0% accuracy through systematic overestimation (Hu et al., 16 Aug 2025). The same study documents high fractions of abnormal samples on existing benchmarks, including 65.85% on GPQA, 53.20% on SuperGPQA, and 47.93% on MMLU-Pro.

COMPOSITE-STEM presents a substantially harder frontier evaluation. Under the adapted multimodal Terminus-2 harness, claude-opus-4.6 achieves 21.4% Pass@1, gemini-3.1-pro 18.6%, grok-4.20-beta 5.7%, and gpt-5.4 4.3% (Waters et al., 10 Apr 2026). The paper attributes the low top score to the combination of expert-authored difficulty, open-ended rubric grading, nontrivial domain tools, a bare environment, and a max_turns=10 interaction budget. It also reports that Solution Error dominates failures for three of the four models.

scBench shows a different but similarly restrictive picture for scientific agents. Across 394 evaluations with three replicates each, Claude Opus 4.6 reaches 52.8% accuracy, Opus 4.5 49.9%, GPT-5.2 45.2%, and Gemini 2.5 Pro 29.2% (Workman et al., 9 Feb 2026). The study’s central empirical claim is that platform choice affects accuracy as much as model choice: cross-model mean accuracy varies by 32.7 percentage points across platforms, exceeding the 23.6-point spread across models. Normalization is the easiest task category, Differential Expression the hardest, and MissionBio the most difficult platform for every model tested.

6. Limitations, controversies, and future directions

The principal controversy surrounding STEM-Bench-style evaluation is whether conventional public benchmarks still measure the intended capability. The STEM transition paper argues that benchmark saturation, overfitting to public benchmarks, structural bias, and high evaluation cost make full-benchmark evaluation both expensive and less effective for distinguishing models (Hu et al., 16 Aug 2025). COMPOSITE-STEM makes a parallel argument from the frontier-evaluation side, noting that expert-grade benchmarks have saturated quickly and that exact-match-only QA misses important aspects of scientific reasoning and multi-validity outputs (Waters et al., 10 Apr 2026).

A second limitation concerns realism. SymPyBench explicitly filters out problems requiring figures or tables, which reduces multimodal realism even as it enables executable verification; the paper lists multimodal expansion, hallucination benchmarking under deliberate omission of critical variables, and broader STEM domains as future work (Imani et al., 5 Dec 2025). 3LM remains focused on middle- and high-school material, acknowledges synthetic biases from LLM-based generation, and notes that its code prompts preserve English identifiers and signatures (Boussaha et al., 21 Jul 2025). The original STEM benchmark covers pre-K through grade 8 and is English-only and U.S.-centric, leaving upper secondary, university-level, and non-English STEM outside its scope (Shen et al., 2024).

A third issue is grader validity. COMPOSITE-STEM mitigates single-judge idiosyncrasy through a five-model jury and audit artifacts, but the paper still notes that LLM-as-jury inherits any systematic biases of the panel and semantic prompts (Waters et al., 10 Apr 2026). scBench addresses grader brittleness through deterministic artifact scoring and tolerance calibration, yet it acknowledges that discretizing biological judgment into checkable artifacts can reject correct but uncommon workflows (Workman et al., 9 Feb 2026).

There are also reproducibility and contamination concerns. COMPOSITE-STEM warns that newly contributed and open-sourced tasks may become contaminated over time, while scBench withholds the full 394-evaluation suite and releases only 30 canonical evaluations to reduce contamination risk (Waters et al., 10 Apr 2026, Workman et al., 9 Feb 2026). SymPyBench reports iterative prompt refinement, code validation loss, and manual review, which improve quality but do not eliminate the possibility of synthetic artifacts (Imani et al., 5 Dec 2025).

A plausible implication is that the next phase of STEM-Bench will extend beyond question answering and even beyond agentic problem solving toward benchmarkable scientific preprocessing and instrumentation tasks. The 4D-PreNet paper is relevant in that respect: it isolates denoising, beam center calibration, and elliptical distortion correction in 4D-STEM, defines explicit corruption models and supervised targets, and reports concrete metrics such as more than 50% MSE reduction in denoising, sub-pixel center localization with average errors below 0.04 pixels, and approximately 7 minutes to preprocess a full $$D_j = \sigma_{S_j} \times \rho_{S_j,\log(P)}, \qquad w_j = \frac{D_j}{\sum_{j=1}^{m} D_j},$$2 4D-STEM cube on a GeForce RTX 4090 (Liu et al., 5 Aug 2025). Although that work is not itself an LLM benchmark, it exemplifies the shift toward executable, quantitatively verifiable scientific evaluation.

Taken together, the literature portrays STEM-Bench not as a single benchmark but as a rapidly diversifying evaluation regime. Its center of gravity has moved from static multiple-choice breadth toward dynamic parameterization, multimodality, executable verification, criterion-based grading, and realistic scientific workflows. The resulting benchmarks are more discriminative and more auditable, but they also expose how far current systems remain from robust, reliable scientific competence.