STEM2Code-Eval: Code-Grounded STEM Evaluation

• STEM2Code-Eval is a code-grounded evaluation framework that transforms STEM artifacts into executable code for precise assessment of visual and computational skills.
• It integrates domain-specific benchmarks such as CodePercept and SciEvalKit, employing execution rate, image fidelity, and code quality metrics to validate performance.
• The paradigm also embeds psychometric assessments like ACE to measure higher-order computational thinking and enhance diagnostic capabilities in STEM education.

Searching arXiv for papers mentioning STEM2Code-Eval and closely related work. STEM2Code-Eval denotes a code-grounded evaluation paradigm in which STEM understanding is assessed through executable code rather than only through natural-language description or downstream problem-solving accuracy. In recent arXiv literature, the term is used in multiple related senses: as a benchmark for STEM visual perception that requires Python-based image reconstruction in "CodePercept: Code-Grounded Visual STEM Perception for MLLMs" (Guan et al., 11 Mar 2026); as the scientific code generation evaluation pipeline within "SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence" (Wang et al., 26 Dec 2025); and as an integration target for the ACE assessment of higher-order computational thinking in block-based programming (Ghosh et al., 2024). A plausible unifying interpretation is that STEM2Code-Eval treats code as a verifiable intermediate representation for measuring STEM competence.

1. Terminological scope and core idea

Across the cited works, STEM2Code-Eval is associated with transforming STEM artifacts into executable representations and then evaluating correctness, completeness, or cognitive proficiency through those representations. In CodePercept, the evaluated object is a Python script that reconstructs a STEM image. In SciEvalKit, the evaluated object is generated scientific code tested through execution and validators. In the ACE integration report, the relevant unit is not free-form Python generation but a psychometrically validated assessment whose scores can be incorporated into a STEM2Code-Eval pipeline (Guan et al., 11 Mar 2026).

Context	What is evaluated	Verification mode
CodePercept	Python reconstruction code for STEM visuals	execution and expert scoring
SciEvalKit	scientific code generation on benchmark datasets	unit tests, validators, reports
ACE integration	higher-order CT skills in visual programming	psychometric analysis and score modeling

The underlying rationale differs by setting but remains structurally similar. CodePercept seeks to isolate perceptual ability from reasoning by requiring executable reconstruction code. SciEvalKit standardizes end-to-end evaluation of scientific code generation across datasets and models. ACE contributes a diagnostic layer for Analyzing, Evaluating, and Creating skills in block-based programming, with explicit recommendations for use as input to student-modeling modules and longitudinal growth models (Ghosh et al., 2024).

2. Benchmark formulation in CodePercept

In CodePercept, STEM2Code-Eval is introduced as a benchmark whose stated goal is to provide a curated set of STEM visuals with hand-verified Python reconstruction code so that MLLMs can be directly evaluated on their raw visual comprehension, independent of downstream deductive reasoning (Guan et al., 11 Mar 2026). The motivation is explicit: prior STEM benchmarks measure end-to-end problem solving, so failure modes are ambiguous, and caption-based proxies capture only problem-relevant facts rather than full visual understanding.

The benchmark covers several image classes: geometry, including solid and planar figures, orthographic projections, nets, and cross-sections; mathematical diagrams, including coordinate-plane graphs, function plots, bar and line charts; physics and chemistry schematics, including circuit diagrams and molecule sketches; and logic puzzles and combinatorial patterns, including domino and graph drawings. The task is to generate a single, self-contained Python script using matplotlib and standard scientific libraries that, when executed, produces a pixel-perfect recreation of the original image. Required elements include imports, a main drawing function, plt.show(), all geometric primitives, exact styling, and full layout.

The construction pipeline is also specified. Source images are drawn from the test splits of six VQA-style STEM benchmarks: DynaMath, LogicVista, MathVerse, MathVision, MathVista, and WeMath. Candidate references are produced through a multi-stage procedure: auto-caption and auto-code generation using Gemini 2.5 Pro and Claude 4.1; an iterative render-repair loop with image-similarity scoring; candidate ranking; and human expert 5-point scoring on style, content, and functionality. The selected release contains 1,000 images, each paired with a human-refined, pixel-perfect Python script.

This design makes the benchmark more demanding than problem-answering benchmarks. Success depends not only on recognizing the main semantic object in an image but on recovering points, lines, labels, colors, spatial relations, axes limits, aspect ratios, annotations, and typography with enough precision for faithful reconstruction. That requirement is central to CodePercept’s claim that code can serve as a perceptual medium rather than merely a reasoning output.

3. Metrics, empirical results, and benchmark limitations

CodePercept defines three principal metrics over the 1,000-example evaluation set (Guan et al., 11 Mar 2026). Execution Rate measures the fraction of generated scripts that run to completion and produce an image file:

$$\mathrm{ExecRate} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\bigl[\text{script}_i\ \text{executes without error}\bigr] \times 100\%.$$

Image Score, denoted $S_{\mathrm{img}}$, is the average expert similarity rating on a 0–100 scale comparing rendered output against the ground-truth image according to geometric completeness, positional accuracy, annotation fidelity, style consistency, and legibility:

$$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$

Code Score, denoted $S_{\mathrm{code}}$, is the average expert rating on a 0–100 scale of the script itself, focusing on pixel-level equivalence of the rendered output irrespective of implementation differences, plus code completeness and clarity:

$$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$

The paper notes that a combined average for summary tables is often reported either with a convention based on $S_{\mathrm{img}}$, $S_{\mathrm{code}}$, and $\mathrm{ExecRate}$, or simply as the arithmetic mean of the three. Concrete examples in the benchmark description include a right-triangle reconstruction using matplotlib.patches.Polygon and a three-bar chart rendered with plt.bar, with verification proceeding through execution plus expert comparison.

Reported results indicate that CodePercept-trained models outperform strong open-source baselines on STEM2Code-Eval. For 8B models, CodePercept-8B-R1 is reported at ExecRate $\approx 93.4\%$, ImageScore $\approx 50.3$, and CodeScore $S_{\mathrm{img}}$0, compared with approximately $S_{\mathrm{img}}$1, $S_{\mathrm{img}}$2, and $S_{\mathrm{img}}$3 for Qwen3-8B. At larger scale, CodePercept-32B-R1 reaches ExecRate $S_{\mathrm{img}}$4, Image $S_{\mathrm{img}}$5, and Code $S_{\mathrm{img}}$6. The paper interprets these results as evidence that the code-grounded paradigm directly addresses STEM perceptual hallucinations by enforcing exact quantitative and spatial semantics.

The stated limitations are equally important. Construction overhead is high because manual annotation and refinement of 1,000 code scripts is expensive. Domain coverage is incomplete, particularly for niche STEM visuals such as 3D molecular surfaces and advanced circuit layouts. Evaluation retains some subjectivity because LLM-based scoring is used for fine-grained assessment. The benchmark is also model-dependent in that it assumes fluency in Python and matplotlib; other rendering toolkits, including TikZ and Plotly, are not yet supported.

4. SciEvalKit’s STEM2Code-Eval pipeline

Within SciEvalKit, STEM2Code-Eval is presented as a structured end-to-end evaluation pipeline for scientific code generation (Wang et al., 26 Dec 2025). Its architecture is divided into a Dataset Layer, a Preprocessing stage, a Model Inference Layer, an Evaluation Testing Layer, and a Report Storage Layer.

At the dataset layer, build_dataset("SciCode") and build_dataset("AstroVisBench") consult a registry in supported_text_datasets and instantiate SciCodeDataset or AstroVisBenchDataset. Each dataset class inherits from TextBaseDataset or a specialized CodeGenBaseDataset and implements load_metadata() to read TSV or JSON lines with problem descriptions, reference code, and unit-test definitions, together with build_prompt(sample) to emit a standardized multi-part message containing the natural-language problem statement and any scaffolding code stubs. Preprocessing includes discipline-aware prompt templates and code cleaning steps such as escaping backticks and normalizing indentation.

At the inference layer, build_model_from_config("gpt-4-code", **kwargs) reads settings such as API versus local vLLM deployment, temperature, max_tokens, and retry_policy from YAML or JSON configuration. The .generate(message, dataset_name, seed=[SEED](https://www.emergentmind.com/topics/semantic-enhanced-efficient-denoiser-seed)) interface returns raw text containing the model’s code answer. infer_data(...) orchestrates batching, parallelism, temperature control, and a fixed random seed for reproducibility.

The evaluation layer executes generated code in a Docker-based sandbox or Python subprocess with limited permissions. Dataset.evaluate(predictions_file) stitches each generated snippet into a .py script or Jupyter cell and runs official unit-test suites using pytest or unittest. Dependencies are benchmark-specific: SciCode tasks import numpy and scipy, while AstroVisBench tasks import astropy and matplotlib. Result aggregation produces test_results.json, metrics.csv or metrics.json, and a final XLSX report with separate sheets for per-benchmark metrics, per-model metrics, and raw logs.

SciEvalKit currently enumerates two supported code-generation benchmarks. SciCode is a multidisciplinary benchmark with 80 research-grade tasks spanning physics, chemistry, biology, materials science, and mathematics, each with a natural-language description, a Python reference solution, and unit tests. AstroVisBench is an astronomy benchmark with 110 Jupyter-notebook-style problems requiring data download, astropy tables, and plotting; it is evaluated through visual-output comparison, such as pixel match or chart properties, together with code tests. Planned or community extensions include Chemical Reaction Modeling, Materials Science Simulation, and Physics Simulation.

The formal metrics include Exact Match Rate, Execution Accuracy, Unit-Test Pass Rate, CodeBLEU, relative error for numeric outputs, and Scientific-Correctness Score. Exact Match Rate and Execution Accuracy are defined as

$S_{\mathrm{img}}$7

and

$S_{\mathrm{img}}$8

Unit-Test Pass Rate is defined over all tests across all samples:

$S_{\mathrm{img}}$9

CodeBLEU is described as borrowed from CodeBLEU and as combining $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$0-gram, AST, data-flow match, and weighted $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$1-gram components. Relative error is defined for numeric results, and Scientific-Correctness Score is defined through custom validators such as an energy-conservation check.

A distinctive feature of SciEvalKit’s formulation is reproducibility. All sampling calls carry a fixed seed parameter; batching and shuffling honor a global SEED; every run creates a timestamped directory; logs include stdout.log, stderr.log, inference_trace.json, test_results.json, and metrics.json; benchmarks live in Git submodules tagged by version; and each run stamps the benchmark submodule’s git SHA into run_metadata.json. This makes STEM2Code-Eval, in the SciEvalKit sense, a benchmarking infrastructure as much as a benchmark.

5. ACE as a psychometric component for STEM2Code-Eval

The paper "Analyzing-Evaluating-Creating: Assessing Computational Thinking and Problem Solving in Visual Programming Domains" presents ACE and, in its detailed report, frames the instrument as designed for integration into the STEM2Code-Eval pipeline (Ghosh et al., 2024). ACE is a novel test focusing on the three higher cognitive levels in Bloom’s Taxonomy: Analyze, Evaluate, and Create. It comprises 21 single-correct multiple-choice questions completed in one 45-minute session, divided equally into seven Analyzing items, seven Evaluating items, and seven Creating items.

All items are grounded in the Hour of Code: Maze Challenge block-based visual programming environment. The tasks use a grid world in which an Avatar must navigate, using sequencing, loops, and conditionals, to a Goal without colliding with Walls. Analyzing items ask students to trace or check given solution code on one or more grids. Evaluating items present buggy or alternative codes and require students to identify errors or equivalences. Creating items require synthesis of new grid configurations by designing Avatar start positions, wall placements, or goal layouts that satisfy a given code.

The item pool was constructed to span sequencing, loops, conditionals, debugging, equivalence, and creative design while targeting the three higher cognitive levels of Bloom’s Taxonomy. Expert review and think-aloud pilots involved five CS educators and six students in grades 3–6, with iterative refinement of wording, visual complexity, and difficulty.

Psychometric analysis is central to ACE’s relevance. Internal consistency across all 21 items is defined through Cronbach’s $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$2:

$$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$3

with $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$4. The observed overall $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$5 is $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$6. Subscale alphas are $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$7 for Analyzing, $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$8 for Evaluating, and $$S_{\mathrm{img}} = \frac{1}{N}\sum_{i=1}^{N} s_i, \qquad s_i\in[0,100].$$9 for Creating. Under a 1-Parameter Rasch model, person reliability is $S_{\mathrm{code}}$0 with $S_{\mathrm{code}}$1.

Construct validity is reported through confirmatory factor analysis of a three-factor model with items loading on Analyzing, Evaluating, and Creating. Fit indices are RMSEA $S_{\mathrm{code}}$2, CFI $S_{\mathrm{code}}$3, and TLI $S_{\mathrm{code}}$4. Composite reliability for each factor is defined as

$S_{\mathrm{code}}$5

and the estimated $S_{\mathrm{code}}$6 exceeds $S_{\mathrm{code}}$7 for all three factors. Convergent validity is supported by a Pearson correlation of $S_{\mathrm{code}}$8 with Hour of Code Maze score, with $S_{\mathrm{code}}$9, and by inter-category correlations above $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$0, also with $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$1.

The results summary shows an overall mean ACE score of $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$2 with $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$3. Grade-level means are $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$4 in grade 3, $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$5 in grades 4, 5, and 6, and $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$6 in grade 7. Subscale means out of seven are $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$7 for Analyzing, $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$8 for Evaluating, and $$S_{\mathrm{code}} = \frac{1}{N}\sum_{i=1}^{N} c_i, \qquad c_i\in[0,100].$$9 for Creating. Item difficulty ranges from Q02, the easiest item with $S_{\mathrm{img}}$0 and approximately $S_{\mathrm{img}}$1 success, to Q17, the hardest item with $S_{\mathrm{img}}$2 and approximately $S_{\mathrm{img}}$3 success. Q17 is flagged as problematic because of a low factor loading of $S_{\mathrm{img}}$4 with $S_{\mathrm{img}}$5 and very high difficulty; revision is recommended.

For STEM2Code-Eval specifically, the report recommends embedding ACE items or analogous task designs into the platform, using the three subscale scores as input features in student-modeling modules, leveraging IRT-based difficulty calibration to personalize item selection, and using repeated administrations for longitudinal tracking across grades 3–7. This makes ACE a psychometric complement to code-based evaluation, extending the framework from benchmark scoring toward learner diagnosis.

6. Intellectual antecedents, common themes, and points of caution

A useful antecedent is PySTEMM, which presents executable concept modeling for K–12 STEM learning through immutable objects and pure functions in Python (D'Souza, 2014). PySTEMM is organized around a Tool that traverses concept classes and instances and a Model Library containing user-written concept definitions, instances, visualization templates, and pure-function helpers. The system produces pictures, narrative descriptions, animations, and graph plots from a single conceptual model, and it demonstrates examples in mathematics, chemistry, physics, and engineering.

PySTEMM does not use the term STEM2Code-Eval, nor does it report a formal user study or quantitative K–12 evaluation. Its relevance is conceptual rather than terminological. The paper presents a proof-of-concept tool in which executable models serve as a common substrate for multiple STEM representations. This suggests an early form of the broader “STEM-to-code” idea later operationalized in evaluation settings.

Several common themes run across the later STEM2Code-Eval usages. First, code functions as a structured, executable representation of STEM artifacts: a rendered figure in CodePercept, a tested scientific program in SciEvalKit, or a block-based program interpretation in ACE. Second, verifiability is emphasized: execution success, unit tests, validators, IRT calibration, and factor-analytic evidence all serve as mechanisms for making evaluation more determinate. Third, the framework shifts attention from final answers to representational fidelity. In CodePercept, a model may understand only the answer-relevant part of a figure yet still fail the reconstruction task. In ACE, a learner may solve routine items yet perform differently on creating new valid task grids.

A common misconception would be to treat STEM2Code-Eval as a single fixed benchmark. The current literature instead uses the term for a family of related evaluation mechanisms: a visual-perception benchmark, a scientific code-generation pipeline, and an educational assessment integration target. Another possible misconception is that code-grounded evaluation removes all ambiguity. The evidence does not support so strong a claim. CodePercept explicitly notes evaluation subjectivity in LLM-based scoring and gaps in domain coverage; SciEvalKit relies on benchmark-specific tests and validators whose adequacy depends on task design; and ACE identifies at least one problematic creating item, Q17, whose loading and difficulty warrant revision (Guan et al., 11 Mar 2026).

Taken together, these works position STEM2Code-Eval as a convergence point between benchmark engineering, executable representation, and educational measurement. In one line of work, it isolates perceptual competence through image reconstruction code; in another, it standardizes scientific code-generation evaluation; and in a third, it incorporates validated computational-thinking diagnostics into a broader assessment pipeline.