---
title: 'Composite-STEM: Frontier AI STEM Benchmark'
url: https://www.emergentmind.com/topics/composite-stem
type: topic
---

# Composite-STEM: Frontier AI STEM Benchmark

COMPOSITE-STEM is a frontier STEM benchmark for AI agents that comprises 70 expert-written tasks in physics, chemistry, biology, and mathematics. It is designed for execution in Harbor/Terminal-Bench-style agent environments with multimodal support, and it combines exact-match grading with criterion-based semantic evaluation under a five-model LLM-jury protocol. Its stated purpose is to address two recurrent weaknesses of prior evaluations: saturation and contamination in static short-answer benchmarks, and an overreliance on tightly constrained outputs that fail to capture scientifically meaningful partial progress in realistic workflows [2604.09836].

## 1. Definition and evaluative scope

COMPOSITE-STEM is framed as an agentic benchmark rather than a conventional question-answer set. The benchmark targets end-to-end STEM problem solving, including long-horizon reasoning, derivations, modeling, experimental design, and multimodal interpretation of reference materials such as figures and images. In this respect, it evaluates not only terminal answers but also whether an agent can navigate a realistic computational environment, use tools, and return outputs that satisfy domain-specific scientific criteria [2604.09836].

The benchmark’s design reflects an explicit shift away from narrow exact-answer regimes. Earlier benchmark styles are described as vulnerable to contamination and saturation, while many rigid-output formats underrepresent partial correctness and methodological adequacy. COMPOSITE-STEM therefore treats scientifically meaningful outputs as first-class evaluation objects, particularly in tasks where correctness cannot be reduced to a single scalar or string without discarding essential structure [2604.09836].

A plausible implication is that COMPOSITE-STEM functions simultaneously as a capability benchmark and as an interface benchmark. It tests not only whether a model “knows” relevant STEM content, but whether an agent can operationalize that knowledge under containerized execution, tool constraints, and rubric-based assessment.

## 2. Task composition and curation

The benchmark contains 70 tasks distributed across four domains:

| Domain | Task count |
|---|---:|
| Physics | 20 |
| Chemistry | 23 |
| Biology | 20 |
| Mathematics | 7 |

The task set was curated by doctoral-level researchers, faculty, postdocs, and industry practitioners with prior benchmark-design experience. Curation was iterative and used Portex Datalab: experts drafted prompts and rubrics, ran frontier models live to inspect failure modes, received detailed trial reports, and then refined both tasks and scoring criteria across multiple review cycles. Tasks were retained when they showed clear discriminative power on state-of-the-art agents and when rubric intent remained unambiguous [2604.09836].

The benchmark is explicitly multimodal. Eighteen of the 70 tasks include reference files mounted under `/app/refs`; 17 of these are image files and one is a PDF. Physics tasks include symbolic derivations, quantum and open-system reasoning, many-body and topological analyses, and device-level or measurement-grounded calculations. Chemistry tasks include synthetic mechanism reasoning, SMILES/InChI conversion, spectroscopy interpretation, and concept-heavy judgment. Biology tasks include imaging diagnosis, MRI engineering reasoning, electron microscopy interpretation, spatial-structure inference, and mechanism-aware analysis. Mathematics tasks emphasize proof-oriented derivation, combinatorial and algebraic reasoning, stochastic-process analysis, and invariant or structural computation [2604.09836].

This composition suggests a benchmark philosophy centered on disciplinary heterogeneity rather than a single dominant skill axis. The inclusion of multimodal biology and chemistry items alongside proof-oriented mathematics and derivational physics makes the benchmark structurally resistant to optimization for any one narrow response pattern.

## 3. Execution environment and agent substrate

COMPOSITE-STEM is executed in Harbor using an adapted multimodal Terminus-2 harness. Each task bundle follows a structured layout that includes `instruction.md` for the prompt, `tests/criteria.json` for grading rubrics, `tests/test.sh` for verifier entry, `app/refs` for reference assets, and optionally `solution/solve.sh`. When a task includes references, the multimodal harness attaches those assets to the first model turn; images are passed natively, while text files are inlined as text [2604.09836].

The reported execution substrate is a Modal sandbox with a single-container trial orchestration model. The runtime envelope is fixed at a 3600 s agent loop timeout, a 1200 s build timeout, 1 CPU, 2048 MB memory, and 10 GB storage. The base environment uses Python 3.12 slim, includes common shell utilities, and installs `litellm>=1.67.0`; agents may further install domain packages at runtime. The paper notes observed use of packages such as `rdkit-pypi`, `scipy`, `sympy`, `numpy`, and `googlesearch-python` [2604.09836].

The benchmark therefore evaluates agents under a constrained but nontrivial systems setting. It is neither a bare language-only protocol nor a fully unrestricted workstation environment. This intermediate regime is technically important: it exposes failures in setup, package management, and workflow planning that are invisible in static benchmark formats.

## 4. Grading architecture: exact match, rubrics, and LLM jury

COMPOSITE-STEM uses the AsymmetryZero grading protocol, which mixes exact-match checks and criterion-based semantic grading. Of the 70 tasks, 35 are graded with exact match, 34 with LLM-jury semantic grading, and one uses a hybrid format. Rubrics contain between 1 and 40 criteria, with an average of 2.6 criteria per task. Each criterion has a weight, a `grader_type` in `{ExactMatch, llm-judge}`, and a `semanticPrompt` for jury-based evaluation [2604.09836].

For `llm-judge` criteria, the jury consists of five models: DeepSeek-v3.2, GLM-5, GPT-oss-120b, Llama-3.3-70b-instruct, and Kimi-K2.5. Each juror receives the task prompt, the candidate submission, and the criterion-specific semantic prompt, and returns a binary pass/fail decision with optional rationale. The criterion outcome is determined by strict majority vote. The raw task score is the weighted sum of criterion outcomes,
$$
S_{\mathrm{raw}} = \sum_{k \in C} w_k r_k,
$$
with normalized Harbor reward
$$
S = \frac{1}{100}\sum_{k \in C} w_k r_k.
$$
Here, $r_k \in \{0,1\}$ is the result for criterion $k$, and the weights are normalized to a 0–100 task scale [2604.09836].

The protocol is designed to preserve auditability. Verifiers emit both `reward.json`, containing the scalar score, and `portex_detail.json`, which stores criterion-level outcomes, judge votes, and rationales. This makes semantic grading inspectable rather than opaque. In exact-match settings, the verifier may normalize a terminal answer format; one chemistry example requires a final line of the form `Answer: 350` [2604.09836].

A common misconception is that rubric-based judging necessarily sacrifices rigor relative to exact match. The benchmark’s architecture rejects that dichotomy: exact match is retained where a unique terminal object is appropriate, while semantic judging is reserved for outputs whose scientific quality depends on validity, completeness, or methodological soundness rather than literal string identity.

## 5. Empirical performance and observed failure modes

The benchmark was evaluated with four frontier models. The reported Pass@1 results are as follows:

| Model | Pass@1 | Avg. time | Avg. episodes |
|---|---:|---:|---:|
| claude-opus-4.6 | 21.4% | 11m30s | 5.6 |
| gemini-3.1-pro | 18.6% | 9m16s | 6.7 |
| gpt-5.4 | 4.3% | 6m05s | 2.6 |
| grok-4.20-beta | 5.7% | 6m03s | 3.3 |

The top-performing model, `claude-opus-4.6`, reaches 21.4% Pass@1, which is presented as evidence that the benchmark remains beyond current frontier-agent reach. The performance gap between the top two models and the weaker pair is reinforced by trajectory statistics: stronger models used more steps, invoked tools more often, and produced submissions that were more likely to be partially correct rather than plainly invalid [2604.09836].

Failure modes are divided into `Solution Error`, meaning a graded but incorrect submission, and `Submission Error`, meaning a missing or invalid output, including runs that exhausted the maximum turn budget of 10. For `claude-opus-4.6`, 54 failed trials were reported, with 63.0% classified as solution errors and 37.0% as submission errors. For `gemini-3.1-pro`, 56 failed trials were reported, with 46.4% solution errors and 53.6% submission errors. `gpt-5.4` and `grok-4.20-beta` failed predominantly via solution errors, at 90.9% and 83.3% respectively [2604.09836].

The paper also reports judge-vote patterns for LLM-graded criteria. Stronger models produced more borderline outputs and thus more nontrivial 3–2 vote splits and higher average vote entropy, whereas weaker models more often elicited near-unanimous rejection. This suggests that COMPOSITE-STEM is sensitive not only to outright correctness but also to gradations of partial scientific adequacy [2604.09836].

An illustrative chemistry case involves counting hydrogens from a large SMILES string. One successful trajectory installed RDKit, parsed the molecule, derived the formula `C280H350Br5Cl2I2N23O75S3`, and returned `Answer: 350`. A weaker trajectory instead built an ad hoc SMILES parser and produced `Answer: 399`. The contrast is diagnostically important: the benchmark rewards credible domain workflow, not merely verbal plausibility [2604.09836].

## 6. Benchmark position, reproducibility, and limitations

COMPOSITE-STEM is situated within a broader shift from static QA benchmarks toward agentic, workflow-level evaluation. Unlike short-form evaluations that emphasize exact match or multiple choice, it foregrounds containerized execution, multimodal inputs, flexible outputs, and criterion-level semantic assessment. Its closest methodological affinities lie with recent agent benchmarks and rubric-based professional evaluations, but its distinctive contribution is the combination of expert-authored frontier STEM tasks, Harbor-based execution, and an auditable five-model jury protocol [2604.09836].

The benchmark is open-sourced. Tasks are hosted on Hugging Face, the Harbor adapter is available on GitHub, and the AsymmetryZero framework is released under the MIT License. This supports reproducibility at the levels of task content, verifier behavior, and criterion-level audit traces. The paper’s intended use case is comparative evaluation of frontier AI agents under fixed environmental constraints rather than informal leaderboard gaming [2604.09836].

The benchmark also states several limitations. It is not a formal external-audit consortium benchmark. It uses a fixed maximum of 10 turns, so longer-horizon runs could alter results. Reported scores are Pass@1 only; repeated trials would be needed to characterize variance. The grading system still depends in part on LLM-as-judge evaluation, although this dependence is mitigated through five-model majority voting, explicit semantic prompts, and preserved vote logs. A plausible implication is that COMPOSITE-STEM should be interpreted less as a final authority on absolute capability and more as a high-resolution instrument for frontier comparative analysis [2604.09836].

In that role, COMPOSITE-STEM occupies a specific niche in the evaluation literature: it measures whether an AI agent can execute realistic STEM problem solving under operational constraints, return scientifically meaningful outputs, and withstand criterion-level scrutiny when exact-match simplifications are no longer adequate.

Source: https://www.emergentmind.com/topics/composite-stem