---
title: 'ASI-Bench: Benchmark for Evaluating AI in Scientific Research'
url: https://www.emergentmind.com/topics/asi-bench
type: topic
---

# ASI-Bench: Benchmark for Evaluating AI in Scientific Research

ASI-Bench is a benchmark for evaluating whether artificial-intelligence systems can conduct project-level scientific research with progressively decreasing methodological guidance. Introduced in “ASI-Bench: At the Dawn of Artificial Superintelligence” [2608.17271], it targets capabilities beyond retrieval and application of existing human knowledge: innovative exploration, method selection, autonomous scientific execution, creation of new task-relevant results, and production of verifiable scientific artifacts. The benchmark contains 60 research projects across 11 scientific domains and evaluates systems under four matched conditions, ranging from a fully prescribed procedure to independent workflow construction with scientifically plausible distractors.

## 1. Purpose and conceptual scope

ASI-Bench is designed around the distinction between executing a known scientific procedure and independently determining how research should be conducted. Existing evaluations commonly test question answering, scientific coding, data analysis, research replication, machine-learning engineering, tool use, hypothesis generation, or open-ended work within a fixed task specification. ASI-Bench instead combines cross-domain research, methodological autonomy, end-to-end execution, verifiable outputs, and a controlled withdrawal of human guidance.

The benchmark operationalizes progress toward artificial superintelligence through four capabilities:

1. **Innovative exploration**: investigating unfamiliar data or scientific problems rather than answering questions with known solutions.
2. **New-knowledge or new-result creation**: constructing models, generating predictions, extracting scientific structure, or producing other scientifically meaningful outputs not copied from the prompt.
3. **Autonomous scientific execution**: understanding an objective, inspecting data, selecting or reconstructing a method, implementing it, running experiments, diagnosing failures, refining the workflow, analyzing results, and generating artifacts.
4. **Verifiable scientific results**: supporting claims through executable code, numerical outputs, files, visualizations, deterministic reference generation, validity gates, and task-specific scorers.

The benchmark does not claim to constitute a complete test of artificial superintelligence. Its more specific target is the transition from procedure-following scientific assistants to systems capable of independent, end-to-end, verifiable research.

The benchmark’s conception is consistent with the distinction between artificial general intelligence and artificial general superintelligence described in “From AGI to ASI” [2606.12683]. That report characterizes ASI as artificial general intelligence with superhuman abilities across virtually all tasks and domains of human interest, potentially exceeding large, well-coordinated collectives of human experts. ASI-Bench addresses only selected capabilities relevant to that broader notion, particularly scientific autonomy, innovation, and execution.

## 2. Dataset construction and scientific domains

ASI-Bench contains 60 project-level research tasks spanning:

1. mathematics;
2. physics;
3. chemistry;
4. biology;
5. astronomy;
6. materials science;
7. earth science;
8. medicine and biostatistics;
9. computer science;
10. robotics; and
11. electrical engineering.

The benchmark was constructed from more than 1,300 candidate research ideas. More than 40 experts participated in construction and review, while 21 researchers contributed tasks retained in the final benchmark. The process involved more than 31,000 human-hours, five review rounds, more than 1,100 review assignments, more than 2,000 task revisions, and more than 1,500 sandbox validation runs.

Each task preserves the same scientific objective, input data, required outputs or artifacts, and evaluation criteria across the four benchmark conditions. The changing variable is the amount and specificity of methodological information supplied to the agent.

The tasks are intended to represent complex scientific investigations rather than isolated questions. They can require multiple interdependent decisions, iterative refinement, recovery from failed attempts, numerical or computational experimentation, and validation. The benchmark does not provide a domain-by-domain task-count table or domain-specific scores in the supplied material.

Task construction incorporates several safeguards:

- **Expert review** examines scientific logic, prompt information, reference artifacts, scoring code, information leakage, unintended shortcuts, and agent trajectories.
- **AI-assisted auditing** supplements human cross-review.
- **Sandbox execution** checks runtime stability, reference reproducibility, artifact generation, scorer behavior, environment consistency, and execution consistency.
- **Scorer validation** relies on explicit validity checks, weighted scoring criteria, declared dependencies, local-testing evidence, runtime information, sandbox provenance, and scorer-replay evidence.

Task contributors are expected to provide a reproducible reference-generation procedure, such as a `generate_gt.py` script, together with dependencies and local evaluation evidence across all four conditions.

## 3. The B1–B4 guidance gradient

The principal experimental design is a four-level methodological-guidance gradient.

| Setting | Information supplied | Primary capability |
|---|---|---|
| **B1** | Scientific background, governing equations or method, implementation procedure, numerical choices, and often parameter choices | Executing a complete prescribed procedure |
| **B2** | Methodological background and intended method class, but not the full procedure | Converting a specified method into a working workflow |
| **B3** | Objective, inputs, constraints, and required outputs only | Independently selecting a method and constructing a workflow |
| **B4** | B3 information plus scientifically plausible but task-irrelevant distractors | Autonomous research under irrelevant information |

### B1: complete methodological guidance

B1 may specify the governing equation, mathematical formulation, numerical representation, solver, discretization, parameter values, implementation steps, and validation requirements. The agent must still write or execute the workflow and produce the required artifacts, but it is not expected to discover the research procedure.

### B2: method specified, procedure withdrawn

B2 retains the intended method or method family while removing detailed implementation instructions. Agents must reconstruct parameter handling, intermediate analyses, implementation decisions, and the precise executable sequence. This setting tests whether a system can transform methodological knowledge into a complete research workflow.

### B3: method selection required

B3 specifies only the scientific objective, available inputs, constraints, and required outputs. The agent must determine the relevant model or explanation, select a method or solver, implement it, analyze the data, validate the result, and produce the required artifacts. The benchmark does not disclose the intended algorithm, solver, or effective evolution strategy.

### B4: autonomy with distractors

B4 retains the B3 requirements and adds unrelated but scientifically plausible model families, numerical methods, analysis techniques, misleading preliminary suggestions, and unrelated background information. The condition tests whether agents remain focused on task evidence while choosing and executing an appropriate workflow.

The B1–B4 structure separates three capabilities that are often conflated: following a complete research recipe, operationalizing a named method, and independently designing the research procedure. B4 adds robustness to irrelevant information without changing the underlying objective or evaluation criteria.

## 4. Research workflow and deliverables

A typical ASI-Bench project requires a complete research loop:

1. understanding the scientific objective;
2. inspecting and analyzing input data;
3. generating a hypothesis, model, or research idea;
4. selecting a method;
5. implementing the method;
6. running experiments or simulations;
7. diagnosing failure and refining the approach;
8. analyzing results scientifically; and
9. producing executable and scientifically verifiable artifacts.

The benchmark’s concrete tasks are built around task data, scientific objectives, executable environments, and reference artifacts. The supplied description does not require a separate literature-search subscore or external literature access for every task.

### Representative dynamical-systems task

One case study concerns **2D Anisotropic Stiff Dynamics**. The agent receives:

- `system_info.json`, containing system parameters and timing metadata;
- `field_evolution.npy`, containing observed spatio-temporal field snapshots; and
- `initial_condition.npy`, containing the initial field for prediction.

The objective is to characterize spatial and temporal dynamics, construct a model reproducing the observations, predict the field at a future target time, and extract physically meaningful diagnostics. Required outputs include a predicted field, a spatial spectrum, physical diagnostics, scientific visualizations, data-analysis results, and complete executable simulation code.

Under B1, the prompt reveals an anisotropic conserved Kuramoto–Sivashinsky-type partial differential equation, Fourier pseudospectral discretization, dealiasing, and ETDRK4 time integration. Under B3, these modeling and numerical details are withheld. The task therefore tests whether the agent can infer a suitable representation and solver from the data and objective rather than merely implement a specified method.

The benchmark’s deliverables are designed to make scientific claims inspectable. A persuasive narrative is insufficient if the code fails, outputs are invalid, or predictions do not satisfy task-specific validity checks. At the same time, executable verification does not fully establish genuine novelty, causal understanding, or scientific importance.

## 5. Scoring and evaluated systems

ASI-Bench assigns a scientific score to each task and guidance level. Scores are macro-averaged over the 60 tasks and are generally averaged over three independent runs. The reported results include B1, B2, B3, B4, an overall score, and differences between guidance conditions.

The supplied material states that scoring uses task-specific validity checks, weighted scoring criteria, and executable reference artifacts. It does not provide the complete per-dimension rubric, numerical weights, normalization equation, gate implementation, formal definition of the overall score, or task-level artifact-combination formula.

The benchmark evaluates 18 agent–model configurations using Codex, Claude Code, Kimi Code, MiMo Code, and OpenHands. The reported backbone models include GPT-5.5, GPT-5.6 Sol, Claude Opus 5, Kimi K3, Claude Opus 4.8, GLM-5.3, GLM-5.2, DeepSeek V4 Flash, Kimi K2.7, MiniMax M3, DeepSeek V4 Pro, and MiMo V2.5 Pro.

The main results are reported without external tool access. Agents nevertheless use the task’s executable environment for coding and sandbox execution. Average resource requirements include 4.35 million tokens and 37.8 minutes per task for B1, and 6.91 million tokens and 49.7 minutes per task for B2. B3 uses approximately 25% more tokens than B1 and 22% more execution time, while B4 uses approximately 30% more tokens and 18% more execution time. Claude Opus 5 is reported from one run; the other configurations generally use three independent runs.

### Overall scores

| Condition | Mean score |
|---|---:|
| B1: full methodological guidance | **50.91** |
| B2: method specified, procedure removed | **29.10** |
| B3: method must be determined | **26.62** |
| B4: B3 plus distractors | **26.99** |
| Overall | **33.41** |

The B2–B1 difference is **−21.82** points, while the B3–B1 difference is **−24.29** points. The B3-to-B4 difference is **+0.36** points. Thus, the largest decline occurs when detailed procedure is withdrawn. Removing the named method produces a further decline from 29.10 to 26.62, but this decline is smaller than the loss caused by removing the procedural recipe.

The strongest configuration is Codex with GPT-5.6 Sol ultra:

- B1: **71.78**;
- B2: **49.57**;
- B3: **51.60**;
- B4: **50.41**; and
- Overall: **55.84**.

It is the only evaluated system to exceed 50 on B3. Increasing GPT-5.6 Sol from xhigh to ultra raises B3 from 40.86 to 51.60, a gain of 10.74 points.

The results also show harness effects. MiMo V2.5 Pro scores 16.17 with MiMo Code and 23.25 with Claude Code. Kimi K2.7 scores 19.72 with Kimi Code and 27.34 with Claude Code. Kimi K3 scores 36.22 with Kimi Code and 37.09 with Claude Code. These comparisons indicate that measured scientific capability depends on the interaction between backbone model and agent harness, not solely on the underlying model.

Cost is not a reliable proxy for scientific performance. GPT-5.6 xhigh with Codex achieves a B3 score of 40.86 at approximately \$684 per run; Claude Opus 5 with Claude Code achieves 40.70 at approximately \$2,728 per run; and GPT-5.6 ultra with Codex achieves 51.60 at approximately \$1,550 per run.

## 6. Findings, limitations, and future development

The central empirical finding is a substantial guidance dependence. Systems perform considerably better when humans specify the scientific procedure than when they must reconstruct or determine it. This pattern suggests that current agents possess greater capability for executing explicit research recipes than for decomposing an open-ended objective into a robust, executable scientific workflow.

The relatively small average difference between B3 and B4 suggests that irrelevant information is not the primary bottleneck in the reported experiments. The principal difficulty is scientific workflow construction, including procedural knowledge, implementation choices, intermediate validation, parameter handling, error recovery, and translation of abstract scientific goals into executable operations.

The benchmark’s design has several limitations:

- **Coverage**: 60 tasks across 11 domains cannot represent the breadth of scientific research.
- **Contamination risk**: public release through a website, GitHub repository, Hugging Face dataset, and leaderboard may expose prompts, artifacts, or solutions to future models.
- **Shortcut risk**: agents might exploit predictable file names, artifact formats, scorer tolerances, reference-like outputs, or task-specific computational shortcuts.
- **Scoring transparency**: the supplied material does not include all task-level weights, equations, or gate definitions.
- **Scientific validity**: executable checks may not fully measure novelty, causal understanding, interpretation quality, or the importance of a result.
- **Cost**: millions of tokens, extended execution times, and multiple independent runs make repeated evaluation expensive.
- **Model–harness confounding**: results reflect the combined effect of model, agent framework, prompting, and execution policy.
- **Limited error decomposition**: the reported material emphasizes aggregate scores and guidance differences rather than a complete per-domain or per-stage error taxonomy.
- **External-tool restriction**: the main experiment excludes external tool access, which may understate systems designed for literature retrieval, specialized scientific databases, laboratory control, or broader research environments.

Future development is intended to be community-driven. Proposed directions include adding scientific domains, increasing methodological diversity, introducing underrepresented research capabilities, improving reference artifacts, refining failure analysis, evaluating new agent architectures, and maintaining an open leaderboard and task-contribution process.

A complete contributed task is expected to contain a non-trivial scientific objective, four B1–B4 prompt variants, explicit input and output specifications, a reproducible reference-generation procedure, validity checks, weighted scoring criteria, software dependencies, and local evaluation evidence across all four conditions.

ASI-Bench’s principal significance lies in making methodological autonomy measurable within a common project. Its results indicate that current AI systems remain substantially more capable of following detailed scientific procedures than of independently selecting, constructing, validating, and executing such procedures. The benchmark therefore shifts attention from performance on known-answer problems toward the more demanding question of whether an AI system can generate scientifically meaningful, executable, and verifiable results when human methodological guidance is progressively withdrawn.

Source: https://www.emergentmind.com/topics/asi-bench