ASI-Bench: At the Dawn of Artificial Superintelligence

Abstract: Artificial superintelligence (ASI) requires AI to move beyond mastering existing knowledge toward exploring the unknown, creating new knowledge, and turning new ideas into verifiable results. However, the capabilities of today's AI systems are still largely built on learning, compressing, and applying existing human knowledge. Accordingly, existing benchmarks primarily test whether AI can produce correct answers based on learned knowledge, or whether it can complete tasks under extensive human guidance. We therefore introduce ASI-Bench, the first benchmark to jointly evaluate AI systems' capabilities of innovative exploration and autonomous scientific execution across general research domains, and the first to progressively withdraw human methodological guidance within the same research project to test how far AI can proceed on its own. Built by over 40 experts with the cost of 31,000+ human hours, ASI-Bench contains 60 project-level research tasks across 11 scientific domains and progressively reduces methodological guidance to test whether AI can independently select methods, conduct research, and produce verifiable results. All tasks undergo expert review, AI-assisted auditing, sandbox execution, and scorer validation. Across 18 state-of-the-art agent--model configurations, the average score drops from 50.91 with full methodological guidance to 29.10 with only the method specified and 26.62 when agents must determine the method themselves. This sharp decline shows that current systems remain heavily dependent on human guidance and are still far from autonomously conducting end-to-end, project-level scientific research. ASI-Bench is open to the world. We invite researchers and builders everywhere to contribute new tasks, challenge the limits of today's AI, and help accelerate humanity's collective path toward artificial superintelligence at https://asibench.apexin.ai/submit.

• The paper develops ASI-Bench to evaluate AI systems' ability to conduct autonomous scientific research, varying from fully specified methods to complete method autonomy, reveal substantial performance drop, mean scores of 50.91(B1), 29.10(B2), 26.62(B3), and 26.99(B4)
• Methodological guidance significantly influences performance: shifting from B1 (full guidance) to B3 (no method guidance) drops the mean score from 50.91 to 26.62 out of 100, highlighting procedural implementation as a key bottleneck
• the mean performance drop is significant in shifting from B1 to B2 (loses 21.82), while from B2 to B3 loses only 2.48, suggesting full administrative guidance impacts performance more than partial

ASI-Bench frames autonomous scientific research as a capability distinct from knowledge retrieval, code generation, or execution of a human-specified workflow. Its central evaluation question is how performance changes when methodological responsibility is progressively transferred from the benchmark designer to the AI system. The benchmark therefore treats scientific autonomy as a composite of method selection, workflow construction, implementation, experimentation, failure recovery, and artifact-based validation rather than as a single reasoning or question-answering skill. The paper’s principal empirical claim is that current agentic systems remain strongly dependent on detailed procedural guidance: across 18 Agent–Model configurations, the mean score falls from 50.91 under full methodological specification to 29.10 when only the method is supplied, and to 26.62 when the system must determine the method itself (2608.17271).

Evaluation objective and benchmark rationale

Existing evaluations cover important but separable components of scientific and technical work. Knowledge benchmarks such as GPQA and Humanity’s Last Exam test advanced disciplinary knowledge, but do not require executable research. SciCode evaluates research-level scientific programming, while ScienceAgentBench and DiscoveryBench target data-driven scientific analysis and discovery (Rein et al., 2023, Tian et al., 2024, Chen et al., 2024, Majumder et al., 2024). PaperBench and related replication benchmarks evaluate end-to-end artifact production, but generally retain a fixed research specification or an existing paper as the target (Starace et al., 2 Apr 2025). Agent benchmarks such as RE-Bench and Terminal-Bench extend horizon length and tool interaction, yet their objectives and solution structures remain more constrained than those of open-ended scientific investigation (Wijk et al., 2024, Merrill et al., 17 Jan 2026).

ASI-Bench’s methodological contribution is to hold the scientific objective, input data, required artifacts, and evaluation criteria constant while varying only the amount of methodological information supplied to the agent. This produces a controlled guidance gradient:

• B1 supplies the full method, equations, implementation details, and procedural steps.
• B2 identifies the methodological approach but leaves the agent to reconstruct the implementation and workflow.
• B3 supplies only the research objective, data, constraints, and required outputs; method selection is autonomous.
• B4 retains the B3 setting while adding plausible but task-irrelevant information to test contextual robustness.

This design separates three capacities that are often conflated: executing a known procedure, operationalizing a named method, and independently selecting and implementing a research strategy. The distinction is important because an agent can perform well when given a complete solver specification while failing to identify the relevant model or numerical procedure from data and objectives alone.

Figure 1: ASI-Bench compares B3 performance across agent configurations and measures the score reduction as methodological guidance is withdrawn from B1 to B4.

The benchmark’s framing is consistent with the limitations identified in automated scientific-discovery systems. Systems such as The AI Scientist and its successors integrate ideation, coding, experimentation, visualization, and writing, but their effective autonomy depends on the extent to which objectives, templates, evaluation procedures, or experimental structures are supplied externally (Lu et al., 2024, Yamada et al., 10 Apr 2025). ASI-Bench consequently evaluates not merely whether a system can complete a research pipeline, but whether it can construct the pipeline that completion requires.

Task structure and scientific coverage

The release contains 60 project-level tasks spanning 11 domains: mathematics, physics, chemistry, biology, astronomy, materials science, earth science, medicine and biostatistics, computer science, robotics, and electrical engineering. The tasks are designed to require heterogeneous data representations, domain-specific modeling decisions, executable computation, and verifiable outputs. They include problems in which an agent must analyze observations, formulate or recover a model, run simulations or experiments, interpret intermediate results, revise failed approaches, and generate scientific artifacts.

Across the benchmark, the authors report more than 2,600 interaction turns and 2,400 execution steps, with more than 35 hours of aggregate agent execution. These figures establish that ASI-Bench is intended to measure sustained project execution rather than isolated tool calls or short-horizon code synthesis. The tasks also contain interdependent decision points: an incorrect modeling assumption can affect numerical stability, downstream diagnostics, and the validity of the final artifact.

Figure 2: Representative ASI-Bench projects span physics, astronomy, electrical engineering, and computer science, illustrating the benchmark’s cross-domain and project-level construction.

The case study of a two-dimensional anisotropic stiff dynamical system illustrates the guidance gradient concretely. In B1, the agent receives the governing conserved anisotropic Kuramoto–Sivashinsky-type PDE, Fourier pseudospectral discretization, dealiasing procedure, and ETDRK4 time integrator. In B2, the prompt describes a stiff fourth-order nonlinear PDE and identifies appropriate solver families, including ETD, IMEX, and semi-implicit spectral methods, but does not prescribe the implementation. In B3, the agent receives only spatiotemporal observations, system metadata, an initial condition, and the prediction objective. It must infer a suitable model and numerical strategy. B4 introduces alternative PDE families, possible analysis methods, and irrelevant conversational material, testing whether the system can maintain a coherent research direction despite misleading contextual information.

The benchmark therefore evaluates more than numerical prediction. Required outputs include predicted fields, spectra, physical diagnostics, visualizations, data-analysis results, and complete executable simulation code. A high score requires a collection of mutually constraining artifacts rather than a single numerical answer. This artifact-oriented design reduces the extent to which an agent can obtain credit through superficial textual reasoning or an isolated forecast.

Construction and validation methodology

The benchmark was constructed from more than 1,300 candidate research ideas and reduced to 60 retained tasks through five review rounds, more than 1,100 review assignments, and over 2,000 revisions. The reported construction effort exceeds 31,000 human-hours. Review covered scientific validity, task specification, the B1–B4 information partition, reference artifacts, evaluation code, information leakage, and agent trajectories.

The validation protocol combines expert review with execution-based checks. Each task is run end-to-end in an isolated sandbox, and the authors report more than 1,500 development sandbox runs. These runs test runtime stability, reproducibility of reference results, artifact generation, scorer behavior, and the absence of unintended shortcuts. This is a substantive strength because scientific benchmarks are vulnerable to evaluation failures that are not apparent from prompt inspection: underdetermined objectives, unstable numerical procedures, hidden assumptions, or scorers that reward syntactically valid but scientifically incorrect outputs.

The benchmark also provides a contribution workflow intended to standardize task expansion. Contributors must specify the scientific objective, four matched prompt conditions, task inputs and outputs, a reference-generation procedure, evaluation gates and weighted scorers, dependencies, and local-testing evidence. The portal’s staged authoring process is designed to make the guidance gradient explicit and auditable rather than leaving prompt difficulty to informal judgment.

Figure 3: The contribution portal guides authors through scientific formulation, B1–B4 prompt construction, evaluation design, runtime configuration, packaging, and local validation.

This infrastructure addresses a recurring problem in scientific-agent evaluation: the benchmark’s scientific content and its evaluation mechanism must be developed together. A task is not fully specified by a research question; it also requires a reproducible reference, executable environment, and scorer that jointly define what constitutes a valid result.

Main performance results

The central result is a large and systematic dependence on methodological guidance. Averaged over all evaluated configurations, B1 performance is 50.91, B2 performance is 29.10, B3 performance is 26.62, and B4 performance is 26.99. The transition from B1 to B2 produces a 21.82-point decrease, whereas the transition from B2 to B3 produces only an additional 2.48-point decrease. B4 is effectively unchanged relative to B3, with a mean difference of only 0.36 points.

Condition	Information supplied	Mean score
B1	Full method and procedure	50.91
B2	Method specified, procedure omitted	29.10
B3	Objective and data only	26.62
B4	B3 plus distractors	26.99

The asymmetry between B1→B2 and B2→B3 is the paper’s most diagnostically important finding. The dominant bottleneck is not necessarily choosing among methods; it is converting methodological knowledge into a complete, executable, and validated research procedure. A method label provides insufficient operational structure for current systems, while full procedural guidance substantially improves performance. The minimal B3-to-B4 change further suggests that distractor information is less damaging than the absence of procedural scaffolding under this benchmark’s conditions.

The strongest configuration is Codex paired with GPT-5.6 Sol at ultra inference effort, which obtains 71.78 on B1, 49.57 on B2, 51.60 on B3, and 50.41 on B4, with an overall score of 55.84. It is the only evaluated system to exceed 50 on B3. Increasing the same backbone’s reasoning setting from xhigh to ultra raises B3 from 40.86 to 51.60, a gain of 10.74 points. This result indicates that inference-time computation can materially improve autonomous workflow construction, but it does not remove the underlying difficulty: even the strongest configuration reaches only moderate performance when the method is not provided.

The paper also reports substantial harness effects. MiMo V2.5 Pro scores 16.17 with MiMo Code and 23.25 with Claude Code, while Kimi K2.7 scores 19.72 with Kimi Code and 27.34 with Claude Code. By contrast, Kimi K3 performs similarly with Kimi Code and Claude Code, at 36.22 and 37.09 respectively. These comparisons imply that the expressed scientific capability is a property of the Agent–Model system, not of the backbone in isolation. Harness architecture can influence planning, file management, execution control, error handling, and persistence, making model-only comparisons potentially misleading.

Figure 4: Removing procedural guidance increases token and time expenditure, while monetary cost exhibits an imperfect relationship with B3 scientific performance.

Computational cost and efficiency

The guidance gradient affects computational cost in a non-monotonic manner. B1 is the least expensive condition, requiring an average of 4.35 million tokens and 37.8 minutes per task. B3 uses 25% more tokens and 22% more execution time than B1, while B4 uses 30% more tokens and 18% more time. B2 is the most expensive condition: 6.91 million tokens and 49.7 minutes per task, corresponding to increases of 59% in tokens and 32% in time relative to B1.

This pattern is notable because B2 supplies the method but not the detailed procedure. Partial guidance can be more computationally expensive than either complete guidance or unrestricted method selection. The likely operational interpretation is that B2 constrains the agent to a prescribed methodological direction while forcing it to reconstruct missing implementation details, producing additional exploration and failed attempts without providing the flexibility of a genuinely open-ended search. The result cautions against treating “method specified” as an intermediate difficulty level with predictable resource requirements.

Performance also does not scale linearly with expenditure. Codex with GPT-5.6 Sol at xhigh obtains a B3 score of 40.86 at approximately $684 per run, close to Claude Opus 5 with Claude Code at 40.70 despite the latter costing approximately$2,728. GPT-5.6 Sol at ultra reaches the best B3 score, 51.60, at approximately $1,550 per run. Thus, greater spending can improve the absolute frontier, but configurations with similar scores can have substantially different costs. The practical consequence is that autonomous-research evaluation should report both scientific performance and resource consumption; a leaderboard ordered only by score would obscure meaningful system-level trade-offs.

Limitations and open questions

The evaluation has several limitations that constrain the interpretation of its results. First, the experiments are conducted without external tool access. The systems can execute within the supplied environments, but the benchmark does not establish how performance would change with web search, literature retrieval, domain databases, laboratory interfaces, or specialized scientific software services. Consequently, the results measure autonomous research under a deliberately closed information regime rather than scientific research in its full institutional setting.

Second, the benchmark contains 60 tasks, and the reported macro-average may conceal substantial domain and task heterogeneity. The paper emphasizes cross-domain coverage but does not, in the supplied content, provide a full statistical decomposition of variance by domain, task type, artifact, or research stage. The extent to which the B1-to-B3 drop is consistent across scientific disciplines therefore remains an open empirical question.

Third, the score is necessarily shaped by benchmark-specific task construction and scoring choices. Expert review, reference generation, sandbox execution, and scorer validation improve reliability, but they cannot guarantee that every task’s intended method gradient is perfectly clean. In particular, B3 tasks may contain latent cues in variable names, file formats, metadata, or data distributions that partially reveal the intended solution. Conversely, some scientific objectives may be underdetermined without background knowledge that the benchmark intentionally withholds. Measuring and reporting cue sensitivity would strengthen the causal interpretation of the guidance gradient.

Fourth, one reported result, Claude Opus 5 with Claude Code, is based on a single run, whereas most configurations use three independent runs. The standard deviations are therefore not uniformly comparable, and the benchmark does not support strong claims about statistical rank ordering for closely performing systems. Finally, the near-equivalence of B3 and B4 may reflect the particular distractor design, agent context-window behavior, or scoring tolerance; it should not be generalized to robustness against adversarial or domain-relevant misinformation without additional experiments.

The benchmark leaves a specific methodological question unresolved: does the large B1-to-B2 degradation primarily reflect deficient procedural planning, weak numerical implementation, poor state tracking, inadequate error recovery, or failures in scientific validation? The aggregate scores establish the existence of the gap but do not fully localize its causes. Stage-level diagnostics, intervention experiments, and human baselines would be needed to distinguish these mechanisms.

Conclusion

ASI-Bench makes methodological autonomy an explicit experimental variable in project-level scientific-agent evaluation. Its matched B1–B4 conditions show that current systems perform substantially better when supplied with complete procedures and degrade sharply when required to operationalize or independently construct those procedures. The mean scores—50.91 in B1, 29.10 in B2, and 26.62 in B3—support the paper’s central conclusion that autonomous scientific execution remains limited, with procedural operationalization emerging as a more consequential bottleneck than method naming alone.

The benchmark’s principal value lies in connecting scientific validity, executable environments, artifact-based scoring, cross-domain coverage, and controlled withdrawal of human guidance. Its future evidentiary strength will depend on enlarging task and domain coverage, adding external-information settings, reporting finer-grained failure analyses, and establishing more rigorous statistical and human baselines. For the present, ASI-Bench provides a clear empirical separation between following a research procedure and independently designing one (2608.17271).