Empirical Scaling Law for Discovery

• The paper presents an empirical scaling law demonstrating a linear increase in state-of-the-art breakthroughs with rising GPU hours.
• It details how autonomous systems like ASI-Arch generate novel neural architectures and emergent design principles through closed-loop innovation.
• The findings imply that optimizing computational resource allocation can accelerate self-reinforcing cycles of discovery in automated research.

An empirical scaling law for discovery characterizes how the process of scientific or architectural innovation responds quantitatively to resource investment, such as computational effort, data volume, or human effort. In recent years, empirical scaling laws—originally formulated to describe performance improvement in machine learning models—have been extended to the “rate of discovery” itself, particularly in the context of AI-driven automated science. This concept is distinguished by its direct measurement of knowledge or capability breakthroughs as a function of resource increase, enabling principled predictions about the acceleration of scientific progress.

1. Definition and Empirical Formulation

An empirical scaling law for discovery specifies the relationship between a measurable rate of innovation—namely, the number of state-of-the-art (SOTA) discoveries, breakthroughs, or novel architectural designs—and the total resources expended. In the context of autonomous AI research systems such as ASI-Arch, this relationship is found to be simply linear:

$$N_\mathrm{SOTA} \approx k \cdot (\mathrm{GPU\,hours}) + b$$

where:

• $N_\mathrm{SOTA}$ is the cumulative count of unique breakthroughs meeting a predefined SOTA criterion,
• $\mathrm{GPU \, hours}$ is the total computational resource consumed,
• $k$ is the empirical rate of discovery per unit compute,
• $b$ is a baseline offset.

This linear scaling reflects that the number of SOTA results discovered is directly proportional to the computational resource invested, up to the observed limits of the exploration (Liu et al., 24 Jul 2025).

2. Architectural Discovery in Autonomous Systems

The scaling law for discovery was established in a framework where an AI-driven autonomous system (ASI-Arch) conducts neural architecture innovation. Unlike traditional Neural Architecture Search (NAS), which searches within human-predefined spaces, ASI-Arch operates as a closed-loop, self-driven system capable of:

• Proposing and justifying new architectural motifs,
• Translating abstract hypotheses into executable implementations,
• Training, evaluating, and robustly verifying model performance autonomously,
• Storing and analyzing emergent design principles.

The empirical scaling law emerged from analyzing the outputs of 1,773 fully autonomously executed experiments, comprising over 20,000 GPU hours and resulting in the discovery of 106 novel SOTA linear attention architectures. The breakthrough density was observed to be well described by the linear law above (Liu et al., 24 Jul 2025).

3. Properties and Emergent Phenomena

A central qualitative outcome is the observation of “emergent design principles” in the discovered architectures—features comparable to those seen in AlphaGo’s Move 37, which introduced previously unanticipated strategies. The empirical scaling law predicts not just quantitative rates of discovery, but also the increasing likelihood of qualitatively novel solutions emerging as resource investment increases.

This process exhibits two significant features:

• Automated Innovation vs. Optimization: Discovery is not limited to “hill-climbing” within a fixed search space but includes the generation and validation of concepts irreducible to prior human knowledge.
• Research Self-Acceleration: The closed-loop process, characterized by the scaling law, constitutes a self-reinforcing cycle of knowledge growth: more discoveries enable richer candidate pools, which in turn increase the diversity and depth of future discoveries, subject only to the resource constraint (Liu et al., 24 Jul 2025).

4. Implications for Computational Science and Policy

The establishment of an empirical scaling law for scientific discovery marks a transition in research methodology:

• Research progress is no longer strictly bounded by human cognitive capacity; it is now in principle a function of scalable compute resources.
• Given sufficient automation, innovation in scientific or technical domains can be accelerated to rates determined by the available computational investment.
• This reframes research management problems as optimization over resource allocation, pipeline design, and verification strategies rather than fundamentally unsolvable creativity bottlenecks.

A plausible implication is that fields previously limited by the pace of human intuition may shift toward self-accelerating, compute-driven progress, provided sufficient mechanization of hypothesis generation, experimentation, and validation can be engineered.

5. Analytical and Practical Applications

The scaling law provides a framework for forecasting and optimizing the allocation of computational resources in autonomous research programs. Decision-makers can:

• Estimate the probable number of impactful discoveries for a given compute budget,
• Set cost–benefit targets for scaling infrastructure,
• Guide the design of experiment prioritization heuristics for maximal discovery yield.

Additionally, establishing the empirical law enables the benchmarking of different approaches to automated innovation and facilitates the comparison of efficiency between systems (Liu et al., 24 Jul 2025).

6. Limitations and Open Questions

The empirical law holds robustly over the measured scale of the experiments conducted but may be subject to saturation or phase transition effects as the space of easy-to-discover innovations is depleted, or as fundamental architectural or theoretical bottlenecks are reached. The generalization of the law across different scientific domains, varying task complexity, or in the presence of adversarial constraints remains an open area of research. Furthermore, the law does not account for the qualitative value or impact of discoveries—only their measured SOTA status.

7. Future Directions

Understanding and extending the empirical scaling law for discovery can enable further developments in:

• Quantitative science-of-science research,
• Self-optimizing closed-loop research systems,
• Autonomous knowledge generation beyond neural architecture design (e.g., materials, mathematics, drug discovery).

Strategic resource allocation, closed-loop system design, and the integration of domain-theoretic knowledge remain areas for ongoing paper. Theoretical models explaining why the linear scaling holds—and under which conditions it may break—are also open to investigation.

---

In summary, the empirical scaling law for discovery provides a formal quantitative link between resource allocation and innovation rate in autonomous research systems. By operationalizing the process of scientific discovery as a scalable function of computational investment, it delineates a pathway for computation-driven, self-accelerating progress in AI and beyond (Liu et al., 24 Jul 2025).