Tool Building as a Path to "Superintelligence"

Abstract: The Diligent Learner framework suggests LLMs can achieve superintelligence via test-time search, provided a sufficient step-success probability $γ$. In this work, we design a benchmark to measure $γ$ on logical out-of-distribution inference. We construct a class of tasks involving GF(2) circuit reconstruction that grow more difficult with each reasoning step, and that are, from an information-theoretic standpoint, impossible to reliably solve unless the LLM carefully integrates all of the information provided. Our analysis demonstrates that while the $γ$ value for small LLMs declines superlinearly as depth increases, frontier models exhibit partial robustness on this task. Furthermore, we find that successful reasoning at scale is contingent upon precise tool calls, identifying tool design as a critical capability for LLMs to achieve general superintelligence through the Diligent Learner framework.

• The paper demonstrates that tool-building significantly bolsters LLM reasoning by maintaining high stepwise success probabilities, even in adversarial settings.
• The methodology combines full-context integration with a stepwise reconstruction game, rigorously testing model performance across increasing reasoning depths.
• Empirical results show that frontier LLMs using external tools outperform small models, emphasizing tool integration as key for achieving superintelligence.

Tool Building as a Path to “Superintelligence”: A Technical Analysis

Introduction and Motivation

The paper "Tool Building as a Path to 'Superintelligence'" (2602.21061) critically examines the empirical and theoretical viability of achieving superintelligence through the Diligent Learner framework. This framework hypothesizes that it is possible for LLMs to reach arbitrarily high capabilities—without changes to model architecture—by leveraging test-time search over reasoning steps, provided the model retains a sufficiently high stepwise success probability, denoted $\gamma$, at each reasoning step.

Despite the theoretical promise, a core open problem persists: does $\gamma$ remain bounded away from zero as the depth of reasoning (i.e., the number of chained inference steps) grows, especially when confronting increasingly complex, out-of-distribution problems? The authors respond to this by constructing an adversarial benchmark—Boolean circuit reconstruction over $\mathrm{GF}(2)$ under a stepwise statistical obfuscation regime—which strictly prevents shortcut exploitation and requires diligent integration of history and newly presented evidence for successful next-step prediction.

Benchmark Construction and Theoretical Guarantees

The benchmark formalizes a stepwise reconstruction game, where the model, at each depth $g$, receives a revealed prefix $P_g$ (the solution-so-far) and a batch of evidence $\mathsf{S}_g$. The challenge is to output the unique correct next monomial in the Algebraic Normal Form (ANF) representing the underlying Boolean function. The benchmark is meticulously engineered so that:

1. No Data-only or History-only Shortcuts: Both the sequence of prior terms and the step-specific evidence, by themselves, are uninformative regarding the next correct step.
2. Statistical Obfuscation: The sampling oracle utilizes the known prefix as a “key” to mask the evidence labels, rendering each new batch statistically uniform unless properly unmasked with the prefix.
3. Unique Continuation: At every reasoning depth, there is a single correct extension, ensuring that $\gamma$ is cleanly operationalized as the probability the model proposes exactly the true next term.

The construction ensures that only a solver equipped to combine both the accumulated solution and the fresh evidence can recover the next step. Data-only or history-only solvers achieve success rates that degrade towards uniform random guessing as depth increases, theoretically approaching $\frac{1}{\binom{p}{d-1}}$.

Figure 2: Only algorithms using both history and step-specific data (Estimator $\mathcal{A}$) sustain high next-step prediction accuracy across depths; all others degrade rapidly with depth.

Empirical Results: LLMs vs. Baselines

Performance of Bayesian and Partial-information Estimators

The authors first empirically validate that Bayesian and partial-information estimators (with access only to data, only to history, or partially both) collapse rapidly in stepwise accuracy as either the reasoning depth $g$ or the payload size $p$ increases. Only the “diligent” estimator (with full access to both the prefix and step evidence) sustains reliable $\gamma_g$.

Figure 4: Probability of successful next-step prediction for different estimators as depth $g$ and payload size $p$ increase; only full-information estimators are robust.

Small LLMs: Depth-induced Degradation

Experiments with small LLMs (Qwen3-2507 family, both 4B and 30B) exhibit a superlinear collapse in $\gamma_g$ with increasing reasoning depth, even though an explicit polynomial-time algorithm exists for the task. Augmented variants (“Thinking”) provide marginal gains at shallow depths, but cannot forestall degradation as depth increases—mirroring the behavior of partial-information solvers.

Figure 1: Small LLMs show sharp declines in stepwise success probability with depth, failing to utilize the full prefix despite the existence of a tractable algorithm.

Frontier LLMs: The Effect of Tool Use

Frontier LLMs (e.g., GPT-5.2, Claude Opus 4.5, Gemini 3 Pro) display qualitatively different behavior. When allowed to invoke tools programmatically (i.e., perform external computations or validations), these models sustain high $\gamma_g$, even at considerable reasoning depths ($g=127$), with only minimal deterioration. Tool use enables offloading of both state-tracking and algorithmic execution, letting the LLM focus on correct constraint specification.

Figure 3: Frontier LLMs, especially with tool integration, maintain high next-step accuracy where all small LLMs have failed.

Further, when models are explicitly instructed not to use tools, their $\gamma_g$ drops sharply with increasing problem size, contrasting with tool-enabled runs. Interestingly, some models (e.g., Opus) exhibit implicit tool-like behaviors even under tool use prohibition, indicating the difficulty of disentangling internal computations from external calls.

Figure 5: Tool calls (T.) in frontier LLMs dramatically stabilize $\gamma_g$ over depth, while performance collapses when tools are disallowed (N.T.).

Theoretical and Practical Implications

Parameter Scaling and Generalization Mechanisms

The results robustly demonstrate that superlinear degradation in $\gamma_g$ with depth is an inherent limitation for small or non-tool-enabled LLMs, attributable to their limited ability to utilize the entire evolving prefix and integrate it with new evidence. This places practical hardness on achieving superintelligence via pure search or scaling alone. In contrast, tool use—externalizing execution and state—unlocks qualitatively superior generalization in line with the Diligent Learner’s theoretical requirements.

Tool Use as an Architectural Bottleneck

These findings elevate tool design to a first-class architectural concern for LLMs aspiring to superintelligence within the Diligent Learner paradigm. The division of labor—models specify constraints, tools execute—is crucial for compositional generalization and for avoiding exponential blowups in search budgets.

Insights for Future Research

• Algorithmic Reasoning: Pure in-context learning or scale increases are unlikely to achieve robust long-horizon reasoning without external state and computation.
• Agentic Architectures: Effective tool-building and tool-use, including differentiable programmatic calls and persistent state, should be prioritized in future LLM architectures.
• Benchmarking: Adversarial, stepwise evaluation protocols (with unique correct continuations and cryptographically robust masking) deliver more fine-grained, diagnostic information about reasoning flaws than end-to-end accuracy metrics.

Conclusion

This work rigorously operationalizes and empirically tests the Diligent Learner hypothesis, establishing that tool-building and tool-use are essential—and perhaps singular—mechanisms for achieving superintelligence-like capabilities in LLMs under deep multi-step reasoning. Without these capabilities, $\gamma_g$ collapses and the theoretical guarantees of search-based superintelligence dissolve. Research must therefore focus on advancing LLM/tool symbiosis, as well as designing new benchmarks that directly measure models’ ability to integrate history, evidence, and external computation across unbounded reasoning depths.