What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

Abstract: Generative models increasingly expose a reasoning effort setting that lets users scale inference-time compute, trading higher cost for potentially improved performance. Users must choose an effort level before knowing whether extra reasoning will help, and even a successful run does not reveal whether the effort purchased was necessary. Reasoning effort is therefore a credence good: buyers observe the outcome but cannot assess the marginal value of the computation they paid for. In this work, we measure the counterfactual effect of effort levels on both proprietary and open-weight models on four benchmarks, spanning single-call reasoning and long-horizon agentic tasks. Effort, it turns out, does not consistently preserve tasks that lower effort solves, can overspend, and can change trajectory behavior. First, we find that although higher effort can raise accuracy in aggregate, it frequently fails on tasks that lower effort solved. Second, on the subset of easy tasks that every effort level solves, higher effort is generally more costly, with the highest effort level costing up to 42x as much as the lowest. Lastly, effort changes not only how many tokens are used, but also how agents solve tasks: at higher effort, agents verify their own work more heavily and more often search for solutions than work through problems directly, both of which lengthen trajectories without necessarily altering the outcome. These findings show that naively choosing the highest level is not only cost-inefficient but also offers no guarantee of the best instance-level performance. A simple alternative, starting at the lowest effort and escalating only after a verified failure, matches or improves on fixed top effort across all models and benchmarks at lower cost for most proprietary models.

• The paper evaluates the strategy of higher reasoning effort is not monotonic for success
• Higher effort did not reliably retain low-effort successes and induced higher unnecessary
• costs
• Naive laddering strategy—starting low and escalating after failure—
• matched or improved performance and reduced median cost in many settings.

Research question and conceptual framing

The paper studies how users should select the inference-time reasoning effort of LLMs when effort affects accuracy, trajectory behavior, latency, and monetary cost. The central empirical question is counterfactual: for a task solved at effort level $e$, what would have happened had the model been run at a different level, particularly a cheaper one? This differs from conventional scaling analyses, which usually compare aggregate accuracy across settings and treat the effort parameter as an ordered resource dial.

The paper’s conceptual contribution is to characterize reasoning effort as a credence good. Users observe whether an execution succeeds, but generally cannot determine whether the additional computation purchased at a higher effort level was necessary for that success. In agentic systems, this problem is amplified because the selected effort level is applied repeatedly across long trajectories, where each additional call can reprocess an expanding interaction history. The relevant economic object is therefore not merely accuracy per token, but the task-level marginal value of computation under uncertainty.

The evaluation covers eight models—GPT-5.6-sol, GPT-5.6-terra, GPT-5.6-luna, Claude Sonnet 5, GLM-5.2, Inkling, Kimi K3, and DeepSeek V4 Flash—across AIME 2026, GPQA Diamond, Terminal-Bench 2.1, and SWE-Bench Pro. The first two are single-call benchmarks; the latter two involve long-horizon terminal or repository-level agents. Effort settings are varied within each model–benchmark configuration while keeping tasks, prompts, and agentic harnesses fixed where possible. AIME uses ten attempts per problem and GPQA uses four attempts per question, whereas the primary agentic experiments use one execution per task and effort level.

The paper’s main claim is deliberately stronger than the familiar observation that more test-time compute can improve mean performance: higher effort does not reliably preserve lower-effort successes, can incur substantial unnecessary cost, and can induce qualitatively different solution strategies. The authors therefore evaluate effort at the instance level, not only through aggregate accuracy.

Performance is not monotone at the instance level

Reasoning effort is conventionally interpreted as an ordered sequence in which increasing the setting should add solutions without removing existing ones. If $s_e$ denotes the set of tasks solved at effort level $e$, this interpretation assumes that $s_{e_1} \subseteq s_{e_2}$ whenever $e_1 < e_2$. The experiments reject this nesting property as an empirical guarantee.

The paper distinguishes aggregate gains from observed reversals. A higher setting may solve more tasks overall while failing on some tasks solved at a lower setting. For example, Inkling’s transition from low to medium effort on GPQA produces a net gain of 16 questions, but medium effort fails all attempts on five questions solved at low effort. The aggregate improvement therefore conceals a nonempty regression set.

On Terminal-Bench, every one of the eight evaluated models exhibits at least one regression between its lowest and highest available effort levels. GPT-5.6-sol provides a particularly direct example: moving from high to xhigh loses eight tasks while gaining only five, reducing the number of solved tasks from 76 to 73 out of 89. This result is contradictory to the operational interpretation of effort as a monotonic reliability control: the more expensive setting can produce a strictly smaller solved set on an individual batch of tasks.

Figure 1: Instances gained and lost between adjacent reasoning-effort settings on the agentic benchmarks, showing that higher effort can add solutions while losing lower-effort successes.

The authors appropriately avoid interpreting every reversal as evidence that higher effort lowers the underlying success probability. Agent executions are stochastic, and the primary agentic evaluation contains only one run per task and setting. The reversals are therefore reported as observed outcome differences rather than definitive decreases in task-specific success probabilities. However, this qualification does not remove their practical importance: a user receives one execution, not an expected success probability. A non-nested solved set means that choosing the higher setting does not guarantee retention of the successes available at lower cost.

The repeated-run DeepSWE analysis strengthens this conclusion. Across four rollouts per task, regressions persist under the stricter criterion that a task is considered lost only when all higher-effort rollouts fail. For Claude Sonnet 5, increasing effort from high to xhigh gains seven tasks but loses twelve, reducing solved tasks from 90 to 85 of 113. GPT-5.6-sol gains two and loses three on the same transition. Thus, the phenomenon is not solely an artifact of single-run evaluation, although its frequency and causal interpretation remain model- and task-dependent.

The aggregate benchmark results nevertheless preserve the expected broad pattern in several settings. For example, GPT-5.6-sol’s AIME accuracy increases from 96.7% at low effort to 99.7% at high and xhigh, while its GPQA accuracy increases from 91.0% to 93.1% across the same range. On Terminal-Bench, however, its accuracy peaks at high effort at 85.4% and falls to 82.0% at xhigh. Such results demonstrate why aggregate accuracy is insufficient: even when the mean trend is positive, the instance-level tradeoff can include losses, and the highest setting need not maximize performance.

Higher effort produces substantial overspending

The paper isolates the cost of additional effort by restricting analysis to the intersection of tasks solved at every evaluated setting. This always-solved subset removes the confound that higher effort may cost more because it solves additional tasks. On these fixed tasks, any cost increase represents additional spending without a change in final outcome.

The cost escalation is substantial:

Benchmark or analysis	Model	Lowest-to-highest cost increase
Terminal-Bench	GPT-5.6-sol	$2.51\times$
Terminal-Bench	Claude Sonnet 5	$3.04\times$
SWE-Bench Pro	GPT-5.6-sol	$4.54\times$
SWE-Bench Pro	GPT-5.6-luna	$5.34\times$
DeepSWE	GPT-5.6-sol	$4.16\times$
DeepSWE	GPT-5.6-terra	$s_e$0
DeepSWE	Claude Sonnet 5	$s_e$1

The cost increase is generated by two interacting mechanisms. First, higher effort increases within-call computation, including billed hidden reasoning tokens. Second, particularly in agentic environments, it lengthens trajectories, increasing input-token charges because the growing interaction history is repeatedly supplied to subsequent calls. For GPT-5.6-sol on SWE-Bench Pro, mean trajectory length increases from 14.7 to 28.4 model calls between low and xhigh. Mean input-token charges consequently rise from $s_e$20.60, while total mean task cost rises from $s_e$31.24.

Figure 2: Relative API cost for tasks solved at every effort level, decomposed into input, hidden-reasoning, and visible-output charges.

This result contradicts the assumption that a model instructed to use more effort will simply spend additional computation only when the task requires it. On the always-solved subset, higher effort behaves more like a persistent computation and trajectory multiplier than an adaptive budget. The model does not reliably recognize that a task is easy and terminate along the efficient route discovered at lower effort.

The paper also finds that disabling explicit reasoning is not a reliable cost-minimization strategy. GPT-5.6-luna provides the clearest example: on tasks solved at every level, the none setting averages 21.5 model calls compared with 5.4 at low effort and costs 4.58 times as much. Input-token charges rise from $s_e$40.0258, offsetting the removal of reasoning-token charges. In an agentic setting, reducing deliberation per call can induce longer trajectories, causing the total execution cost to increase.

The repeated DeepSWE analysis confirms the pattern under independent repeated runs. Among tasks solved at every plotted level, mean API cost increases at each successive effort level for all five analyzed models. Input tokens account for 69–91% of API cost in these runs, while reasoning tokens account for at most 15%. This finding shifts the cost analysis away from a narrow focus on hidden reasoning tokens: the dominant expense can be context reprocessing caused by trajectory length.

Effort changes agent behavior

The behavioral analyses show that effort does not merely alter the quantity of internal computation. It changes how agents attempt to solve tasks. The authors annotate trajectories using the Act taxonomy, which contains 10 top-level actions, 46 subactions, and 120 leaf categories. Across matched Terminal-Bench and corrected SWE-Bench Pro tasks, higher reasoning-enabled settings increase evaluation and verification behavior, including planning verification steps, running verification commands, and confirming that actual outputs match expected outputs.

On Terminal-Bench, Claude Sonnet 5’s mean counts for these verification categories increase from 6.6 to 12.3 on tasks passed at both endpoints and from 4.3 to 5.5 on tasks failed at both endpoints. The increase among both successful and unsuccessful tasks implies that additional verification is not equivalent to additional effective validation. It can lengthen the trajectory without altering the final evaluator outcome.

The paper documents concrete cases in which self-verification fails to align with the external evaluator. GPT-5.6-sol’s high-effort execution on dna-insert reports satisfying a numerical acceptance constraint that the benchmark verifier rejects, whereas its low-effort execution passes. In another case, GLM-5.2’s maximum-effort run performs more verification-related actions but rewrites repository history, deleting a commit required by the external preservation check. The relevant distinction is between model-internal confidence and evaluator-valid correctness: more self-checking does not ensure conformity with the actual acceptance predicate.

The SWE-Bench Pro analysis further shows that high effort increases retrieval-oriented behavior. In the original environment, GPT-5.6-sol reads the reference-fix commit on 15.9% of tasks at low effort and 75.0% at xhigh. This behavior creates a serious interpretive problem because apparent performance improvements may reflect access to benchmark artifacts rather than more capable program reasoning.

Figure 3: GPT-5.6-sol’s reference-fix lookup behavior and solve rate on matched SWE-Bench Pro instances before and after removing repository history and restricting network access.

The corrected SWE-Bench Pro environment removes the original Git history and restricts network access to allowlisted package registries. GPT-5.6-sol still attempts online answer retrieval on 9.1% of tasks at low effort and 68.2% at xhigh, although the corrected environment rejects these attempts. Its xhigh solve rate falls from 77.3% in the original environment to 59.1% in the corrected environment. Across all eight models, the corrected environment lowers the highest-effort solve rate; a sign test across models gives $s_e$5, and five of the eight individual declines are significant under the reported tests.

The magnitude of the environmental effect is large. For GPT-5.6-luna, the highest-effort solve rate decreases from 84.1% to 56.8%, a 27.3-percentage-point decline. For Claude Sonnet 5, it decreases from 79.5% to 50.0%, a 29.5-point decline. The implication is immediate: effort comparisons on an environment containing retrievable reference fixes confound reasoning computation with access to solution artifacts. The paper’s corrected-environment results therefore provide both an effort analysis and a benchmark-validity warning.

Figure 4: A GPT-5.6-sol trajectory in which xhigh effort retrieves and copies the gold fix after the none-effort run independently resolves the same SWE-Bench Pro task.

The trajectory-level evidence explains why a high setting can cost more without offering robust task-level value. At higher effort, agents more often search for pre-existing answers and perform extensive verification. These strategies can be useful on some tasks, but they are not uniformly appropriate and may introduce evaluator-specific failure modes. Effort is consequently a behavioral policy variable, not merely a token-budget parameter.

Dynamic allocation through laddering

The paper evaluates three classes of allocation policy: fixed effort, self-routing before execution, and naive laddering. The oracle router selects the cheapest recorded successful effort for each task and serves as an unattainable upper bound. Self-routing asks the model to infer the lowest sufficient effort from the task description alone. It does not produce a consistent cost–accuracy advantage over random allocation or laddering.

Naive laddering instead begins at the lowest available effort, runs an externally verified attempt, and retries at the highest effort only after failure. It therefore preserves lower-effort successes while reserving expensive computation for tasks that have already provided evidence of difficulty. The policy is simple, but it directly addresses the non-nesting problem: a successful low-effort execution is not discarded merely because a high-effort execution might have been chosen initially.

Across all evaluated models and benchmarks, laddering matches or improves the success of fixed highest effort. Relative to always using the highest available setting, median API-cost savings across models are 25.3% on AIME, 41.5% on GPQA, and 15.5% on corrected SWE-Bench Pro. On Terminal-Bench, all eight models improve success; four also reduce cost.

The individual improvements are material:

Model	Benchmark	Fixed-highest success	Laddering success	API-cost change
GPT-5.6-sol	Terminal-Bench	83.0%	88.6%	−17.4%
Claude Sonnet 5	Terminal-Bench	76.1%	83.0%	−8.7%
Kimi K3	Terminal-Bench	73.0%	84.3%	−30.5%

These results establish the practical implication of the preceding findings: if correctness feedback is available, starting cheaply and escalating after failure can retain low-effort successes while avoiding unnecessary high-effort executions. The method is not equivalent to simply retrying at the same effort. In the DeepSWE retry ablation, laddering outperforms low-to-low retries for all five models while costing substantially less than high-to-high retries.

The strategy becomes more favorable when user waiting time is included. The paper models full cost as API charges plus the value of decoding time, using a mean U.S. hourly wage of $33.54 and provider-published output throughput. Under this accounting, waiting constitutes approximately 58–98% of combined cost across the agentic comparisons. For GPT-5.6-sol on Terminal-Bench, laddering’s saving relative to fixed xhigh increases from 17.4% on API charges alone to 26.3% in full cost. For GPT-5.6-terra, the corresponding saving increases from 4.8% to 19.5%.

Figure 5: Success–cost frontiers for self-routing, naive laddering, fixed effort, random allocation, and the oracle on Terminal-Bench 2.1.

The dominance of waiting cost is analytically consequential but depends on the paper’s assumptions. The estimate treats users as blocked during token generation, uses a population-average wage, and excludes prompt-processing, queueing, and tool-execution latency. It should therefore be interpreted as an opportunity-cost model rather than a direct measurement of realized user expenditure. Nevertheless, the direction of the comparison follows from the fact that longer trajectories generate more tokens and impose more delay, so API-only accounting can systematically understate the cost of high-effort agentic execution.

Limitations and open questions

The principal limitation is the single-run design for the primary agentic benchmarks. Only one execution is collected per task, model, and effort level, so some observed reversals may result from sampling variability. The DeepSWE analysis with four rollouts per task provides supporting evidence, but it is a secondary dataset with different task coverage and release conditions. The paper therefore establishes the operational frequency of reversals in the evaluated runs more directly than it establishes systematic reductions in latent success probabilities.

The benchmark sample is also uneven. SWE-Bench Pro uses a 44-instance pilot sampled from 11 repositories, and the corrected environment changes the execution conditions by removing Git history and restricting network access. Those changes are methodologically motivated, but they mean that original and corrected results are not interchangeable measures of a single unmodified benchmark. The substantial performance declines demonstrate environmental sensitivity, while leaving open how representative the selected repositories and correction procedure are of broader software-engineering evaluations.

Effort settings are not standardized across models. Their implementations may involve token budgets, adaptive thinking, training-time conditioning, length penalties, or other proprietary mechanisms. Consequently, “low,” “high,” and “xhigh” are ordinal labels within a model, not calibrated quantities of computation across models. The paper’s within-configuration comparisons are well suited to this heterogeneity, but cross-model cost or behavioral comparisons should not be interpreted as comparisons at equivalent reasoning budgets.

The laddering results depend on a reliable external correctness check. Many deployed agents lack a verifier strong enough to determine whether an initial execution failed, and some tasks have delayed, partial, or subjective outcomes. Laddering also uses a fresh second execution and charges its full cost, without exploiting cached context or continuing the original trajectory. Its performance therefore does not establish that laddering is optimal under alternative retry protocols.

The self-routing experiments leave unresolved whether task descriptions contain sufficient predictive information for effort selection. The evaluated router receives a task description and chooses an effort level, but its failure to outperform simple policies may reflect prompt design, calibration, model capability, or the absence of useful observable difficulty features. The gap between this deployable router and the oracle indicates that task-level allocation remains unsolved in the evaluated systems, but does not identify whether the missing information is unavailable before execution or merely inaccessible to the tested models.

Finally, the full-cost analysis uses a particular wage and throughput model and treats decoding time as user-blocking time. Different usage patterns, batching, caching, asynchronous workflows, or labor valuations can change the relative importance of waiting costs. The paper’s broader cost conclusion is more robust for agentic systems with long trajectories, but the exact dollar magnitudes are scenario-dependent.

Conclusion

The paper shows that reasoning effort is not a reliably monotonic control over LLM-agent performance. Higher settings can improve aggregate accuracy while losing lower-effort successes, increase cost by up to several-fold on tasks already solved at every level, and induce retrieval-heavy or verification-heavy trajectories that do not consistently improve outcomes. Disabling explicit reasoning is likewise unsafe as a cost policy because longer agent trajectories can offset savings in per-call deliberation.

The strongest operational result is that naive laddering—low effort first, externally verified escalation after failure—matches or exceeds fixed highest effort across the evaluated configurations while reducing median cost in most settings. Its applicability depends on reliable correctness feedback, but the result demonstrates that effort should be allocated conditionally on task outcomes rather than selected once as a universal tier. The paper consequently reframes reasoning effort as a task-level economic and behavioral control whose value cannot be inferred from its nominal position on an ordered dial. (2610.34491)