Scaling Test-Time Compute for Agentic Coding

Abstract: Test-time scaling has become a powerful way to improve LLMs. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge is no longer generating more attempts, but representing prior experience in a form that can be effectively selected from and reused. We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories. Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes while discarding low-signal trace details. This representation enables two complementary forms of inference-time scaling. For parallel scaling, we introduce Recursive Tournament Voting (RTV), which recursively narrows a population of rollout summaries through small-group comparisons. For sequential scaling, we adapt Parallel-Distill-Refine (PDR) to the agentic setting by conditioning new rollouts on summaries distilled from prior attempts. Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0. For example, by using our method Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified (mini-SWE-agent) and 46.9% to 59.1% on Terminal-Bench v2.0 (Terminus 1). Our results suggest that test-time scaling for long-horizon agents is fundamentally a problem of representation, selection, and reuse.

• The paper introduces a unified test-time scaling framework (PDR+RTV) that leverages compact rollout summaries to enhance agentic coding performance.
• It employs Recursive Tournament Voting for parallel aggregation and Parallel-Distill-Refine for sequential refinement to improve candidate selection and consistency.
• Empirical results show substantial pass@1 improvements and efficiency gains across multiple coding benchmarks and model families.

Scaling Test-Time Compute for Agentic Coding: A Technical Analysis

Introduction and Motivation

The paper "Scaling Test-Time Compute for Agentic Coding" (2604.16529) addresses the critical challenge of leveraging additional inference-time compute to enhance the performance of long-horizon, agentic LLM (LM) systems for program synthesis and software engineering tasks. In contrast to short-form tasks, where outputs are concise and direct comparison is feasible, agentic coding involves complex, multi-step interactions with external environments such as shells and codebases, leading to lengthy, noisy trajectories. Established test-time augmentation techniques fail to transfer directly due to these trajectory complexities and the difficulty of reusing or ranking “raw” rollouts.

The authors hypothesize that the core bottleneck in inference-time scaling for agentic coding is the representation of agent experience. They propose to summarize agentic rollouts into compact, structured artifacts that capture hypotheses, progress, and failure modes, and to use these as substrates for both parallel aggregation and sequential refinement.

Methodological Advancements

The proposed test-time scaling framework introduces two orthogonal, complementary dimensions of inference-time augmentation for agentic coding: parallel aggregation and sequential refinement, unified by the use of structured rollout summaries.

1. Compact Structured Summaries

Agentic rollouts, comprising sequences of actions, terminal observations, and internal reasoning, are first transformed into bounded, structured summaries via LM-based summarization. These summaries are designed to abstract the salient aspects of the agent’s trajectory while filtering low-signal noise, thus serving as reusable and comparable representations.

2. Parallel Aggregation via Recursive Tournament Voting (RTV)

RTV is a parallel aggregation protocol for selection among a population of candidate rollouts without access to ground truth labels. Proceeding recursively, RTV partitions the population into small groups (optimal $G=2$), applies a voting-based comparison using the LM itself as judge, and retains the winning candidate from each group. Successive rounds narrow down to a single, aggregate best attempt.

Figure 1: RTV recursively narrows the candidate population by staging small-group comparisons over structured summaries until a single best rollout is selected.

Ablation studies reveal that using compact summaries rather than entire rollouts as comparison inputs yields significantly superior final selection performance, especially in late tournament rounds.

Figure 2: Structured summaries as substrates outperform raw rollouts for parallel aggregation on both coding benchmarks.

Additionally, aggregating multiple independent “votes” per comparison ($V=8$) further enhances decision reliability.

3. Sequential Refinement via Parallel-Distill-Refine (PDR)

Sequential scaling is instantiated by adapting the PDR protocol, which generates fresh rollouts conditioned on summaries of prior attempts. Each new rollout is initialized with a context window containing $K$ sampled summaries from the previous generation, facilitating targeted reuse of distilled agent experience.

A key finding is that conditioning on multiple high-quality prior summaries, preferentially selected by RTV, further improves downstream rollout quality and consistency.

Figure 3: K-sampled refinement contexts (bottom) yield broader pass counts in iteration 1 compared to single-summary refinement (top).

4. Unified Test-Time Scaling Pipeline

The composite pipeline orchestrates the following:

1. Iteration 0: Generate $N$ independent rollouts and summarize.
2. RTV: Select top $K$ summaries.
3. Iteration 1: Produce $N$ new rollouts conditioned on the selected $K$ summaries.
4. Final RTV: Aggregate the new rollout population to select a final output.

   Figure 4: Unified PDR+RTV pipeline combining parallel aggregation and sequential reuse.

This framework robustly balances exploitation (focusing on strong subpopulations) and exploration (preserving candidate diversity through broader context windows at each iteration).

Empirical Results

Evaluations are performed across two demanding coding agent benchmarks—SWE-Bench Verified and Terminal-Bench v2.0—using multiple leading LLMs, including Claude-4.5-Opus, Gemini-3.1-Pro, Claude-4.5-Sonnet, Gemini-3-Flash, and GPT-5-0825. The main results demonstrate consistent and substantial improvements over single-attempt and vanilla best-of-$N$ baselines, validated on challenging long-horizon tasks.

Figure 5: Core performance improvements: Claude-4.5-Opus rises from 70.9%→77.6% on SWE-Bench Verified and 47.0%→59.1% on Terminal-Bench v2.0. Gemini-3.1-Pro achieves similar gains.

Figure 6: Standalone RTV improves pass@1 scores by 5-6% (SWE-Bench Verified) and 8-12% (Terminal-Bench v2.0) across model families.

Sequential scaling with high-quality refinement contexts enables agents to increasingly solve tasks that were previously infeasible for all initial rollouts, a previously unobserved phenomenon in test-time scaling for agentic models.

A detailed analysis shows that the quality of the refinement context—specifically, the number of passing rollouts in the prior $K$ summaries—is a strong predictor of iteration-1 success rates. Furthermore, the unified PDR+RTV pipeline not only increases solution rates but also reduces average steps per rollout—indicating efficiency gains in agent search.

Analytical Insights

Transition matrices over pass counts reveal a marked shift towards higher numbers of passing rollouts after each refinement iteration, especially when RTV-selected contexts are used. Parallel aggregation continues to add value post-refinement because even refined populations retain diversity that can be exploited through recursive small-group selection.

Figure 7: Pass count transition matrices from iteration 0→1 consistently show net improvement (mass above diagonal) under PDR+RTV.

Figure 8: Pass@1 evolution across RTV rounds confirms monotonic performance gains as tournament voting proceeds.

RTV group-level comparison accuracy reflects the nontrivial judgment challenge: while aggregate accuracy is above random, there remains a gap between current LM-as-a-judge capability and an oracle selector, highlighting an avenue for future improvement via explicit selection training or external judges.

Implications and Future Outlook

Practically, this work establishes a general recipe for inference-time scaling in agentic LLMs that is robust to trajectory complexity and agent autonomy. It signals a shift away from naive sampling and towards intelligent orchestration of compute through carefully managed representation, selection, and reuse of prior agentic experience.

For theoretical research, the findings align with emerging evidence that long-horizon agent performance is increasingly bottlenecked by context management and experience curation, rather than raw generation capacity. The demonstrated benefits suggest that continued advances in trajectory summarization and experience selection may yield further substantial returns.

Future research directions include:

• Extending summaries to cover external persistent agent artifacts (e.g., code patches, scripts, test generation outputs).
• Training dedicated LM-based judges or external aggregation modules for higher fidelity selection in RTV.
• Developing active selection and context management strategies that exploit rollout diversity more systematically.
• Integrating with advanced agentic memory architectures or meta-learning controllers for rollouts.

Conclusion

The paper presents a representation-centric unification of test-time scaling strategies tailored for agentic coding tasks, demonstrating strong, systematic improvements across multiple model families and task benchmarks. The PDR+RTV framework leverages structured rollout summaries to enable reliable selection and efficient information reuse, decisively outperforming prior state-of-the-art inference-time scaling methods in the agentic regime. This work not only advances current test-time augmentation methodology but also sets the stage for a new generation of agentic LLM systems that can accumulate, distill, and reuse experience at inference-time with high efficiency and robustness.