SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness

Abstract: As coding agents move from supervised code completion to unattended, around-the-clock exploration, their work expands from isolated predictions into long trajectories of reasoning, tool use, and feedback. Token efficiency therefore becomes important for scaling recursive self-improvement. We take an RSI-inspired approach at the harness layer, scaling auto-research loops across increasingly numerous and diverse environments for harness rollouts. At this scale, the process yields reusable improvements that transfer beyond their development setting, moving automated harness discovery toward production-level outcomes. Four mechanisms survive selection and form SoL-Pi, spanning action execution, context compaction, observation handling, and delegated reading. On the 51-task EdgeBench evaluation, SoL-Pi achieves performance comparable to Pi across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by 44.7-49.0% and API cost by about one third. In other words, estimated hourly savings are $8.75-$13.50 relative to native Codex and Claude Code harnesses, and $4.36-$5.71 relative to Pi.

• The paper presents 'SoL-Pi,' an automated search procedure for discovering reusable harness mechanisms that optimize token efficiency and cost, using GPT-5.6 Sol and Opus 5 backward conditions.
• The SoL-Pi harness incorporates four mechanisms—Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer—each addressing different aspects of task execution and cost management.
• Across multiple backends and benchmarks, the SoL-Pi harnew reduces tokens by 44.7%–49.0% with minimal performance loss while lowering API costs by nearly 33.5%.

Research objective and central contribution

“SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness” (2609.20519) studies token efficiency as a property of the agent harness rather than solely of the underlying model, inference server, or model-selection policy. The harness mediates tool invocation, context construction, observation processing, compaction, delegation, verification, and termination. The paper argues that these control-plane decisions can produce substantial cost differences during long-horizon execution without modifying model parameters.

The central contribution is an automated search procedure for discovering reusable harness mechanisms. An optimizing research agent analyzes trajectories generated by a separate task-solving agent, proposes executable changes, and evaluates them in automatically verified development environments. Candidate mechanisms must satisfy two fixed criteria: capability metrics must remain within predeclared tolerances, and at least one efficiency metric must improve. Candidates that pass both gates are retained according to a nondominance criterion. The resulting mechanisms are integrated into a frozen harness, SoL-Pi, and evaluated on held-out tasks that do not feed information back into the search process.

The scale of the search is substantial but does not by itself establish a scaling law. The authors report approximately 152 proposed directions, 535 executable development environments, more than 3,000 runs, and over 60,000 agent–environment interactions. The search is organized around six proposal families—context, progress, tools, delegation, prompt and policy, and improvement and evaluation—and uses isolated search lineages so that failed experiments do not contaminate other candidates.

The paper’s methodological claim is therefore narrower than a general theory of recursive self-improvement: automated, capability-constrained search over harness implementations can discover mechanisms that transfer across tasks and model backends, provided that development feedback and final validation remain strictly separated.

Search design and evaluation isolation

SoL-Pi uses a broad-to-deep search architecture. The outer stage generates and screens a large hypothesis pool. The inner stage independently implements, reviews, tests, and refines selected hypotheses. Each lineage follows an iterative cycle of proposal, implementation, fixed-budget evaluation, trajectory inspection, and retention or rejection. An implementation reviewer operates before evaluation, and failed reviews trigger revision rather than immediate candidate acceptance.

The separation between search and validation is a particularly important aspect of the design. Development trajectories guide mechanism discovery, while held-out results are observed only after the candidate, implementation, metrics, tolerances, and acceptance rule have been frozen. The authors explicitly prevent held-out failures from returning to the optimization loop. This addresses a central threat in harness evolution: an apparently general improvement may instead be a patch for the tasks used during search.

Figure 1: Development feedback guides mechanism refinement, while held-out evaluation occurs only after the harness is frozen and never returns to the search loop.

The search environments consist of 495 repository-derived tasks and 40 verifier-driven synthetic tasks. Repository tasks are constructed from pre-fix GitHub repositories with hidden regression tests whose failure and subsequent repair are verified automatically. Verifier-driven tasks admit multiple solution paths under executable success criteria, reducing dependence on a single reference patch. EdgeBench is excluded from mechanism discovery and is used only for frozen evaluation.

Figure 2: Repository-derived and verifier-driven environments support development search while EdgeBench remains isolated for final validation.

This separation is methodologically stronger than evaluating a modified harness on the same task pool used to generate it. Nevertheless, the paper does not provide a controlled decomposition of the value of each search-design choice—for example, the marginal contribution of breadth, lineage isolation, independent review, or the fixed acceptance rule. The search scale is reported descriptively, and the authors explicitly avoid interpreting it as evidence for a breadth- or depth-based scaling law.

The four retained mechanisms

The final SoL-Pi harness combines four mechanisms that intervene at different points in the agent–environment loop.

Action Fusion combines a file mutation with an immediately subsequent test, build, or execution command in a single tool request. The mechanism removes an intermediate model round trip when the follow-up command does not require inspecting the mutation result. Commands that require such inspection remain separate. In the Action Fusion lineage, prompt-only triggering proved unreliable; the final implementation exposed the fused action directly through the tool schema.

Online Context Compact makes context compaction conditional on estimated future savings. At plan-step completion, the harness estimates the remaining number of model requests from observed request rates and unfinished plan steps. It compares projected input savings against the cost of rewriting the prompt cache, accounts for unrecovered rewrite costs from earlier compactions, and invokes native compaction only when the savings margin is sufficient or the context approaches its limit.

ObservationPack archives large tool outputs locally. Results above 10 KiB are transmitted in full for the next two provider requests; from the third request onward, the harness supplies a stable handle, original size, and a 1 KiB excerpt containing complete head and tail lines. Exact chunks remain retrievable through the handle. Smaller observations are not transformed.

Evidence-Preserving Reducer processes sufficiently large build and test logs using a lower-cost model. It generates a compact evidence receipt, which is deterministically checked for schema validity, source-hash consistency, exit status, exact quotations, and actual size reduction. Any verification failure, suspected credential exposure, or absence of compression causes fallback to the original output. File reads and search results bypass the reducer.

The mechanisms are deliberately nonredundant in their intended operation: Action Fusion reduces action-level round trips; the reducer compresses semantically structured logs; ObservationPack controls repeated transmission of large observations; and Online Context Compact manages accumulated trajectory context.

Figure 3: The retained mechanisms reduce overhead through action fusion, conditional compaction, archived observations, and verified evidence extraction, with explicit fallback paths.

The interaction between ObservationPack and Evidence-Preserving Reducer is important. The reducer runs first, and ObservationPack recognizes its receipt marker and avoids repacking verified evidence. This ordering reduces the risk that a second compression layer will obscure or duplicate information. It also illustrates that the final efficiency result depends on integration behavior, not merely on the sum of isolated component savings.

EdgeBench performance and cost

The primary EdgeBench evaluation uses 51 public tasks from a larger 134-task benchmark. Eleven tasks are used for one-way acceptance of frozen candidates, while 40 are reserved for final evaluation. The paper reports two SoL-Pi operating points:

• SoL-Pi [Efficiency]: the complete four-mechanism stack.
• SoL-Pi [Performance]: the highest-scoring single-mechanism configuration selected separately for each backend.

Under GPT-5.6 Sol, the complete stack reduces recorded token traffic from 2.1538 billion tokens for Pi to 1.0990 billion, a 49.0% reduction. Average score decreases from 44.833 to 42.003, corresponding to 93.7% of Pi’s score. API cost falls from $1,339 to $894, and token cost per score point decreases from 0.5855 to 0.4174, a 28.7% improvement.

The performance-oriented configuration, ObservationPack alone, produces a different trade-off. It raises average score from 44.833 to 47.208, a 5.3% improvement, while reducing token traffic by 6.1% and token cost per score point by 9.8%. The paper therefore does not show that the full four-mechanism stack dominates every alternative on task quality; rather, it demonstrates a clear efficiency–performance frontier. The complete stack is the preferred cost point, whereas a single mechanism can provide the best measured score.

GPT-5.6 Sol configuration	Total tokens	API cost	Average score	Cost per score
Pi	2.1538B	$1,339	44.833	0.5855
SoL-Pi [Efficiency]	1.0990B	$894	42.003	0.4174
SoL-Pi [Performance]	2.0224B	$1,271	47.208	0.5280

The comparison with native harnesses is also material. Codex with GPT-5.6 Sol records 3.0537 billion tokens, costs $1,787, and achieves an average score of 34.738. Thus, the complete SoL-Pi stack uses approximately 64.0% fewer tokens and costs approximately 50.0% less than Codex in the reported EdgeBench setting, while achieving a substantially higher score. Relative to Claude Code with Opus 5, the teaser reports an API-cost reduction of 54.3% for the complete stack.

Figure 4: SoL-Pi retains comparable task performance to Pi while substantially reducing recorded token traffic and API cost relative to native and baseline harnesses.

The cost accounting includes input, cache-read, cache-write, and output traffic. This is important because context optimization can reduce cache reuse while still lowering total cost. Under GPT-5.6 Sol, the full stack reduces cache-read traffic from 2.1326 billion to 1.0605 billion tokens but increases cache-write traffic from 0.0141 billion to 0.0316 billion. The net API cost nevertheless falls by $445. The result supports evaluating end-to-end task cost rather than optimizing cache-read volume as an isolated metric.

Transfer across models and benchmarks

SoL-Pi is searched using GPT-5.6 Sol and then applied to Opus 5 without further search or adaptation. On Opus 5, Pi achieves an average score of 44.756 with 2.3697 billion recorded tokens and $1,741 in API cost. The complete SoL-Pi stack achieves 42.224, 1.3101 billion tokens, and $1,158. This represents a 44.7% token reduction and a 33.5% API-cost reduction relative to Pi, while retaining 94.3% of Pi’s average score.

The transfer result is positive but qualified. Mechanism activation is backend-dependent: trigger rates and trigger intensities are generally lower on Opus 5 than on GPT-5.6 Sol. The mechanisms therefore transfer as executable policies, but not with invariant behavioral frequency. The aggregate efficiency result remains favorable because the triggered instances still tend to save tokens.

The single-mechanism performance point again differs by backend. ObservationPack is best under GPT-5.6 Sol, whereas Action Fusion is best under Opus 5, achieving an average score of 50.482 with a cost of $1,605. This backend dependence indicates that the mechanisms interact with model-specific action patterns, context usage, and tool behavior.

Figure 5: Trigger frequency, activation intensity, and efficiency gains vary across GPT-5.6 Sol and Opus 5, demonstrating backend-dependent mechanism use.

The broader evaluations show a similar trade-off. On 63 CPU-only Terminal-Bench 4 tasks, Codex and Pi each solve 18 tasks, while SoL-Pi solves 15. SoL-Pi reduces total cost from $286.45 for Pi to $211.12, a 26.3% reduction, and lowers cost per solved task from $15.91 to $14.07, an 11.6% reduction. Here, efficiency improves despite a lower completion count; the result should not be interpreted as an unambiguous capability improvement.

On six IMO 2026 problems formalized and verified in Lean 4, SoL-Pi and Pi each pass three problems. SoL-Pi costs $62.69 in total and $20.90 per passed problem, compared with $75.95 and $25.32 for Pi. Codex passes five problems at $114.47, or $22.89 per passed problem. Thus, SoL-Pi provides the lowest cost per successful formalization in this small evaluation, but not the highest pass count.

Component behavior and integration effects

Add-one ablations show that each mechanism reduces total token traffic relative to Pi under both evaluated backends. Under GPT-5.6 Sol, Online Context Compact produces the lowest single-component cost at $935, while ObservationPack produces the highest average score at 47.208. Under Opus 5, ObservationPack yields the lowest single-component cost at $1,176, whereas Action Fusion produces the highest average score at 50.482.

The complete stack reduces cost further than any isolated mechanism: $894 under GPT-5.6 Sol and $1,158 under Opus 5. However, the interaction analysis is not a conventional factorial ablation. Each standalone and full-stack comparison uses its own triggered-task subset and corresponding disabled baseline. Consequently, the reported larger full-stack gains are consistent with complementarity but do not identify causal interaction effects independently of trigger selection.

Figure 6: Full-stack integration increases measured efficiency gains on the evaluated triggered-task subsets, although the subset-specific comparison does not isolate interaction effects causally.

ObservationPack becomes more selective in the full stack, plausibly because the Evidence-Preserving Reducer removes some large, compressible logs before ObservationPack can activate. This is a concrete example of mechanism composition changing activation distributions rather than merely adding independent savings.

The paper also analyzes Action Fusion as a discovery lineage. Oracle analysis identifies repeated adjacent actions and projects an 11.5% token reduction under full triggering. Subsequent iterations move from unreliable prompt-based triggering to explicit tool-schema support, followed by prompt and schema refinement. The lineage contains 27 recorded iterations, including 18 prompt-optimization explorations. Trigger rate becomes an intermediate metric alongside task score, allowing the search to optimize both whether opportunities are recognized and whether the resulting fused action remains correct.

Figure 7: Action Fusion progresses from trajectory-level opportunity detection to schema-level implementation, trigger-rate optimization, and held-out retention.

Agent-swarm evaluation

The paper evaluates whether harness efficiency remains useful in a coordinated multi-agent setting. A two-hour kernel-optimization experiment compares a single Codex agent, a Codex coordinator with 20 Pi workers, and a Codex coordinator with 20 SoL-Pi workers. All workers begin from the same frozen starter requiring 147,734 simulated cycles, and final candidates undergo independent correctness verification.

The single agent reaches 1,333 cycles at $39.20. The Pi swarm reaches 1,366 cycles at $82.12, while the SoL-Pi swarm reaches 1,127 cycles at $60.11. Relative to the Pi swarm, SoL-Pi improves the final result by approximately 17.5% and reduces API cost by 26.8%. The SoL-Pi swarm passes all eight speed thresholds, whereas the Pi swarm passes seven. The single agent remains the least expensive configuration, so the result does not establish that larger swarms are cost-optimal. It does indicate that reducing per-worker overhead can improve the effectiveness of a fixed-budget swarm.

Figure 8: In a two-hour coordinated kernel-optimization experiment, SoL-Pi workers achieve the best verified kernel result and lower cost than the Pi baseline swarm.

Limitations and open questions

The evaluation has several restrictions that constrain the strength of the conclusions. First, the main held-out evidence comes from 40 final EdgeBench tasks drawn from the 51 public tasks used in the evaluation setting, rather than the full 134-task benchmark. The authors also use 11 EdgeBench tasks for one-way frozen-candidate acceptance, so the held-out partition is not entirely independent of all EdgeBench-related decision-making, even though held-out results do not feed back into search.

Second, the mechanisms are searched primarily from GPT-5.6 Sol trajectories. Opus 5 transfer is therefore a cross-backend test of a Sol-trained harness, not multi-backend harness training. The lower trigger rates and intensities on Opus 5 show that transfer is behaviorally incomplete, even though aggregate efficiency remains favorable.

Third, the cost results depend on API prices stated for August 17, 2026 and on the particular cache accounting conventions used by the providers. The conclusions about token traffic are more stable than the absolute dollar savings, but the reported hourly savings—$1,741 in API cost. The complete SoL-Pi stack achieves 42.224, 1.3101 billion tokens, and $013.50 relative to native harnesses and $1,741 in API cost. The complete SoL-Pi stack achieves 42.224, 1.3101 billion tokens, and $15.71 relative to Pi—remain price- and workload-dependent.

Fourth, the component analysis does not provide a full factorial design. The full-stack gains are measured on configuration-specific triggered subsets, and the authors explicitly characterize the evidence for complementarity as descriptive. A stronger interaction claim would require matched trajectories or factorial evaluation with controlled activation.

Finally, the paper does not demonstrate recursive compounding. Although the authors propose using SoL-Pi to make subsequent harness search cheaper, no second search generation is reported. The “recursive efficient improvement” hypothesis therefore remains an untested research question. Similarly, the search includes more than 3,000 runs and 60,000 interactions, but the study does not quantify how performance changes as a function of search breadth, lineage depth, environment diversity, or research budget.

Conclusion

SoL-Pi presents a carefully structured approach to automated harness optimization in which token efficiency is treated as an emergent property of action execution, context management, observation storage, and evidence processing. Its four retained mechanisms reduce token traffic by 44.7–49.0% relative to Pi across GPT-5.6 Sol and Opus 5, reduce API cost by roughly one third, and preserve most of Pi’s EdgeBench score. The strongest task-quality result comes from selected single mechanisms rather than the full efficiency stack, establishing a measurable performance–cost trade-off rather than universal dominance.

The paper’s most consequential methodological result is the combination of broad executable search, independent refinement lineages, capability-constrained selection, and strict held-out isolation. The empirical evidence supports transfer across evaluated tasks, a second model backend, and a multi-agent swarm, while leaving open whether the search process itself exhibits a reproducible breadth–depth scaling law or can produce recursively compounding improvements.