Rethinking the Evaluation of Harness Evolution for Agents
Abstract: We revisit the evaluation of automatic harness evolution for LLM agents. Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark. This protocol raises two fundamental concerns. First, harness evolution is itself an iterative search procedure that repeatedly evaluates and revises candidate harnesses using task feedback. As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets to determine whether its gains arise from improved harness design or from additional search alone. Second, because the search and the final evaluation share the same benchmark, the reported gains risk overfitting to that specific task set. To address these concerns, we conduct an extensive evaluation comparing harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and also evaluate evolved harnesses on held-out tasks to assess whether the discovered improvements generalize. Experiments on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 show that automatic harness evolution does not consistently outperform simple test-time scaling methods and exhibits limited generalization. Our results raise important questions about the effectiveness of automatic harness evolution and highlight the need for fairer evaluation protocols and benchmarks for automatic harness design. Our code is available at https://github.com/rethinking-harness-evolution.
Summary
- The paper introduces a unified framework that separates genuine harness redesign benefits from improvements due to increased search budget.
- It compares four approaches—parallel sampling, sequential refinement, harness evolution, and harness scaling—demonstrating that test-time search methods often outperform harness modifications.
- Findings indicate that harness evolution yields marginal benefits and overfits evaluation tasks, highlighting challenges in achieving cross-task generalizability.
Rigorous Evaluation Protocols for Harness Evolution in LLM Agents
Background and Motivation
Automatic harness evolution aims to optimize the scaffolding—prompts, tools, memory, verification, and control logic—that surrounds fixed LLM agent policies, enhancing performance without retraining the base model. Existing protocols generally utilize unit test cases from public benchmarks for both search and final evaluation, which raises two core concerns: (1) untracked conflation of improvements due to search budget allocation versus genuine harness redesign, and (2) potential overfitting to the evaluation task set, limiting generalizability. The paper systematically interrogates these issues, contextualizing harness evolution with strong test-time scaling baselines and introducing rigorous metrics for discriminating between reusable harness improvements and ephemeral gains from additional sampling.
Unified Budget Framework and Methodological Taxonomy
Harness evolution and test-time scaling algorithms are operationalized under a unified compute budget and inference framework. Four approaches are delineated: parallel sampling, sequential refinement, harness evolution (across task batches), and harness scaling (per-instance adaptation). All methods are described in terms of what aspect is revised (trajectory or harness), what feedback is consumed (agent self-judgment or unit test cases), and how inference budget is allocated. This formalization distinguishes harness-level and solution-level optimization strategies and facilitates fair baseline comparisons.
Figure 1: Test-time scaling and automatic harness evolution algorithms. We compare four approaches under a unified budget, emphasizing what feedback the agent receives, how it spends its budget, and whether it modifies the trajectories or the harness.
Parallel sampling exploits multiple independent rollouts and aggregates via self-judgment or correctness signals. Sequential refinement iteratively improves trajectories, conditioning on prior attempts. Harness evolution leverages a meta agent to iterate on the harness itself using batches of tasks and accumulated experience. Harness scaling adapts harnesses per-instance, effectively providing single-task harness optimization, also driven by a meta agent. The explicit budget encodings expose whether reported gains stem from harness generalization or merely increased test-time exploration.
Experimental Results and Contradictory Findings
The evaluation protocol utilizes Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 models. Three settings are inspected: no unit test feedback, access to unit tests, and generalization to held-out tasks.
Without Unit Test Feedback
LLM agents revising harnesses or trajectories based only on self-generated feedback fail to consistently outperform simple test-time scaling. Parallel sampling yields the highest pass@1 improvement (68.2 to 72.3 averaged across models). Harness evolution produces negative gains for GPT-5.4, with degradation from 75.3 to 69.7. Harness scaling, while competitive on specific models, does not dominate parallel approaches. These outcomes underscore the inherent noisiness of harness revision without strong external success signals.
With Unit Test Feedback
Provision of unit test cases boosts all methods. However, automatic harness evolution and harness scaling do not surpass parallel sampling or sequential refinement. For pass@1, parallel sampling achieves an average of 86.0, outstripping harness evolution (75.8). For pass@5, sequential refinement yields 91.8 versus harness evolution’s 86.2. Harness evolution does not reliably convert failures to successes and is outperformed by solution-level optimization, indicating that repeated trajectory sampling is a more productive utilization of additional compute.
Generalization to Held-Out Tasks
Harnesses evolved on one set of tasks and selected based on validation performance produce minimal gains (average of +0.6 pass@1) when transferred to unseen test tasks. This starkly contrasts with improvements seen when harnesses are evaluated on the same instances used for search, demonstrating significant overfitting and limited cross-task generalization.
Harness Modification Analysis
Qualitative inspection reveals that both harness evolution and harness scaling meta agents make reasoned, multifaceted edits—enriching prompts with task-specific advice, enforcing runtime constraints in middleware, correcting tool parameters, and codifying workflow efficiencies. Harness scaling is particularly aggressive in encoding task-specific operational history and verification logic into the harness.
Figure 2: Examples of harness modifications made in Harness Scaling.
Despite the rationality and breadth of these edits, overall improvement remains marginal compared to test-time search baselines. The majority of modifications act as mnemonic aids (memorizing fixes per task) rather than distilling broadly applicable harness strategies. Hard failures, often outside prompt/middleware/tool control, are unaffected by harness evolution, while context bloat from accumulated artifacts can counteract minor gains.
Benchmark Sensitivity and Future Directions
Terminal-Bench 2.1 tasks are not highly sensitive to harness configuration, and agents attain near-optimal performance using minimal setups. The residual headroom for improvement is limited, and modifying harnesses yields diminishing returns relative to model reasoning capacity. Harness evolution’s utility is therefore hypothesized to be task-dependent. Its effectiveness should be revisited on benchmarks where agent success is gated more strongly by specialized harnesses, high domain complexity, and ample unsolved tasks.
Practical and Theoretical Implications
The rigorous comparison protocol and contradictory findings recalibrate expectations around harness evolution, revealing that existing protocols can conflate genuine harness design advances with test-time exploration effects. The poor generalization of evolved harnesses and their underperformance relative to parallel scaling baselines call for careful benchmarking and evaluation metric refinement. Practically, efforts should prioritize test-time search and solution-level exploration when compute is unconstrained. Theoretically, further research in harness evolution must address overfitting, context bloat, and scale up to benchmarks with greater harness sensitivity.
Conclusion
Automatic harness evolution, when evaluated fairly against test-time scaling baselines under unified compute budgets and inference constraints, exhibits limited advantage and poor generalization to novel tasks. Gains often arise from repeated sampling rather than generalizable harness redesign. This necessitates more rigorous, disentangled evaluation protocols and benchmarks to accurately measure reusable harness improvements for LLM agents (2607.12227).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about
This paper looks at how to fairly test a new idea called “automatic harness evolution” for AI agents. A “harness” is the setup around an AI that tells it how to work: the instructions it gets, the tools it can use, how it keeps notes, how it checks its work, and when to stop. Researchers have started letting AIs automatically edit this setup to try to perform better on tasks. The authors ask a simple question: are the reported gains from smarter harnesses, or just from letting the AI try more times?
The main questions the paper asks
- Are improvements from “harness evolution” actually better harness designs, or just the result of giving the AI more chances to search for answers during testing?
- If you evolve a harness using one set of tasks, will it still work well on new, unseen tasks (does it generalize), or does it mostly “memorize” the tasks it trained on?
- How should we compare harness evolution to simpler, well-known methods that just give the AI more attempts or more chances to refine answers at test time?
How they tested their ideas (in everyday language)
The team tested different ways to help an AI agent solve command-line tasks (like fixing code or running programs in a terminal) using a benchmark called Terminal-Bench 2.1.
They controlled the “budget,” which you can think of as:
- how many tries the AI gets,
- how much feedback it gets (like hints or a checker that says “right” or “wrong”).
They compared four approaches, all given the same total budget:
- Parallel sampling: Like taking multiple shots at the answer at the same time and then picking the best one. Think of trying several solutions in parallel and choosing the winner.
- Sequential refinement: Like writing an answer, reading it, fixing mistakes, and repeating. Each try learns from the previous one.
- Harness evolution: Instead of only changing the answer, the AI also changes its “game plan” (the harness) across many tasks—editing prompts, tools, and rules to hopefully make future tasks easier.
- Harness scaling: Like harness evolution but tailored per task. The AI repeatedly tweaks the harness for that one specific problem while trying again.
They tested under two conditions:
- Without a “unit test” checker: The AI had to judge its own answers. (A unit test is an automatic program that checks if a solution is correct.)
- With a unit test checker: The AI gets a clear “pass/fail” signal and can also select the best attempt by using the checker.
Finally, they also tested generalization: evolve a harness on one set of tasks, then measure performance on a separate, held-out set of tasks to see if the improvements carry over.
What they found and why it matters
- Without unit tests (no clear right/wrong feedback), simple methods often did better:
- Trying multiple answers at once (parallel sampling) was the most reliable boost.
- Letting the AI rewrite its harness (harness evolution) didn’t consistently beat these simple methods and sometimes made things worse. Self-feedback was noisy, so changes to the harness could amplify early mistakes.
- With unit tests (clear right/wrong feedback), simple methods still led:
- Both parallel sampling and sequential refinement outperformed harness evolution on key scores.
- Big gains mostly came from making multiple attempts and picking the best one, not from changing the harness.
- Poor generalization:
- A harness evolved on one group of tasks barely helped on new tasks. This suggests the evolved harness was “overfitting”—like studying the exact questions on a practice test instead of learning the general skills.
- Why these results likely happen:
- Many harness edits “memorized” fixes for the specific tasks seen during evolution instead of teaching general strategies.
- Some remaining hard tasks weren’t fixable by harness tweaks alone; they needed stronger reasoning from the model itself.
- Bigger, more detailed prompts and rules can also create “context bloat,” making it harder for the model to focus.
What this means going forward
- Compare fairly: When testing harness evolution, we should always compare it to strong, simple baselines (like giving more attempts or refining answers) under the same number of tries and the same feedback. Otherwise, we might credit harness edits for gains that really come from extra attempts.
- Avoid overfitting: Don’t use the same tasks to both tune the harness and measure final performance. Use separate sets to see if improvements truly transfer.
- Pick the right problems: Harness evolution may shine on tasks where the setup really matters (e.g., specialized tools and workflows), not tasks where performance mostly depends on the model’s raw reasoning.
- Bottom line: Today, automatic harness evolution did not consistently beat simpler “try more/think more” strategies and didn’t generalize well. Better evaluation protocols and more harness-sensitive benchmarks are needed to fairly measure progress.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper surfaces several issues but also leaves multiple concrete questions and limitations unresolved. The following list summarizes what remains missing or unexplored and can guide follow‑up research:
- Benchmark scope: Results are limited to Terminal‑Bench 2.1 (CLI coding tasks). It is unknown whether conclusions hold for harness‑sensitive domains (e.g., web, multi‑modal, robotics, data pipelines) where tools, workflows, and middleware are more pivotal.
- Cross‑benchmark transfer: Generalization is only tested via splits within the same benchmark. Transfer of evolved harnesses to entirely different suites (e.g., SWE‑Agent‑style code repair, web agents, research agents) remains unmeasured.
- Model coverage: Evaluations use Claude Opus 4.6, GPT‑5.4, and GPT‑5.4 mini. Effects on open‑source models, larger/smaller closed models, or mixing different models for policy vs. meta‑agent are not explored.
- Meta‑agent identity: The meta‑agent is instantiated by the same underlying model as the policy. Whether a stronger/different model as the meta‑agent (or an ensemble) changes outcomes is unknown.
- Harness evolution algorithm breadth: Only AHE (with its explore component disabled) is instantiated. Comparative studies with fully featured AHE, Meta‑Harness, AEVO, and other harness optimizers are missing.
- Search‑space characterization: The paper does not quantify or bound the harness edit/search space (prompt length, tool set size, middleware complexity) or how this space affects discoverability and overfitting.
- Cost and compute accounting: Although a unified budget K is used, the paper does not report token/step counts, wall‑clock time, or per‑method latency, leaving unclear whether budgets were truly matched in practice and how cost‑performance trade‑offs compare.
- Scaling curves: Only a single budget regime (K=5, m=1, up to 128k tokens) is tested. Performance as a function of compute (e.g., K∈{1,2,5,10,20}) and the point at which harness evolution might overtake test‑time scaling are not analyzed.
- Variance and statistical significance: Results are averaged over only two runs; no confidence intervals or statistical tests are provided. Stability across seeds and sensitivity to stochasticity remain unclear.
- Selection without unit tests: The non‑oracle selection uses a self‑judge. Alternatives (e.g., majority voting, external verifiers, learned rerankers, tool‑based heuristic checks) are not tested and may differentially benefit methods.
- Feedback granularity: Only binary unit‑test outcomes are used when available. The impact of finer‑grained signals (process rewards, stepwise checks, partial credit, static analyzers, lints) on stabilizing harness evolution is unstudied.
- Summarizer Φ design: The summarization function’s architecture, memory window, and ablations are not analyzed, leaving open how experience summarization quality affects evolution vs. scaling.
- Compute allocation strategies: The paper defines harness scaling and compares it to trajectory‑level scaling, but does not explore hybrid or adaptive schedulers that allocate compute between trajectory candidates, refinement depth, and harness edits based on online signals.
- Overfitting mitigation: While overfitting to search tasks is observed, the paper does not test countermeasures (e.g., cross‑validation across rounds, regularization of prompt growth, early stopping, edit penalties, stochastic task subsampling).
- Edit quality metrics: There is no quantitative analysis of harness “quality” beyond pass@k (e.g., prompt/middleware/tool complexity, context length bloat, readability/maintainability, brittleness), nor trade‑offs between harness size and performance.
- Failure taxonomy quantification: The discussion provides qualitative patterns (e.g., memorizing fixes), but lacks a quantitative failure‑class analysis linking specific edit types to success/failure deltas across task categories.
- Robustness and safety: Potential risks from middleware/tool edits (e.g., issuing unsafe commands, brittle heuristics, or environment‑dependent behavior) are not assessed; no safety metrics or guardrails are evaluated.
- Environment drift and reproducibility: Although TB2.1 repairs some drift, the robustness of evolved harnesses to changing dependencies, timeouts, and system configurations is not tested; reproducible environment snapshots and sensitivity analyses are absent.
- Human‑in‑the‑loop baselines: There is no comparison against modest manual harness engineering (e.g., 30–60 minutes of expert edits), which could contextualize the value of automation vs. human effort.
- Instance leakage controls: For harness scaling, safeguards to prevent task‑specific information from leaking into reusable artifacts are not described; policies for isolating per‑instance edits need clarification.
- Learning the meta‑agent: The meta‑agent operates zero‑shot; whether fine‑tuning or RL (e.g., PRM/RLHF‑style signals) for meta‑editing yields better generalization is an open question.
- Theoretical guidance: There is no formal analysis relating task harness‑sensitivity, model capacity, and expected gains from harness evolution vs. test‑time scaling, leaving practitioners without criteria for when to invest in each.
- Benchmark design: The paper hypothesizes that TB2.1 may be “too easy” or not harness‑sensitive, but does not propose or validate concrete benchmark properties or tasks that reliably reward better harness design.
- Compositional generalization: Whether evolved harness components (tools, middleware checks, prompt modules) compose to help unseen combinations of subtasks is not investigated.
- Release artifacts: While code is promised, the paper does not quantify or catalog the actual harness edits produced (e.g., diff statistics, categories), limiting replicability of qualitative claims.
Practical Applications
Practical Applications Derived from the Paper’s Findings
Below are actionable applications organized by immediacy, reflecting how teams can use the paper’s evaluation insights, baselines, and methodology to improve real-world LLM agent systems.
Immediate Applications
- Agent evaluation protocol overhaul for fair comparison
- What to do: Adopt a unified budget protocol that matches feedback types (e.g., unit tests vs. self-judging), inference budgets, and task overlap when comparing agent improvements. Always benchmark harness evolution against simple test-time scaling baselines (parallel sampling, sequential refinement) under the same budget.
- Sectors: Software, AI platforms, research engineering.
- Tools/products/workflows: “Agent Evaluation Kit” (scripts to enforce matched budgets, pass@k reporting, overlap checks), CI-triggered evaluation jobs, A/B harness tests.
- Assumptions/dependencies: Access to verifiers or unit tests for target tasks; ability to instrument compute budgets; reproducible task suites.
- Prioritize test-time scaling before harness redesign in production agents
- What to do: Shift extra compute to parallel sampling and sequential refinement to improve pass@k quickly; use self-judging when no unit tests are available and oracle selection when they are.
- Sectors: Software engineering agents (code assistants, CLI agents), web agents, customer support copilots.
- Tools/products/workflows: “Self-Judge Orchestrator” (aggregates parallel candidates), verifier-driven reranking, pass@k dashboards.
- Assumptions/dependencies: Stable cost budgets; sufficiently strong base models; reliable self-judging prompts or external verifiers.
- Instance-specific harness scaling for per-task adaptation
- What to do: Apply harness scaling (task-level harness updates guided by previous rollouts and unit-test feedback) when verifiable signals exist; use it to fix workflow inefficiencies and add targeted checks for recurring failures.
- Sectors: DevOps agents, data engineering agents, ETL pipelines, RPA.
- Tools/products/workflows: “Harness Tuner” (runtime meta-agent that edits prompts/middleware/tools per task), verifier-integrated refinement loops.
- Assumptions/dependencies: Access to rapid feedback (tests/verifiers); guardrails to prevent overfitting and context bloat; compute budget for iterative runs.
- Benchmark hygiene and anti-overfitting controls
- What to do: Separate search/optimization tasks from final evaluation tasks; maintain train/validation/test splits for agent harness work; explicitly report overlap and leakage risk in papers and product metrics.
- Sectors: Academia (evaluation studies), enterprise model governance.
- Tools/products/workflows: “Agent Benchmark Splitter” (creates disjoint splits), governance templates for evaluation documentation.
- Assumptions/dependencies: Sufficient task volume and diversity; clear benchmark metadata.
- Compute budget accounting and transparency
- What to do: Instrument and disclose inference budgets (e.g., K trajectories, token limits) and feedback types used; enforce budget parity across baselines in internal and external evaluations.
- Sectors: Finance (cost control), AI platform ops (quota management), enterprise AI governance.
- Tools/products/workflows: “Compute Budget Manager” (tracks per-run K, tokens, retries), cost dashboards, budget guardrails in CI.
- Assumptions/dependencies: Logging infrastructure; billing integration; consistent parameterization across runs.
- Guardrails against context bloat and brittle harness edits
- What to do: Monitor harness growth; prune non-generalizable prompts/memory; implement middleware constraints (timeouts, output truncation) conservatively to avoid degrading strong models.
- Sectors: Software agents, knowledge workers using AI automation.
- Tools/products/workflows: Prompt memory managers, “Harness Diff & Audit” (tracks what changed and why), automated linting for harness components.
- Assumptions/dependencies: Telemetry on token usage; policies for memory retention; tooling to diff prompt/tool/middleware edits.
- Unit-test–driven agent orchestration for reliability
- What to do: Integrate unit tests or external verifiers to drive sequential refinement loops; gate acceptance on pass@k outcomes; use verifier-informed retries rather than blind harness edits.
- Sectors: Software QA, data validation, compliance workflows.
- Tools/products/workflows: Verifier plugins, pass@k gating policies, “Fail-Fast Verifier Router.”
- Assumptions/dependencies: High-quality tests; deterministic environments; alignment between tests and task instructions.
- Reporting standards in research and procurement
- What to do: Require disclosure of feedback sources, test overlap, and compute budgets when claiming harness improvements; evaluate claims against strong test-time scaling baselines.
- Sectors: Academia, policy, enterprise procurement.
- Tools/products/workflows: Evaluation checklists, reproducibility statements, procurement RFP templates including compute disclosure.
- Assumptions/dependencies: Organizational buy-in; accessible evaluation artifacts.
Long-Term Applications
- Domain-specific, harness-sensitive benchmarks with real headroom
- What to build: New benchmarks where specialized tools/workflows materially affect success (e.g., robotics manipulation routines, healthcare EHR tasks with constrained tools, complex web navigation under compliance).
- Sectors: Robotics, healthcare, web agents, energy operations.
- Tools/products/workflows: Community benchmark suites; standardized verifiers; task libraries emphasizing harness variability.
- Assumptions/dependencies: Multi-organization collaboration; data access; safety/compliance review.
- Generalizable harness design via meta-learning
- What to build: Meta-agents that learn reusable harness components across diverse distributions rather than memorizing task-specific fixes; focus on transfer and robustness metrics.
- Sectors: AI frameworks, agent platform vendors.
- Tools/products/workflows: Cross-domain training pipelines, meta-RL or supervised meta-learning on harness edits, “Harness Library” components.
- Assumptions/dependencies: Large, diverse task corpora; reliable correctness signals; scalable training budgets.
- Standards and governance for agent evaluation
- What to build: Sector-wide guidelines (akin to NIST-style evaluations) mandating separation of optimization and measurement, budget parity, and generalization tests for agent claims.
- Sectors: Policy/regulation, compliance-heavy industries (finance, healthcare).
- Tools/products/workflows: Certification programs, audit protocols, conformance tests.
- Assumptions/dependencies: Regulator engagement; industry consortiums; consensus on metrics.
- Adaptive compute schedulers for agents
- What to build: Decision systems that allocate compute dynamically between parallel sampling, sequential refinement, and harness scaling based on task signals, expected ROI, and verifier availability.
- Sectors: Cloud AI ops, energy-efficient compute, cost-sensitive enterprises.
- Tools/products/workflows: RL-based schedulers, “Test-Time Compute Optimizer,” per-task policy gradients over allocation strategies.
- Assumptions/dependencies: Telemetry, accurate reward signals, policy safety constraints.
- Harness evolution grounded by richer process rewards
- What to build: Use process reward models, structured verifiers, or step-wise correctness signals to guide harness changes; reduce overfitting by grounding revisions in external, generalizable feedback.
- Sectors: Complex reasoning agents (research copilots, math, law).
- Tools/products/workflows: PRMs, step-wise verifiers, “Harness-Change Evaluator” scoring incremental edits against process criteria.
- Assumptions/dependencies: Availability and reliability of process-level feedback; model capabilities to parse process signals.
- Structured harness DSLs, versioning, and audits
- What to build: A domain-specific language for prompts/tools/middleware with explicit constraints, versioning, and diff-able changes; integrate with CI/CD and compliance.
- Sectors: Enterprise AI engineering, regulated industries.
- Tools/products/workflows: “Harness DSL,” version control integrations, policy linters for harness edits.
- Assumptions/dependencies: Standardization across vendors; integration with existing DevOps stacks.
- Automated agent CI pipelines focused on generalization
- What to build: “Harness CI” that automatically runs disjoint splits, reports pass@k under matched budgets, flags overfitting, and suggests when to prefer test-time scaling over harness edits.
- Sectors: Software platforms, agent tooling providers.
- Tools/products/workflows: CI plugins, evaluation orchestrators, policy-based gating in production.
- Assumptions/dependencies: Reliable benchmarks; engineering investment; governance support.
- Safety, fairness, and reproducibility audits for agent claims
- What to build: Auditing frameworks to ensure claims are not inflated by task overlap or compute discrepancies; include reproducibility assessments and bias checks for agent workflows.
- Sectors: Healthcare, finance, education, public sector.
- Tools/products/workflows: Audit templates, reproducibility badges, third-party evaluation services.
- Assumptions/dependencies: Access to evaluation artifacts; independent auditors; stakeholder incentives.
- Practitioner education and daily-use guidance
- What to build: Curricula and best-practice guides that teach when to use parallel sampling vs. sequential refinement vs. harness scaling; emphasize held-out testing and compute transparency.
- Sectors: Education, professional training, power users automating workflows.
- Tools/products/workflows: Course modules, lab exercises using open benchmarks, “Agent Ops Playbook.”
- Assumptions/dependencies: Public datasets; teaching resources; community adoption.
Each application’s feasibility depends on factors the paper highlights: availability of reliable correctness signals (unit tests/verifiers), matched and disclosed compute budgets, disjoint task splits to prevent overfitting, sufficiently capable base models, and organizational commitment to transparent, fair evaluation.
Glossary
- Agent policy: The parameterized decision-making function that maps observations to actions for an agent. "Let denote the agent policy"
- Agentic test-time scaling: Allocating extra computation at inference for an agent to actively search or refine solutions. "As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets"
- Automatic harness evolution: Methods that iteratively revise the external scaffold around an LLM agent using feedback to improve performance. "We revisit the evaluation of automatic harness evolution for LLM agents."
- Compute budget: The fixed amount of inference/computation a method can expend during evaluation or search. "Every method below consumes a compute budget and returns a final trajectory for each task ."
- Experience store: A repository of tasks, harness versions, rollouts, and outcomes used to inform subsequent edits. "The map summarizes an experience store into a form that a downstream agent can consume, such as recurring failures, redundant attempts, or cost, so that subsequent refinement or editing steps can diagnose why earlier attempts underperformed and avoid repeating edits."
- Finalization gates: Middleware checks that prevent completion when required outputs are missing or unverified. "finalization gates that block completion when deliverables are missing or unverified."
- Harness: The external scaffold specifying prompts, tools, memory, verification routines, and control logic for agent interaction. "A harness defines the prompts, tools, memory, verification routines, and control logic through which a model observes tasks and acts"
- Harness engineering: The practice of designing and tuning the harness to improve agent performance without changing model weights. "Prior work shows that harness engineering can substantially affect agent performance even with a fixed model"
- Harness scaling: Instance-specific adaptation of the harness during a single task by spending extra inference budget on harness revisions. "we introduce harness scaling (Figure~\ref{fig:framework} (d)), a harness-level analogue of test-time scaling."
- Held-out tasks: Evaluation tasks not seen during search, used to test generalization. "and also evaluate evolved harnesses on held-out tasks to assess whether the discovered improvements generalize."
- Interaction horizon: The depth or number of iterative steps in a refinement process at inference time. "allocates the budget to depth, extending the interaction horizon so that each trajectory builds on the last."
- Long-term memory: Persistent information stored in the harness (e.g., rules, notes, artifacts) that the agent can reuse across steps or tasks. "evolves prompts, tools, middleware, skills, and long-term memory through an observability-driven loop."
- Meta agent: A higher-level controller that reads summarized experience and proposes the next harness update. "A meta agent drives the search."
- Middleware: Runtime components between the model and tools that enforce behaviors, budgets, and checks. "the meta agent escalates to runtime enforcement through middleware, such as turn budget trackers that remind the agent to ship output once turn or time thresholds are reached, truncation of oversized tool outputs, and finalization gates that block completion when deliverables are missing or unverified."
- Parallel sampling: A test-time scaling strategy that generates multiple independent candidate trajectories and selects among them. "Parallel sampling (Figure~\ref{fig:framework} (a)) is a representative test-time scaling method that generates multiple trajectories in parallel"
- Pass@1: The fraction of tasks solved by the top (single) attempt. "Average pass@1 without access to unit test feedback, averaged across Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini."
- Pass@5: The fraction of tasks solved when selecting among multiple attempts (up to five). "Because oracle selection allows the best candidate among multiple sampled trajectories to be identified, reporting pass@5 becomes meaningful alongside pass@1, and we therefore report both metrics under this setting."
- Reranking: Selecting the best candidate among multiple outputs using an external verifier or criterion. "aggregates them by voting or verifier reranking."
- Rollouts: Individual executions of an agent on a task that produce trajectories and outcomes. "we sample rollouts per task under the current harness,"
- Seed harness: The initial harness configuration that serves as the starting point for evolution or scaling. "starting from a seed harness ."
- Self judge: An internal model-based evaluator used to score and select among candidates when no external tests are available. "Let be a self judge realized by the agent itself, and we return the highest scoring candidate,"
- Sequential refinement: A test-time scaling method where each attempt conditions on previous attempts to iteratively improve the solution. "Sequential refinement~\citep{madaan2023self} (Figure~\ref{fig:framework} (b)) allocates the budget to depth, extending the interaction horizon so that each trajectory builds on the last."
- Summarization map: A function (often using the same model) that compresses stored experiences into a digestible summary for the meta agent. "We also define a summarization map , realized by the same underlying model as the policy."
- Task distribution: The probability distribution over tasks from which evaluation or training tasks are drawn. "let denote a task drawn from a task distribution ."
- Terminal-Bench 2.1: A benchmark suite of command-line tasks used to evaluate code agents and harness methods. "We evaluate these methods on Terminal-Bench~2.1 with multiple models"
- Test-time scaling: Improving model outputs by spending additional inference-time computation without changing model parameters. "Test-time scaling refers to improving a LLM's outputs by allocating additional computation at inference, without modifying the model weights"
- Trajectory: The sequence of states, actions, and observations produced by an agent during task execution. "Executing the agent on using the harness yields a trajectory , the full sequence of states, actions, and observations,"
- Unit test case: A verifiable test used as feedback and as an oracle for selecting successful trajectories. "When a unit test case is available, we define an outcome "
- Verifier feedback: Signals from automated checks/tests on benchmark tasks that guide harness search or refinement. "harness search uses verifier feedback from benchmark tasks, and the final harness is then evaluated on the same public benchmark,"
- Verifier reranking: Using a verifier to score and rank multiple sampled answers to choose the final output. "aggregates them by voting or verifier reranking."
Collections
Sign up for free to add this paper to one or more collections.