Papers
Topics
Authors
Recent
Search
2000 character limit reached

Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design

Published 20 Oct 2026 in cs.CL and cs.SE | (2610.28380v1)

Abstract: Agent harnesses are executable code that specifies the prompts, tools, and control flow around a LLM. A single global harness can struggle with heterogeneous tasks that require different execution strategies. In this work, we introduce Object-Oriented Harness (OO-Harness), a harness optimization framework that evolves a hierarchy of executable harness classes using object-oriented design. During training, an evolution agent uses execution records to create a family of specialized harnesses, each implemented as a subclass of the base or an existing specialized harness via inheritance. At test time, a harness router uses routing lessons accumulated during training to select a specific harness for each task. Across three long-horizon benchmarks spanning business workflows, workspace tasks, and tool use, OO-Harness achieves the best performance among the compared methods, with an absolute improvement of up to 15% over the base. Extensive ablation studies examine the contributions of its core components.

Summary

  • The paper demonstrates that an object-oriented approach to agent harnesses, using inheritance and a cluster-based evolution and routing strategy, outpaces global optimization and specialist-only methods with an absolute improvement of up to 15.79pp% across three benchmarks.
  • The study discovers clustering, object-oriented inheritance, and task evaluation mechanisms lead to effective harness specialization that preserves the success of base executions while allowing targeted improvements.
  • Trajectory-aware routing significantly boosts performance by up to 1.3% in true performance metrics and shows object-oriented inheritance is key to effective classifier maintenance and reliability.

Problem formulation and motivation

Agent harnesses encode the executable policy surrounding a fixed language-model executor: prompts, tool interfaces, context management, middleware, file backends, subagent configuration, termination logic, and completion checks. The paper argues that optimizing a single global harness is structurally problematic for heterogeneous long-horizon tasks. A modification that improves document generation, API error recovery, or spreadsheet verification may interfere with unrelated workflows. Conversely, independently optimized specialists may duplicate code and fail to preserve broadly useful behavior.

"Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design" (2610.28380) addresses this tension by representing harness adaptation as a hierarchy of executable classes. The base harness remains available, while specialized subclasses inherit existing behavior and selectively override methods or add new ones. A router then chooses among the base and specialized harnesses for each task. This representation separates shared mechanisms from task-conditional interventions without requiring every change to be applied globally.

The paper contrasts three adaptation structures: a single global harness, a collection of independent specialists, and a hierarchy whose specialists share implementation through inheritance.

Figure 1

Figure 1: Global, independent-specialist, and inheritance-based organizations for harness adaptation.

The central claim is not merely that task-specific harnesses can outperform a global harness, but that object-oriented structure provides a useful search space for evolving them. Inheritance allows later candidates to extend accepted mechanisms through super() calls, while routing allows the system to avoid applying a learned intervention when the base execution is already adequate.

Method

Discovery, evolution, and deployment

The method separates harness evolution from evaluation. Discovery tasks are used to initialize clusters, generate candidate subclasses, and accumulate design and routing lessons. Evaluation is performed on disjoint held-out tasks, with the executor weights fixed in both phases.

Figure 2

Figure 2: Harnesses evolve on discovery tasks and are evaluated on disjoint held-out tasks.

This split is important because global harness optimization can otherwise exploit the same tasks used for final scoring. The paper explicitly distinguishes its protocol from an evaluation setting in which search and final evaluation share tasks. The resulting claims therefore concern held-out generalization rather than only discovery-set improvement.

The initial base execution produces records containing trajectories, outcomes, errors, and reflections. An evolution agent clusters discovery tasks according to potentially shared execution needs. These clusters are not required to correspond to benchmark domains. They may instead represent recurrent failure modes, such as missing state-changing operations, repeated API errors, incomplete artifact creation, or failure to finalize within the tool-call budget.

For each evolution iteration, the agent selects a target cluster and a parent class, proposes a subclass, and evaluates it only on the target cluster. A candidate is retained when it improves the cluster score and satisfies benchmark-specific acceptance checks. Accepted classes are added to the hierarchy and may become the active harness for the cluster. Failed proposals remain available as evidence for subsequent revisions. Clusters may also be split, merged, or reassigned as execution evidence accumulates.

This cluster-local evaluation is a computational design choice with direct implications. A candidate need not be rerun over the complete discovery set, reducing recorded search time and API usage. However, acceptance is based on target-cluster improvement rather than universal improvement, so the method depends on routing and cluster formation to prevent locally beneficial changes from being applied inappropriately.

At deployment, the hierarchy, class descriptions, and routing lessons are frozen. For each new task, the router observes the task description and an observable summary of the base trajectory. It selects either a specialized harness or the base result. If it selects the base, the original execution is reused rather than rerun. If it selects a specialist, the complete subclass executes as a single instance, with inherited and overridden methods composing the final behavior.

Figure 3

Figure 3: Training-time subclass evolution and frozen test-time routing.

The use of the base trajectory is a distinctive aspect of the method. A task description may indicate the desired artifact or application, but it may not reveal whether the current execution has encountered repeated query-parsing failures, omitted a required write, or produced an incomplete file. The trajectory therefore supplies an execution-state signal for selecting interventions whose applicability depends on observed failure.

Object-oriented harness representation

The hierarchy contains complete executable harness classes rather than independent fragments executed sequentially. A child inherits methods from its parent, overrides selected methods, and can invoke the parent implementation through super(). It may also add new methods for validation, recovery, or artifact inspection. The executor model itself remains fixed; adaptation occurs in the code around it.

The paper gives several concrete examples. In AutomationBench, one lineage begins with a missing-write completion check, adds a near-budget reminder, and then incorporates guidance for tool errors and repeated unproductive results. In Workspace-Bench, a specialized file backend converts textual content into a binary Word document rather than merely writing text to a path ending in .docx. In Toolathlon-GYM, a descendant checks saved spreadsheet cells and can request a targeted correction when percentage-valued cells appear to contain fractional rather than percentage-point values.

These examples clarify the division between inherited and specialized behavior. A child can preserve a parent’s context-management or completion-check logic while narrowing when it applies, adding a new condition, or introducing a post-execution artifact check. The hierarchy thus supports incremental specialization rather than repeated reconstruction from the base.

Lessons and routing

The evolution process maintains two forms of summarized evidence. Design lessons describe successful changes, failed approaches, and mechanisms associated with discovery outcomes. Routing lessons describe observable conditions under which a given harness should or should not be selected. Both are frozen before held-out evaluation.

The router is instructed to choose the base when no specialist clearly matches or when a learned anti-pattern suggests that specialization would be harmful. This conservative option is central to the paper’s generalization argument: a specialist is not assumed to be universally better merely because it improved its discovery cluster.

Experimental design

The evaluation spans three long-horizon benchmarks with distinct execution environments:

Benchmark Workload Discovery tasks Held-out tasks Primary metric
AutomationBench Business workflows and API-mediated state changes 300 300 Mean partial credit
Workspace-Bench File and application workflows 288 100 Micro-averaged rubric pass
Toolathlon-GYM Multi-tool and multi-application tasks 251 252 Strict PASS

All methods use the same Gemini 3.5 Flash executor. Harness evolution uses Claude Opus 4.8 for up to 40 iterations, while the main router uses Claude Sonnet 5. The comparisons include the unevolved base, global optimization with Meta-Harness and Agentic Harness Engineering, specialized harnesses from Adaptive Auto-Harness, and inference-time scaling through parallel sampling or sequential self-refinement.

This comparison isolates harness evolution from executor improvement. It also tests whether additional inference alone can reproduce the gains attributed to learned execution policies. The paper reports three independent executor runs for the main comparison, although several ablations are single-run experiments.

Main empirical results

The proposed method achieves the highest primary score on all three benchmarks. Its held-out results are summarized below.

Method AutomationBench Mean Workspace-Bench Mean Toolathlon-GYM PASS
Base 61.62 25.45 31.15
Meta-Harness 63.92 29.56 33.45
Agentic Harness Engineering 64.18 38.43 36.23
Adaptive Auto-Harness 75.40 38.62 34.32
Object-Oriented Harness 77.41 39.48 37.45

The absolute improvement over the base is 15.79 percentage points on AutomationBench, 14.03 points on Workspace-Bench, and 6.30 points in Toolathlon-GYM PASS. Relative to Adaptive Auto-Harness, the gains are 2.01 points on AutomationBench, 0.86 points on Workspace-Bench, and 3.13 points on Toolathlon-GYM.

Figure 4

Figure 4: Hierarchical optimization shows faster early gains, shorter recorded search time, and higher held-out scores than global Meta-Harness optimization.

The result is significant because the principal advantage is not confined to one benchmark family. AutomationBench favors trajectory-dependent recovery from API and completion failures; Workspace-Bench exposes backend and artifact-format requirements; Toolathlon-GYM combines heterogeneous applications and tool protocols. The consistent ranking supports the claim that inheritance and selective routing provide a useful representation across different harness interfaces.

The method also produces the lowest mean final-trajectory step count on AutomationBench: 27.99 steps, compared with 28.89 for Adaptive Auto-Harness, 30.05 for Meta-Harness, 30.61 for Agentic Harness Engineering, and 31.27 for the base. This indicates that the performance gain is not explained simply by longer successful trajectories on that benchmark. On Toolathlon-GYM, it also reduces steps to 37.25, compared with 39.85 for Meta-Harness and 41.23 for Adaptive Auto-Harness.

The inference-scaling baselines are inconsistent. Parallel best-of-five improves Workspace-Bench Mean to 37.71 and Toolathlon-GYM Mean to 75.69, but lowers AutomationBench Mean to 60.24. Sequential self-refinement similarly fails to improve the primary metrics consistently. The implication is specific: additional base executions or self-refinement do not substitute reliably for learning when the required intervention concerns executable control flow, backend semantics, or completion verification.

Routing and inheritance ablations

Trajectory-informed routing

The routing ablation holds the learned harness catalogue fixed and varies only the router’s evidence. Task-only routing receives descriptions and metadata; trajectory-based routing additionally observes the base execution.

Benchmark and metric Base Task only Task plus trajectory
AutomationBench Mean 61.30 64.88 77.52
Workspace-Bench Mean 24.57 40.03 40.56
Toolathlon-GYM PASS 31.35 33.33 37.70

Trajectory evidence improves AutomationBench by 12.64 points over task-only routing and Toolathlon-GYM PASS by 4.37 points. The Workspace-Bench difference is only 0.53 points. The contrast is diagnostically useful. Workspace-Bench clusters largely correspond to visible output types such as documents and spreadsheets, so task metadata often identifies the appropriate specialist. AutomationBench and Toolathlon-GYM contain cross-domain failure modes, such as missing writes, query errors, or incomplete finalization, that are not reliably inferable from the task description. In those settings, the base trajectory exposes the mechanism requiring intervention.

Figure 5

Figure 5: Learned clusters reflect task types on Workspace-Bench and execution needs across domains on AutomationBench, with routing lessons linking observable failures to specialized harnesses.

The routing design also preserves strong base executions. In one AutomationBench run, 134 of 300 tasks reused the base, while 166 were routed to specialists. The tasks routed to specialists had an original base mean reward of 44.40%, compared with 77.15% for tasks whose base result was reused. Their strict base PASS rates were 14.25% and 36.57%, respectively. This separation indicates that routing is not merely applying a specialist indiscriminately; it preferentially directs intervention toward tasks with weaker base behavior.

Inheritance structure

The inheritance ablation compares three search strategies on Workspace-Bench: Base-reset, which starts every candidate from the base; Flat, which copies previously accepted code into a new base-derived class; and Object-Oriented Harness, which inherits from the currently deployed parent.

Figure 6

Figure 6: Inheritance-based evolution outperforms copy-based reuse and restarting from the base on Workspace-Bench.

Object-Oriented Harness reaches 40.56% micro rubric pass, exceeding Flat by 4.96 points. Flat itself outperforms Base-reset, establishing that carrying forward accepted mechanisms is beneficial. The additional gap between Flat and inheritance supports a narrower claim: the organizational structure of reuse matters, not only the existence of copied behavior. Inheritance avoids duplicating parent code, permits explicit super() composition, and allows descendants to modify a targeted seam without reconstructing the full harness.

This ablation should not be interpreted as a universal proof that inheritance dominates all alternative program representations. The variants conduct separate searches, and the result depends on the supplied harness interfaces and evolution-agent prompts. It does, however, establish that under the paper’s controlled Workspace-Bench setting, inheritance produces a stronger learned catalogue than the two copy-based alternatives.

Learned specializations and out-of-distribution behavior

The qualitative analyses connect score improvements to concrete executable changes. A Workspace-Bench document specialist intercepts writes intended for Word documents and renders a binary document, while retaining fallback behavior if rendering fails. An AutomationBench lineage adds increasingly targeted checks for required writes, imminent budget exhaustion, and tool errors. A Toolathlon-GYM specialist inspects saved workbooks rather than relying only on the model’s textual completion claim.

These interventions operate at different levels of the harness: backend transformation, runtime control, context guidance, and artifact verification. The result is evidence that the method is not merely evolving prompts. It changes the operational semantics surrounding the executor.

The learned clusters also need not align with benchmark-provided taxonomy. AutomationBench clusters can span business domains when the same execution failure recurs across them. Workspace-Bench clusters more closely follow task types because output requirements imply backend and artifact constraints. Toolathlon-GYM clusters can span multiple application groups when they share completion or finalization needs.

The paper further evaluates an extreme OOD split in which Toolathlon training and test application families are disjoint. After evolution on 254 training tasks, the router selects the base for all 49 held-out tasks. Reusing the base preserves its performance, whereas applying the learned global Meta-Harness decreases PASS.

Figure 7

Figure 7: In the disjoint-application OOD setting, selecting the base preserves performance while forcing the evolved global harness reduces PASS.

This is a deliberately conservative outcome rather than a claim that specialization transfers to every unseen application. The implication is that retaining the base as an explicit routing option can prevent negative transfer when the learned specialists have no demonstrated applicability. The result also exposes a distinction between adaptation and generalization: a global change that improves discovery tasks can be harmful when its assumptions do not hold in a new application family.

Limitations and open questions

The method depends on the host benchmark’s APIs, scaffold, tool contracts, filesystem semantics, and execution environment. Evolved subclasses can become invalid or unreliable when interfaces change, services experience outages, rate limits intervene, or private credentials and permissions are configured differently. The paper acknowledges that deployment therefore requires secure credential handling and environment-specific maintenance.

The evaluation also leaves several methodological questions open. First, the principal systems use substantial hosted-model computation for evolution, routing, execution, and in some settings judging; the reported benchmark scores do not provide a complete cost-normalized comparison. Cluster-local candidate evaluation may reduce search cost, but the total cost of trajectory collection, lesson synthesis, routing calls, and specialist reruns remains important.

Second, routing is evaluated with frozen catalogues and observable base trajectories, but the reliability of router confidence is not established. In the task-only ablation, the confidence threshold is self-reported rather than calibrated against specialist success. A specialist selected on the basis of an incorrect applicability judgment can impose additional execution cost or degrade a previously adequate base result.

Third, inheritance introduces lineage and maintenance dependencies. A change to a parent method can affect multiple descendants, while copied alternatives may avoid such coupling at the cost of duplication. The experiments demonstrate an advantage for inheritance in one Workspace-Bench ablation, but do not fully characterize hierarchy depth, parent-selection errors, conflicting overrides, or the effect of refactoring large class families.

Finally, the OOD experiment shows that the router can conservatively select the base, not that evolved specialists acquire robust capabilities on unseen applications. The open technical question is how to determine, without held-out grader feedback, when a specialized mechanism is semantically transferable across environments and when its assumptions should be rejected.

Conclusion

Object-Oriented Harness formulates harness evolution as hierarchical program search over executable subclasses. Its main contribution is the combination of inheritance-based code reuse, cluster-local evolution, trajectory-informed routing, and explicit base reuse. Across AutomationBench, Workspace-Bench, and Toolathlon-GYM, the method obtains the highest reported primary scores, with improvements of up to 15.79 percentage points over the base harness. The ablations attribute these gains to both inheritance and execution-aware routing, while the OOD analysis demonstrates the value of preserving the base when learned specializations are not justified. The remaining challenge is to make the hierarchy, routing decisions, and environment-specific assumptions reliable under changing interfaces and broader distribution shift.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

1. What is the paper about?

This paper presents a system called Object-Oriented Harness, or OO-Harness.

The system helps an AI agent complete difficult, multi-step tasks. For example, an agent might need to:

  • Use several computer tools
  • Edit files
  • Follow a business process
  • Check whether its work is finished
  • Recover from errors

A harness is the code that controls how an AI model performs these tasks. It decides what instructions to give the model, which tools it can use, how it remembers information, and how it checks its work.

The main idea of the paper is simple:

Instead of creating one harness for every task, build a family of specialized harnesses and choose the best one for each task.

This is similar to having one basic vehicle, plus special versions designed for racing, carrying heavy objects, or driving in snow.

2. What questions does the research ask?

The researchers wanted to find out:

  1. Can an AI automatically create specialized harnesses for different kinds of tasks?
  2. Can these harnesses share useful code instead of being built completely from scratch?
  3. Can another AI, called a router, choose the right harness for each new task?
  4. Does this approach work better than using one general harness for everything?
  5. Does looking at the agent’s previous actions help the router make better choices?

The paper also studies whether this approach can save time and computer costs during training.

3. How was the research done?

The basic idea: object-oriented programming

OO-Harness uses a programming technique called object-oriented design.

In everyday terms, imagine creating a basic character in a video game. The basic character can walk, jump, and collect objects. You can then create special versions:

  • A swimmer who knows how to move through water
  • A climber who can climb walls
  • A fighter who has extra combat abilities

The special characters inherit the basic abilities but can also change or add new ones.

OO-Harness works in a similar way:

  • It begins with a base harness, called H0H_0.
  • It creates specialized harnesses as children of the base harness.
  • A specialized harness can reuse existing code.
  • It can also override an old behavior or add a new behavior.

For example, a specialized harness might add instructions that remind the agent to save a file before finishing or help it recover from a tool error.

Training process

The researchers used the following process:

  1. They ran the base harness on a set of training-like tasks called discovery tasks.
  2. They examined the agent’s actions, errors, and results.
  3. They grouped similar tasks into clusters. A cluster is a group of tasks that seem to need the same kind of improvement.
  4. An evolution agent created a new specialized harness for one cluster.
  5. The new harness was tested on that cluster.
  6. If it improved the results, it was kept. If not, it was rejected.
  7. The system saved lessons about which changes worked and when each harness should be used.

This is somewhat like testing different study methods with groups of students. If a method helps a particular group, it is kept for that group instead of being forced on everyone.

Choosing a harness during testing

When the system receives a new task, a harness router chooses what to do.

The router can consider:

  • The task description
  • The available specialized harnesses
  • Lessons learned during training
  • The agent’s early actions, called its trajectory

A trajectory is like a video recording of the agent’s attempt. It shows what tools the agent used, what errors occurred, and whether it seems to be stuck.

The router can then:

  • Use the original base harness
  • Choose a specialized harness
  • Reuse the result from the base harness if it already looks good

Importantly, the researchers kept the main LLM fixed. They changed the surrounding harness code, not the model itself.

Benchmarks

The system was tested on three challenging benchmarks:

Benchmark Main type of task
AutomationBench Business workflows
Workspace-Bench Working with files and applications
Toolathlon-GYM Tasks requiring several tools

The researchers separated the tasks used for training from the tasks used for final testing. This is important because it checks whether the system can handle new tasks rather than simply memorizing the training examples.

4. What were the main findings?

OO-Harness performed best overall

OO-Harness achieved the highest main score on all three benchmarks.

Its results were:

  • AutomationBench: mean score of 77.41%
  • Workspace-Bench: mean score of 39.48%
  • Toolathlon-GYM: strict success rate of 37.45%

Compared with the original base harness, this represented improvements of up to about 15.79 percentage points.

On AutomationBench, OO-Harness also used the fewest steps in its final attempts. This suggests that it was not only more successful but sometimes more efficient.

Specialized harnesses were better than one global harness

The researchers compared OO-Harness with systems that create only one improved harness for all tasks.

The results showed that a single change can help one type of task but hurt another. For example:

  • Extra memory may help with complicated tasks but make simple tasks harder.
  • Adding more tools may help some tasks but cause confusion in others.
  • A strategy that works for spreadsheets may not work well for business workflows.

OO-Harness avoids this problem by keeping different versions for different needs.

Routing based on the agent’s actions helped

The researchers compared two routers:

  • One that looked only at the task description
  • One that looked at both the task description and the agent’s early actions

The second router generally performed better.

For example, if the agent repeatedly made errors while using an API, the router could notice this and choose a harness designed to repair or prevent those errors.

On AutomationBench, using the agent’s trajectory improved the score from 64.88% to 77.52% in one routing experiment.

However, the benefit was not equally large on every benchmark. Sometimes the task description alone gave enough information.

Inheritance improved performance

The researchers also tested different ways of creating specialized harnesses:

  • OO-Harness: specialized harnesses inherit from earlier harnesses
  • Flat copies: each new harness copies code directly from the base
  • Base-reset: each new harness starts over from the original base

On Workspace-Bench, the inheritance-based approach achieved a rubric pass rate of 40.56%, which was 4.96 points higher than the flat-copy approach.

This suggests that inheritance helped the system organize and reuse improvements more effectively.

Keeping the base harness was useful

The system did not always force a task to use a specialized harness.

In one AutomationBench experiment:

  • About 44.7% of tasks reused the base harness.
  • The remaining tasks used specialized harnesses.

Tasks that already worked well with the base harness were often left alone. Tasks that performed badly were more likely to be sent to a specialist.

This is important because changing something that already works can accidentally make it worse.

The system handled unfamiliar tasks more safely

The researchers also tested tasks from application families that were not seen during training. These are called out-of-distribution, or OOD, tasks.

For unfamiliar tasks, the router chose the original base harness instead of using a specialized harness learned from unrelated applications. This preserved performance better than forcing all tasks to use one globally changed harness.

5. Why are these findings important?

The research suggests that AI agents may work better when they are not controlled by one universal set of instructions.

Different tasks often need different strategies:

  • Writing a document may require careful formatting.
  • Editing a spreadsheet may require checking formulas.
  • Using an API may require recovering from errors.
  • Completing a business workflow may require confirming that all required actions were finished.

A system with specialized harnesses can respond to these differences.

The object-oriented design also makes the system more organized. Instead of creating completely separate programs, it can reuse common behavior and add only the changes needed for a particular situation.

6. Possible impact and limitations

Potential impact

OO-Harness could make AI agents more useful for:

  • Office and business automation
  • Software tools
  • File and document management
  • Research workflows
  • Complex tasks involving many applications
  • Systems that must learn from previous successes and failures

It could also reduce the cost of training because a new harness is tested mainly on the tasks it is meant to improve, rather than on every task each time.

Limitations

The system still depends on its computer environment and the tools it uses. Problems may occur if:

  • An API changes
  • A service becomes unavailable
  • The system reaches a rate limit
  • Permissions or passwords are not configured correctly
  • A specialized harness is sent to the wrong task

The researchers also tested specific benchmarks, so more research is needed to see how well the approach works in real-world settings.

Conclusion

The paper’s central message is that AI agents should not necessarily use one identical strategy for every task.

OO-Harness builds a family of related harnesses. Each one specializes in a different type of problem, while still sharing useful code with the others. A router then chooses the best harness for each task, sometimes using evidence from the agent’s early actions.

Overall, the method performed better than the compared systems on three difficult benchmarks. The results suggest that organizing AI behavior like a family of specialized computer programs could make agents more accurate, efficient, and adaptable.

View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The paper’s main practical contribution is a deployable architecture for adapting language-model agents to heterogeneous tasks: a shared base harness, specialized subclasses, and a router that selects among them using task descriptions and observable execution trajectories. The following applications could be implemented with existing agent frameworks, provided that the relevant tools, APIs, graders, and execution logs are available.

  • Enterprise workflow automation — business operations, consulting, and finance
    • Deploy a catalogue of specialized agent harnesses for tasks such as report generation, spreadsheet analysis, document preparation, data extraction, and multi-step business-process execution.
    • A router could preserve the base agent for routine tasks while selecting specialists when the initial execution reveals problems such as incomplete writes, repeated tool errors, failed calculations, or imminent tool-call limits.
    • A practical product would be an agent orchestration layer containing:
    • a common executor interface;
    • a versioned hierarchy of harness classes;
    • a routing service;
    • logs of successful and failed execution patterns; and
    • rollback mechanisms for unsafe specializations.
    • Dependencies: access to structured business applications, reliable APIs, task-level completion checks, secure credentials, and sufficient historical execution data.
  • Workspace copilots for documents, spreadsheets, and files — productivity software
    • Create separate harnesses for document writing, spreadsheet editing, file organization, and cross-application workflows.
    • For example, a spreadsheet specialist could add verification of formulas and cell writes, while a document specialist could enforce formatting, citation, or completeness checks.
    • The system could inherit common authentication, planning, and tool-use logic while overriding only the workspace-specific methods.
    • Dependencies: stable application interfaces, permission-aware file access, observable intermediate state, and safeguards against unintended edits or data loss.
  • API and software-operation assistants — software engineering and IT
    • Use routing lessons to detect execution failures such as malformed queries, authentication errors, missing parameters, or incomplete deployment steps.
    • Specialized harnesses could provide query normalization, retry policies, schema validation, test execution, or near-budget completion reminders.
    • This could support:
    • database administration assistants;
    • cloud-resource provisioning agents;
    • incident-response tools;
    • code-repair agents; and
    • CI/CD workflow assistants.
    • Dependencies: sandboxed execution, least-privilege credentials, deterministic validation tests, and human approval for production-impacting actions.
  • Customer-support and service-desk automation — telecommunications, retail, and public services
    • Build distinct harnesses for billing questions, account changes, troubleshooting, refunds, and escalation.
    • The router could identify when a standard dialogue strategy is insufficient—for example, after repeated failed lookups or an unresolved authentication issue—and select a specialist with additional recovery or escalation logic.
    • A shared parent harness could enforce privacy, disclosure, and handoff policies across all service-specific subclasses.
    • Dependencies: accurate customer records, identity verification, policy-compliant tools, comprehensive audit logs, and human escalation paths.
  • Quality assurance for long-horizon agents — AI engineering and research
    • Use the framework as a practical testing and optimization workflow for existing agents rather than retraining the underlying LLM.
    • Teams could cluster failures by execution need—for example, “fails to commit required writes,” “does not verify calculations,” or “uses too many tool calls”—and evolve targeted subclasses.
    • This provides an alternative to applying one global prompt or global workflow change that may improve one task type while degrading another.
    • Dependencies: representative discovery and held-out test sets, reliable task-level metrics, reproducible environments, and code review for evolved harnesses.
  • Cost-aware agent serving — cloud software and platform engineering
    • Since the system can reuse the base result when the initial trajectory appears adequate, it can avoid rerunning tasks through a specialist unnecessarily.
    • A production router could combine task type, trajectory quality, estimated risk, latency, and API cost to decide whether to:
    • accept the base result;
    • invoke a specialized harness;
    • request human review; or
    • terminate the task safely.
    • This is particularly relevant where specialized execution is more expensive than base execution.
    • Dependencies: calibrated routing confidence, reliable cost estimates, latency requirements, and monitoring for routing failures.
  • Agent-development education and academic experimentation — computer science and AI research
    • The object-oriented harness representation can serve as a teaching and experimentation framework for:
    • prompt and tool orchestration;
    • inheritance and specialization in agent systems;
    • execution-trace analysis;
    • automated program repair; and
    • evaluation under distribution shift.
    • Researchers can compare global optimization, flat specialization, base-reset strategies, and hierarchical inheritance under a common interface.
    • Dependencies: open implementations, reproducible benchmark splits, transparent execution records, and evaluation that separates discovery tasks from held-out test tasks.
  • Policy and governance workflows — government and regulated organizations
    • Specialized harnesses could enforce different procedures for benefits processing, records management, procurement, regulatory reporting, or legal-document preparation.
    • A common parent could implement universal controls—logging, consent checks, data minimization, and escalation—while subclasses add domain-specific verification.
    • The base-reuse behavior is useful for unfamiliar cases: the system need not force an out-of-distribution task through an inappropriate specialist.
    • Dependencies: regulatory approval, explainable routing decisions, data residency controls, human oversight, and formal authorization boundaries.
  • Personal productivity assistants — daily life
    • A consumer assistant could route among harnesses for calendar management, travel planning, shopping, household budgeting, email drafting, or file organization.
    • Specialized execution policies could require confirmation before purchases, detect incomplete bookings, verify dates and totals, or avoid sending messages without user approval.
    • Dependencies: explicit user permissions, secure integrations, privacy-preserving logs, reversible actions, and conservative defaults for financial or social actions.

Long-Term Applications

The following applications are plausible extensions of the paper’s findings but require broader validation, stronger safety mechanisms, continual adaptation, or scaling beyond the reported benchmark settings.

  • Continually self-improving enterprise agent platforms — cross-domain automation
    • A production platform could maintain a persistent hierarchy of harnesses that evolves as new tasks and failure patterns appear.
    • New subclasses might be proposed from deployment traces, evaluated in a sandbox, and promoted only after passing regression tests across affected task clusters.
    • This would enable agents to adapt to changing workflows without modifying the underlying LLM.
    • Dependencies: safe continual-learning pipelines, version control, regression testing, drift detection, approval workflows, and defenses against feedback contamination or malicious tasks.
  • Federated harness evolution across organizations — cloud and platform ecosystems
    • Multiple deployments could contribute anonymized design and routing lessons to a shared parent harness while retaining organization-specific subclasses for local policies and tools.
    • For example, common error-recovery logic could be shared across customers, while credentials, compliance rules, and application-specific procedures remain isolated.
    • Dependencies: privacy-preserving telemetry, legal agreements, standardized harness interfaces, secure aggregation, and protection against leaking proprietary workflows.
  • Robotics and embodied agents — manufacturing, logistics, and domestic robotics
    • A robot could select different harnesses for navigation, manipulation, inspection, maintenance, or human interaction.
    • A trajectory-informed router could respond to physical execution evidence such as repeated grasp failures, localization uncertainty, low battery, or incomplete manipulation states.
    • Inheritance could preserve shared safety and control procedures while allowing task-specific overrides.
    • Dependencies: real-time guarantees, reliable state estimation, hardware abstraction layers, simulation-to-real transfer, physical safety validation, and substantially richer trajectory observability than in the paper’s software benchmarks.
  • Healthcare workflow agents — clinical administration and decision support
    • Specialized harnesses could support scheduling, chart summarization, coding, laboratory-result retrieval, prior authorization, and discharge-document preparation.
    • A base harness could enforce common privacy and provenance rules, while subclasses add domain-specific validation and escalation.
    • The approach should initially remain limited to administrative or decision-support functions rather than autonomous diagnosis or treatment.
    • Dependencies: clinical validation, patient-data protection, auditability, integration with health-record systems, clinician oversight, and regulatory approval.
  • Financial and legal agents with procedural specialization — regulated professional services
    • The framework could support separate harnesses for investment analysis, compliance checks, contract review, due diligence, tax workflows, and litigation-document preparation.
    • Each subclass could encode different source-verification, calculation, approval, and record-retention requirements.
    • A router could avoid transferring specialized behavior to unfamiliar legal jurisdictions, financial products, or document types.
    • Dependencies: jurisdiction-specific validation, traceable evidence, liability allocation, secure data handling, and mandatory human sign-off for consequential decisions.
  • Energy and infrastructure management — utilities and industrial control
    • Specialized harnesses could manage forecasting, maintenance scheduling, outage response, asset inspection, and demand optimization.
    • Trajectory-based routing could detect incomplete control actions, inconsistent sensor data, or failure to satisfy operational constraints.
    • The architecture may eventually support adaptive agents coordinating heterogeneous tools across grid, building, and industrial systems.
    • Dependencies: high reliability, real-time operation, certified control interfaces, adversarial-security testing, fail-safe behavior, and strict separation between advisory and autonomous control.
  • Open-ended software engineering agents — large-scale development
    • A hierarchy could include specialists for debugging, testing, refactoring, dependency upgrades, security review, deployment, and documentation.
    • Parent harnesses could provide shared repository navigation and test execution, while subclasses introduce domain-specific planning and verification.
    • Over time, routing could be based on repository structure, observed build failures, code ownership, and change risk.
    • Dependencies: secure repository access, reproducible builds, robust test coverage, protection against supply-chain attacks, and prevention of self-modifications that bypass review.
  • Learning and tutoring systems — education
    • An educational platform could maintain harnesses for explanation, Socratic questioning, formative assessment, writing feedback, mathematics, programming, and accessibility support.
    • The router could use observed student interaction patterns—such as repeated misconceptions or abandoned steps—to select a more appropriate instructional strategy.
    • Inheritance would allow all educational harnesses to share age-appropriate language, privacy, and content-safety controls.
    • Dependencies: pedagogical validation, reliable learner modeling, teacher oversight, protection of minors, cultural and curricular adaptation, and evidence that routing improves learning rather than merely task completion.
  • Policy simulation and public-sector decision support — governance
    • Agencies could use specialized harnesses for scenario analysis, legislative research, emergency planning, and public communication.
    • A hierarchy would allow shared evidence-handling and transparency rules while adapting workflows to different policy domains.
    • Routing lessons could help identify when a task falls outside the system’s validated domain and should be escalated.
    • Dependencies: representative public-sector data, nonpartisan evaluation, transparency of automated recommendations, accountability mechanisms, and safeguards against discriminatory outcomes.
  • Formal verification and safety certification for agent hierarchies — academic and industrial research
    • Future work could combine object-oriented harness evolution with static analysis, type checking, policy verification, and model-based testing.
    • Each subclass could be required to preserve inherited safety invariants while proving or testing the correctness of its overrides.
    • This would turn the current empirical acceptance rule—improvement on a task cluster—into a stronger promotion process that also checks non-regression and safety properties.
    • Dependencies: formal specifications for agent behavior, verifiable tool interfaces, scalable test generation, and methods for validating language-model-driven decisions.
  • General-purpose operating systems for heterogeneous agents — long-term AI infrastructure
    • The paper suggests a possible agent runtime in which harnesses function like reusable classes or plugins, with:
    • inheritance for shared capabilities;
    • routing for task specialization;
    • execution traces for diagnosis;
    • frozen releases for reproducibility; and
    • sandboxed evolution for future versions.
    • Such a platform could support agents spanning software, robotics, enterprise applications, and personal devices while avoiding a single globally optimized workflow.
    • Dependencies: standardized interfaces, cross-environment portability, robust identity and permission management, observability, benchmarking beyond the three reported suites, and evidence that the approach remains effective under real-world distribution shift.

The practical feasibility of all applications depends on a central assumption of the paper: the agent’s execution environment must expose enough information to recognize failure modes and enough control to modify prompts, tools, memory, and execution logic. The reported gains—up to 15.79 percentage points over the base harness—are promising, but they come from controlled benchmarks with fixed executors and frozen post-training hierarchies. Deployment in high-stakes settings therefore requires additional testing for reliability, security, API changes, cost, fairness, and performance on genuinely novel task families.

View Paper Prompt  View All Prompts 

Glossary

  • Ablation study: An experiment that removes or varies individual components to measure their contributions. “Extensive ablation studies examine the contributions of its core components.”
  • Agent harness: Executable code that defines the prompts, tools, and control flow surrounding a LLM. “Agent harnesses are executable code that specifies the prompts, tools, and control flow around a LLM.”
  • API: An application programming interface that allows software components or services to communicate. “Evaluating revisions on target clusters instead of rerunning the full discovery set can reduce API calls and associated cost per candidate.”
  • Base trajectory: The sequence of actions and observations produced when the base harness executes a task. “The harness router uses the task, an observable base trajectory, and routing lessons to select a specialized harness or retain the base result.”
  • Best-of-KK selection: A strategy that generates KK candidate outputs and chooses the best according to an evaluation criterion. “We evaluate parallel best-of-KK selection (K∈{2,5}K\in\{2,5\}) and sequential self-refinement (K=2K=2; \citealp{madaan2023selfrefine}).”
  • Cluster score: The performance score of a candidate harness on the tasks assigned to a particular cluster. “Let Sc(H′)S_c(H') denote its cluster score and $S_c^{\mathrm{current}$ the stored score under current harness assignments.”
  • Code workspace: An environment in which an agent can inspect, modify, and execute source code and related files. “At each iteration, the evolution agent works in a code workspace with filesystem access to harness source, task clusters, and execution records.”
  • Context management: The handling, organization, and limitation of information supplied to an agent during execution. “These workloads differ in the tool use, context management, and completion checks they require.”
  • Discovery set: A collection of tasks used to develop or optimize a system before evaluation on held-out tasks. “Meta-Harness, for example, evolves candidate programs using discovery-set feedback and selects a global harness based on performance on that set.”
  • Distribution shift: A change between the data or task distribution used during development and that encountered during evaluation or deployment. “Such selection does not guarantee that the same changes remain beneficial under distribution shift.”
  • Executor: The LLM or software component that performs the task under the direction of a harness. “A harness implements the prompts, tools, and execution rules around an executor, the LLM MM that carries out tasks.”
  • Execution interface: A standardized interface through which different harness implementations can be run. “At test time, polymorphism allows the executor to run a selected harness through a common execution interface.”
  • Execution record: Stored information about candidate code, trajectories, outcomes, and reflections from harness executions. “Execution records E\mathcal{E} store candidate code, discovery-task trajectories, outcomes, and reflections from successful and unsuccessful proposals.”
  • Execution strategy: A particular method for controlling tools, prompts, context, and actions while completing a task. “A single global harness can struggle with heterogeneous tasks that require different execution strategies.”
  • Filesystem access: The ability of a program or agent to read, create, modify, and inspect files in a filesystem. “At each iteration, the evolution agent works in a code workspace with filesystem access to harness source, task clusters, and execution records.”
  • Generalization: The ability of a system developed on some tasks or data to perform well on new, unseen tasks. “Separating discovery from evaluation is essential for assessing generalization beyond the search set.”
  • Grader feedback: Evaluation information produced by an automated or model-based task assessor. “Grader feedback supports evolution on training tasks only.”
  • Held-out task: A task excluded from training or optimization and reserved for evaluation. “This differs from Meta-Harness's TerminalBench-2 discovery experiment, which uses the same tasks for search and final evaluation.”
  • Heterogeneous task: A task that differs substantially from others in its requirements, domain, tools, or execution strategy. “A single global harness can struggle with heterogeneous tasks that require different execution strategies.”
  • Inheritance: An object-oriented mechanism through which a subclass reuses and extends the methods and behavior of a parent class. “Each specialized harness inherits from the base or an existing specialized harness, overriding inherited methods or adding new ones.”
  • Instance: A concrete runtime object created from a class, containing the state and behavior defined by that class. “The harness router selects a complete class whose methods execute within one instance; ancestors are not separate agent runs.”
  • Long-horizon benchmark: An evaluation suite involving tasks that require many sequential actions or decisions to complete. “Across three long-horizon benchmarks spanning business workflows, workspace tasks, and tool use, achieves the best performance among the compared methods.”
  • Mean partial credit: An evaluation metric that assigns credit for partial completion rather than only complete success. “Mean denotes mean partial credit on AutomationBench, micro-averaged rubric pass rate on Workspace-Bench, and mean reward on Toolathlon-GYM.”
  • Micro-averaged rubric pass rate: The proportion of all individual rubric requirements satisfied across all evaluated tasks. “Mean denotes mean partial credit on AutomationBench, micro-averaged rubric pass rate on Workspace-Bench, and mean reward on Toolathlon-GYM.”
  • Middleware: Software positioned between an application or agent and underlying services, often providing reusable processing or integration functions. “Where supported by the scaffold, it can also configure middleware, file backends, and subagents.”
  • Method override: A subclass implementation that replaces or specializes a method inherited from a parent class. “Each specialized harness inherits from the base or an existing specialized harness, overriding inherited methods or adding new ones.”
  • Object-oriented design: A programming paradigm that organizes software around objects and classes with encapsulated behavior and relationships such as inheritance. “In this work, we introduce Object-Oriented Harness (), a harness optimization framework that evolves a hierarchy of executable harness classes using object-oriented design.”
  • Out-of-distribution (OOD): Describing data or tasks that differ from the distribution used for training or development. “Finally, we construct an out-of-distribution (OOD) setting with disjoint training and test application families, and examine Base reuse on the selected test tasks.”
  • Parent class: A class from which another class inherits methods and behavior. “It selects a target cluster and parent class, states the intended change, and implements an executable subclass.”
  • Polymorphism: The ability to use a common interface to execute objects of different classes according to their specialized implementations. “At test time, polymorphism allows the executor to run a selected harness through a common execution interface.”
  • Prompt: An instruction or input supplied to a LLM to guide its behavior. “Agent harnesses are executable code that specifies the prompts, tools, and control flow around a LLM.”
  • Rebasing: In version-control systems, the operation of replaying changes onto a different base revision. “Adaptive Auto-Harness shares and specializes workspace contents through Git branching and rebasing.”
  • Reflection: An analysis of an agent’s previous execution used to identify successes, failures, or possible improvements. “Execution records E\mathcal{E} store candidate code, discovery-task trajectories, outcomes, and reflections from successful and unsuccessful proposals.”
  • Routing lesson: A learned description of when a particular harness should be selected for a task. “Routing lessons describe when a harness applies.”
  • Scaffold: A supporting software framework that provides the interfaces and infrastructure required by an implementation. “The resulting class must remain compatible with the scaffold's execution interface.”
  • Self-refinement: An iterative process in which a system revises its own output using feedback or evaluation. “We evaluate parallel best-of-KK selection (K∈{2,5}K\in\{2,5\}) and sequential self-refinement (K=2K=2; \citealp{madaan2023selfrefine}).”
  • Subclass: A class derived from another class that inherits and may extend or replace its behavior. “The evolution agent iteratively selects a cluster and parent harness, creates a subclass, and evaluates it across the cluster's tasks.”
  • Subagent: An agent invoked by another agent to perform a delegated task or analysis. “It can inspect files, delegate analysis to subagents, and write or edit candidate code.”
  • Task cluster: A group of tasks organized according to shared execution needs, capabilities, or failure patterns. “Task clusters C\mathcal{C} group discovery tasks with related execution needs.”
  • Test-time scaling: Increasing computation or the number of generated attempts during evaluation rather than changing the trained system. “Since \ uses a base trajectory for routing, we also compare with test-time scaling of the base harness to assess gains from additional inference.”
  • Trajectory: The sequence of actions, observations, tool calls, and intermediate states produced during task execution. “We use a train/test split: harnesses evolve on discovery tasks and are evaluated on disjoint held-out tasks.”
  • Tool use: An agent’s use of external software, APIs, or other resources to accomplish a task. “These workloads differ in the tool use, context management, and completion checks they require.”
  • Trajectory-informed routing: Selecting a harness using information from the execution trajectory of a preliminary or base run. “The trajectory-based router also observes the base execution.”
  • Unsupervised test-time scaling: Increasing inference or generating multiple candidate executions during testing without using grader feedback to train the system. “Parallel (K=2K{=}2)‡^{\ddagger} & 61.46±2.3861.46_{\pm2.38}”
  • Workflow: A structured sequence of operations required to complete a task or business process. “FutureX evaluates agents on forecasting real-world events using external information, while APEX-Agents evaluates cross-application workflows in investment banking, management consulting, and corporate law.”
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 7 tweets with 229 likes about this paper.

Sign Up for Free