Papers
Topics
Authors
Recent
Search
2000 character limit reached

Evolving Agent Interfaces

Published 20 Oct 2026 in cs.AI | (2610.28601v1)

Abstract: Computer-use agents are limited by the interfaces humans give them. An agent may type a spreadsheet cell by cell when a single line of code would do, or print all two thousand lines of a file through a shell to find one function. Both the interfaces and the knowledge of when to use each are built by hand, which cannot keep pace with new applications, workflows, and models. We argue that interface design should itself be an agent capability. We introduce Agent S4, an autonomous computer-use scientist that builds Evolving Agent Interfaces from task experience. A meta-agent examines task agents' trajectories on synthetic training tasks, without success labels, and constructs and revises interfaces and usage knowledge in a repository. Population-based evolution then selects which repositories to develop further by their performance on validation tasks the meta-agent never sees, favoring changes that generalize beyond the seen tasks. On OSWorld V2, Agent S4, with Qwen3.8-27B and the evolved repository of interfaces, sets a new state of the art in open-source computer-use agents, reaching 21.3% binary score, 3.8 times the same model with fixed interfaces. This is above both GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%), at about 4.3% and 1.5% of their respective costs. On ScienceBoard, the evolved interfaces improve the success rate from 41.4% to 58.6% with the same model. These results suggest that useful interfaces can be evolved from experience, making autonomous interface design a source of improvement complementary to advances in models and harnesses. We open-source our code at https://github.com/simular-ai/agent-s.

Summary

  • The paper demonstrates that evolving interfaces for computer-use agents significantly improves their performance and efficiency, achieving a 3.8-fold increase in binary success on OSWorld V2 tasks while reducing task-solving cost from $1.39 to $1.10.
  • Agent S4, a population-based evolutionary system, constructs and refines interfaces using synthetic tasks, crossover, and mutation techniques, selecting the best repositories through held-out validation tasks.
  • The paper’s findings indicate that interface evolution facilitates both intermediate progress and complete-task reliability, supporting the agent’s efficiency in long-horizon and specialized task environments, such as scientific software.

Problem formulation and central claim

“Evolving Agent Interfaces” (2610.28601) argues that the interface through which a computer-use agent observes and manipulates its environment should be treated as an evolvable agent component rather than as a fixed, human-designed prerequisite. The paper targets a specific failure mode of current computer-use agents: a capable model may still be forced to perform inefficient, error-prone operations because it can access an application only through screenshots, mouse clicks, keystrokes, or a narrow collection of manually specified tools. In such settings, the model often compensates with additional inference and interaction steps instead of obtaining a more appropriate representation of the task state.

The paper defines an interface as a mechanism connecting an agent to part of a computer through an action language and an observation language. A GUI interface, for example, accepts pointer and keyboard actions and returns screenshots; a shell or application API accepts structured requests and returns textual or structured observations. An Evolving Agent Interface is a reusable executable interface constructed or revised from task experience, together with supporting knowledge about its capabilities, configuration, workflows, and appropriate usage.

These components are stored in a persistent repository RR. For a fixed task agent AA, the objective is to select a repository that maximizes expected task performance over a target task distribution. The important restriction is that the task agent’s model and harness remain fixed during evolution. Agent S4 changes the repository of interfaces and usage knowledge, not the underlying LLM, invocation mechanism, permissions, or task budget.

The paper’s central methodological claim is that an autonomous meta-agent can infer useful interface modifications from task trajectories even when those trajectories carry no ground-truth success labels. The evidence used to construct interfaces comes from synthetic tasks, while held-out benchmark tasks are used only by the evolutionary controller to select parents and ultimately choose a repository. This asymmetric separation is intended to reduce direct overfitting by the meta-agent to benchmark instructions and evaluator feedback.

Agent S4

Agent S4 implements repository evolution as population-based search. The population begins with a seed repository containing 15 general-purpose interfaces in approximately 550 lines of Python and shell. These interfaces provide basic GUI control, accessibility-tree access, window inspection, browser-target discovery, Office-file extraction, desktop configuration, and command dispatch.

At each construction step, the controller selects one or two parent repositories using binary tournaments based on held-out validation performance. A mutation modifies one repository by adding, revising, or removing interfaces, observations, workflows, or documentation. A crossover combines two distinct repositories, potentially selecting complementary capabilities or reconciling alternative implementations. The descendant is evaluated and added to the population alongside its parents, regardless of its score. This preserves earlier branches for later recombination rather than enforcing a strictly monotonic lineage.

Figure 1

Figure 1: Agent S4 uses trajectory analysis, mutation, crossover, and held-out validation to evolve repositories of executable interfaces and usage knowledge.

The meta-agent receives the selected repositories, their synthetic-task trajectories, and a lineage notebook. It can inspect action sequences, screenshots, tool outputs, errors, and observed artifacts. It is explicitly instructed not to interpret synthetic evaluator outputs as rewards: the synthetic tasks use constant-zero evaluators, so the construction evidence consists of ungraded behavioral traces rather than success labels. The meta-agent must therefore distinguish observed failures from hypotheses about missing capabilities.

The resulting repository can change both the agent’s action space and its observation space. Examples include bulk spreadsheet operations, direct browser access through Chrome DevTools Protocol, document parsing without launching the originating application, application-specific commands, task-level verification routines, and reusable workflows. Documentation is treated as part of the interface: capability listings, signatures, docstrings, and usage notes determine whether the task agent can discover and correctly invoke the evolved operations.

The held-out validation protocol is a key design choice. Validation tasks, trajectories, diagnostics, and scores are inaccessible to the meta-agent. They influence only parent selection and final repository selection. On OSWorld V2, selection prioritizes the number of fully solved validation tasks and breaks ties using mean partial score. On ScienceBoard, selection uses validation success. This arrangement does not eliminate benchmark dependence—the controller still performs repeated selection against benchmark-derived validation tasks—but it prevents the interface builder from directly optimizing against their task content or evaluator outputs.

Experimental design

The evaluation uses OSWorld V2 and ScienceBoard. OSWorld V2 contains 108 long-horizon desktop and web workflows and reports both binary success and partial progress. ScienceBoard contains 169 tasks across Celestia, ChimeraX, GRASS GIS, KAlgebra, Lean, and TeXstudio. Repository evolution uses 30 synthetic training tasks and 15 official validation tasks for OSWorld V2, and 48 synthetic training tasks and 15 validation tasks for ScienceBoard.

All primary Agent S4 experiments use Qwen3.8-27B with a modified Prime Agent harness. The harness provides a persistent IPython kernel, code execution, a 131,072-token context window, a 32,000-token output limit, automatic context compaction, and a 300-response task budget. The repository is installed in the task environment and is accessed through capability discovery, documentation queries, and interface calls.

This setup is technically consequential. The evolved-agent conditions are not merely the fixed Qwen CUA agent with a larger prompt; they use an IPython-backed, code-execution-oriented harness in which repository functions return outputs directly to the model. The paper reports this distinction explicitly: external OSWorld reference results use a different benchmark harness, generally involving screenshot observations and PyAutoGUI-style actions. Consequently, comparisons with GPT-5.5 and Claude Opus 4.8 are informative but not controlled model-only comparisons. The controlled comparisons are primarily between repositories and evolution strategies using the same Qwen model and Prime Agent harness.

The OSWorld evolution experiments compare population evolution with and without a scratch virtual machine, as well as linear evolution with scratch-VM access. A scratch VM allows the meta-agent to open applications and test interface implementations while constructing repositories. Without scratch access, it must reason from repository code and saved trajectories. The construction budget is 30 descendants in each OSWorld run and 30 descendants in the ScienceBoard run.

Main OSWorld V2 results

The strongest reported result is the performance of Qwen3.8-27B with the repository evolved without scratch-VM access. On all 108 OSWorld V2 tasks, it achieves:

Condition Binary success Partial score Cost per task
Qwen CUA, fixed interface 5.56% 20.69% $1.39
Evolved repository, scratch VM 19.44% 51.06% $1.27
Evolved repository, no scratch VM 21.30% 46.90% $1.10

The no-scratch repository raises binary success from 5.56% to 21.30% for the same Qwen3.8-27B model, a 3.8-fold relative increase, while reducing estimated task-solving cost from $1.39 to$1.10. Its binary score exceeds the reported scores of GPT-5.5 at 13.0% and Claude Opus 4.8 at 20.6%, while costing approximately 4.3% and 1.5% as much, respectively. The scratch-VM repository obtains the strongest partial score, 51.06%, but not the strongest binary score. This divergence shows that interface evolution can improve intermediate progress and complete-task reliability through different mechanisms.

The result is also more pronounced than the seed-repository improvement. Relative to using no repository, the seed raises binary success from 15.74% to 19.44% while leaving partial score almost unchanged, from 44.93% to 44.90%. Evolution without scratch access then raises binary success to 21.30% and partial score to 46.90%. Thus, the reported OSWorld gain is not attributable solely to exposing the model to a prebuilt collection of helpers; the evolutionary process contributes additional performance under the paper’s controlled setup.

The paper interprets the gain as reducing indirection between the agent’s intended computation and the computer’s exposed interface. A fixed-interface agent may write shell or application code through a terminal opened and controlled via the GUI, then inspect the result in a screenshot. An evolved repository can instead expose a direct operation whose request and output are already represented in the agent’s code-execution context. This can remove coordinate grounding, window management, screenshot parsing, download dialogs, and repeated setup.

The paper nevertheless reports no uniform reduction in trajectory length. Across all OSWorld tasks, the mean number of executed IPython calls changes only from 163.2 for the seed repository to 162.1 for the evolved repository. Among tasks receiving full credit under each respective condition, the means are 111.6 and 103.7 calls, but the successful subsets differ. On the 11 tasks solved by both repositories, the evolved condition actually averages more calls, 103.2 versus 87.8. The implication is that the repository primarily changes the probability and quality of successful execution, not that every task becomes shorter.

Generalization to scientific software

ScienceBoard tests whether the method can discover interfaces for specialized applications that are poorly covered by generic computer-use toolkits. The results are:

Condition Success on 169 tasks
Qwen CUA 41.4%
Agent S4 seed repository 55.03%
Agent S4 evolved repository 58.58%

The evolved repository solves 99 of 169 tasks, compared with 93 for the seed repository and approximately 70 for the Qwen CUA baseline. It also reduces average model calls from 61.7 to 58.9. The repository grows from 15 seed functions to 64 functions, adding application-specific functionality for ChimeraX, GRASS GIS, KAlgebra, and Lean. The largest application-level improvement occurs on KAlgebra, where success increases from 48% to 68%.

Figure 2

Figure 2: Across the evolution runs, repository growth consists both of entirely new interfaces and of extensions to interfaces introduced earlier.

The ScienceBoard result supports the paper’s claim that evolution can adapt an interface repository to a domain without requiring manually engineered coverage for every application. For example, the evolved KAlgebra interface can add curves and read them back through a structured operation; the GRASS GIS interface can execute modules against the live mapset and return textual statistics; and the Lean interface can expose proof-checking and completion status.

The qualitative examples clarify the mechanism. In a KAlgebra plotting task, the seed agent spends 68 model steps locating the expression interface through erroneous GUI clicks, whereas the evolved agent calls a plotting helper and completes the task in 11 steps. In a GRASS GIS raster-statistics task, the evolved interface returns values such as the cell count and sum directly from the live mapset, reducing a 40-step seed trajectory to 21 steps. These are single-episode illustrations rather than controlled estimates of individual interface effects, and the paper explicitly does not treat them as causal ablations.

What the evolved repositories contain

The paper classifies evolved functionality into four overlapping categories. System-wide interfaces expose desktop state, windows, processes, and background jobs. Application-specific interfaces operate on objects native to a named application, such as spreadsheet ranges, musical scores, molecular selections, or 3D scenes. Cross-application interfaces operate on shared data or platforms, such as PDFs, Office documents, or logged-in browser pages. Task-specific interfaces package a procedure together with a completion check, such as compiling a Lean proof and reporting whether it succeeds.

The no-scratch OSWorld population run introduces four new interfaces and 127 extensions. These include 29 system-wide extensions, 63 application-specific extensions, and 40 cross-application extensions, with overlapping labels permitted. The scratch-VM run produces 12 new interfaces and 253 extensions, including substantial growth in application-specific and cross-application capabilities. The ScienceBoard run introduces seven new interfaces and 91 extensions, mostly application-specific.

Figure 3

Figure 3: The OSWorld lineage shows that high-performing repositories can arise by recombining earlier branches rather than by repeatedly extending only the latest variant.

The lineage analysis provides evidence for retaining a population rather than using only linear evolution. In the scratch-VM OSWorld run, a crossover at construction step 24 combines repositories from steps 2 and 6 and reaches the highest mean partial validation score, 50.2%, compared with 35.8% for the seed. Its binary validation success is 26.7%, tying the best observed value. This result supports the paper’s claim that earlier repository variants remain useful sources of complementary capabilities.

The apparent advantage of population search is metric-dependent. With scratch access, population evolution and linear evolution both reach 19.44% binary success, but population evolution reaches 51.06% partial score compared with 47.17% for linear evolution. Without scratch access, population evolution reaches the highest binary score, 21.30%, but a lower partial score, 46.90%. The experiments therefore do not establish a uniformly dominant search strategy.

Mechanistic case studies

The browser examples are especially representative of the paper’s interface thesis. In one MailHub task, the fixed-interface baseline reads messages through repeated GUI interaction, handles download warnings, scrolls through the banking interface, and transcribes information from screenshots. The evolved repository exposes a browser API that sends requests from an already authenticated tab and returns mailbox and bank data as structured text. The task agent can then use curl and pdftotext to process attachments without repeatedly navigating the GUI.

Figure 4

Figure 4: The task agent discovers repository capabilities, reads usage documentation, and invokes evolved interfaces through code execution.

A related CDP case compares the seed and evolved repositories on the same task. The seed agent implements and debugs its own CDP client, requiring 299 tool cells, 68 image-bearing results, and 97.5 minutes before receiving a score of 0.50. The evolved agent uses reusable CDP helpers, requires 167 tool cells and 14 image-bearing results, takes 51.1 minutes, and receives a score of 1.00. The paper correctly treats this as an observed case outcome rather than an isolated causal estimate: the full task also contains file, image, and email operations, and the repositories differ in more than CDP access.

Figure 5

Figure 5: A reusable CDP implementation eliminates much of the setup and debugging required to retrieve the same authenticated page content.

Other examples show the same pattern in application-specific environments. An evolved XMind interface constructs and exports a map through direct operations after the seed agent reaches its step limit attempting menu navigation. A Zotero interface returns collections and items as data rather than requiring the agent to navigate plugin dialogs. A Word-document interface lists tracked changes and accepts or rejects them in batches instead of requiring manual XML reconstruction. These interfaces do not merely store task plans; they alter the granularity and semantics of available actions and observations.

Validation noise and repository selection

The ScienceBoard run includes repeated validation because long-horizon computer-use outcomes exhibit substantial stochastic variation. Every five construction steps, the three highest-scoring repositories are re-evaluated and ranked by their running mean. This denoising procedure can change repository selection materially.

Two repositories obtain a first-evaluation score of 73.33% but fall to means of 60.00% and 66.67% after re-evaluation. The selected repository, if-016, obtains 73.33%, 66.67%, 73.33%, and 73.33% across four evaluations, for a mean of 71.67%. Thus, a single validation rollout can overestimate repository reliability, and repeated evaluation is not merely an implementation detail; it changes the evolutionary fitness signal.

Figure 6

Figure 6: Repeated validation favors repositories with consistent performance rather than those benefiting from a high first evaluation.

The paper applies denoising to ScienceBoard but not OSWorld V2 because repeated validation is substantially more expensive there. This creates an asymmetry between the benchmark-specific search procedures. It also leaves open how much of the ScienceBoard result depends on repeated validation relative to the interface mutations themselves, since there is no full ablation of the denoising operation on the final benchmark score.

Limitations and open questions

The experiments support the usefulness of evolving repositories, but several limitations constrain the scope of the claims. First, the strongest cross-model comparisons on OSWorld V2 are not matched-harness comparisons. Agent S4 uses a Prime Agent variant with persistent IPython execution and direct code-oriented interaction, whereas the reported GPT-5.5 and Claude results use different benchmark interfaces and action backends. The comparisons establish practical system-level performance and cost, but not that repository evolution alone makes Qwen intrinsically more capable than those models under identical interaction surfaces.

Second, the validation controller still uses official benchmark tasks and scores for selection. Although the meta-agent never sees validation instructions, trajectories, diagnostics, or scores, the evolutionary process can select repositories according to a finite benchmark-specific validation set. The paper’s asymmetric protocol limits direct task leakage, but it does not constitute distribution-free evidence of generalization.

Third, the search budget is small relative to the repository design space: 30 descendants for each run, with 30 synthetic OSWorld tasks, 48 synthetic ScienceBoard tasks, and only 15 validation tasks per benchmark. The paper does not report confidence intervals, multiple independent evolution seeds, or statistical significance tests for the headline full-benchmark comparisons. Long-horizon nondeterminism and small validation sets make these omissions consequential.

Fourth, synthetic training trajectories are generated from agent-authored tasks, with task construction involving other LLMs and human review. The results therefore depend on the coverage, realism, and failure modes of these synthetic scenarios. The paper shows that synthetic experience can produce useful interfaces, but it does not establish which properties of the synthetic task distribution are necessary.

Finally, interface evolution introduces software maintenance and security questions that are outside the evaluation. The meta-agent can write executable code, access application sessions, and create operations that aggregate or transform data. The harness retains permissions and budget control, but the paper does not provide a systematic analysis of interface correctness, regressions, privilege boundaries, malicious repository changes, or compatibility across application versions. The central open empirical question is how reliably evolved repositories can be maintained when task distributions, application states, or harness implementations change.

Conclusion

“Evolving Agent Interfaces” (2610.28601) presents Agent S4, a population-based system that constructs executable interfaces and usage knowledge from ungraded synthetic task trajectories while selecting repository variants through held-out validation. With Qwen3.8-27B and a fixed Prime Agent-based harness, the evolved repositories improve OSWorld V2 binary success from 5.56% to 21.30% and ScienceBoard success from 41.4% for Qwen CUA to 58.58%. The qualitative and lineage analyses indicate that the gains arise from changing both the operations available to the agent and the observations returned by those operations, with population search preserving useful alternatives for later recombination. The evidence is strongest for repository-level improvement under the paper’s specific harness and validation protocol; broader claims require controlled comparisons across harnesses, repeated evolution runs, and stronger tests of robustness to distribution and software changes.

Whiteboard

Explain it Like I'm 14

1. ¿De qué trata el artículo?

El artículo presenta una idea llamada Agent S4: un sistema que permite que un agente de inteligencia artificial mejore sus propias “herramientas” para usar computadoras.

Normalmente, los investigadores humanos deciden cómo puede interactuar una IA con una computadora. Por ejemplo, pueden darle herramientas para:

  • hacer clic y escribir con el teclado;
  • leer capturas de pantalla;
  • usar una terminal;
  • editar una hoja de cálculo;
  • abrir aplicaciones.

El problema es que estas herramientas suelen ser fijas. Si una tarea sería más fácil usando una herramienta nueva, la IA no puede crearla por sí sola.

Agent S4 intenta cambiar esto. Observa cómo trabaja un agente, detecta sus dificultades y crea o modifica herramientas para que pueda resolver mejor tareas futuras.

Una comparación sencilla sería esta: en vez de darle a un estudiante siempre el mismo estuche de herramientas, el sistema observa qué problemas tiene y le ayuda a construir una regla, una calculadora o una herramienta especial cuando las necesita.

2. Objetivos y preguntas de investigación

El artículo intenta responder principalmente a estas preguntas:

  1. ¿Puede una IA diseñar y mejorar sus propias herramientas para usar computadoras?
  2. ¿Puede aprender de sus experiencias, incluso cuando no recibe una respuesta clara de “correcto” o “incorrecto”?
  3. ¿Las herramientas creadas por la IA ayudan a resolver tareas nuevas, y no solo las tareas que ya ha visto?
  4. ¿Puede una IA relativamente pequeña rendir mejor si tiene mejores herramientas?
  5. ¿Qué tipo de herramientas nuevas son más útiles?

La idea importante es que mejorar las herramientas puede ser tan valioso como crear un modelo de IA más grande o más inteligente.

3. ¿Cómo realizaron la investigación?

Agentes que cumplen tareas

El estudio utiliza un agente de uso de computadoras. Este agente recibe una instrucción, como:

  • modificar una hoja de cálculo;
  • buscar información en un navegador;
  • trabajar con un programa científico;
  • editar un documento;
  • completar una prueba matemática.

El agente observa la computadora y realiza acciones para completar la tarea.

Interfaces

Una interfaz es la forma en que el agente se comunica con la computadora.

Por ejemplo:

  • una interfaz gráfica permite hacer clic, escribir y leer capturas de pantalla;
  • una interfaz de terminal permite enviar comandos y recibir texto;
  • una interfaz de hoja de cálculo puede leer o cambiar muchas celdas de una sola vez;
  • una interfaz para un programa científico puede ejecutar directamente una función del programa.

La diferencia puede compararse con escribir una dirección letra por letra en un mapa frente a decirle directamente al mapa: “llévame a esta dirección”. Ambas opciones funcionan, pero una es mucho más rápida y clara.

Un repositorio de herramientas

Las interfaces y las instrucciones sobre cómo usarlas se guardan en un repositorio. Este repositorio contiene:

  • programas y herramientas;
  • explicaciones sobre lo que hace cada herramienta;
  • consejos sobre cuándo utilizarla;
  • procedimientos reutilizables.

El agente puede consultar este repositorio durante sus tareas.

El meta-agente

Agent S4 incluye otro agente llamado meta-agente. Este no intenta resolver directamente las tareas principales. Su trabajo es estudiar cómo se comportó el agente principal.

Por ejemplo, puede observar que el agente:

  • modifica una hoja de cálculo celda por celda, aunque podría cambiar muchas celdas de una vez;
  • usa muchas capturas de pantalla para encontrar información;
  • tiene problemas para comprobar si terminó correctamente;
  • repite muchas veces el mismo procedimiento.

A partir de estas observaciones, el meta-agente puede crear una herramienta nueva, mejorar una existente o eliminar una que no sea útil.

Evolución de una población de repositorios

El sistema crea diferentes versiones del repositorio, como si fueran distintas versiones de un invento.

Hay dos formas principales de modificarlos:

  • Mutación: cambiar una sola versión, añadiendo, eliminando o mejorando herramientas.
  • Cruce: combinar ideas de dos versiones diferentes para crear una nueva.

Después, el agente principal prueba cada versión en tareas. Las mejores versiones tienen más posibilidades de ser utilizadas como base para futuras mejoras.

Esto se parece a una competición entre varios diseños de bicicletas: se prueban, se conservan las que funcionan mejor y se combinan sus mejores características para crear nuevos modelos.

Tareas de entrenamiento y validación

Los investigadores separaron las tareas en varios grupos:

  • Tareas de entrenamiento: sirven para que el meta-agente descubra problemas y proponga cambios.
  • Tareas de validación: se usan para decidir qué repositorios funcionan mejor, pero el meta-agente no puede verlas directamente.
  • Tareas finales: se utilizan para medir el resultado final del sistema.

Esta separación es importante porque evita que el sistema simplemente memorice las tareas utilizadas durante el desarrollo. Es parecido a estudiar con unos ejercicios y después hacer un examen con preguntas nuevas.

Además, muchas tareas de entrenamiento no tenían una etiqueta que dijera claramente si el agente había tenido éxito. El meta-agente debía aprender observando las acciones y los resultados de los intentos.

Pruebas realizadas

El artículo utiliza dos pruebas principales:

  • OSWorld V2: tareas variadas de uso de computadoras, como trabajar con documentos, sitios web y aplicaciones de escritorio.
  • ScienceBoard: tareas relacionadas con programas científicos, como astronomía, mapas, moléculas, álgebra, demostración de teoremas y escritura científica.

El modelo principal utilizado fue Qwen3.8-27B. Los investigadores mantuvieron igual el modelo y el sistema que lo controla, y cambiaron principalmente las interfaces disponibles.

4. Principales resultados

Mejora en OSWorld V2

En OSWorld V2, el agente con interfaces fijas consiguió completar completamente el 5,56 % de las tareas.

Cuando utilizó el repositorio evolucionado por Agent S4, completó completamente el 21,30 % de las tareas.

Esto representa una mejora de aproximadamente 3,8 veces con el mismo modelo de IA.

Sistema Tareas completadas por completo
Qwen con interfaces fijas 5,56 %
Qwen con repositorio evolucionado 21,30 %

También mejoró la puntuación parcial, que mide cuánto avanzó el agente aunque no terminara la tarea:

  • interfaces fijas: 20,69 %;
  • interfaces evolucionadas: 46,90 %.

Comparación con modelos más costosos

El sistema evolucionado obtuvo mejores resultados que algunos modelos comerciales mucho más caros:

  • Agent S4 con Qwen: 21,3 %;
  • GPT-5.5: 13,0 %;
  • Claude Opus 4.8: 20,6 %.

El coste aproximado por tarea también fue mucho menor:

  • Agent S4: alrededor de 1,10 dólares;
  • GPT-5.5: alrededor de 25,50 dólares;
  • Claude Opus 4.8: alrededor de 72,40 dólares.

Esto no significa necesariamente que Agent S4 sea mejor en todos los tipos de problemas. Significa que, en esta prueba concreta, un modelo abierto y menos costoso funcionó muy bien cuando recibió mejores herramientas.

Mejora en ScienceBoard

En ScienceBoard, los resultados también mejoraron:

  • repositorio inicial: 55,03 % de éxito;
  • repositorio evolucionado: 58,58 % de éxito.

El agente resolvió:

  • 93 de 169 tareas con el repositorio inicial;
  • 99 de 169 tareas con el repositorio evolucionado.

Las mejoras fueron especialmente importantes en algunas aplicaciones. Por ejemplo, en KAlgebra, el éxito subió del 48 % al 68 %.

El repositorio creó herramientas específicas para programas como:

  • ChimeraX, para visualizar moléculas;
  • GRASS GIS, para trabajar con mapas;
  • KAlgebra, para hacer cálculos;
  • Lean, para comprobar demostraciones matemáticas.

¿Qué tipo de herramientas creó?

Las nuevas herramientas pertenecían a varias categorías:

  • Herramientas generales del sistema: observar ventanas o iniciar programas.
  • Herramientas específicas de una aplicación: trabajar directamente con objetos de una hoja de cálculo o un programa de diseño.
  • Herramientas para varias aplicaciones: leer archivos o datos que pueden utilizarse en distintos programas.
  • Herramientas específicas para una tarea: realizar un procedimiento completo y comprobar si terminó bien.

Un ejemplo importante aparece en una tarea relacionada con correos electrónicos y cargos bancarios. El agente con interfaces fijas tuvo que abrir mensajes, hacer clic muchas veces, esperar descargas y desplazarse por páginas.

El agente con la interfaz evolucionada pudo:

  1. leer directamente los datos del correo;
  2. descargar los archivos;
  3. convertir los documentos a texto;
  4. buscar los cargos bancarios;
  5. comparar la información.

Así evitó muchos pasos lentos y propensos a errores.

No siempre fue mejor tener un entorno de prueba

Los investigadores también permitieron, en algunos experimentos, que el meta-agente utilizara una máquina virtual para probar sus herramientas.

Esto mejoró la puntuación parcial en algunos casos, pero la mejor puntuación de tareas completamente terminadas se obtuvo sin ese entorno de prueba. Esto muestra que más información no siempre produce automáticamente mejores resultados.

5. ¿Por qué son importantes estos resultados?

El artículo sugiere que mejorar las herramientas de un agente puede ser una forma poderosa de mejorar su rendimiento.

Muchas veces el problema no es que la IA “no sepa” qué hacer. El problema es que solo puede hacerlo de una manera incómoda. Por ejemplo, puede saber que necesita cambiar una hoja de cálculo, pero tener que hacer clic y escribir en cada celda por separado.

Una interfaz mejor permite que el agente:

  • haga más acciones en menos pasos;
  • reciba información más clara;
  • compruebe mejor sus resultados;
  • cometa menos errores;
  • gaste menos tiempo y dinero.

Por eso, el artículo defiende que las interfaces no deberían diseñarse una sola vez y dejarse sin cambios. También deberían poder aprender y evolucionar a partir de la experiencia.

6. Limitaciones del estudio

Aunque los resultados son prometedores, el trabajo tiene algunas limitaciones:

  • Solo se probó en determinados conjuntos de tareas.
  • Las tareas de entrenamiento fueron sintéticas, es decir, creadas especialmente para el experimento.
  • El éxito puede variar entre diferentes intentos de la misma tarea.
  • El sistema todavía necesita investigadores que definan el entorno, las reglas y los conjuntos de evaluación.
  • Un buen resultado en OSWorld V2 o ScienceBoard no garantiza que funcione igual de bien en todas las computadoras o situaciones reales.
  • El artículo compara costes estimados, por lo que esas cifras pueden cambiar según el proveedor y la forma de realizar los cálculos.

7. Implicaciones y posible impacto

La principal conclusión es que un agente de IA puede mejorar no solo sus respuestas, sino también las herramientas que utiliza para pensar y actuar.

En el futuro, esto podría permitir crear agentes que:

  • aprendan a manejar programas nuevos;
  • construyan herramientas para tareas repetitivas;
  • trabajen mejor con aplicaciones científicas o profesionales;
  • se adapten a diferentes usuarios y computadoras;
  • mejoren sin que una persona tenga que programar manualmente cada nueva función.

En términos sencillos, Agent S4 intenta que las IA no sean solo usuarias de herramientas diseñadas por humanos, sino también inventoras y mejoradoras de sus propias herramientas. Si esta idea continúa desarrollándose, podría hacer que los agentes informáticos sean más rápidos, baratos, flexibles y capaces de resolver tareas complejas en el mundo real.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge Gaps, Limitations, and Open Questions

The paper demonstrates promising gains from evolving agent interfaces, but leaves the following issues unresolved:

  • Limited benchmark coverage: Evaluation is restricted to OSWorld V2 and ScienceBoard, so it remains unclear whether the method generalizes to other operating systems, web environments, enterprise software, coding tasks, mobile interfaces, or real-world workflows.
  • Small evolution datasets: Repository evolution uses only 30 synthetic OSWorld tasks and 48 synthetic ScienceBoard tasks, leaving uncertainty about whether such limited experience is sufficient for broader interface discovery.
  • Dependence on synthetic task quality: The effectiveness of Agent S4 may depend strongly on the coverage, realism, and diversity of agentically generated training tasks. The paper does not quantify how omissions or systematic biases in these tasks affect the evolved repository.
  • Potential validation overfitting: Although validation tasks are hidden from the meta-agent, they are repeatedly used for parent selection and repository selection. The final improvements may therefore overfit the 15-task validation sets, especially given the small number of candidates and repeated evaluations.
  • No independent held-out evaluation: The paper reports full benchmark results, but does not provide a separate, independently sampled test distribution designed specifically to measure generalization beyond the benchmark and its validation tasks.
  • Unclear statistical reliability: Most reported results appear to come from single runs or single rollouts per task. Confidence intervals, significance tests, random-seed sensitivity, and per-task variance are not systematically reported.
  • Run-to-run instability: The paper shows substantial evaluation variance on ScienceBoard, but does not fully characterize how often evolution produces beneficial, neutral, or harmful repositories across independent evolution runs.
  • Incomplete ablation of evolution components: The effects of mutation, crossover, tournament selection, lineage retention, population size, pruning, operator schedules, and repository denoising are not isolated individually.
  • Limited comparison with simpler alternatives: The experiments do not adequately compare Agent S4 with strong non-evolutionary baselines such as supervised interface design from the same trajectories, retrieval-based tool selection, automated tool synthesis without population search, or repeated prompt and workflow optimization.
  • Unclear causal source of improvements: It is not established how much of the performance gain comes from new executable interfaces, usage instructions, reusable workflows, improved observations, task-specific checks, or merely increased access to programmatic execution.
  • Seed repository confounding: The seed repository itself substantially improves performance over the no-repository condition. Consequently, the reported gains attributed to evolution may partly reflect the manually engineered seed rather than autonomous interface discovery.
  • Manual design remains in the pipeline: Synthetic task construction, the seed repository, benchmark-specific selection criteria, prompts, harness configuration, and implementation constraints all involve human design. The extent to which Agent S4 reduces rather than relocates human engineering effort is unresolved.
  • Evolution cost is excluded from the main cost comparison: Reported task-solving costs omit repository-construction costs, meta-agent inference, VM usage, repeated validation, engineering effort, and infrastructure. The total cost and break-even point of evolution are therefore unknown.
  • Scalability to larger repositories is untested: The ScienceBoard repository grows from 15 to 64 functions, while the OSWorld repositories receive many extensions. The paper does not measure whether repository size eventually harms context length, tool discovery, selection accuracy, latency, or task performance.
  • Repository maintenance is unexplored: There is no systematic analysis of how evolved interfaces behave when applications, APIs, file formats, operating systems, or authentication states change.
  • Interface conflicts and regressions are insufficiently studied: Adding or revising interfaces may introduce overlapping capabilities, ambiguous tool choices, incompatible assumptions, or regressions on previously solved tasks. The population mechanism does not guarantee backward compatibility.
  • Limited evidence for compositional generalization: It remains unclear whether evolved interfaces can be reliably composed in novel combinations, rather than primarily solving tasks that resemble the capabilities represented in the synthetic training set.
  • No evaluation of transfer across models: The repository is evolved for a fixed Qwen3.8-27B task agent. Whether the same interfaces and usage knowledge transfer to smaller, larger, open-source, or closed-source models is not tested.
  • No evaluation of transfer across harnesses: Because the model and harness are fixed during evolution, it is unknown whether the repository remains effective under different context-management strategies, tool-execution systems, action budgets, or permission models.
  • Meta-agent capability dependence is unclear: The method uses a powerful fixed model and substantial construction budgets, but the paper does not examine how evolution quality changes with weaker meta-agents, different reasoning settings, or constrained coding abilities.
  • Scratch-VM effects are not explained: Scratch access increases partial performance in one comparison but not binary success. The paper does not identify which interface properties benefit from execution-based testing or why scratch access can reduce final binary performance.
  • Evaluator and verifier dependence remains unresolved: Training is described as verifier-free, but validation and final selection rely on benchmark evaluators. The method’s behavior when reliable evaluators are unavailable, noisy, delayed, or adversarial is not established.
  • Security and safety risks are underexplored: Evolved interfaces can execute shell commands, access files, interact with logged-in websites, and potentially bypass intended GUI or permission boundaries. The paper does not evaluate malicious interfaces, accidental data leakage, privilege escalation, or unsafe tool behavior.
  • Permission-boundary preservation is unclear: The relationship between evolved interfaces and the harness’s permission model is not fully analyzed. It is unknown whether generated interfaces can unintentionally circumvent restrictions imposed by the harness.
  • Robustness to adversarial or ambiguous tasks is unknown: The experiments do not test prompt injection, malicious webpages, misleading application state, corrupted files, ambiguous instructions, or adversarial task artifacts.
  • Human usability is not evaluated: The paper studies agent performance but does not assess whether evolved interfaces are understandable, inspectable, debuggable, or reusable by human developers and operators.
  • Interface quality is measured mainly through task success: There is little direct evaluation of reliability, latency, interpretability, maintainability, error recovery, observability, or API ergonomics of the generated interfaces.
  • Long-term repository dynamics are unknown: Evolution is run for only 30 construction steps. It remains unclear whether continued evolution converges, oscillates, accumulates redundant code, or gradually degrades repository quality.
  • Cross-domain reuse is not demonstrated: Separate evolution runs are conducted for OSWorld V2 and ScienceBoard. The paper does not test whether a single repository can support multiple domains or whether capabilities learned in one domain transfer to another.
  • Benchmark contamination risks are not fully characterized: Although synthetic-task authors were instructed not to overlap with official tasks, the method uses benchmark construction principles and taxonomies. The degree of semantic or capability overlap, and its effect on measured generalization, remains uncertain.
  • Reported comparisons may not be fully controlled: Results for competing frontier models are taken from another study, potentially involving differences in prompts, harnesses, batching, environment versions, evaluator settings, or cost assumptions.
  • Cost estimates rely on assumptions: The reported costs depend on token pricing and an assumed 80% input-cache hit rate. Sensitivity to cache rates, provider pricing, infrastructure costs, and meta-agent expenses is not reported.
  • Partial-score and binary-score trade-offs are unresolved: Scratch-enabled evolution improves partial score but not binary success, while the no-scratch repository shows the reverse pattern. The paper does not determine which objective better reflects practical usefulness or how repositories should optimize both simultaneously.
  • Failure mechanisms are not systematically categorized: Qualitative case studies illustrate successful interface use, but there is no comprehensive analysis of failures caused by incorrect tool selection, faulty interface implementation, stale observations, API errors, or misuse of evolved knowledge.
  • Automatic correctness of generated interfaces is unverified: The paper does not report unit tests, formal checks, sandbox validation, differential testing, or independent code review for interfaces created by the meta-agent.
  • Reproducibility details may be insufficient: Exact prompts, model sampling parameters, environment states, repository versions, randomness controls, and complete lineage artifacts are not presented in the main text, making independent replication difficult.
  • The optimal repository objective is underspecified in practice: The formal objective focuses on expected task score, but does not incorporate construction cost, execution cost, safety, latency, repository complexity, maintainability, or robustness, all of which may materially affect the best repository in deployment.
  • Human-designed interfaces are not directly optimized under the same budget: The paper does not establish whether a comparable amount of human effort, trajectory analysis, and interface engineering could achieve similar or better results than Agent S4.
  • Real deployment feedback is absent: All evolution occurs offline from preconstructed tasks. It remains open whether Agent S4 can safely and effectively learn from continuously arriving user tasks, nonstationary environments, incomplete feedback, and changing organizational requirements.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Cost-efficient enterprise computer-use agents — software, operations, and automation. Organizations can deploy the open-source Agent S4 framework with a fixed LLM and a repository of executable interfaces to automate browser, spreadsheet, document, email, file-management, and desktop workflows. The paper shows that replacing screenshot-based interaction with structured operations—such as bulk spreadsheet edits, browser data extraction, direct PDF parsing, or application APIs—can substantially improve task completion while reducing model-call costs. Potential products and workflows: internal IT assistants, finance and expense-processing agents, procurement assistants, document-processing pipelines, and browser-based research tools. Dependencies: reliable application APIs or scriptable interfaces, sandboxed execution, authentication management, and task-specific validation. The reported results are benchmark-based and do not establish reliability for unsupervised production use.
  • Agent interface optimization for existing models — AI infrastructure and software engineering. Model providers and agent-platform developers can use Agent S4 as an offline optimization layer that improves an agent without changing its underlying model or general harness. A development team could collect agent trajectories, identify repeated inefficient behavior, evolve interface repositories, and deploy the selected repository as a versioned tool layer. Potential tool: a repository compiler or “interface optimizer” integrated into frameworks such as code-execution agents, browser agents, or desktop automation systems. Dependencies: representative training tasks, held-out validation tasks, reproducible evaluation, and controls preventing the meta-agent from modifying unsafe permissions or core execution infrastructure.
  • Application-specific interfaces for office and business software — productivity and enterprise IT. Companies can build structured interfaces for recurring operations in spreadsheets, word processors, presentation tools, email systems, accounting platforms, CRM systems, and project-management applications. Instead of asking an agent to navigate menus and enter data cell by cell, an interface can expose high-level actions such as “update this range,” “find all invoices matching these charges,” or “apply and verify tracked changes.” Potential products: spreadsheet bulk-edit tools, automated reconciliation assistants, document-review agents, and cross-application reporting workflows. Dependencies: stable software versions, permission boundaries, audit logs, and safeguards against incorrect edits or unauthorized data access.
  • Web research and information retrieval — knowledge work, finance, and administration. The evolved browser interface demonstrates a practical workflow in which agents retrieve structured page data, mailbox contents, attachment URLs, and dashboard records rather than repeatedly scrolling through visual pages. This can support expense verification, supplier comparison, market research, compliance checks, and evidence collection. Potential workflow: browser API calls → attachment download → text or table extraction → cross-source consistency check → human approval. Dependencies: website terms of service, login/session security, anti-automation mechanisms, changing page structures, privacy requirements, and source-quality verification.
  • Scientific software assistants — research and laboratory computing. Research groups can evolve interfaces for domain-specific applications such as molecular visualization, astronomy, geographic information systems, algebra systems, theorem provers, and scientific typesetting. The ScienceBoard results indicate that application-specific interfaces can improve performance on specialized software, particularly when visual workflows are replaced by commands that expose the application's own objects and state. Potential tools: ChimeraX command assistants, GRASS GIS analysis agents, KAlgebra batch calculators, Lean proof-checking workflows, and automated TeXstudio editing assistants. Dependencies: domain-expert review, correct interpretation of scientific outputs, compatibility with local installations, and independent verification of calculations or proofs.
  • Reusable interface and workflow libraries — academia and open-source development. Researchers can use the released code to create repositories containing executable interfaces, usage documentation, reusable workflows, and task-specific checks. These repositories can be versioned like software libraries and shared across agents or research groups. Potential artifact: a domain repository with entries such as capability descriptions, example calls, known failure modes, and automated regression tests. Dependencies: clear licensing, secure code review, standardized repository schemas, and benchmarks that measure generalization rather than memorization.
  • Evaluation and regression testing for computer-use systems — AI safety and quality assurance. The paper’s separation of synthetic training tasks from hidden validation tasks can be adopted as a practical testing workflow. Teams can generate representative tasks, use trajectory data for interface development, and reserve unseen tasks for model and repository selection. Repeated validation can identify repositories that appear strong on one run but are unstable across runs. Potential workflow: synthetic task generation → trajectory analysis → repository evolution → repeated held-out evaluation → deployment gate. Dependencies: high-quality task generation, leakage prevention, meaningful success evaluators, and variance-aware metrics. Synthetic tasks alone may fail to represent real operational edge cases.
  • Human-in-the-loop desktop assistants — daily life and accessibility. Evolved interfaces could support users who have difficulty performing complex GUI actions by allowing an assistant to execute structured tasks such as organizing files, filling forms, comparing bills, preparing documents, or managing calendars. The agent could present a concise plan and request confirmation before consequential actions. Dependencies: explicit user consent, accessible explanations, reversible operations, protection of personal data, and reliable confirmation of the target and scope of each action.
  • Policy and governance tooling for AI agents — public-sector and regulated environments. Regulators and organizational compliance teams can use the repository-and-harness distinction to define what an agent is allowed to do. The harness can enforce permissions, budgets, network restrictions, and approval requirements, while the evolving repository determines how the agent interacts within those boundaries. Potential tool: a policy-aware agent gateway that records interface calls, blocks prohibited operations, and requires approval for financial, legal, medical, or account-changing actions. Dependencies: enforceable access controls, complete audit trails, sector-specific regulation, and the ability to inspect generated interface code.

Long-Term Applications

  • Self-improving enterprise automation platforms — cross-application operations. A mature version of Agent S4 could continuously discover new interfaces for an organization’s software ecosystem and maintain repositories that evolve as applications, workflows, and failure modes change. For example, an agent could learn a more efficient workflow spanning email, ERP, spreadsheets, document storage, and procurement systems. Potential product: an enterprise “agent operating layer” that maintains application adapters, workflow knowledge, verification routines, and lineage records. Dependencies: continual monitoring, robust rollback, compatibility testing, stable APIs, protection against repository drift, and reliable detection of harmful or over-specialized changes.
  • Autonomous interface engineering for newly released software — software development and robotics. The central innovation could generalize beyond desktop applications to automatically create interfaces for new software systems, simulators, APIs, or robot-control environments. A meta-agent might inspect documentation and trajectories, construct tools, test observation formats, and evolve higher-level actions such as “grasp object,” “run simulation,” or “inspect sensor state.” Potential applications: robotic manipulation, industrial control, digital-twin environments, game agents, and simulation-based engineering. Dependencies: formal safety constraints, deterministic or bounded execution, simulator fidelity, hardware validation, and much stronger guarantees than those required for benchmark computer use.
  • Scientific discovery agents with domain-specific tool ecosystems — research and engineering. Evolving interfaces could allow agents to build persistent, specialized toolchains for computational biology, materials science, astronomy, climate modeling, and engineering design. The agent might combine data retrieval, simulation, visualization, analysis, and result verification into reusable scientific workflows. Potential workflow: retrieve data → preprocess through a domain interface → run simulation → visualize results → test statistical or formal conditions → produce a reproducible report. Dependencies: scientific validity, provenance tracking, reproducibility, expert supervision, uncertainty quantification, and safeguards against generating plausible but incorrect conclusions.
  • Adaptive interfaces for healthcare and other high-stakes sectors — healthcare, law, and finance. In principle, the approach could evolve interfaces for electronic health records, medical imaging, insurance systems, legal case-management platforms, or financial operations. Task-specific observation and verification interfaces could reduce clerical work and expose relevant structured state more reliably than screenshots. Dependencies: regulatory approval, privacy-preserving infrastructure, validated clinical or financial rules, human authorization, explainability, comprehensive auditability, and extensive prospective testing. The paper does not provide evidence sufficient for autonomous high-stakes deployment.
  • Personalized agent interfaces — education and consumer technology. An agent could evolve interfaces based on an individual’s recurring workflows, preferred applications, accessibility needs, and error patterns. Students might receive interfaces for laboratory software or mathematics tools, while consumers might obtain assistants tailored to budgeting, scheduling, or household administration. Dependencies: privacy-preserving learning, prevention of harmful personalization, portability across devices, user control over retained knowledge, and evaluation across diverse users rather than only synthetic task distributions.
  • Economic and policy assessment of agent efficiency — labor and public policy. The paper’s cost-per-task comparisons suggest a framework for evaluating whether interface improvements, rather than larger models, deliver better productivity gains. Policymakers and organizations could measure task completion, total inference cost, human review time, error rates, and energy consumption together. Potential tool: a standardized agent-efficiency dashboard comparing fixed-interface and evolving-interface systems across sectors. Dependencies: comparable workloads, transparent cost accounting, inclusion of infrastructure and repository-construction costs, and metrics that capture safety and quality rather than success alone.
  • Verified self-modifying tool repositories — AI safety and dependable autonomy. Future systems could combine population-based interface evolution with static analysis, sandbox testing, formal specifications, capability restrictions, and cryptographic signing. Only repositories that pass security and behavioral checks would be eligible for deployment. Potential architecture: meta-agent proposes interface changes → isolated test environment executes them → policy and security checks run → repeated validation measures robustness → human or automated approval signs the repository. Dependencies: effective specifications for interface behavior, robust adversarial testing, protection from benchmark overfitting, and methods for detecting subtle data exfiltration or privilege escalation.
  • General-purpose autonomous computer-use infrastructure — broad digital labor. In the longer term, repositories could become a transferable layer between foundation models and the digital world, allowing agents to discover when to use GUI actions, shell commands, application APIs, document parsers, or task-specific procedures. This could reduce dependence on manually engineered interfaces and make agents more adaptable to new software environments. Dependencies: stronger generalization beyond OSWorld and ScienceBoard, improved success rates, stable long-horizon behavior, resistance to distribution shift, scalable evolution costs, and rigorous protection against unsafe autonomous actions. The current results demonstrate promise but remain limited by benchmark size, model and harness choices, and the absence of production-scale reliability evidence.
View Paper Prompt  View All Prompts 

Glossary

  • Agent-computer interface (ACI): The interaction layer that determines how an agent observes and acts on a computer. “SWE-agent showed that this agent-computer interface shapes performance”
  • Asymmetric evaluation: An evaluation design in which the system constructing candidates cannot access the data used to select among them. “Only training trajectories are exposed to the meta-agent.”
  • Binary success: A metric indicating whether a task was completed fully rather than partially. “Binary success measures complete task execution”
  • Binary tournament selection: A selection method that compares two randomly chosen candidates and retains the one with the better score. “For each parent, binary tournament selection draws two eligible variants uniformly and chooses the one with the higher validation score.”
  • Context compaction: The reduction or summarization of an interaction context so that it fits within a model’s context window. “including any context compaction”
  • Crossover: An evolutionary operation that combines features or ideas from two parent candidates to create a descendant. “The descendant need not contain the union of its parents.”
  • Directed acyclic graph (DAG): A graph whose edges have direction and that contains no directed cycles. “Parent links form a directed acyclic graph, with one parent per mutation and two per crossover.”
  • Denoising: Repeated evaluation intended to reduce the influence of noisy or unusually high single-run scores. “In the case of Scienceboard, we also run a Denoise operation after every 5 steps.”
  • Distillation: The transfer or compression of capabilities, behavior, or knowledge into another representation or system. “Distillation can reorder what the interface allows, but it cannot extend it.”
  • Evolution budget: The limit on the number of repository-construction iterations performed during evolutionary search. “evolution budget NN”
  • Evolutionary baseline: A comparison method that applies a simpler evolutionary strategy against which the proposed strategy is evaluated. “We compare our setup against evolutionary baselines such as linear evolution”
  • Fitness: A numerical measure used to evaluate and compare candidate solutions during optimization. “Validation rollouts provide a separate selection score for each candidate RR”
  • Generalization: The ability of a system or learned artifact to perform effectively on tasks or domains not used directly during its construction. “These results suggest that useful interfaces can be evolved from experience”
  • Ground-truth success label: An authoritative annotation indicating whether a task attempt succeeded. “Training trajectories have no ground-truth success labels.”
  • Harness: A software framework that manages model invocation, execution, permissions, budgets, and context preparation. “A harness HH manages model invocation, request execution, task budgets, permissions PHP_H, and preparation of the model's context”
  • Held-out validation: Evaluation on tasks withheld from the data used to construct or modify a candidate. “held-out validation guides subsequent selection”
  • Implicit-state inference: The deduction of hidden or unobserved environmental state from available evidence. “These targets guided the representation of capabilities such as cross-source reasoning, visual-spatial precision, and implicit-state inference.”
  • Interface evolution: The iterative revision of an interface based on observed task trajectories and failures. “We call this process interface evolution.”
  • Lineage: The record of parent–descendant relationships among successive candidate repositories. “The updated population supplies candidates for the next iteration.”
  • Meta-agent: An agent that analyzes task-agent behavior and constructs or revises the interfaces and knowledge used by task agents. “The meta-agent studies these trajectories and forms hypotheses about how changes to actions, observations, or usage knowledge could help the task agent.”
  • Mutation: An evolutionary operation that modifies a single parent candidate by adding, revising, or removing components. “Starting from one parent RpR_p, the meta-agent adds, revises, or removes interfaces and knowledge.”
  • Observation language: The set of possible information returns produced by an interface after an agent’s request. “a language of requests AI\mathcal{A}_I and a language of observations OI\mathcal{O}_I”
  • Open-ended computer use: Computer interaction involving a broad and changing range of tasks, applications, and workflows. “Computer use is open-ended, and each new family of tasks or model release exposes new failure modes.”
  • Operator schedule: A predefined rule that determines which evolutionary operation is applied at each iteration. “At each step, an operator schedule chooses mutation or crossover.”
  • Partial score: A metric measuring intermediate task progress even when the complete task is not finished. “Binary success measures complete task execution, partial score measures intermediate progress”
  • Population-based evolution: An evolutionary search strategy that maintains multiple candidate solutions and develops selected candidates over successive iterations. “Population-based evolution then selects which repositories to develop further by their performance on validation tasks”
  • Programmatic access: The ability to interact with an application or data through code rather than manual graphical operations. “Agents with fixed interfaces already seek programmatic access”
  • Repository: A persistent collection of executable interfaces and supporting usage knowledge. “It contains executable interfaces and supporting knowledge such as capability descriptions, reusable workflows, configuration, and notes about when and how to use them.”
  • Rollout: One recorded execution of an agent attempting a task from an initial state under specified constraints. “Each rollout τ∼Rollout⁡(A,R′,C,x)\tau\sim\operatorname{Rollout}(A,R',C,x) starts from task xx's initial state within its budget”
  • Scratch virtual machine: An isolated virtual computer that allows an agent to inspect applications and test interface code during construction. “A scratch VM with the same AMI as the benchmark tasks lets the meta-agent inspect running applications and try interface code during construction.”
  • Selection pressure: The tendency of an evolutionary procedure to favor candidates with higher evaluation scores. “Validation performance guides parent selection.”
  • Synthetic task: An artificially authored task designed to provide experience for training or system development. “We constructed 30 synthetic tasks for OSWorld V2 and 50 for ScienceBoard”
  • Task distribution: A probability distribution describing the kinds of tasks a system is intended to solve. “Let D\mathcal{D} be the target distribution of tasks”
  • Task trajectory: A chronological record of an agent’s interactions, including requests, interfaces, and observations, while attempting a task. “The task agent uses candidate repositories of Evolving Agent Interfaces to attempt synthetic tasks, producing trajectories without ground-truth success labels.”
  • Test-time compute: Computational resources or additional sampling used while solving an individual task rather than during model training. “current methods compensate with test-time compute instead of a better interface.”
  • Validation score: The numerical performance of a candidate on held-out validation tasks, used to guide evolutionary selection. “The selected repositories and their saved training trajectories provide the starting point for the next descendant.”
  • Verifier-free: Conducted without an automated procedure that directly checks whether a task attempt succeeded. “without success labels or task verifiers”
  • Visual-spatial precision: The ability to reason accurately about locations, geometry, and spatial relationships in visual environments. “These targets guided the representation of capabilities such as cross-source reasoning, visual-spatial precision, and implicit-state inference.”
  • Wall-clock limit: A restriction based on elapsed real-world time rather than computation steps or model calls. “New OSWorld V2 training and validation rollouts also have a two-hour wall-clock limit.”
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 25 tweets with 787 likes about this paper.

Sign Up for Free