Evolving Agent Interfaces
This talk explores a fundamental shift in how we build computer-use agents. Instead of forcing capable models to work through fixed, human-designed interfaces like screenshots and mouse clicks, Agent S4 treats the interface itself as an evolvable component. Using population-based search over ungraded task trajectories, the system constructs executable interfaces and usage knowledge that transform both what agents can do and what they can observe, achieving dramatic performance gains on complex desktop and scientific software benchmarks.Script
A capable language model forced to control a computer through screenshots and mouse clicks will waste steps on operations it could perform directly if only it had the right interface. Agent S4 treats that interface not as a fixed constraint, but as something the system can build and improve.
The core idea is a repository that stores executable interfaces, their capabilities, and their usage instructions. These aren't just wrapper scripts. They change the action language the agent speaks and the observation language it receives, exposing operations like bulk spreadsheet edits or direct browser access through Chrome DevTools Protocol instead of forcing the model to reinvent them through GUI automation.
Agent S4 evolves these repositories through population-based search. It selects parent repositories using held-out validation scores, then a meta-agent studies their task trajectories to construct descendants through mutation or crossover. Crucially, the meta-agent never sees validation task instructions or success labels. It infers useful interface changes from ungraded behavioral traces on synthetic tasks, while benchmark performance guides only parent selection.
On OSWorld version 2, the evolved repository raises binary task success from 5.56 percent to 21.30 percent for the same Qwen model, a nearly 4-fold increase, while reducing cost per task from one dollar thirty-nine to one dollar ten. On ScienceBoard, covering specialized applications like ChimeraX and GRASS GIS, success climbs from 41.4 percent to 58.58 percent as the repository grows from 15 seed functions to 64, adding application-specific capabilities the meta-agent discovered from task experience.
The lineage analysis reveals that high-performing repositories often arise from crossover, recombining capabilities from earlier branches rather than extending only the latest variant. One crossover at step 24 combined repositories from steps 2 and 6 and achieved the highest partial validation score observed in that run, supporting the claim that population search preserves useful alternatives for later innovation.
Agent S4 shows that the interface between model and machine is not a fixed prerequisite but an evolutionary target, one that can be shaped by task experience to match the granularity and semantics the agent actually needs. To explore the full paper and create your own video explanations of cutting-edge research, visit EmergentMind.com.
