Programming with Pixels: Computer-Use Meets Software Engineering (2502.18525v1)

Abstract: Recent advancements in software engineering (SWE) agents have largely followed a $\textit{tool-based paradigm}$, where agents interact with hand-engineered tool APIs to perform specific tasks. While effective for specialized tasks, these methods fundamentally lack generalization, as they require predefined tools for each task and do not scale across programming languages and domains. We introduce $\texttt{Programming with Pixels}$ (PwP), an agent environment that unifies software development tasks by enabling $\textit{computer-use agents}$-agents that operate directly within an IDE through visual perception, typing, and clicking, rather than relying on predefined tool APIs. To systematically evaluate these agents, we propose $\texttt{PwP-Bench}$, a benchmark that unifies existing SWE benchmarks spanning tasks across multiple programming languages, modalities, and domains under a task-agnostic state and action space. Our experiments demonstrate that general-purpose computer-use agents can approach or even surpass specialized tool-based agents on a variety of SWE tasks without the need for hand-engineered tools. However, our analysis shows that current models suffer from limited visual grounding and fail to exploit many IDE tools that could simplify their tasks. When agents can directly access IDE tools, without visual interaction, they show significant performance improvements, highlighting the untapped potential of leveraging built-in IDE capabilities. Our results establish PwP as a scalable testbed for building and evaluating the next wave of software engineering agents. We release code and data at https://programmingwithpixels.com

Insights into the "Programming with Pixels" Approach to Software Engineering Agents

The paper, "Programming with Pixels: Computer-Use Meets Software Engineering," introduces an innovative paradigm in software engineering agent environments—Programming with Pixels (PwP). Unlike traditional tool-based paradigms that rely on hand-engineered APIs tailored for specific tasks, PwP proposes an agent framework where computer-use agents interact with integrated development environments (IDEs) using visual perception, typing, and clicking. This presents a step towards developing general-purpose software engineering agents capable of achieving diverse tasks without being constrained to predefined functionalities.

Overview and Methodology

PwP emphasizes two pivotal characteristics: expressiveness and tool interaction. By allowing agents to perceive the IDE's visual state and utilize basic actions such as typing and clicking, PwP facilitates the performance of any task achievable by a human within an IDE. Agents can naturally and potentially fully exploit the existing rich set of tools within the IDE—such as debuggers, linters, and code completion suggestions—without relying on specially crafted APIs. This flexibility implies a scalable design that reduces the need for hand-engineered tool chains and broadens the potential of software engineering tasks across multiple languages and domains.

The paper introduces PwP-Bench, a robust benchmark suite derived from existing software engineering benchmarks, which evaluates agent performance across various tasks. These encompass code generation, UI development, pull request resolution, and DevOps workflows, covering multiple programming languages and modalities. The authors report that general-purpose computer-use agents often approach or exceed the performance of specialized tool-based agents, suggesting substantial promise for the PwP paradigm.

Key Findings

The paper's experiments reveal that using general IDE environments allows agents to perform a wide range of software engineering tasks. However, the analysis identifies key limitations in current models: challenges with visual grounding and underutilization of IDE tools that could simplify agent tasks. In scenarios where agents leverage direct access to IDE tools instead of visual interaction, there are noteworthy performance improvements, underscoring the latent potential embedded within IDE functionalities.

A significant finding of the paper is that some current models like Claude show better performance, hinting at the prospects for advancements in training models to optimize their efficacy within the PwP framework. Nonetheless, substantial opportunities exist for enhancing visual grounding and the strategic use of available IDE tools.

Implications and Future Directions

Programming with Pixels sets the stage for evolving the capabilities of software engineering agents from highly specialized systems to versatile, general-purpose solutions. By enabling computer-use agents to operate through familiar interfaces within IDEs, this approach challenges the prevailing tool-based paradigm. It suggests a shift towards more human-like interaction with software development environments, where agents seamlessly interact with any available tool via basic interface actions.

For future work, the paper proposes training approaches to enhance agent interactions with IDE tools, thereby capitalizing on the full spectrum of built-in capabilities. This progression signifies a noteworthy direction in developing sophisticated AI agents capable of more intuitive and extensive software engineering tasks.

In conclusion, the PwP framework redefines how software engineering processes can be approached, promoting a paradigm shift towards general-purpose computing agents that interact more naturally with complex development environments. By addressing the identified limitations in current models, PwP holds promise as a foundation for the future evolution of software engineering agents.