IDE-Bench: AI IDE Agent Evaluation

• IDE-Bench is a comprehensive benchmarking framework that evaluates AI-powered IDE agents performing real-world software engineering tasks using Dockerized simulations.
• It employs a three-stage pipeline—including container setup, agent execution with 17 tools, and a grading system—to ensure reproducibility and robust performance assessment.
• The multi-metric evaluation covers pass rates, iteration efficiency, token utilization, and behavioral analysis to clearly reveal both strengths and current limitations of state-of-the-art models.

IDE-Bench is a comprehensive benchmarking framework specifically designed to evaluate AI-powered Integrated Development Environment (IDE) agents, particularly LLMs, on real-world software engineering tasks. Unlike traditional code benchmarks oriented toward static coding solutions, IDE-Bench employs a Dockerized, tool-driven interface that simulates modern, agent-centric IDE workflows. The framework enables rigorous assessment of agents' capacity for structured collaboration on practical engineering problems, supports reproducibility, and minimizes risks of training data contamination by utilizing novel, private codebases. IDE-Bench is distinguished by its systematic task coverage, multi-metric evaluation protocol, and its ability to reveal both strengths and current limitations of state-of-the-art LLM-based IDE agents (Mateega et al., 28 Jan 2026).

1. Architectural Design and Workflow

IDE-Bench is structured around a three-stage evaluation pipeline that emulates the lifecycle of an IDE-native, chat-driven agent:

1. Dockerized Task Container
   • Instantiates an isolated Ubuntu 24.04 container based on each repository’s Dockerfile, ensuring repeatability and uniform dependency environments.
   • Initializes git for change tracking. The task description is loaded as the agent’s initial context.
2. Agent Harness (LiteLLM Interface)
   • Exposes 17 discrete tools (e.g., file navigation, structured editing, codebase search, MERN stack integration) to the agent via OpenAI-style JSON function calls.
   • Agents operate within a self-loop of up to 100 reasoning iterations, receiving tool responses and sequentially issuing further instructions.
   • Logs comprehensive trajectories, including dialog, tool invocation sequences, and edit diffs.
3. Grader System
   • Executes the repository’s test suite (./run_tests.sh) after agent termination.
   • Uses a git-diff pipeline to semantically compare the agent’s code changes with a golden reference patch.
   • Computes a suite of quantitative metrics, capturing accuracy, efficiency, and behavioral traits.

Workflow Sequence:

Container Setup → Agent Execution (iterated tool calls) → Grading & Metric Extraction.

2. Evaluation Metrics and Statistical Formulations

The framework implements a multi-faceted metric protocol designed for depth and rigor:

• Task Resolution Rate (pass@k):

$$\mathrm{pass@}k = \frac{1}{N}\sum_{i=1}^N \max_{1\leq j \leq k} S_{i,j}$$

where $N$ is the number of tasks, $k$ is the number of runs per task, and $S_{i,j}\in\{0,1\}$ indicates test suite success.

• Per-Test Pass Rate:

$$\mathrm{TestPass}_i = \frac{1}{k}\sum_{j=1}^k \frac{s_{i,j}}{T_i}$$

with $T_i$ being the total test cases for task $i$ and $s_{i,j}$ the count passed in run $j$.

• Iteration Efficiency:

$$\mathrm{ExplorationFraction} = \frac{E}{I},\ \mathrm{ProductiveFraction} = \frac{P}{I},\ \mathrm{WasteFraction} = \frac{N}{I}$$

where $I$ is total iterations, $E$ exploratory, $P$ productive, and $N$ non-productive.

• Variance, Consistency, and ICC:

$$\sigma = \frac{1}{N}\sum_{i=1}^N \sqrt{\frac{1}{k}\sum_{j=1}^k (s_{i,j}-\bar s_i)^2}$$

$$\mathrm{ICC} = \frac{O_{\mathrm{between}}}{O_{\mathrm{between}}+O_{\mathrm{within}}},\quad R = \frac{O_{\mathrm{between}}}{O_{\mathrm{within}}}$$

• Token and Cost Efficiency:

$$\mathrm{Efficiency}_m = \frac{\mathrm{pass@}5_m}{\tau_m}$$

where $\tau_m$ represents mean tokens (in thousands) per successful solution.

• Intent–Modification Correlation:

Pearson correlation between intended and productive file edits:

$$\rho_{I,M} = \frac{\sum_i (I_i-\bar I)(M_i-\bar M)}{\sqrt{\sum_i(I_i-\bar I)^2}\,\sqrt{\sum_i(M_i-\bar M)^2}}$$

This multi-dimensional regime quantifies not only pass/fail outcomes but also partial credit, behavioral explorations, model consistency, and resource efficiency.

3. Task Suite Composition and Coverage

IDE-Bench defines 80 tasks distributed across eight repositories, each aligning with one of the representative modern tech stacks: C, C++, Java, and MERN (MongoDB/Express/React/Node.js). Every repository includes ten expert-crafted tasks reflecting realistic private codebase scenarios:

• Categories: Feature implementation, bug fixing, refactoring, and performance optimization.
• Examples:
  • C/C++: Buffer-overflow remediation, file I/O abstraction, profiling hook optimization.
  • Java: Pagination for device logs (Javalin + Thymeleaf), service-layer computation bug fixes.
  • MERN: Signature-validation middleware repair, retry queueing logic, JWT authentication correction.
  • Python: Vectorized bandwidth aggregation, AST-based linting, cyclomatic complexity calculation.

All repositories, except one demonstration sample, remain unpublished to enforce strict data contamination controls.

4. Dockerized Tool Ecosystem and Agent Interface

The agent operates via a tool ecosystem that abstracts typical developer actions within an IDE context. The seventeen provided tools fall into five categories:

Category	Example Tools	Purpose
File System & Navigation	read_file, list_dir, codebase_search, grep_search	Code search, inspection
Structured Editing	edit_file (REPLACE/INSERT/DELETE), search_replace, write_file	Code mutations
Execution & Testing	run_terminal_cmd	Build/test tasks, logging
MERN Full-Stack Test	api_call, database_query, websocket_test, ui_test	Application-level integration verification
Specialized	edit_notebook, web_search, create_diagram	Documentation, visualization extensions

Agents issue tool invocations as JSON function calls, including an “explanation” field. The harness serializes command execution, communicates results, and strictly enforces context, codebase, and security boundaries (e.g., no access to test or golden diff files).

System prompts demand a reasoning-first approach, requiring search actions to precede file edits and enforcing justification for all tool usage.

5. Contamination Avoidance and Reproducibility

To preclude training data leakage:

• All but one repository are unpublished and undiscoverable by LLMs.
• Task and golden diff specifications are scrubbed from agent-visible environments prior to execution.
• Containerization via unified Dockerfiles freezes OS and dependency configurations.
• The full suite is available solely under a non-commercial research license.

This robust protocol ensures future evaluations are not compromised by LLM training set contamination, supporting stable longitudinal performance tracking.

6. Quantitative Assessment and Behavioral Findings

Empirical results reveal stratified agent performance and non-trivial behavioral patterns.

• Frontier model cluster (pass@5: 85–95%): GPT 5.2 (95%), Claude Sonnet 4.5 (88.75%), Claude Haiku 4.5 (87.5%), Claude Opus 4.5 (86.25%), Codex Max (85%). Retry improvement is minimal (≤1.25 percentage points).
• Middle tier (pass@5: 70–80%): Notable for large variance between first-pass and retry outcomes, indicating non-deterministic behavior or local model optima.
• Lower tier (pass@5 < 50%): Agents such as Grok Code Fast and Llama 4 series underperform substantially on project-level tasks.
• Token Efficiency: Grok 4.1 Fast achieves the highest token efficiency, with a token-normalized pass@5 of 0.37, compared to 0.15 for GPT 5.2.
• Iteration Dynamics: Successful completions average a median of 21 iterations per run, with productive steps accounting for a minority of iterations (e.g., Grok 4.1 at 32.7%, Opus 4.5 at 0.9%).
• Core Tool Patterns:
  • read_file→edit_file (37%)
  • edit_file→read_file (55.9%)
  • run_terminal_cmd→run_terminal_cmd (66.2%)
  • codebase_search self-chaining (81.5%)
• Failure Taxonomy (multiple failure modes may co-occur):
  • Premature editing (63.0%)
  • Thrashing/backtracking (28.2%)
  • Context loss (27.6%)
  • Tool-call failures (9.1%)
  • Syntax-error loops (3.6%)
  • Timeouts (4.1%)

Agent consistency, measured as ICC and reliability ratio, indicates that models such as Claude Opus 4.5 and Claude Sonnet 4.5 (ICC ≥ 0.7) exhibit greater predictability versus models with broader performance variance.

7. Limitations and Prospective Extensions

Key limitations include:

• Incomplete stack coverage (notably absence of Go, Rust, mobile frameworks).
• Hard iteration budget (100) may penalize agents that require longer deliberative reasoning.
• The current agent–tool interface models structured function calls but omits fully interactive IDE features (e.g., autocompletion, semantic navigation, incremental linting).

Proposed future directions:

• Broader toolset integration (type checking, debugging consoles).
• Simulation of collaborative multi-agent workflows (pair programming).
• Development of richer semantic diff and compliance metrics.
• Adaptive iteration protocols and cost-aware early termination prediction.
• Expansion of task suites under tightly controlled release protocols.

IDE-Bench establishes a multidimensional, tool-oriented evaluation baseline for AI agents in realistic developer-in-the-loop settings. Its methodology and findings mark a significant advancement in the assessment of LLMs as engineering collaborators, with ongoing expansion planned to increase ecological validity and domain generalization (Mateega et al., 28 Jan 2026).