SWE-Interact: Interactive SWE Benchmark

• SWE-Interact is a benchmark that recasts long-horizon software engineering tasks as interactive, user-driven workflows with staged requirement disclosure.
• It simulates real-world developer interactions by employing a persona-driven user simulator that critiques and incrementally refines code.
• Empirical results show a significant performance drop in interactive tasks compared to fully specified ones, underscoring challenges in iterative goal discovery.

SWE-Interact is a benchmark and testbed for multi-turn, interactive, user-driven software engineering tasks that recasts long-horizon SWE evaluation as a realistic developer workflow with a simulated human in the loop. Instead of giving a coding agent a complete specification upfront, it begins with a vague or incomplete request, then progressively reveals requirements as a persona-driven, tool-using user simulator inspects the agent’s workspace, critiques intermediate implementations, and introduces new constraints. The underlying tasks, environments, and verifiers remain those of established long-horizon SWE benchmarks, but the interaction protocol shifts the problem from autonomous implementation toward interactive goal discovery and iterative refinement. On the benchmark’s 75 adapted tasks, the strongest models solve roughly half of the single-turn baseline tasks but only about a quarter of the corresponding SWE-Interact tasks, indicating that user-driven interaction defines an orthogonal capability axis rather than a minor variation of standard SWE evaluation (Raghavendra et al., 29 Jun 2026).

1. Problem formulation and benchmark scope

SWE-Interact starts from the observation that frontier SWE benchmarks typically assume that the full requirements are specified upfront and that the agent solves the task autonomously in one long session. SWE-Interact changes only the interaction protocol: the base task, repository, and verifier are inherited from existing benchmarks, but the agent must now infer missing requirements, adapt to evolving instructions, and revise its own prior work over many turns. This changes the central question from “can the agent implement a full specification?” to “can the agent discover the goal while collaborating with a user?” (Raghavendra et al., 29 Jun 2026)

The benchmark is built from 75 tasks drawn evenly from three existing sources: 25 from SWE-Bench Pro, 25 from SWE-Atlas (Refactoring), and 25 from DeepSWE. Each original task already includes a detailed natural-language problem description, a Docker-based environment, and a verifier suite based on unit tests and/or rubric-based checks. SWE-Interact preserves those verifiers exactly. The full task goal is placed into a hidden goal block for the simulator, while the solver sees only the simulator’s messages and the usual repository tools. This design isolates interaction difficulty from underlying repository difficulty.

A single SWE-Interact instance can be viewed informally as a base SWE task

$\mathcal{T} = (E, G, V),$

where $E$ is the environment, $G$ is the full hidden goal specification, and $V$ is the verifier, augmented with a simulator

$\mathrm{UserSim}(G, U, E),$

where $U$ denotes persona instructions. The task therefore remains verifier-grounded, but the path to the goal is mediated by dialogue and workspace inspection rather than by an upfront specification.

2. User simulator architecture and interaction protocol

Each trial involves two agents: the solver agent and the user simulator. The simulator is not a fixed-response oracle. It holds the full hidden task specification internally, begins from a vague top-level request, progressively reveals requirements, inspects the agent’s actual code changes, and responds in natural language. The solver has access to its usual coding scaffold plus an MCP-hosted ask_user tool; the simulator has its own tool-calling harness and can run commands such as git, grep, sed, and find inside the agent’s workspace, but in the first version it cannot modify code or run tests (Raghavendra et al., 29 Jun 2026).

The interaction protocol has three characteristic phases. In the initial phase, the simulator sends a short, underspecified request and deliberately avoids over-specification; if the agent asks design questions whose answer does not matter to the hidden goal, the simulator responds with variants of “your call” or “use your judgment.” In the implementation critique phase, after the agent proposes a plan or commits an implementation, the simulator inspects the actual diff and reveals one specific missing requirement at a time. In the visible contract review phase, once most of the hidden task has been surfaced, the simulator checks whether externally visible APIs, signatures, lifecycle behavior, and invariants have been implemented, and only then gives a brief approval such as “yeah looks good.”

This protocol is deliberately asymmetric. The user simulator privately knows the entire target specification, but the solver must reconstruct that specification over time from sparse, staged feedback. The resulting trajectories are long-horizon in both code and dialogue: the average trial has about 7 user messages, while the long tail reaches 27 user messages, 332 user tool calls, and more than 1000 agent steps.

3. Task construction and grounding in real coding interaction

The 75 tasks were manually selected from the source benchmarks for cases with complex, layered requirements, where it is natural for a human to begin with a high-level description and then add constraints after seeing an initial implementation. The benchmark therefore emphasizes tasks whose requirements can be meaningfully staged, rather than tasks that are trivially local or fully determined by an initial one-shot specification.

The simulator design is grounded in analyses of SWE-Chat, a dataset of real human–coding-agent interactions. Two empirical regularities from that dataset shape SWE-Interact. First, the most common interaction mode is “vibecoding”, in which the agent writes more than 99% of the code and the user communicates tersely and often vaguely. Second, the most common user persona is the “Expert Nitpicker”: a senior-developer style persona that is brief, casual, direct, strongly attentive to exact APIs and externally visible behavior, and inclined to reveal requirements incrementally rather than upfront. SWE-Interact explicitly models this persona in its simulator prompt (Raghavendra et al., 29 Jun 2026).

For each selected task, the hidden goal block is extracted from the original benchmark specification and then embedded into the persona prompt. The simulator therefore knows the full task but is instructed to disclose it only through the vibecoding pattern: terse opening, minimal early detail, one missing requirement per critique turn, and strong emphasis on externally visible interfaces. This yields trajectories that resemble real code-review or pair-programming exchanges more closely than static benchmark prompting.

A representative pattern appears in the paper’s kcp-go multiplexing example. The initial user request asks only for “a multiplexing layer” and a quick plan. Only after the agent commits code does the simulator begin to disclose exact constructor signatures, configuration surface requirements, metrics naming conventions, and additional public methods. The interaction is thus neither free-form brainstorming nor rigid checklist-following; it is a staged hand-off of an already-defined goal.

4. Evaluation protocol and trajectory metrics

The primary metric is the same verifier-grounded resolve rate used by the base benchmarks. If $V(\text{final\_repo}) \in \{0,1\}$ denotes pass/fail under the original benchmark verifier, then over $N$ tasks,

$$\mathrm{ResolveRate} = \frac{\sum_{i=1}^{N} V(\text{final\_repo}^{(i)})}{N}.$$

SWE-Interact therefore does not redefine correctness. A run is successful only if the final repository state passes the original tests or rubric checks.

The benchmark also records Steps/trial, Tokens/trial, and Cost/trial, since multi-turn interaction substantially increases the computational burden. For multi-turn resolve rates, the paper reports Wilson 95% confidence intervals; for example, GPT 5.5 achieves 24.7% with interval [18.5, 32.1], and Opus 4.8 achieves 26.7% with interval [20.2, 34.3] (Raghavendra et al., 29 Jun 2026).

To measure how well an agent reconstructs hidden goals over time, SWE-Interact introduces goal discovery scoring. For each task, GPT 5.5 generates a weighted rubric of atomic goal items,

$\{(w_j, r_j, \text{achieved\_when}_j)\}_{j=1}^{M},$

with integer weights satisfying $E$0. Claude Sonnet 4.6 then grades the agent’s PLAN.md and each implementation checkpoint. The score at checkpoint $E$1 is

$E$2

This tracks latent-goal recovery separately from final verifier success.

A further set of metrics measures revision efficiency. Let $E$3 be lines added plus deleted in revision $E$4, and let $E$5 be the final patch size. Then

$E$6

which quantifies extra edit volume beyond the final patch, and

$E$7

which measures how much of the total code churn happens after the first implementation checkpoint. Lower values indicate earlier convergence and less rework.

5. Empirical results and observed failure modes

The central empirical result is a large performance gap between fully specified single-turn tasks and their multi-turn SWE-Interact counterparts. The reported resolve rates are as follows (Raghavendra et al., 29 Jun 2026):

Model	Single-turn baseline	SWE-Interact
Opus 4.8	50.7%	26.7%
GPT 5.5	48.0%	24.7%
Gemini 3.5 Flash	29.3%	17.3%
Kimi K2.6	25.3%	14.7%
Sonnet 4.6	21.3%	18.8%

The strongest models therefore lose roughly half of their baseline success when the same tasks are presented as user-driven, progressively specified sessions. Multi-turn trials are also much more expensive: steps increase by about 3×–4×, tokens by about 2×–4.5×, and cost by about 2×–3.5×.

Trajectory analysis shows a consistent split between stronger and weaker models. GPT 5.5, Opus 4.8, and Sonnet 4.6 tend to plan well under vague instructions, maintain long interactions until more requirements are surfaced, integrate successive corrections more effectively, and produce lower ChurnOverhead and lower LateChangeShare. Yet they still fail frequently through over-agentic coding, forgotten earlier requirements, and technical mistakes in implementation. Weaker models such as Kimi K2.6 and Gemini 3.5 Flash tend to struggle more with the initial underspecification, ask fewer useful clarifying questions, terminate earlier, and rework code more reactively.

The paper audits 287 failed trajectories and groups failures into four main categories. About one third are technical implementation bugs, where the relevant requirement was known but the final code still failed. About one third are forgotten requirements, where a requirement surfaced earlier in the conversation was absent from the final patch. About 14% are misinterpretation or bad assumption, where the agent misunderstood a user requirement instead of clarifying it. About 12% are missing user requirement, where the simulator never surfaced a requirement needed for the hidden verifier. This breakdown matters because it indicates that most failures are not caused by the simulator withholding information; they are caused by memory, integration, and execution failures on the solver side.

Goal-discovery scores refine that picture. Strong models often achieve plan coverage above 80–90% despite the vague initial prompt, and with user feedback their average goal coverage recovers to above 90% over the trajectory. This suggests that many failures occur after substantial goal recovery: the remaining gap is often not discovering what to do, but implementing it correctly and preserving earlier constraints.

6. Position within interactive SWE benchmarking

SWE-Interact belongs to a broader shift from static SWE evaluation toward interaction-aware benchmarks, but its design emphasis is distinctive. SWE-Together reconstructs 109 repository-level tasks from 11,260 real user–agent coding sessions, uses a reactive user simulator anchored in original follow-up turns, and evaluates both final repository correctness and the number of corrective feedback turns required (Wu et al., 29 Jun 2026). Dialogue SWE-Bench converts all 500 SWE-Bench Verified tasks into under-specified dialogues by replacing the initial specification with a short user utterance and adding a message_user action, together with automatic naturalness and coherence judgments (King et al., 12 Jun 2026). ToM-SWE addresses a different but related dimension by pairing a SWE agent with a persistent theory-of-mind partner that models long-term user goals and preferences across sessions; on its stateful benchmark, it reports 59.7% task success versus 18.1% for OpenHands (Zhou et al., 24 Oct 2025).

Within that landscape, SWE-Interact’s defining feature is the combination of four properties: long-horizon repository tasks, a persona-driven tool-using user, latent requirements disclosed over time, and verifier-blind approval, meaning that the simulator’s approval does not determine success. The benchmark therefore tests something narrower and more interaction-specific than general dialogue quality, but more user-realistic than static issue clarification. A plausible implication is that these benchmarks are complementary rather than substitutable: SWE-Together emphasizes replay of real user sessions, Dialogue SWE-Bench emphasizes dialogue capability and dialogue quality metrics, ToM-SWE emphasizes persistent user modeling, and SWE-Interact isolates the ability to recover a hidden task contract through iterative code-aware interaction.

The benchmark also has an infrastructural implication. SWE-World describes a Docker-free “world model” that preserves the standard agent–environment interaction loop while replacing physical execution with learned surrogates, and its details explicitly characterize it as a natural backbone for interactive code-editing frameworks like SWE-Interact (Sun et al., 3 Feb 2026). This suggests a path toward scaling user-driven SWE evaluation without keeping full Docker execution in the inner loop.

7. Limitations and future directions

The current benchmark adopts only one dominant interaction style: the “Expert Nitpicker” in “vibecoding” mode. The authors explicitly note that real users are more diverse, including novices, verbose spec-writers, product stakeholders, and more adversarial or non-cooperative collaborators. The current simulator can inspect code and diffs, but it cannot edit code, run tests, or take more complex actions. Some failures are attributable to simulator limitations, notably the missing user requirement category, where a hidden requirement needed by the verifier was never surfaced (Raghavendra et al., 29 Jun 2026).

Coverage is also limited. The task set contains 75 tasks across three source benchmarks, substantial enough to reveal clear interaction effects but still narrow relative to the diversity of real-world software engineering. The source tasks and scaffolds constrain languages, domains, and agent interfaces, and the paper notes that each model was run with its own provider-specific coding scaffold, which complicates strict model-only comparisons.

The evaluation is still centered on binary repository correctness plus a small number of trajectory metrics. SWE-Interact adds important process measurements—goal discovery and churn—but the paper explicitly identifies room for richer metrics such as partial credit, user-satisfaction proxies, cost–benefit tradeoffs, and more explicit measures of requirement recall across time. Future work proposed in the paper includes expanding persona coverage, allowing the user simulator to run tests or make code suggestions, refining simulator realism and reliability, and scaling the benchmark to more languages, larger systems, and multimodal requirements.

At a conceptual level, SWE-Interact’s main contribution is not a new verifier or a new task source, but a redefinition of what SWE capability is being measured. It demonstrates that strong performance on fully specified repository tasks does not reliably transfer to interactive goal discovery and iterative refinement with a user in the loop, and it makes that gap measurable with repository-grounded, verifier-preserving experiments (Raghavendra et al., 29 Jun 2026).