SWE-Together: Evaluating Coding Agents in Interactive User Sessions

Abstract: Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.

• The paper introduces SWE-Together, a novel benchmark that converts real multi-turn coding sessions into reproducible repository-level coding tasks.
• It employs a trajectory-conditioned user simulator and layered metrics to capture both task correctness and the amount of corrective user intervention.
• Experimental results reveal significant differences among coding agents, with lower corrective feedback correlating with higher accuracy and efficiency.

SWE-Together: Evaluating Coding Agents in Interactive User Sessions

Motivation and Problem Formulation

SWE-Together addresses two critical shortcomings of current coding agent benchmarks: static, single-turn task design and limited evaluation focused only on outcome correctness. Existing benchmarks such as SWE-Bench and Terminal-Bench present agents with a fixed, fully-specified request, then judge them based on executing predefined tests. This protocol omits the interactive nature of real-world coding assistance, in which user intent is gradually revealed, requirements are clarified, and corrections are issued throughout multi-turn sessions. As frontier coding agents approach benchmark saturation and marginal gains, interaction quality and effort parity become central measures of practical utility.

The second limitation concerns evaluation granularity. Pure correctness metrics fail to capture the cost and burden imposed on users—the amount of feedback, clarification, and correction needed to achieve success. Two agents may produce the same correct outcome while requiring vastly different degrees of human intervention. Therefore, an advanced benchmark must quantify both the trajectory of interaction and the user effort required, moving beyond static patch assessment.

Session-to-Task Construction Pipeline

SWE-Together reconstructs reproducible repository-level tasks from multi-turn coding-agent sessions sourced from four real-world datasets: DataClaw, Pi-staging, Hyperswitch, and SWE-chat. The pipeline consists of three stages:

1. Deterministic Eligibility Filtering: Raw traces are filtered to retain only multi-turn sessions with concrete code edits and identifiable repository context. Criteria exclude sessions lacking substantive agent actions, insufficient user interaction, or non-reproducible code changes.
2. Viability Screening: An LLM-based judge determines whether each candidate session can be reconstructed as a self-contained, locally executable task. Sessions relying on external state, such as deployment operations or private credentials, are excluded.
3. Sandboxed Task Construction: Each viable candidate is converted to a benchmark task in an isolated sandbox. A task-generation agent clones the repository at a fixed commit, identifies relevant setup and test commands, and generates task artifacts, including the initial user instruction and a task-specific user-simulation prompt.

This process yields a high-precision benchmark suite, with only $0.97\%$ ($109/11,260$) of raw sessions surviving as executable tasks.

Figure 1: The session-to-task pipeline: deterministic filtering, viability screening, and reproducible sandboxed task generation.

Figure 2: Workflow for constructing sandboxed tasks from normalized sessions, including repository verification and user-simulation prompt creation.

Interactive User Simulation

To model real collaborative coding workflows, SWE-Together employs a trajectory-conditioned, session-anchored user simulator. After each agent turn, the simulator decides whether to inject a user-facing message or remain silent, based on the evolving agent trajectory and original session anchors. Its action space includes silence (no-op), clarification questions, redirections, new requirements, and requests for external artifact inspection.

This design ensures that interventions occur only when warranted by agent behavior, replicating real user feedback patterns rather than generic or fixed responses. Each simulator is tailored to reflect the specific objectives, constraints, and intervention conditions observed in its underlying original session.

Figure 3: The user simulator consults session anchors and agent state at each replay checkpoint to emit structured actions, closely mirroring original user intent.

Evaluation Metrics and Protocol

Evaluation proceeds along two axes: final repository correctness and user-simulator behavior.

• Task Correctness: Scoring is implementation-agnostic, prioritizing behavioral completeness using deterministic verifiers and an agentic rubric judge. The rubric is fixed per task, derived offline, and employs weighted goals to allow partial credit.
• User Correction: Measures the corrective steering required by the agent, weighted by explicit corrections and minor nudges. A multi-label taxonomy distinguishes between strong correction, soft nudge, and non-corrective acts.
• Intent Coverage: Quantifies simulator fidelity, auditing how faithfully simulator messages cover the original user's atomic intents and remain within scope. Weighted recall and precision are combined, with recall receiving higher emphasis due to its direct impact on task presentation.

The metrics include pass@1, stable solve rate (SSR), joint success rate ($\mathrm{pass}^{2}$), mean judge score, User Correction, output tokens, and wall-clock time per task.

Figure 4: Joint diagnostics of correctness and interaction quality: final patch state, intent coverage, and corrective user interventions.

Experimental Results

Seven frontier models were evaluated on SWE-Together's 109 tasks. Claude Opus 4.8 yielded the highest scores: $63\%$ pass@1, $59\%$ SSR, $52\%$ joint success, $0.801$ mean judge, and the lowest corrective steering ($1.38$ User Correction per trial). GPT-5.5 was second, notable for highest efficiency ($29.9$k tokens, $10.7$ min/task), while MiniMax-2.7 ranked lowest across correctness and required the most correction.

Notably, all models' pass@1 fell well below the original reference patch baseline ($109/11,260$0), emphasizing persistent gaps in agent capability even under realistic interaction. Strong negative correlations (Pearson $109/11,260$1 for pass@1, $109/11,260$2 for SSR) between User Correction and correctness metrics validate the hypothesis that superior models impose less feedback burden.

Figure 5: pass@1 and SSR across seven models and varying user input budgets, highlighting model differences and comparison to reference patch accuracy.

Figure 6: Strong inverse relation between User Correction and capability across task subsets: stronger models need less corrective intervention.

Figure 7: Stable solve rate versus efficiency: output tokens and wall-clock time per task for all model cohorts.

User Simulator Consistency and Quality Analysis

Intent Coverage scores were stable ($109/11,260$3–$109/11,260$4) across model cohorts, evidencing consistency in simulator fidelity regardless of agent behavior. GPT-5.5 scored slightly lower ($109/11,260$5), attributed to its capability in swiftly resolving or reshaping interaction, not simulator drift. Human annotators failed to reliably distinguish simulated trajectories from real user sessions, with a $109/11,260$6 Turing pass rate (95% CI $109/11,260$7), confirming indistinguishability.

Comparative Positioning and Theoretical Implications

SWE-Together occupies a unique position among coding-agent benchmarks by combining real task provenance, repository-level multi-turn agent-environment interaction, and interactive user feedback replay, all grounded in actual user-agent sessions. Previous benchmarks either synthesized feedback from static datasets, lacked real interaction trajectories, or omitted repository context. SWE-Together’s design allows agent evaluation in scenarios closely reflecting production workflows and developer experience.

By providing user-centric diagnostics and interaction analysis, SWE-Together supports fine-grained performance differentiation among frontier agents even as outcome correctness metrics saturate in other benchmarks. Practically, these findings inform the deployment of coding assistants: minimizing corrective feedback translates directly into lower user burden, improved productivity, and enhanced adoption potential.

Theoretically, SWE-Together establishes a new standard for benchmarking collaborative AI agents, underscoring the need for verifiable, reproducible, and interaction-sensitive evaluation protocols. It sets the groundwork for advancing agent architectures and training methodologies that explicitly optimize for reduced intervention and greater intent understanding.

Conclusion

SWE-Together delivers a robust benchmark for evaluating coding agents under interactive, realistic conditions. By transforming multi-turn coding sessions into verifiable tasks and incorporating a state-conditional user simulator, it offers comprehensive metrics that capture both correctness and interaction burden. Empirical results reveal pronounced differences in agent reliability, capability, and efficiency, with corrective effort serving as a key diagnostic. The framework's fidelity and indistinguishability from real user trajectories support its suitability for evaluating collaborative coding agents at scale. The benchmarks established by SWE-Together are poised to guide practical improvements in coding assistant deployment and stimulate further research into interaction-aware evaluation and training protocols for software engineering agents.