SWE-Together: Interactive Coding Benchmark

• SWE-Together is an interactive multi-turn evaluation benchmark that models collaborative coding workflows using 109 repository-level tasks derived from real user-agent sessions.
• It employs an LLM-based user simulator to replay iterative, user-steered interactions, enabling a detailed analysis of both code correctness and corrective feedback.
• The framework utilizes comprehensive metrics, including pass@1, SSR, and user correction counts, to quantify agent performance and optimize collaborative efficiency.

SWE-Together is an interactive multi-turn evaluation framework and benchmark for coding agents, designed to model, replay, and quantify realistic collaborative development workflows between human users and AI assistants. Motivated by the limitations of static, single-turn agent evaluation, SWE-Together reconstructs a suite of 109 repository-level tasks from authentic user–agent sessions. By pairing these with a reactive LLM user simulator and a comprehensive set of outcome and process metrics, SWE-Together enables quantitative and qualitative analysis of agent performance as experienced in practical, iterative coding scenarios (Wu et al., 29 Jun 2026).

1. Motivation and Rationale

SWE-Together addresses fundamental shortcomings in prior benchmarks such as SWE-Bench and Terminal-Bench, which present agents with fully specified tasks in a single turn and assess only final code correctness via static test suites. Real software engineering workflows are inherently interactive, with user goals clarified, corrected, or expanded over many turns, and with agent progress requiring continuous steering. As a result, single-turn benchmarks cannot capture evolving requirements, corrective feedback loops, or the human effort necessary for solution convergence. SWE-Together thus measures not only final task correctness, but also the incremental user workload required for successful agent collaboration, leading to a more accurate proxy for end-user experience (Wu et al., 29 Jun 2026).

2. Benchmark Construction and Task Design

The SWE-Together corpus consists of 109 verifiable, repository-level tasks derived from a pool of 11,260 real user–agent coding sessions. Task construction follows a structured three-stage pipeline:

1. Deterministic Eligibility Filtering: Removes sessions lacking multi-turn user messages, concrete code edits, or identifiable public repositories, ensuring realistic steering is present and work is reproducible.
2. Viability Screening: Applies an LLM-based screener to session summaries to filter sessions that depend on external state, lack local reproducibility, or are coupled to ephemeral services.
3. Deterministic Task Construction: For eligible sessions, an orchestrator creates an isolated sandbox, checks out start and end repository states, synthesizes deterministic verifiers (unit tests or shell checks), and codifies user intents and correction triggers.

Each task includes an initial user request, a deterministic verifier, a structured “session analysis” (listing atomic user intents with precise triggers over output or file state), and a simulation prompt suitable for LLM-based user modeling. Eligibility criteria require recoverable repository states, a clear and decomposable user goal, and a deterministically verifiable final outcome (e.g., code diff or executable behavior) (Wu et al., 29 Jun 2026).

3. LLM-Based User Simulation

A distinctive component of SWE-Together is its anchored, reactive user simulator, which enables agent evaluation under interactive, user-steering conditions:

• Architecture: At each replay checkpoint—following a candidate agent turn—the harness composes a context from the initial user goal ("session anchor"), pre-enumerated follow-up intents and triggers, a concise summary of the agent’s recent actions, and memory of prior interventions.
• Action Space: The simulator, via an LLM prompt, selects one action:
  1. no-op (silent),
  2. question (clarification),
  3. redirect (return to earlier requirement),
  4. new-requirement,
  5. check-external (prompt for additional agent checks).

• Intent Preservation: The simulator emits follow-up only if the evaluated agent triggers a corresponding feedback condition, preserving both the content and relative order of the original user steering, but adapting to the agent’s distinct trajectory.

This architecture preserves intent alignment and user-effort reproducibility, enabling cross-agent and cross-run comparability in a controlled multi-turn setting (Wu et al., 29 Jun 2026).

4. Evaluation Protocol and Metrics

SWE-Together evaluates both objective solution quality and interaction process diagnostics:

• Final Repository Correctness:
  • Task-level judge score ($j_{t,r}\in[0,1]$) computed by an “agentic rubric judge” against a frozen rubric.
  • Thresholded success ($s_{t,r} = \mathbb{I}[j_{t,r}\ge\tau],\,\tau=0.85$).
  • pass@1: Marginal per-run success.
  • Stable Solve Rate (SSR): Fraction of tasks solved robustly across replicates.
  • pass^2: Strict joint success across both runs.
  • Mean Judge Score: Averaged across all tasks and replicates.
• User Effort Metrics:
  • UserCorrection: Weighted count of explicit corrections and nudges per trial:

  $$\mathrm{UserCorrection} = N_{\textrm{correction}} + 0.2\,N_{\textrm{nudge}}$$ - Process Analysis: Each simulator intervention is labeled, enabling breakdown by interaction types.

This dual scoring system enables quantitative assessment of both agent correctness and the human supervision burden required for task completion (Wu et al., 29 Jun 2026).

5. Experimental Results and Key Findings

SWE-Together has been used to evaluate a suite of seven frontier coding agents, providing direct head-to-head comparisons:

Model	pass@1 (%)	SSR (%)	pass^2 (%)	MeanJudge	UserCorrection
Claude Opus 4.8	63	59	52	0.801	1.38
GPT-5.5	58	55	48	0.763	1.59

• Strong performance–effort correlation: Pearson correlation between pass@1 and UserCorrection ≈ –0.92, indicating that improved agent capability directly reduces corrective workload.
• Efficiency trade-offs: GPT-5.5 is the most token- and time-efficient, while Claude Opus 4.8 yields the highest correctness.
• Rich qualitative diagnostics: Transcripts reveal that advanced agents can often resolve tasks within a single turn or minimal feedback, while less capable agents require more steering (e.g., multiple clarifications, redirection).
• Generalization: By simulating workflows reconstructed from real-world use, the benchmark assesses not only raw code output accuracy but also agent responsiveness to clarification, persistence, and incremental correction (Wu et al., 29 Jun 2026).

6. Relationship to Skill Priors and Interactive Architectures

SWE-Together provides an empirical substrate for evaluating frameworks that bridge agentless (pipeline) and agentic (multi-turn) paradigms. Results from Kimi-Dev demonstrate that strong skill priors—instilled via agentless training in code localization, patch editing, and self-reflection—facilitate rapid adaptation to the multi-turn interaction regime of SWE-Together. Lightweight supervised fine-tuning (SFT) on agentic trajectories is sufficient to bridge the gap, supporting both strong pass@1 and rich multi-turn dialog policies (Yang et al., 27 Sep 2025). This suggests that agent performance on SWE-Together integrates both the atomic competencies of traditional code-LMs and the higher-order interactive behaviors required in practical development (Wu et al., 29 Jun 2026, Yang et al., 27 Sep 2025).

7. Future Directions and Open Challenges

Key avenues for extension include:

• Broadening coverage to ambiguous or exploratory coding tasks not reducible to code diffs.
• Enabling richer simulator interventions, such as mid-turn interruptions or direct file edits.
• Modeling qualitative user behavioral dimensions (e.g., tone, satisfaction) beyond corrective content.
• Incorporating multimodal and UI-driven feedback channels.
• Leveraging SWE-Together as an engine for continual improvement and alignment of coding assistants with true developer workflows (Wu et al., 29 Jun 2026).

By tightly integrating criteria for both solution quality and user-effort, SWE-Together advances the empirical study of collaborative intelligence in software engineering. Its impact is further amplified when combined with architectures (e.g., ToM-based agents (Zhou et al., 24 Oct 2025)) and training regimes (e.g., skill-prior pipelines (Yang et al., 27 Sep 2025)) designed to reduce human steering and optimize for practical, multi-turn development productivity.