SWE-chat Dataset Overview

• SWE-chat is a continuously updated dataset that records multi-turn interactions between real software developers and AI coding agents in open-source settings.
• It employs an automated GitHub pipeline with transcript parsing and per-line authorship attribution to ensure data provenance and filter out non-human sessions.
• Empirical analysis reveals low survival of agent-generated code, a predominance of collaborative and vibe coding sessions, and increased vulnerability rates in agent-only commits.

SWE-chat is a large-scale, continuously updated dataset capturing interactions between real software developers and AI coding agents in open-source settings. Collected from opt-in users of Entire.io’s CLI logging, SWE-chat provides the first empirical dataset of its scale to systematically analyze how coding agents are used “in the wild,” including the effectiveness, limitations, and security implications of agent-generated code in actual development workflows (Baumann et al., 22 Apr 2026). By recording complete multi-turn human–agent transcripts with granular, per-line authorship attribution, SWE-chat enables rigorous investigation of coding agent behavior, user adoption strategies, outcomes, and failure modes in contemporary software engineering practice.

1. Collection Pipeline and Provenance

SWE-chat is built through continuous, automated harvesting of public GitHub repositories whose maintainers use the Entire CLI. The collection pipeline executes the following steps:

1. Discovery: Public GitHub repositories are identified by the presence of an “entire/checkpoints/v1” branch, which indicates participation in Entire.io’s session logging ecosystem.
2. Parsing: Each session is demarcated by structured “transcript.json” files in these branches, guaranteeing a full record of alternating user prompts, agent tool calls, responses, and meta-events.
3. Filtering: The function IsBotSession(session) ensures that only human-involved sessions—excluding synthetic or zero-human cases—are retained.
4. Authorship Linkage: For every session, the pipeline parses granular commit metadata, extracting the “shadow-branch” diffs generated by Entire to establish line-level authorship tags distinguishing “agent” from “human” contributions.

A high-level formalization of the data collection loop is provided:

$$p_{\mathrm{human}} = \frac{\text{human‐authored lines}}{\text{total committed lines}}$$0

The result is a “living dataset” with automatic discovery, filtering, and provenance guarantees, suitable for longitudinal analysis.

2. Dataset Scale, Coding Patterns, and Metrics

As of April 2026, SWE-chat represents:

Statistic	Value
Real user sessions	5,975
User prompts	63,000+
Agent tool calls	355,000+
Total log events	≈2.7 million

Key corpus-wide ratios define the empirical scope:

• $$p_{\mathrm{agent}} = \frac{\text{agent‐authored lines}}{\text{total generated lines}}$$
• $$p_{\mathrm{human}} = \frac{\text{human‐authored lines}}{\text{total committed lines}}$$

On aggregate, agents author 55.8% of committed lines. However, session-level analysis reveals bimodal coding patterns:

Coding Mode	Definition	Fraction of Sessions
Human-only	$s = 0\%$	22.7%
Collaborative	$0\% < s < 99\%$	36.5%
Vibe coding	$s \geq 99\%$	40.8%

Here, “vibe coding” (where agents write virtually all code) and “human-only” sessions together comprise 63.5% of all sessions, demonstrating clear task partitioning.

3. Data Format, Authorship Tracing, and Metadata

Each session record includes:

• session_id, repository URL, commit SHA
• anonymized user_id, session start/end times
• Turn-level data:
  • prompt_text, prompt_language
  • agent_response_text, token usage (in/out/cache)
  • Ordered sequence of tool_calls:
  • tool_type (read, edit, bash, git, AskUserQuestion)
  • arguments (file paths, shell commands, queries)
  • result snapshots, start/end timestamps
• commit_diff: file hunks, before/after lines
• authorship_tags: For each line in the “after” commit snapshot, a tag $\in$ {base, agent, human}

Precise authorship is achieved by sequentially replaying agent file-edit tool calls and marking all edited lines as “agent.” Upon commit, any remaining added or modified lines are attributed to the human via a shadow-branch diff. This yields unambiguous provenance for every committed line and enables downstream study of agent code retention, success, and rejection.

4. Empirical Insights on Coding Agent Efficacy

Empirical analysis leverages several quantitative and qualitative metrics:

• Survival rate: The proportion of agent-generated lines that remain unmodified in the final commit.

$$\mathit{survival\_rate} = \frac{\text{agent lines in final commit}}{\text{agent lines generated during session}} \approx 44.3\%.$$

Thus, less than half of all agent-proposed code survives unaltered.

• User pushback events: In 44% of session turns, users issue corrections, failure reports, or direct interruptions, indicating persistent friction even when agents are the primary authors.
• LLM-evaluated success: Most sessions receive at least 50/100, but a minority exhibit failure, scored $<\!15$.

These findings demonstrate that, despite increasing adoption, agents remain inefficient in practice, and user oversight is both frequent and necessary.

5. Failure Modes and Security Vulnerability Analysis

Low-success sessions typically involve one or more of the following agent failure patterns:

• Misidentification of code context (e.g., repeatedly changing the incorrect parameter)
• Generation of irrelevant (“junk”) commits
• Planning pathologies (agent stalls, infinite loops interrupted by the user)

Security is evaluated using Semgrep on pre- and post-commit snapshots (restricted to files modified in the session). The introduced vulnerability rate is defined as:

$$\text{vuln\_rate}_\text{introduced} = \frac{\#\text{findings}_\mathrm{post} - \#\text{findings}_\mathrm{pre}}{\#\text{added lines}} \times 1000$$

Coding Mode	Vulnerabilities Introduced (per 1,000 lines)
Human-only	0.08
Collaborative	0.14
Vibe coding	0.76

Agent-only sessions thus introduce approximately nine times the vulnerabilities of pure human commits and nearly five times that of collaborative sessions.

6. Example Interaction Patterns

Representative session excerpts:

• Low-success (Score 10/100):

Multiple unsuccessful attempts to fix the wrong animation parameter, ending with no commit:

$$p_{\mathrm{human}} = \frac{\text{human‐authored lines}}{\text{total committed lines}}$$1

• User correction in collaborative mode:

$$p_{\mathrm{human}} = \frac{\text{human‐authored lines}}{\text{total committed lines}}$$2

These examples illustrate both agent misunderstanding and active user intervention.

7. Research Implications and Future Directions

SWE-chat establishes an evidence-driven foundation to:

• Develop realistic agent benchmarks capturing multi-turn workflows, code comprehension, debugging, and git usage, moving beyond static, patch-oriented evaluations.
• Advance adaptive agent interaction, such as improved timing for clarifying questions (observed at $<2\%$ of agent turns in practice) and uncertainty estimation.
• Support data-driven simulator design for scalable agent evaluation using observed user behaviors instead of human-in-the-loop studies.

As an extensible, growing corpus, SWE-chat is poised to enable longitudinal research into the evolution of developer–agent collaboration and guide the design and benchmarking of both next-generation coding agents and their evaluation frameworks (Baumann et al., 22 Apr 2026).