---
title: SWE-chat Dataset Overview
url: https://www.emergentmind.com/topics/swe-chat-dataset
type: topic
---

# SWE-chat Dataset Overview

SWE-chat is a large-scale, continuously updated dataset capturing interactions between real software developers and AI coding agents in open-source settings. Collected from opt-in users of Entire.io’s CLI logging, SWE-chat provides the first empirical dataset of its scale to systematically analyze how coding agents are used “in the wild,” including the effectiveness, limitations, and security implications of agent-generated code in actual development workflows [2604.20779]. By recording complete multi-turn human–agent transcripts with granular, per-line authorship attribution, SWE-chat enables rigorous investigation of coding agent behavior, user adoption strategies, outcomes, and failure modes in contemporary software engineering practice.

## 1. Collection Pipeline and Provenance

SWE-chat is built through continuous, automated harvesting of public GitHub repositories whose maintainers use the Entire CLI. The collection pipeline executes the following steps:

1. **Discovery**: Public GitHub repositories are identified by the presence of an “entire/checkpoints/v1” branch, which indicates participation in Entire.io’s session logging ecosystem.
2. **Parsing**: Each session is demarcated by structured “transcript.json” files in these branches, guaranteeing a full record of alternating user prompts, agent tool calls, responses, and meta-events.
3. **Filtering**: The function `IsBotSession(session)` ensures that only human-involved sessions—excluding synthetic or zero-human cases—are retained.
4. **Authorship Linkage**: For every session, the pipeline parses granular commit metadata, extracting the “shadow-branch” diffs generated by Entire to establish line-level authorship tags distinguishing “agent” from “human” contributions.

A high-level formalization of the data collection loop is provided:

```latex
\begin{algorithmic}[1]
\Input \text{GitHub API credentials, Entire CLI indicator}
\Output \mathcal{D} \text{ = SWE-chat database of sessions}
\State \textbf{Repeat periodically:}
\State\quad \text{repos} \gets \Call{GitHubSearch}{"filename:entire/checkpoints/v1"}
\ForAll{repo in repos}
  \If{\Call{HasBranch}{repo, "entire/checkpoints/v1"}}
    \State \text{clone}(repo)
    \ForAll{checkpointDir in repo/entire/checkpoints/v1}
      \State session \gets \Call{ParseTranscript}{checkpointDir/transcript.json}
      \State commitInfo \gets \Call{ReadCommitMetadata}{checkpointDir}
      \If{\neg \Call{IsBotSession}{session}}
        \State \Call{StoreSession}{\mathcal{D}, session, commitInfo}
      \EndIf
    \EndFor
  \EndIf
\EndFor
\end{algorithmic}
```

The result is a “living dataset” with automatic discovery, filtering, and provenance guarantees, suitable for longitudinal analysis.

## 2. Dataset Scale, Coding Patterns, and Metrics

As of April 2026, SWE-chat represents:

| Statistic                           | Value                |
|--------------------------------------|----------------------|
| Real user sessions                   | 5,975                |
| User prompts                         | 63,000+              |
| Agent tool calls                     | 355,000+             |
| Total log events                     | ≈2.7 million         |

Key corpus-wide ratios define the empirical scope:

- $p_{\mathrm{agent}} = \frac{\text{agent‐authored lines}}{\text{total generated lines}}$  
- $p_{\mathrm{human}} = \frac{\text{human‐authored lines}}{\text{total committed lines}}$

On aggregate, agents author 55.8% of committed lines. However, session-level analysis reveals bimodal coding patterns:

| Coding Mode                | Definition                                      | Fraction of Sessions |
|----------------------------|------------------------------------------------|---------------------|
| Human-only                 | $s = 0\%$                                      | 22.7%               |
| Collaborative              | $0\% < s < 99\%$                               | 36.5%               |
| Vibe coding                | $s \geq 99\%$                                  | 40.8%               |

Here, “vibe coding” (where agents write virtually all code) and “human-only” sessions together comprise 63.5% of all sessions, demonstrating clear task partitioning.

## 3. Data Format, Authorship Tracing, and Metadata

Each session record includes:

- session_id, repository URL, commit SHA
- anonymized user_id, session start/end times
- Turn-level data:
  - prompt_text, prompt_language
  - agent_response_text, token usage (in/out/cache)
  - Ordered sequence of tool_calls:
    - tool_type (read, edit, bash, git, AskUserQuestion)
    - arguments (file paths, shell commands, queries)
    - result snapshots, start/end timestamps
- commit_diff: file hunks, before/after lines
- authorship_tags: For each line in the “after” commit snapshot, a tag $\in$ {base, agent, human}

Precise authorship is achieved by sequentially replaying agent file-edit tool calls and marking all edited lines as “agent.” Upon commit, any remaining added or modified lines are attributed to the human via a shadow-branch diff. This yields unambiguous provenance for every committed line and enables downstream study of agent code retention, success, and rejection.

## 4. Empirical Insights on Coding Agent Efficacy

Empirical analysis leverages several quantitative and qualitative metrics:

- **Survival rate**: The proportion of agent-generated lines that remain unmodified in the final commit.
  $$
  \mathit{survival\_rate} = \frac{\text{agent lines in final commit}}{\text{agent lines generated during session}} \approx 44.3\%.
  $$
  Thus, less than half of all agent-proposed code survives unaltered.
- **User pushback events**: In 44% of session turns, users issue corrections, failure reports, or direct interruptions, indicating persistent friction even when agents are the primary authors.
- **LLM-evaluated success**: Most sessions receive at least 50/100, but a minority exhibit failure, scored $<\!15$.

These findings demonstrate that, despite increasing adoption, agents remain inefficient in practice, and user oversight is both frequent and necessary.

## 5. Failure Modes and Security Vulnerability Analysis

Low-success sessions typically involve one or more of the following agent failure patterns:

- Misidentification of code context (e.g., repeatedly changing the incorrect parameter)
- Generation of irrelevant (“junk”) commits
- Planning pathologies (agent stalls, infinite loops interrupted by the user)

Security is evaluated using Semgrep on pre- and post-commit snapshots (restricted to files modified in the session). The introduced vulnerability rate is defined as:
$$
\text{vuln\_rate}_\text{introduced} = \frac{\#\text{findings}_\mathrm{post} - \#\text{findings}_\mathrm{pre}}{\#\text{added lines}} \times 1000
$$

| Coding Mode    | Vulnerabilities Introduced (per 1,000 lines) |
|----------------|----------------------------------------------|
| Human-only     | 0.08                                         |
| Collaborative  | 0.14                                         |
| Vibe coding    | 0.76                                         |

Agent-only sessions thus introduce approximately nine times the vulnerabilities of pure human commits and nearly five times that of collaborative sessions.

## 6. Example Interaction Patterns

Representative session excerpts:

- **Low-success (Score 10/100):**  
  Multiple unsuccessful attempts to fix the wrong animation parameter, ending with no commit:

  ```
  User:“Fix the slow appearance of history list card blocks…”
  Agent:Read{HistoryListView.swift}→Edit{…stagger delay…}
  User:“That’s the wrong parameter—the container animation is slow.”
  Agent:Edit{…item timing…}
  User:“You changed the item timing again… verify which parameter controls the container!”
  ```
- **User correction in collaborative mode:**  
  ```
  Agent:“PrepareCommitMsg doesn’t have access to transcriptPath…”
  Agent:Read{hooks.ts}→Edit{…}
  User:“But we do have session.CondensedTranscriptLines…? Please use that.”
  ```

These examples illustrate both agent misunderstanding and active user intervention.

## 7. Research Implications and Future Directions

SWE-chat establishes an evidence-driven foundation to:

- **Develop realistic agent benchmarks** capturing multi-turn workflows, code comprehension, debugging, and git usage, moving beyond static, patch-oriented evaluations.
- **Advance adaptive agent interaction**, such as improved timing for clarifying questions (observed at $<2\%$ of agent turns in practice) and uncertainty estimation.
- **Support data-driven simulator design** for scalable agent evaluation using observed user behaviors instead of human-in-the-loop studies.

As an extensible, growing corpus, SWE-chat is poised to enable longitudinal research into the evolution of developer–agent collaboration and guide the design and benchmarking of both next-generation coding agents and their evaluation frameworks [2604.20779].

Source: https://www.emergentmind.com/topics/swe-chat-dataset