---
title: 'SWE-Dev Dataset: Agentic Software Benchmarks'
url: https://www.emergentmind.com/topics/swe-dev-dataset
type: topic
---

# SWE-Dev Dataset: Agentic Software Benchmarks

The SWE-Dev dataset refers to several distinct large-scale, executable benchmarks in software engineering, most commonly denoting (1) a dev set associated with the APEX-SWE benchmark for measuring agentic software engineering capabilities [2601.08806], (2) a platform focused on large-scale, annotated agent trajectories and synthesized test suites for Python bug-fixing [2506.07636], and (3) a dataset targeting autonomous feature-driven development evaluation at scale [2505.16975]. Each variant advances research on autonomous software development and evaluation of Large Language Models (LLMs) and multi-agent systems by constructing verifiable, real-world task suites and supporting reproducible, containerized execution.

## 1. Data Scope and Composition

The SWE-Dev datasets serve as benchmarks for diverse software engineering tasks, varying in design and primary objective:

**A. APEX-SWE Dev Set ("SWE-Dev") [2601.08806]:**
- 50 tasks: 25 integration (“greenfield” end-to-end system construction) and 25 observability (diagnose & fix production bugs).
- Realistic contexts, with integration tasks requiring orchestration of services such as AWS LocalStack, EspoCRM, MailHog, Medusa, Zammad, Mattermost, and PostgreSQL.
- Observability tasks are derived from production bug reports (GitHub issues→PRs, containerized environments, synthetic logs/dashboards).

**B. SWE-Dev for Agent Training and Bugfix [2506.07636]:**
- 38,000 issue→patch instances from 4413 public Python repositories (≥5 GitHub stars, ≥3 PRs).
- Each instance: issue description, pre-patch source snippet, gold patch, test suite (synthesized and extracted), and 1–N agent trajectories.
- 4630 executable unit-test files (2,097 synthesized “fail-to-pass” pytests; remainder extracted from PRs).
- 19,300 agent trajectories (multi-step ReAct/OpenHands rollouts).

**C. SWE-Dev for Feature-Driven Development [2505.16975]:**
- 14,000 training and 500 test instances (250 easy, 250 hard), Python-only.
- Tasks from 1,086 PyPI packages: per-task averages of 64 source files (20,200 LOC), PRD length 1,833 tokens, ~200 LOC ground-truth edit, ~6 unit tests.
- Each repo is containerized for strict reproducibility.

| Dataset Variant                        | Instances/Tasks      | Domain          | Task Focus         |
|----------------------------------------|----------------------|-----------------|--------------------|
| APEX-SWE Dev Set [2601.08806]          | 50                   | Mixed           | Integration/Debug  |
| SWE-Dev (Agent Bugfix) [2506.07636]    | 38,000               | Python          | Bug fixing         |
| SWE-Dev (Feature Dev) [2505.16975]     | 14,500 (14k+500)     | Python          | Feature addition   |

## 2. Task Design and Underlying Motivation

**Integration Tasks** in the APEX-SWE dev set simulate greenfield (from-scratch) construction of cloud-centric business workflows, emphasizing realistic interoperation across APIs and infrastructure-as-code. These are designed by experienced engineers based on common industry patterns, and stacked environments such as AWS emulation and enterprise SaaS products are provided for full-fidelity validation.

**Observability Tasks** simulate the complex, information-incomplete debugging flows that dominate production engineering, using logs, dashboards (Grafana/Loki), and fragments of chat/discussion for context. Each task is locked down via a full, containerized snapshot, including failing and gold-standard patches.

The SWE-Dev datasets for bug-fix and FDD are inherently grounded in practical, developer-authored source code and actual or synthesized test suites. The bug-fix corpus [2506.07636] centers on the issue→patch workflow, while FDD [2505.16975] masks complete features (6 functions/task on average) and enforces verification through authentic unit tests.

## 3. Data Generation, Test Synthesis, and Trajectory Curation

The SWE-Dev corpus uses LLM-driven synthetic test pipelines for expanding coverage beyond the limitations of human-authored tests. The test-generation methodology [2506.07636] entails:
- Extracting (i) bug-report text, (ii) faulty function snippet, (iii) import/dependency metadata.
- Transforming issues into one or more Gherkin “Given–When–Then” scenarios via Llama-3.1-70B-Instruct.
- Compiling scenarios into pytest functions using Qwen2.5-Coder-32B.
- Executing synthesized tests to ensure they fail pre-patch and pass post-patch, discarding those not meeting both criteria.
- Selection based on a sharp threshold: candidates with fail→pass accuracy ≥0.9 (or probabilistically weighted via $P_{acc}$ for lower scores).

Agent trajectories leverage ReAct-style iterations under OpenHands, producing multi-action (exec, patch, etc.) rollouts examined both for reward-weighted fine-tuning and as negative examples in offline RL protocols (KTO, OREO).

The feature-driven variant constructs environments per-task in Docker, verifying test-harness integrity before adding to the dataset. Masked features match PRD prompts, and only tasks for which all original tests pass on ground truth are included [2505.16975].

## 4. Data Structures and Schema

**APEX-SWE dev set [2601.08806]:** Each task is a JSON object with fields:
- `task_id`, `task_type`, `prompt.md`, `context_files` (multi-file), `test_driver`, `test_patch`, `golden_patch`.
- Integration: Code/configuration snippets, working deployment passing side-effect unit tests.
- Observability: Patch files (.diff) for regression-free fixes, synthetic logs, full environment specs.

**Bug-fix & Feature instances:**
- One JSON per issue/task, with fields for repo URL, issue or PRD text, source snippet, reference patch, associated tests, and cyclomatic complexity.
- Trajectories: action sequence, final score, and metadata.

All container environments, test execution traces, and reproducible instructions are provided to guarantee on-demand re-evaluation.

## 5. Evaluation Protocols and Metrics

The principal evaluation metric for SWE-Dev variants is Pass@k — the empirical probability that at least one of $k$ sampled generations passes all executable correctness checks. The formula used:
\[
\text{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}
\]
where $n$ is number of samples, $c$ is count of passing generations. Aggregate Pass@1 and Pass@3 are reported.

Other critical metrics:
- Instruction Following Rate (IFR) over filesets in FDD [2505.16975].
- RL reward as test-pass rate (fraction of test assertions satisfied).
- Classical classification metrics (precision/recall/F1) for test-case “fail→pass” classification [2506.07636].
- Cost and call counts for multi-agent system (MAS) benchmarking in FDD.

All metrics are strictly computed with containerized, developer-authored or LLM-synthesized test suites, guaranteeing factual reproducibility.

## 6. Statistical Properties and Coverage

The SWE-Dev family yields a diverse, challenging dataset space:
- Bug-fix corpus [2506.07636]: 38k issues, mean input length ≈4,100 tokens (σ²≈(1,400)²), patches at μ=15 LOC, pre-patch complexity μ≈8.1.
- FDD [2505.16975]: avg. 64 source files/repo, 20k LOC per sample, 6 functions/feature, ≈6 tests/sample.
- APEX-SWE dev set [2601.08806]: 
  - Integration: stacks include LocalStack (56%), EspoCRM (35%), MailHog (33%), Mattermost (32%), Medusa (31%), Zammad (26%).
  - Observability: language coverage — Go (30%), Python (25%), TypeScript (25%), Java (10%), C++ (10%); ~615 log lines/task.

Feature development and complex integration/observability set new frontiers for agentic LLM evaluation, revealing substantial unsolved challenges; for instance, state-of-the-art models achieve only 21–29% Pass@3 on hard FDD tasks [2505.16975].

## 7. Licensing, Access, and Usage

All SWE-Dev variants are fully open-source:
- APEX-SWE dev set: Released under CC-BY 4.0, accessible via HuggingFace Datasets (`mercor/APEX-SWE`, split="dev"), with scripts/evaluation at https://github.com/Mercor-Intelligence/apex-evals [2601.08806].
- SWE-Dev for bug-fix: Apache 2.0 license, hosted at https://github.com/THUDM/SWE-Dev [2506.07636].
- SWE-Dev for feature development: Available at https://github.com/justLittleWhite/SWE-Dev [2505.16975].

All datasets provide Docker configurations and code for test execution, enabling direct reproducibility. Python code snippets for installation, loading, and batch evaluation are included in documentation. Dataset schemas are published with executable environments and include sample notebooks for both inference and training.

---

The SWE-Dev suite thus constitutes a foundational resource for advancing empirical research on LLM-based software engineering, providing data at scale, strict verification regimes, and environments for reliable, fine-grained benchmarking [2601.08806, 2506.07636, 2505.16975].

Source: https://www.emergentmind.com/topics/swe-dev-dataset