---
title: 'SWE-rebench: Swedish QA & Software Benchmarks'
url: https://www.emergentmind.com/topics/swe-rebench-dataset
type: topic
---

# SWE-rebench: Swedish QA & Software Benchmarks

SWE-rebench refers to a family of datasets and methodologies for constructing, evaluating, and benchmarking knowledge and agentic capacity in both factual-question answering (particularly for Sweden-specific facts) and interactive software engineering tasks. Several distinct datasets share this name, anchored by two primary streams: (1) a diagnostic QA benchmark for Sweden-related factual knowledge [2510.21360], and (2) large-scale, contamination-controlled datasets for issue-resolving tasks in software engineering [2505.20411, 2504.02605, 2512.10218]. This entry provides a comprehensive technical account of SWE-rebench’s design, statistics, protocols, and implications within these domains.

## 1. Sweden-Related Factual Knowledge Benchmark

SWE-rebench (“A Diagnostic Benchmark for Sweden-Related Factual Knowledge”) targets evaluation of LLM factual recall for Sweden-specific domains insufficiently covered by conventional (often US-centric) benchmarks [2510.21360]. The benchmark comprises 1,293 manually authored and verified fill-in-the-blank QA pairs, with Swedish original and human-edited English translations.

### Dataset Structure

- **Subsets**:  
  - *Sommarpratare*: 1,190 instances related to “Sommar i P1” radio hosts (2018–2024), covering a range of biographical and anecdotal facts.
  - *Sports Events*: 102 instances spanning “En Svensk Klassiker” races and other Swedish sports.
- **Schema**: Each item records: unique id, subset, Swedish and English Q/A, and optional cutoff_date for time-sensitive items.
- **Example**:
  - Niche (Figure 1):  
    question_sv: “Vilket cykelmärke kör Stig Johansson på Vätternrundan?”  
    answer_sv: “Husqvarna”  
    question_en: “What is the brand of the bike that Stig Johansson rides at Vätternrundan?”  
    answer_en: “Husqvarna”
  - Mainstream (Figure 1):  
    question_sv: “I vilken sångtävling representerade Toussaint ‘Tusse’ Chiza Sverige?”  
    answer_sv: “Eurovision”

### Annotation and Validation

Two independent annotators constructed and peer-verified all Q/A pairs using high-quality sources (Wikipedia, press); only “minimal” answers—sufficient for unambiguous closed-book evaluation—are retained. Units on numerical items are explicit.

### Limitations

Bias toward personalities from media and endurance sports is present; no multiple-choice or distractor candidates. No domain coverage in politics or business.

## 2. Evaluation Metrics and Benchmarks for QA

Model outputs are assessed using string-normalized Exact Match (EM), token-level F1, and token-level Recall—the latter chosen as the primary diagnostic given the dataset’s minimal-answer design.

#### Formal Definitions (`LaTeX` notation)

- Accuracy:  
  $Accuracy = \frac{\text{Number of correct predictions}}{\text{Total number of questions}}$
- Precision:  
  $Precision = \frac{|\text{Tokens}_{\text{pred}} \cap \text{Tokens}_{\text{gold}}|}{|\text{Tokens}_{\text{pred}}|}$
- Recall:  
  $Recall = \frac{|\text{Tokens}_{\text{pred}} \cap \text{Tokens}_{\text{gold}}|}{|\text{Tokens}_{\text{gold}}|}$
- F1:  
  $F_{1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

Cross-lingual factual consistency is quantified:

- $P_{\text{en}\mid\text{sv}} = \frac{\#\{\text{samples with correct\_en} \cap \text{correct\_sv}\}}{\#\{\text{samples with correct\_sv}\}}$
- $P_{\text{sv}\mid\text{en}} = \frac{\#\{\text{samples with correct\_en} \cap \text{correct\_sv}\}}{\#\{\text{samples with correct\_en}\}}$

### Model Performance

Smaller models with substantial Swedish pretraining achieve recall scores similar to those of three-times larger multilingual models. For example, EuroLLM-9B matches Gemma-3-27B (recall_sv/en ≈ 23–25%). Continued Swedish pretraining improves Swedish factual recall (e.g., AI Sweden LLaMA-3-8B, recall_sv 23.1%) but induces partial catastrophic forgetting of previously known facts in English. Cross-lingual consistency ranges 62–78% for best models.

## 3. Multilingual Issue-Resolving Dataset for SWE (Multi-SWE-bench)

A distinct SWE-rebench branch, sometimes referred to as Multi-SWE-bench, focuses on evaluating agentic reasoning and code-editing capabilities across seven programming languages [2504.02605]. It addresses the inadequacy of Python-only SWE-bench for multilingual and cross-ecosystem LLM benchmarking.

### Dataset Characteristics

- **Total instances**: 1,632 manually verified issue-resolving tasks.
- **Languages**: Java, TypeScript, JavaScript, Go, Rust, C, C++.
- **Proportions**: Go (26.2%), JavaScript (21.8%), Rust (14.6%), TypeScript (13.7%), C/C++ (~7.85% each), Java (7.84%).
- **Structure**: Each JSON object logs repo metadata, commit hashes, issue description, patch diffs (test_patch, fix_patch), before/after file contents, test-case transitions (ANY→FAILED→PASSED), Docker recipe, and run commands.

### Annotation Pipeline

A five-stage process ensures semantic correctness and reproducibility:
1. Repo selection—high popularity, CI support.
2. PR crawling and metadata extraction.
3. Containerized build environments.
4. Semantic filtering via test transitions—retaining only PRs with test failures resolved in the patch.
5. Dual manual annotation (>80% accuracy per language, 68 annotators).

### RL Extension (Multi-SWE-RL)

An RL-ready dataset (4,723 tasks) omits manual verification, retaining Docker environments and reward structures (Δpassed = number of tests passing post-patch − pre-patch). Rewards: +1 if at least one failing test is fixed.

### Evaluation Metrics

- **Resolved Rate**: Proportion of issues whose patch passes all tests.
- **Pass@k**: Fraction for which at least one out of k samples fixes the issue.
- **Exact-match Rate**: Generated diffs identical to ground-truth fix_patch.

## 4. Automated Interactive SWE Task Pipeline (Python-Focused)

The original SWE-rebench pipeline [2505.20411, 2512.10218] automates large-scale, contamination-controlled task extraction for interactive Python SWE agent evaluation.

### Pipeline Stages

1. **Task Collection**: PR–issue matching from GitHub Archive (∼10M PRs), filtered for Python, permissive license, and substantive test modification.
2. **Install Configuration**: LLM-driven (Qwen2.5-72B-Instruct) extraction of install/test recipes via README/Dockerfile parsing; error-based refinement.
3. **Execution Verification**: Each task containerized; verification proceeds if patch transitions at least one test from fail→pass, no regressions.
4. **Instance Annotation**: Fine-tuned LLM predicts binary task labels—difficulty, clarity, test-correctness (F1 = 0.82/0.76/0.65 on validation).

#### Funnel Rates

| Stage                       | Input      | Output   | Rate   |
|-----------------------------|------------|----------|--------|
| PR–Issue Matching           | 10M PRs    | 450K     | 4.5%   |
| Filtering (1–15 files/tests)| 450K       | 153K     | 34%    |
| Build & Execution Verification| 153K     | 21,336   | 14%    |
| Automated Annotation        | 21,336     | 21,336   | 100%   |

Resulting in 21,336 valid Python interactive tasks from 3,468 repos.

### Dataset Statistics

| Metric                  | Mean    | p50  | p75  | p95   |
|-------------------------|---------|------|------|-------|
| Issue Length (words)    | 141.7   | 91   | 173  | 412.3 |
| Files Edited            | 3.46    | 2    | 4    | 10    |
| Lines Edited            | 142.2   | 37   | 112  | 500   |
| Fail→Pass Tests         | 14.56   | 2    | 5    | 37    |
| Pass→Pass Tests         | 85.81   | 22   | 64   | 351   |
| Total Tests             | 105.4   | 31   | 82.3 | 428   |
| Difficulty Score        | 1.13    | 1    | 2    | 2     |
| Issue Clarity Score     | 1.04    | 1    | 2    | 3     |
| Test Correctness Score  | 1.38    | 2    | 2    | 3     |

Lower scores indicate ease, clarity, or correctness.

## 5. Contamination Controls and Benchmark Comparisons

SWE-rebench benchmarks are constructed and refreshed explicitly to avoid overlap with LLM pretraining data (“contamination”). Every instance’s timestamp is tracked, and evaluation is split by temporal cut-offs matching major LLM releases; all pre-cut-off tasks are flagged for contamination [2505.20411, 2512.10218].

- **SWE-Bench-Verified** comprises 500 issues drawn mostly before late 2023 and exhibits extensive contamination [2512.10218].
- **SWE-rebench** (as used by Prathifkumar et al.) provides temporally fresh, >21,000 issues, predominantly Python, yielding substantially lower model performance (e.g., DeepSeek-V3-0324: 39.7% on Verified vs 21.3% on Rebench). A plausible implication is that static benchmarks dramatically inflate evaluations when agents can simply recall memorized data.

## 6. Benchmark Evaluation Protocols and Model Comparison

All agents are evaluated using standardized minimal ReAct scaffolding, repeated seeds $(k=5)$, and uniform context limits (≤128K tokens). Key outcome measures include resolved rate $R$ and pass@5 metric:

\[
R = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\text{task}_i\text{ solved}\}\times100\%
\]
\[
\text{pass@}5 = 1 - \prod_{j=1}^5(1 - r_{ij}), \quad r_{ij}\in\{0,1\}
\]

Performance drops between contaminated and decontaminated sets are consistent across both proprietary (GPT-4.1: 31.1%→26.7%) and open-source models (LLaMA-3.3-70B: 18.1%→11.2%).

## 7. Utility, Community, and Extensions

SWE-rebench datasets are openly available; code and pipelines reside on GitHub and HuggingFace. The platform supports progressive community extensions (e.g., new PR seeds added per quarter for Multi-SWE-bench) and RL-oriented benchmark design using curated or agentlessly generated tasks [2504.02605].

Potential future directions—beyond current Swedish factual QA and Python-centric SWE—include coverage in business/politics/other languages, explicit external–coverage annotation (e.g., English Wikipedia overlap), and expansion of agentic SWE pipeline to JavaScript, Java, and C++ domains using analogous methods.

---

SWE-rebench therefore constitutes a family of benchmarks and methodologies central to current research on LLM factuality (especially for underrepresented cultural domains) and robust, contamination-resilient evaluation of agentic reasoning in software engineering. Its multifaceted construction, detailed annotation, and strict contamination control protocols reflect contemporary best practice for QA and code-interacting model evaluation [2510.21360][2505.20411][2504.02605][2512.10218].

Source: https://www.emergentmind.com/topics/swe-rebench-dataset