---
title: 'Multi-SWE-bench: Multilingual Benchmark'
url: https://www.emergentmind.com/topics/multi-swe-bench
type: topic
---

# Multi-SWE-bench: Multilingual Benchmark

Multi-SWE-bench is a multilingual, repository-level benchmark suite for evaluating Large Language Models (LLMs) and agentic frameworks on real-world software issue resolution tasks. It addresses the critical deficiency of prior benchmarks—most notably SWE-bench—which focused almost exclusively on Python and thus could not measure cross-language generalization, real-world robustness, or multilingual competence in software engineering agents. Multi-SWE-bench coordinates its task collection, annotation protocols, infrastructure, and evaluation criteria to provide a reliable and reproducible standard for measuring LLM-driven bug fixing, feature addition, and code optimization across Java, TypeScript, JavaScript, Go, Rust, C, and C++. Its open-source production pipeline and rapidly growing RL training dataset ecosystem (Multi-SWE-RL) lay the groundwork for scalable, cross-paradigm autonomous software engineering research [2504.02605].

## 1. Scope, Motivation, and Design Rationale

The inception of Multi-SWE-bench is motivated by the limitations observed in prior benchmarks such as SWE-bench [2504.02605], which relies solely on Python repositories. While SWE-bench demonstrated the viability of evaluating LLMs on repository-level GitHub issue resolution—a process that simulates practical bug fixing and feature implementation—its language restriction precludes meaningful generalization to real-world software practice, which is inherently multi-lingual and multi-paradigmatic.

Multi-SWE-bench expands this paradigm to seven major programming languages (Java, TypeScript, JavaScript, Go, Rust, C, C++), each selected for ecosystem prevalence, practical impact, and technical diversity. This design enables comprehensive evaluation across language boundaries, divergent build systems, runtime constraints, and type systems (dynamic/static, low-/high-level). Importantly, the benchmark adopts strict annotation and filtering protocols. From an initial pool of 2,456 PR-based candidates, 1,632 high-quality instances were retained after dual annotator verification and expert QA, ensuring high signal in issue clarity, test coverage, and absence of severe repository-level ambiguities [2504.02605].

## 2. Task Collection, Annotation, and Dataset Composition

Task construction in Multi-SWE-bench follows a five-phase protocol:

1. **Repository selection**: Only repositories with ≥500 GitHub stars, active maintenance, and continuous integration are considered to ensure industrial relevance and runnable builds.
2. **PR crawling**: All merged PRs linked to issues and modifying test files are candidate tasks. Metadata such as base commits, diff patches, and issue text are extracted.
3. **Environment determination**: Automated Dockerfile generation is performed by parsing repository CI workflows and documentation to ensure assemblable, platform-stable test harnesses for each language and project.
4. **Test-case filtering**: Each PR undergoes a three-state test transition analysis (base, test.patch, fix.patch) to detect true fail-to-pass transitions (demonstrable bug fix or feature addition) and to exclude non-reproducible or flaky cases.
5. **Manual verification**: 68 trained annotators apply a standardized questionnaire on issue description clarity, test coverage adequacy, and repository health, with cross-language QA over 1,632 instances.

Instance-level statistics reveal substantial task variety. Language distribution is: Java 128, TypeScript 224, JavaScript 356, Go 428, Rust 239, C 128, C++ 129. Each entry includes an issue description, ground-truth fix and test diffs, Docker build context, and logs. Patch and test suite sizes are reported per repository, with increased length and complexity for “hard” instances [2504.02605].

## 3. Evaluation Methodologies and Metrics

Multi-SWE-bench supports three canonical evaluation frameworks, all ported to the multilingual setting:

- **MagentLess** (“Agentless”): A deterministic workflow comprising fault localization, followed by code repair without iterative patch selection.
- **MSWE-agent**: A multi-turn, agent-based pipeline enabling iterative exploration, error analysis, patch creation, and controlled regression testing.
- **MopenHands**: An open-ended agent interface integrated with multi-language prompts and “git diff” support, allowing free-form codebase manipulation.

All three methodologies support a uniform metric regime:

- **Resolved Rate (RR)**: Percentage of tasks in which all previously failing tests pass post-patch, with no new regressions.
- **Exact Match (EM)**: Fraction of tasks for which the generated patch is bitwise identical to the ground-truth fix.
- **Success Location (SL)**: File-level localization accuracy, i.e., whether the model touches any ground-truth file.
- **Code Execution Success Rate (CESR)**: Patch-execution reliability over multiple samples.

Resolution rates are substantially lower for new languages than for Python even when using state-of-the-art LLMs. For instance, using GPT-4o and MagentLess, RR is 36.2% for Python but only 2.2% for TypeScript and 1.4% for JavaScript [2504.02605].

## 4. Comparative Results and Empirical Findings

Performance on Multi-SWE-bench reveals several consistent trends and failure modes:

- **Language-dependency**: All tested models and frameworks exhibit a sharp performance drop when transitioning from Python to other languages, with average RRs ≤15% in non-Python cases. Java is the highest among new languages, but C, C++, TypeScript, and JavaScript remain challenging (most <7%).
- **Framework-specific strengths**: MagentLess tends to yield higher file-level localization rates, while agent-based methods such as MopenHands excel in edit steps, yielding better overall RR across most languages.
- **Complexity sensitivity**: Success probability decreases with increased instance complexity, measured by patch size or number of files touched. For “hard” instances (estimated ≥1 h to solve), RR approaches zero for all new languages except Python [2504.02605].
- **Issue type dependency**: Bug fix tasks are resolved more frequently than new features or feature optimizations. This trend is observed across all languages and agent types (see Table 11 in [2504.02605]).

Resource consumption analysis shows deterministic workflows are more token- and cost-efficient, but agentic methods provide better debugging coverage at higher cost.

## 5. Expansion, Open-Source Community, and Multi-SWE-RL

To drive rapid data growth and foster broader agent development, the Multi-SWE-bench creators have released the full benchmark pipeline and an RL-targeted extension called Multi-SWE-RL. The latter is a collection of 4,723 containerized, reinforcement-signal–rich instances, supporting research in RL-based autonomous bug fixing and patch generation [2504.02605]. The data schema encapsulates detailed signals such as test pass/fail, compile status, and code coverage deltas.

Community contributions are encouraged through open-source CI/CD pipelines, HuggingFace datasets, and transparent documentation for adding new tasks. Contributor incentives include quarterly paper releases and authorship opportunities.

Related efforts, such as SWE-bench Multimodal (visual/JS-centric tasks) [2410.03859], SWE-bench-java [2408.14354], the automated dataset generation of SWE-Bench++ [2512.17419], and SPICE (automated clarity/test/evaluation labeling) [2507.09108], enrich the multilingual and modal breadth of evaluation and training resources in this ecosystem.

## 6. Limitations, Benchmark Mutation, and Future Directions

Despite its breadth, Multi-SWE-bench reflects several inherited limitations:

- **Overestimation on formal prompts**: Mutation analysis of the TypeScript subset, using IDE telemetry to mutate GitHub issue descriptions into realistic, terse user-style queries, demonstrates that formal benchmarks systematically overestimate agent capabilities—often by >50% for leading models [2510.08996].
- **Modality and coverage constraints**: Multi-SWE-bench is primarily text/code-based. Multi-modal reasoning and vision–code integration are treated separately (e.g., SWE-bench Multimodal [2410.03859]).
- **Oracle dependency**: All evaluation depends on repository-supplied test suites, which may provide incomplete functional specification.
- **Manual labor and scalability**: Expert annotation is expensive; attempts like SPICE [2507.09108] and programmatic pipelines in SWE-Bench++ [2512.17419] aim to automate labeling and reduce curation cost while preserving label fidelity.
- **Language and task extension**: Increasing coverage of underrepresented languages (e.g., PHP, Ruby, Kotlin), expanding to infrastructure and UI tasks, and integration of security and style checks remain key open directions.
- **Reinforcement learning infrastructure**: The Multi-SWE-RL community and pipeline are positioned to drive research toward fully autonomous RL-driven debugging and software agent learning.

A plausible implication is that sustained, collaborative expansion—combining open-source data production, annotation automation, mutation frameworks, and RL-infrastructure—will continue to advance the empirical measurement and practical capabilities of multilingual software engineering agents.

Source: https://www.emergentmind.com/topics/multi-swe-bench