SWE-bench Lite-S: Python Function Benchmark

Updated 30 March 2026

SWE-bench Lite-S is a rigorously filtered subset of SWE-Bench, focusing on real-world, function-level Python edits for precise LLM evaluation.
It consists of 300 curated GitHub issues with strict leakage and test adequacy checks, ensuring robust and cost-efficient assessment.
Empirical evaluations report high localization and resolution rates from agents like GPT-5.2 and Claude-4.5, guiding reliable benchmark evolution.

SWE-bench Lite- $S$ refers to a rigorously filtered subset or evaluation regime within the SWE-Bench family of benchmarks, focused on assessing the capability of LLMs and code-generation agents to perform fine-grained software engineering tasks on real-world Python projects. The term “Lite- $S$ ” is not an official benchmark release but functions as a commonly used shorthand in the literature to distinguish (a) function-level or single-file evaluation splits from (b) broader or less stringently curated variants. Across several high-impact papers, including those introducing agentic methods and evaluation pipelines, Lite- $S$ crystallizes as a reference point for rigorous, difficulty-normalized, and cost-efficient assessment of LLM-driven repository-level code modification.

1. Benchmark Definition and Construction

SWE-bench Lite, from which Lite- $S$ is derived, is a Python-only benchmark comprised of 300 curated GitHub issues—each coupled with a repository snapshot, a natural-language issue description (e.g., bug, feature, or configuration task), and a gold patch representing the developer’s actual fix (He et al., 1 Mar 2026, Yu et al., 10 Jun 2025, Aleithan et al., 2024). These issues are drawn from 11 or 12 widely-used open-source repositories. The Lite benchmark is selected for breadth (covering diverse real-world problems) and tractability (single-file or at most low-complexity multi-file patches).

The Lite- $S$ subset emerges from further filtering, with criteria varying by paper:

In (He et al., 1 Mar 2026), Lite- $S$ refers to the 274 instances in which the ground-truth patch modifies an existing Python function—i.e., only tasks that stress function-body localization and edit rather than configuration files or global script changes.
In (Xia et al., 2024) and (Aleithan et al., 2024), Lite- $S$ is used for a 252-instance subset, filtered to exclude issues with (i) exact patch leakage in descriptions, (ii) insufficient or misleading descriptions, or (iii) solution steps that contradict the developer patch.
In (Pan et al., 2024), Lite- $S$ (sometimes called “SWE-Bench Lite”) denotes the main 300-instance single-file subset, but after further filtering for tasks validated against executable Docker environments, 230 instances remain.

Key Construction Principles

Source: All issues are taken as-is from open-source Python repositories with issue and PR traceability.
Patch granularity: Excludes multi-file or cross-module edits, focusing on tractable, functionally targeted interventions.
Test suite alignment: Each issue is coupled with tests that fail on the old code and pass on the gold patch.
Manual curation: Benchmark maintainers and researchers have manually screened for solution leakage and insufficient test coverage.

2. Filtering Schemes and Data Quality

Stringent filtering to arrive at Lite- $S$ has become crucial to ensure evaluation integrity:

Leakage checks: Instances where the gold patch or its functional equivalent appears verbatim or stepwise in the issue description are removed (Aleithan et al., 2024, Aleithan et al., 2024).
Description adequacy: Issues lacking critical reproduction information or containing misleading steps are excluded (Xia et al., 2024).
Function-level focus: For localization metrics at the function level (Acc@5), only issues modifying at least one function in the codebase are selected (He et al., 1 Mar 2026).
Test sufficiency: Test augmentation and oracle filtering are used to mitigate cases in which original tests are too weak (e.g., via UTGen and intramorphic oracles in (Yu et al., 10 Jun 2025)).

Relevant statistics from (Aleithan et al., 2024):

Statistic	Value	Description
$N_{\text{total}}$	300	Full SWE-bench Lite size
$N_{\text{leak}}$	18 (6%)	Instances with solution leakage
$N_{\text{suspicious}}$	8 (2.67%)	Weak-test cases
$N_{\text{filtered}}$	274	Remaining after leakage/weak-test removal

The solution-leakage rate is $R_{\mathrm{leak}} = 0.06$ , and the adjusted resolution rate after filtering becomes $R_{\mathrm{after}} = 28/274 \approx 10.22\%$ .

3. Task Types and Evaluation Metrics

Lite- $S$ is engineered to target function-level or fine-grained repository edits, with a strong focus on real-world localization and repair. Typical tasks involve single-file bug fixes or small feature additions; large, ambiguous, or multi-file changes are excluded (Pan et al., 2024).

Metrics

End-to-end resolve rate (ResolveRate):

$\mathrm{ResolveRate} = \frac{\#\text{issues resolved end-to-end}}{\#\text{total issues}}$

A patch is considered resolved if it passes all tests.

Localization accuracy (LocAcc@k):

$\mathrm{LocAcc@k} = \frac{\#\text{instances with all ground-truth locations in top-}k}{\#\text{total instances}}$

File-level (Acc@3) and function-level (Acc@5) are routinely reported.
- F $\to$ P rate (reproduction success): Fraction of issues for which a new failing test can be generated and then passes on the developer patch (Chen et al., 21 Oct 2025).
- Oracle-based correctness: Augmented testing with LLM-generated or verification tests to detect weak or insufficient originals (Yu et al., 10 Jun 2025).

Instance Complexity

Mean lines edited = 10.6; always a single file modified (Pan et al., 2024).

4. Empirical Findings and Leaderboard Effects

Reported results on Lite- $S$ and comparable subsets reveal both incremental LLM/agent progress and the sometimes dramatic impact of benchmark curation:

Resolution/Localization (He et al., 1 Mar 2026):
- File-level Acc@3 (GPT-5.2): $92.0\%$ ; Claude-4.5: $97.0\%$
- Function-level Acc@5 (on 274 tasks): GPT-5.2 $70.4\%$ , Claude-4.5 $87.6\%$
- ResolveRate (GPT-5.2): $59.0\%$ , Claude-4.5 $71.3\%$
Baseline solve rates (Xia et al., 2024):
- Agentless, Lite- $S$ (252 tasks): $28.17\%$
- CodeR, Lite- $S$ : $28.57\%$
- Aider, Lite- $S$ : $26.59\%$

Test-augmentation and parser improvements have triggered leaderboard changes for up to 40.9% of agents (Yu et al., 10 Jun 2025). After filtering solution leaks and weak tests, the “true” end-to-end solve rate for leading toolkits can drop from 18% to approximately 10% (Aleithan et al., 2024).

5. Evaluation Methodologies and Automation Pipelines

State-of-the-art agentic frameworks evaluated on SWE-bench Lite- $S$ employ various localization and modification strategies:

Two-agent systems (e.g., SWE-Adept in (He et al., 1 Mar 2026)): decouple localization (agent-directed DFS over code dependencies) and resolution (iteration, adaptive planning, and fine-grained checkpoint tracking).
Simplified agentless pipelines (Xia et al., 2024): rely on static sequences (localize, repair, validate) rather than dynamic tool use.
Automated regression-minimization (Chen et al., 21 Oct 2025): TestPrune prunes regression suites from ~9,000 to 9–11 tests, yielding 27× speedups and several point increases in resolve rate.
LLM-driven test augmentation (Yu et al., 10 Jun 2025): UTGen covers file-, function-, and line-level localization with direct LLM prompting for test synthesis; acts as an oracle in UTBoost to identify passing but functionally incorrect fixes.

These frameworks, when applied to Lite- $S$ , sharply surface the impact of evaluation methodology — showing “true” agent capability versus artifacts arising from insufficient or leaky data.

6. Critical Limitations and Benchmark Design Implications

Solution leakage and weak tests are recurrent in original SWE-bench Lite, necessitating post hoc curation, additional test synthesis, and leaderboard rescoring (Aleithan et al., 2024, Yu et al., 10 Jun 2025).
Difficulty normalization: Removing trivialized or ill-specified problems yields a more accurate reflection of open LLM-agent capabilities on unsolved or truly ambiguous real-world software issues (Xia et al., 2024, Pan et al., 2024).
Evaluation harness robustness: Multi-line parsing and test-log auditing are critical, as parser deficiencies have wrongly labeled over half of Lite instances in some settings (Yu et al., 10 Jun 2025).
Recommendations for future variants (Aleithan et al., 2024, Yu et al., 10 Jun 2025):
- Systematic leak-proofing of text fields.
- Incorporation of automated coverage/mutation analysis.
- Explicit report of both unfiltered and filtered pass rates, with denominator definitions.

A plausible implication is the increasing institutionalization of rigorous secondary curation and meta-evaluation practices for all SWE-bench variants, with Lite- $S$ and similar subsets serving both as development baselines and as “sanitization” checkpoints for agent research.

7. Extensions and Research Ecosystem

SWE-bench Lite- $S$ serves as a foundational subset for rapid iterative testing in environments such as SWE-Gym (Pan et al., 2024), for test-augmentation frameworks like UTBoost (Yu et al., 10 Jun 2025), and for task-normalized leaderboards aiming to more faithfully represent LLM progress (Aleithan et al., 2024, Xia et al., 2024). While more stringent derivatives (e.g., SWE-Bench Pro, SWE-Bench+) exist, Lite- $S$ strikes a deliberate trade-off — fast, cost-efficient evaluation with enough complexity to surface real model weaknesses but enough tractability to permit statistical power and automation pipeline iteration.

The measured effects of curation, test augmentation, and leakage filtering in Lite- $S$ directly shape consensus on “true” LLM agent capability, informing both methodology and benchmark evolution across the software engineering research community.