LFTBench: Long-Input APR Benchmark

• LFTBench is a long-input automated program repair benchmark featuring 200 real bugs from AtCoder tasks with failure-inducing inputs typically around 1MB in size.
• The benchmark isolates the impact of long failure tests by showing that full test inputs can worsen repair quality, while reduced inputs enhance success rates.
• Constructed with rule-based task selection and low training leakage, LFTBench rigorously evaluates LLM-based repair methods against complete hidden test suites.

LFTBench is a benchmark for automated program repair (APR) with LLMs in the long-input regime, introduced in “Input Reduction Enhanced LLM-based Program Repair” (Yang et al., 21 Jul 2025). It is described as the first long-input APR benchmark and contains 200 real bugs from 20 programming tasks, each paired with a failure-inducing input whose median size is 1 MB (Yang et al., 21 Jul 2025). The benchmark is designed for a setting in which the failing test is crucial for reasoning about the root cause of a bug, yet very large failing inputs can trigger the “lost-in-the-middle” problem and degrade repair quality rather than improve it (Yang et al., 21 Jul 2025). By combining large official hidden tests, accepted reference solutions, buggy C++ submissions, and full-suite validation, LFTBench isolates the interaction between prompt length, failure evidence, and repair success (Yang et al., 21 Jul 2025).

1. Motivation and problem setting

LFTBench is motivated by a gap in existing APR evaluation. The paper argues that a fair study of input reduction for APR requires two properties that prior datasets lack: “(1) long failure-inducing test inputs and (2) low risk of training leakage” (Yang et al., 21 Jul 2025). Existing APR benchmarks such as Defects4J, Human-EvalFix, and TutorCode are said to contain only tiny tests, “usually a few hundred characters,” and therefore do not reveal how APR systems behave when failing inputs reach kilobytes or megabytes (Yang et al., 21 Jul 2025).

This design target is closely tied to the benchmark’s central systems question: whether including a long failure-inducing input in the prompt helps or harms LLM-based repair (Yang et al., 21 Jul 2025). In conventional APR settings, test information is usually assumed to be beneficial. LFTBench is constructed precisely to examine the opposite possibility: when official hidden tests are very large, the artifact meant to guide repair can dominate the prompt budget and bury the bug-triggering evidence in a long context (Yang et al., 21 Jul 2025). The benchmark is therefore not merely a new bug corpus; it is a targeted stress test for long-context APR.

A common misconception is that LFTBench is simply another general-purpose APR benchmark. The paper’s framing is narrower. It repeatedly positions the benchmark as the first APR benchmark that focuses on long test inputs, and more specifically as the first long-input APR benchmark (Yang et al., 21 Jul 2025). This suggests that its main contribution lies in isolating a failure mode of LLM-based repair that short-input benchmarks cannot expose.

2. Corpus construction and benchmark composition

LFTBench is built from AtCoder Beginner Contest tasks rather than from older widely used software-repair corpora (Yang et al., 21 Jul 2025). The source selection is deliberate. Codeforces was rejected because it exposes only small samples rather than full hidden tests, whereas AtCoder “released the full test archives for every Beginner Contest (ABC) up to ABC 377,” giving access to large official hidden cases and a trusted oracle (Yang et al., 21 Jul 2025). To reduce contamination risk, the benchmark uses tasks from ABC 361 to 377, which the paper states are all after the knowledge cutoffs of the four evaluated LLMs (Yang et al., 21 Jul 2025).

Task inclusion is rule-based rather than cherry-picked. The construction filters tasks by requiring that the largest official test file be at least 4 KB and that the task difficulty be between C and F (Yang et al., 21 Jul 2025). For each selected task, the authors “collect the ten most recent C++ submissions that failed on a large test before July 1, 2025,” producing 200 benchmark bugs (Yang et al., 21 Jul 2025). The benchmark therefore covers buggy programs in C++ and uses competitive-programming-style algorithmic problems as its task domain.

Difficulty band	Task IDs	Bugs
C	361C, 366C, 368C, 375C, 376C, 377C	60
D	362D, 364D, 365D, 367D, 369D, 370D, 371D, 376D	80
E/F	363E, 372E, 373E, 374E, 376E, 377F	60

The resulting benchmark contains 6 C tasks, 8 D tasks, and 6 E/F tasks, with 10 buggy codes per task, for a total of 200 buggy programs (Yang et al., 21 Jul 2025). Input scale is a defining characteristic. The paper states that the median failing-input size is over 1 MB, that each bug is paired with a failure-inducing input whose median size is 1 MB, and that the largest single file exceeds 8 MB (Yang et al., 21 Jul 2025). This size regime is what gives LFTBench its diagnostic value for long-context prompting.

The benchmark also incorporates a low-leakage construction logic. The evaluated standard models are reported with the following knowledge cutoffs: GLM-4-9B-chat, Oct. 2023; Qwen2.5-Coder-7B, Mar. 2024; Qwen2.5-Plus, Mar. 2024; and DeepSeek-V3, Jun. 2024 (Yang et al., 21 Jul 2025). Because ABC 361–377 postdate these cutoffs, the benchmark is designed to be “leakage-free” or at least low-leakage-risk (Yang et al., 21 Jul 2025).

3. Benchmark instance structure and formalization

The paper defines a benchmark instance through the repair setting used by ReduceFix. It states that the system receives five inputs: the problem description $P$, a correct reference solution $A$, a buggy submission $s_w$, the hidden test suite $I$, and one failure-inducing input $i_0$ (Yang et al., 21 Jul 2025). In this sense, each LFTBench instance couples a task statement, accepted reference, buggy C++ submission, one long concrete failing input, and the official hidden suite used as oracle.

The reduction target is formalized as follows:

$$i^{*}= \arg\min_{i \preceq i_0} |i| \quad\text{s.t.}\quad A(i)\neq s_w(i),$$

where $i \preceq i_0$ means $i$ is a subsequence of $i_0$, obtained by deleting elements while preserving order (Yang et al., 21 Jul 2025). This formalization makes the benchmark more specific than “bug plus test.” Each instance is a bug paired with a long failure-inducing input that can be minimized while preserving disagreement between the accepted and buggy programs.

The prompt-visible repair artifact includes both input and output discrepancy. In the prompt template, the repair model sees the problem description, the buggy code, the failing case input, “Your Output” from the buggy program, and “Expected Output” from the reference program (Yang et al., 21 Jul 2025). The visible failing case is therefore concrete and execution-grounded rather than symbolic.

Correct repair is evaluated against the full official test suite rather than only against the shown failing case. The success condition is defined as

$\forall\,i\in I,\qquad A(i)=\hat{s}(i).$

A candidate patch counts as correct only if it behaves identically to the accepted reference solution over the entire hidden AtCoder archive for that task (Yang et al., 21 Jul 2025). This strong oracle is a central property of LFTBench: it mitigates the risk that a repair method merely overfits the displayed counterexample.

4. Evaluation protocol, prompt conditions, and metrics

LFTBench is used in a protocol that separates reduction from repair. For reduction, the main metric is the compression rate

$A$0

where $A$1 and $A$2 are the sizes of the original and reduced inputs (Yang et al., 21 Jul 2025). A reduction is considered successful when it finishes within 60 seconds and still preserves the bug (Yang et al., 21 Jul 2025).

For repair evaluation, the paper defines pass@k as

$A$3

Although the typesetting is described as slightly malformed in the provided text, the intended meaning is explicit: pass@k is the fraction of bugs in benchmark set $A$4 fixed within $A$5 attempts (Yang et al., 21 Jul 2025).

The main repair experiments compare three prompting conditions: Baseline, which omits the failure-inducing input; Origin Test, which includes the full original failure-inducing input; and Reduced Test, which includes the reduced input produced by ReduceFix (Yang et al., 21 Jul 2025). The evaluated models are Qwen2.5-Coder-7B-Instruct, GLM-4-9B-chat, Qwen-Plus, and DeepSeek-V3 (Yang et al., 21 Jul 2025). Repair hyperparameters include temperature 0.8, up to 10 candidate patches per bug, 10-second compilation timeout, and 5-second execution timeout per test case (Yang et al., 21 Jul 2025).

Because even reduced inputs may remain large, the protocol also includes prompt-budget handling. When $A$6 or related output exceeds a line budget $A$7, the system truncates the displayed file by keeping the first $A$8 lines and the last $A$9 lines (Yang et al., 21 Jul 2025). This affects prompt presentation only; the full files remain used for execution and validation (Yang et al., 21 Jul 2025). That distinction is important because the benchmark is explicitly about long inputs, but correctness remains tied to full execution semantics.

5. Empirical characteristics and benchmark behavior

LFTBench is empirically difficult in a way that short-input APR benchmarks do not reveal. The central finding is that “Origin Test” often performs worse than “Baseline,” meaning that adding the full long failure-inducing input can reduce repair accuracy (Yang et al., 21 Jul 2025). This is the benchmark’s signature phenomenon: the raw test input can be informative in principle but harmful in practice when it is too long.

Reduction effectiveness on LFTBench is strong. Qwen-Plus generated a syntactically valid reducer.py for all 200 buggy codes, and 95% successfully reduced the input while preserving failure (Yang et al., 21 Jul 2025). Broken down by task difficulty, reducer success was 1.00 over 60 samples for C tasks, 0.96 over 80 samples for D tasks, 0.88 over 60 samples for E/F tasks, and 0.95 overall over 200 samples (Yang et al., 21 Jul 2025). By contrast, the pure-LLM one-shot reduction baseline succeeded on only 40% overall (Yang et al., 21 Jul 2025).

Compression statistics are correspondingly large. Among the 190 successful reductions, reported compression rates were 84.5% mean and 100.0% median for C tasks, 97.0% mean and 100.0% median for D tasks, 83.0% mean and 99.9% median for E/F tasks, and 89.1% mean and 100.0% median overall (Yang et al., 21 Jul 2025). The paper explains that, under the definition of $s_w$0, 100% means “the input was reduced to almost nothing” (Yang et al., 21 Jul 2025). This suggests that many LFTBench bugs are triggered by small core counterexamples hidden inside very large official tests.

Repair results show the same pattern. For Qwen2.5-Coder-7B-Instruct overall, pass@10 was 20.0 for Baseline, 19.0 for Origin Test, and 25.5 for Reduced Test (Yang et al., 21 Jul 2025). For GLM-4-9B-chat overall, the corresponding values were 8.5, 6.5, and 10.0 (Yang et al., 21 Jul 2025). For Qwen-Plus, they were 59.0, 59.0, and 61.0; for DeepSeek-V3, 66.5, 63.0, and 67.0 (Yang et al., 21 Jul 2025). The abstract summarizes the strongest relative gain as “up to 53.8% relative to a prompt that includes the original test, and by 17.6% compared with omitting the test entirely” (Yang et al., 21 Jul 2025).

Difficulty-stratified results make LFTBench especially diagnostic on D tasks. For Qwen2.5-Coder-7B-Instruct on D tasks, pass@10 rose from 25.0 for Baseline to 23.8 for Origin Test and 36.2 for Reduced Test (Yang et al., 21 Jul 2025). For GLM-4-9B-chat on D tasks, pass@10 changed from 13.8 to 11.2 to 20.0 (Yang et al., 21 Jul 2025). The paper further reports that Mann-Whitney-Wilcoxon testing yields a two-sided $s_w$1 for comparing ReduceFix to Origin Test (Yang et al., 21 Jul 2025).

The ablation studies clarify that LFTBench exposes both a length effect and an information-quality effect. Using Qwen2.5-Coder-7B-Instruct, prompt lengths averaged 3.1 KB for Baseline, 30.6 KB for Origin Test, 3.1 KB for Diff Lines, 6.4 KB for Reduced Test, and 36.4 KB for Reduced + Origin Test (Yang et al., 21 Jul 2025). Reduced Test achieved overall pass@10 of 25.5, compared with 20.0 for Diff Lines and 19.0 for Reduced + Origin (Yang et al., 21 Jul 2025). The paper’s interpretation is that compact but semantically complete failure evidence is more useful than either sparse diffs or concatenated long context.

The benchmark also transfers into an existing APR pipeline. With MAX_RETRY $s_w$2 and conversation window length $s_w$3, replacing the full failure-inducing input with the reduced one improved ChatRepair on LFTBench from pass@1 9.2 to 12.1, pass@5 22.6 to 29.0, and pass@10 30.5 to 37.0 (Yang et al., 21 Jul 2025). This is the reported 21.3% relative gain in pass@10 (Yang et al., 21 Jul 2025).

6. Scope, limitations, and significance

LFTBench is purpose-built rather than broad-spectrum. Its domain is AtCoder C++ competitive programming tasks, and the paper explicitly notes that it does not cover general software repositories or other programming languages (Yang et al., 21 Jul 2025). The benchmark is therefore specialized to a setting with algorithmic problems, accepted reference programs, released hidden tests, and concrete failure-inducing input files. This specialization is a limitation if one seeks a universal APR benchmark, but it is also what enables controlled study of megabyte-scale failure inputs.

Another common misconception is that a benchmark of this kind mainly measures prompt engineering. LFTBench is more tightly specified than that. Each instance is grounded in an accepted reference solution, a buggy submission, a concrete failing input, and the full official hidden suite, with repair counted as exact behavioral agreement over the entire suite (Yang et al., 21 Jul 2025). That evaluation design is stricter than checking whether a generated patch fixes only the displayed failing case.

The benchmark also has explicit validity and reproducibility strengths. Its task selection is rule-based, its leakage-control rationale is tied to task dates relative to model cutoffs, and its oracle comes from the official AtCoder archive (Yang et al., 21 Jul 2025). The paper says, “We release the first APR benchmark, LFTBench,” and the construction pipeline is described clearly enough to reproduce in principle, although the exact repository URL is not included in the provided text (Yang et al., 21 Jul 2025).

Its limitations are equally clear. LFTBench is intentionally narrow in implementation language; it is built around AtCoder tasks in a narrow date window; and it is designed to study long failure-inducing inputs rather than to serve as a general benchmark for all APR settings (Yang et al., 21 Jul 2025). A plausible implication is that LFTBench should be read primarily as a stress benchmark for long-context program repair, not as a replacement for broader APR corpora.

Within that scope, its significance is straightforward. LFTBench makes long failure-inducing inputs a first-class evaluation object in APR. It shows that raw long tests can underperform even no-test prompting, that minimized counterexamples often preserve the critical bug signal, and that reduction substantially improves repair success on a corpus of 200 real bugs validated against full hidden suites (Yang et al., 21 Jul 2025). For LLM-based APR, it establishes long-input handling as a benchmark dimension in its own right.