Refine After Generation: Toward Correct and Concise Patches in LLM-based Program Repair

Abstract: LLMs have advanced automatic program repair (APR) to the point where agentic systems routinely resolve real-world, repository-level issues. Yet the generated patch has received little scrutiny beyond whether it passes tests. In this paper, we identify patch verbosity as a major yet overlooked concern in LLM-based APR. Characterizing 28 state-of-the-art approaches on SWE-bench Verified, we find that even successful patches are consistently larger and more complex than developer patches, with the median approach producing 121.78% more total changes, 80.91% more net changes, and 43.99% higher cyclomatic complexity. We further show that this verbosity is rooted in capability-oriented design choices such as iterative refinement and broad context, and can hardly be reduced by surface-level controls such as output format or minimality prompts. Motivated by these findings, we formulate post-generation patch refinement and propose RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation. RECAP's refiner is trained via supervised fine-tuning and direct preference optimization with distilled reasoning traces, on a dataset of patch pairs we construct from multiple sources. Across four host systems, prompting, commit-untangling, and minimality-aware baselines reduce patch size only by sacrificing 49 to 217 resolved instances. In contrast, RECAP achieves a substantially better size-correctness tradeoff, cutting average total changes from +242.14% to +4.24% and net changes from +348.24% to -39.75% relative to developer patches while preserving or improving resolution by up to 42 instances. Our results indicate that minimality cannot be simply reduced to syntactic compression, and that decoupling minimization from generation offers a practical path to more reviewable repairs.

• The paper shows that all 28 evaluated LLM-based repair systems produce patches larger than developer patches, with median increases of 121.78% in total changes and 43.99% in cyclomatic complexity.
• The paper introduces RECAP, a specialized post-generation adapter trained with SFT and DPO that reduces average patch inflation from 242.14% to 4.24% while preserving or improving resolution across four SWE-bench systems.
• The paper finds that DPO is essential for preventing over-aggressive simplification, while refinement can also recover previously failing repairs, demonstrating that patch verbosity threatens correctness as well as reviewability.

Motivation and problem

LLM-based automatic program repair (APR) has matured to the point where agentic systems routinely resolve real-world repository-level issues on SWE-bench. The paper argues that the resulting artifact—the patch—has been evaluated almost exclusively by whether it passes tests, while its size, scope, and complexity for downstream review have been neglected. This matters because empirical software engineering evidence consistently links smaller patches to higher acceptance rates, lower fault-proneness, and lower review effort. The paper identifies patch verbosity as a systemic property of state-of-the-art LLM-based APR, and argues that it cannot be fixed by prompt engineering or output-format choices, motivating a dedicated post-generation refinement stage.

Characterizing verbosity (RQ-1)

The authors collect resolved patches from 28 leaderboard approaches on SWE-bench Verified (as of November 2025) and compare each against the corresponding developer (gold) patch using five metrics: total changes (added + deleted LOC), net changes, edited files, Halstead Time, and cyclomatic complexity (CC), all expressed as per-instance relative deltas against gold.

The headline result is stark: all 28 approaches produce successful patches larger than gold, with the median approach at +121.78% total changes, +80.91% net changes, +13.06% edited files, and +43.99% cyclomatic complexity. Notably, LOC inflation and operator-level complexity can diverge: Halstead Time deltas range from −14812% (Zai) to +901% (EntroPO) with a near-zero median, partly because large patches often rewrite rather than add, and partly because auxiliary edits to tests and reproduction scripts inflate LOC. The four systems exceeding +1000% total changes are precisely the four most auxiliary-edit-heavy (e.g., Zai with 320 auxiliary-edit instances and +15143% total changes), showing that raw LOC alone is an unreliable verbosity measure.

The factor analysis (RQ-1.2) yields a conclusion that contradicts common practice: surface-level controls are weak. Structured output formats (S/R, XML, JSON) versus Git diffs show no significant difference in total changes; explicit minimality instructions produce patches (+208.83% total) statistically indistinguishable from no instruction at all (+191.39%), while only implicit guidance yields smaller patches (+49.65%). By contrast, capability-oriented design choices are strongly and significantly associated with verbosity: iterative refinement (+406.46% vs. +95.60% total changes, $p<0.001$) and top-ranked resolution (+223.39% vs. +118.09%, $p<0.001$) both inflate patches. The implication is a capability–concision tradeoff that constraining the generator cannot resolve, which motivates placing refinement downstream of generation rather than inside it.

The RECAP adapter

RECAP (REfine for Correct And concise Patches) is a lightweight, plug-and-play adapter inserted between a host framework's patch generation and its downstream validation/ranking. It comprises three components: a collector that normalizes heterogeneous framework artifacts into a unified $(B, C, p, T)$ context (issue description, code context, candidate patch, optional tests) using an LLM-generated, framework-adaptive extraction script; a filter that decides whether a candidate should be refined; and a refiner, a small open-source LLM (primarily Qwen-3.5-27B) that rewrites the candidate in SEARCH/REPLACE blocks.

Training data totals 5,540 instances from three sources: 1,227 function-level pairs from the ACPR dataset (lower-CCR solution as verbose input, higher-CCR as target), 533 tangled-commit pairs from the CCS dataset (tangled commit as input, atomic fix as target), and 3,780 synthetic SWE-bench instances created by inflating gold patches via seven categories of verbosity patterns (over-complex solutions, irrelevant edits, code overwriting, overfitting fixes, documentation noise, reinvented abstractions, cosmetic noise) plus rejection sampling, with 20% "as-is" examples to prevent blind refinement. Reasoning traces are distilled from a stronger teacher model. Training proceeds in two phases: SFT with a granularity-based curriculum (function → commit → repository), then DPO over 822 preference pairs contrasting gold patches with the SFT model's own over-large samples.

Three deployment modes trade safety against concision: UR (unconditional refinement), JGR (LLM-as-Judge decides refinability when tests are unavailable), and OGR (oracle-guided, accepting the refined patch only if it still passes the test suite).

Refinement effectiveness (RQ-2)

Across four host systems (Agentless, SWE-agent, Moatless, OpenHands), the baselines fail systematically. Prompting Claude-4.5-Sonnet—the far stronger general-purpose model—with the same refinement input shrinks patches but loses 29 to 44 resolved instances per host. Atomizer (commit untangling) loses 49 to 217 instances because untangling never guarantees the extracted fix is self-contained. AdaPatcher and QiMeng-PRepair, trained as minimality-aware function-level generators, lose 49 to 102 instances because their locality assumptions do not transfer to repository-level patches. This validates the paper's central claim that repository-level patch refinement is a distinct task, not a repurposing of untangling or function-level minimization.

RECAP achieves a markedly better tradeoff. Averaged across hosts, UR and JGR cut total changes from +242.14% to +4.24% and +9.76% respectively, and net changes from +348.24% to −39.75% and −34.84%, while preserving or improving resolution (up to +24 on SWE-agent under UR). OGR is deliberately less aggressive (+18.55% total, −7.39% net) but guarantees no resolved instance is dropped, and in fact improves all four hosts by +23 to +42 resolved instances. The gains scale with host verbosity: SWE-agent's extreme patches (+690.91% total, +1158.45% net) collapse to +24.69% and −33.17% under UR. Even Agentless, already near-gold, is pushed below the gold baseline.

Two caveats are worth noting. First, textual reduction does not imply proportional complexity reduction: Halstead Time remains positive for SWE-agent and OpenHands under UR even as LOC falls, echoing the RQ-1 finding that verbosity is multi-dimensional. Second, the transition analysis shows refinement is not purely subtractive—the refiner recovers 18 to 43 previously failing instances per host (I→C), strongest on the most verbose hosts, indicating that extraneous edits actively interfere with correctness. Verbosity is thus a correctness risk, not merely a review burden.

Ablation (RQ-3)

The ablation isolates the roles of the training phases. Prompting the base model without any fine-tuning already performs shallow cleanup but is unreliable, dropping 42–50 resolved instances on three hosts. SFT alone teaches concision so well that it over-applies it: it achieves the best Halstead Time on every host and the lowest average total changes among trained variants (+15.21%), yet resolves only 276 instances on average—below even the untrained baseline's 293. Judged on a single minimality metric, the SFT-only model would look superior while being the weakest overall, a concrete instance of the paper's thesis that the most aggressive simplifier is not the best patch. DPO supplies the missing restraint by contrasting gold patches with the model's over-aggressive samples, recovering average resolution to 332 (gains of +84, +70, and +68 on the three most verbose hosts) without sacrificing concision; the full model retains the best average total changes (+4.24%), net changes (−39.75%), and edited-file scope.

Generalizability

Re-instantiating the refiner with Gemma-4-4B and Mistral-3-14B-Reasoning shows the gains stem from the refinement task and training pipeline rather than a specific backbone: all three models reduce patch size on nearly every metric, and even a 4B model remains viable, enabled by long-context capacity for repository-level inputs. Qualitative analysis further shows refinement beyond line deletion, including removal of redundant scaffolding branches and consolidation of four near-identical conditionals into a single loop, consistent with the observed cyclomatic-complexity reductions.

Limitations and open questions

The authors are explicit about scope. RQ-1 classifies design factors from public artifacts and treats the analysis as association, not causation; undocumented settings may affect the classification. Correctness claims are bounded by SWE-bench's test suites, which establish plausibility rather than semantic correctness, and gold patches serve only as relative reference points since they are not guaranteed minimal. The synthetic training pipeline depends on LLM-based judging, inflation, and rejection sampling, which may introduce noisy supervision, mitigated but not eliminated by multi-source data. Evaluation is confined to SWE-bench Verified and Python; transfer to other languages, proprietary repositories, and settings lacking executable oracles remains untested. Open questions left by the paper include whether refinement transfers beyond SWE-bench-style settings, how to guarantee correctness without a test oracle (the JGR mode relies on an imperfect judge), and whether the capability–concision tension observed in RQ-1 can be addressed at generation time rather than post hoc.

Conclusion

The paper makes two contributions of independent interest. Empirically, it establishes that patch verbosity is systemic across 28 state-of-the-art APR systems—median +121.78% total changes and +43.99% cyclomatic complexity over gold—and is driven by capability-oriented design rather than controllable surface choices. Methodologically, it formulates post-generation patch refinement and shows that a small specialized refiner trained with SFT and DPO dominates prompting, untangling, and minimality-aware baselines on the size–correctness tradeoff, reducing average total changes from +242.14% to +4.24% while improving resolution by up to 42 instances. The ablation's finding that DPO is essential to restrain an over-aggressive SFT simplifier reinforces the paper's broader conclusion: minimality is not syntactic compression, and decoupling minimization from generation is a practical path to reviewable repairs.