Calibration-Guided Automated Repair

• Calibration-guided automated repair is a design approach that uses precise, behavior-based signals to steer patch generation beyond mere compilability.
• It leverages diverse cues like runtime errors, static analysis, and discriminative test generation to align repairs with intended functionality and avoid overfitting.
• Recent experiments show that calibration improves repair quality by reducing trivial patches and increasing both genuine fixes and correction speed.

Searching arXiv for the cited works and closely related papers on calibration-guided automated repair. Calibration-guided automated repair denotes a family of repair methods in which patch generation, fault localization, validation, or search is steered by signals that are more closely aligned with intended behavior than raw compilability or coarse test outcomes alone. Taken together, recent work suggests a shift from open-loop generate-and-validate pipelines toward repair systems calibrated by semantic judges, discriminative tests, runtime symptoms, debugger state, static-analysis specifications, intermediate postconditions, model-counting heuristics, or domain-specific repair patterns. In this sense, calibration is not a single algorithmic primitive but a design principle for aligning repair with stronger evidence about correctness, intent preservation, and non-triviality (Sun et al., 19 Sep 2025, Le-Anh et al., 13 Apr 2026, Xin et al., 2024).

1. Conceptual foundations

A recurring premise in automated program repair (APR) is that weak validation signals induce weak repairs. The static-analysis integration work identifies patch overfitting as a central failure mode: patches are accepted because they pass a limited test suite, not because they are truly correct for all relevant inputs. Its core claim is that tests often provide only a weak specification of intended behavior, and that stronger, pattern-based correctness specifications can both validate and guide repair (Al-Bataineh et al., 2021).

The same concern appears in compilation repair. CCrepairBench defines compilation-error repair as producing code that must both compile and preserve the original intent rather than “cheating” via trivial edits such as deleting broken code. Its two-stage evaluation environment therefore treats compilation as necessary, not sufficient, and adds an LLM-as-a-Judge that classifies repairs as “Genuine Fix,” “Trivial Deletion,” “Excessive Modification,” or “Invalid Fix” (Sun et al., 19 Sep 2025).

Test-driven APR has been recalibrated in related ways. Repair-R1 reverses the usual “repair first, then validate with tests” order by requiring the model to generate discriminative tests before repair, so that repair is conditioned on executable evidence that passes the correct program and fails the buggy one (Hu et al., 30 Jul 2025). SpecTune similarly argues that final pass/fail outcomes are only macro-level signals; its response is to introduce localized postconditions at execution checkpoints and to filter them through consistency and discriminative criteria before using them to guide repair (Le-Anh et al., 13 Apr 2026).

Taken together, these works suggest that calibration-guided repair is best understood as an attempt to close the gap between plausibility and correctness by replacing or augmenting coarse end signals with richer behavioral evidence.

2. Why calibration is needed

The strongest empirical motivation for calibration-guided repair is the mismatch between surface success and actual repair quality. In CCrepairBench, removing or weakening the semantic reward leads to a regime in which the model can achieve very high compilation success while producing mostly trivial deletions; with zero semantic reward, 96% of successful compilations were due to deletion. This result is used to show that compile-only feedback is vulnerable to reward hacking and that semantic guidance is required to steer repair toward genuine fixes (Sun et al., 19 Sep 2025).

Test-based APR exhibits an analogous pathology. The static-analysis integration paper evaluates plausible patches produced by GenProg and SCRepair on integer-overflow and termination benchmarks and reports that none of the plausible patches were actually correct under the stronger specifications proposed there. For integer-overflow repair, the paper formalizes the desired property as

$${Spec}_{IO} = \forall_{e_s \in {split}(e')} (isOverflow(e_s) = {false}) \land e' \equiv e,$$

so the patch must remove overflow risk while preserving semantic equivalence. For termination bugs it uses a composite specification requiring both termination and preservation of intended behavior, again making explicit that “passes tests” is not an adequate definition of correctness (Al-Bataineh et al., 2021).

The Art of Repair reinforces this point from a different angle. Under a 10-patch budget, it defines a patch as plausible if it compiles successfully and passes all tests, yet its manual inspection of 3,298 plausible patches found 3,167 confirmed correct and 131 overfitting to the tests, implying about 4.0% overfitting among the inspected plausible patches. This result does not negate test-based validation, but it does show that plausibility remains an imperfect proxy for correctness (Ruiz et al., 5 May 2025).

A related issue is that automatically generated guidance can itself be unreliable. SpecTune reports that, with $(\theta,\gamma)=(0.80,0.80)$ on DeepSeek-V3.1, 43.39% of generated postconditions fall in a non-consistent region, 45.05% in a trivial region, and only 11.56% in an acceptable region. The need for calibration therefore applies not only to patch candidates but also to the intermediate signals used to guide them (Le-Anh et al., 13 Apr 2026).

3. Runtime- and feedback-centered repair

One major lineage of calibration-guided repair is debugger-centered APR. ROSE operates from a debugger stopping point and a short symptom description rather than from a full test suite. The symptom can be an unexpected exception, an assertion failure, a line that should not be executed, or a variable that should not hold a certain value. Fault localization uses a lightweight dynamic-slice-like analysis with FAIT flow analysis, while validation uses SEEDE to reconstruct or partially reexecute the failing situation and compare repaired and original traces. The validation score blends a problem-specific score with a closeness term, returning score * 0.95 + closeness * 0.05, and candidate repairs are ranked by semantic priority together with suggester-specific syntactic priority. On QuixBugs, ROSE suggested correct repairs for 17 of 40 bugs; on the selected Defects4J subset, it suggested correct repairs for 16 of 32 bugs, with median times of about 5 seconds and about 29 seconds respectively (Reiss et al., 2022).

PracAPR generalizes the same debugger-centered idea into an interactive IDE repair architecture. It assumes that the program is already suspended at a location where a problem is observed and that the developer can provide a problem specification such as “an exception is unexpected,” “a line should not be executed,” or “a variable should not hold a certain value.” Localization is performed by abstract-interpretation-based flow analysis that computes a partial backward slice from the symptom, runtime values, and current stack. For local repair, it augments the prompt with buggy location and context, failing input and output, coverage, key program states, promising patterns and fix ingredients, and user guidance. For multi-location bugs, it derives 8 partial patch relationships—DU, OA, RIF, DIF, EOH, SU, ONPF, and FU—and maps them to iterative repair, simultaneous repair, local repair, or pattern-based methods. The ROSE precursor included the correct repair location for 89% of the bugs tested; its validation gave a top-5 rank for all correct repairs; and ROSE-based repair could fix 36/40 QuixBugs and 37/60 Defects4J bugs in seconds, with debugging time reduced by about 16.5% in a user study (Xin et al., 2024).

RESTORE introduces a different but related feedback loop: failed patch validation is not discarded but reused as localization evidence. Built on JAID, it performs partial validation on failing tests first, treats candidate patches as higher-order mutants, and uses mutation-based suspiciousness to recalibrate the ranking of repair locations. This retrospective fault localization is motivated by the observation that validation can consume 92.8% of JAID’s runtime in some settings. On Defects4J, RESTORE reports 97 faults with at least one valid fix and 41 faults with at least one correct fix, together with 3.1× speedup in total runtime, 3.4× speedup in time to first valid fix, 2.3× speedup in time to first correct fix, 57% fewer candidates checked until first valid fix, and 36% fewer until first correct fix (Xu et al., 2019).

These systems illustrate a common principle: repair becomes more targeted when it is anchored to the observed failure context, whether by a debugger state, a symptom description, or the behavior of partially successful mutants.

4. Learning calibrated repair signals

The most explicit learning-based formulation appears in CCrepairBench. Its reinforcement-learning environment uses a deterministic GCC compilation check followed by an LLM-as-a-Judge whose reliability was meta-evaluated against a panel of five human software experts on 100 sampled cases using Macro-F1 and inter-rater reliability-style comparisons. The appendix reports an expert score of 0.592 versus judge score of 0.602, and this is used to justify the judge as a scalable proxy for human semantic assessment. The terminal reward is

$R = S_{\text{judge}} + S_{\text{compile}},$

with

$$S_{\text{judge}} = \begin{cases} 0.5 & \text{if classification is ``Genuine Fix''} \ 0 & \text{otherwise} \end{cases}$$

and

$$S_{\text{compile}} = \begin{cases} 0.5 & \text{if } S_{\text{judge}} > 0 \text{ and code compiles} \ 0 & \text{otherwise}. \end{cases}$$

The maximum reward is therefore $1.0$, but only when a patch is both semantically judged correct and compilable. The policy objective is

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ R_T(\tau) \right],$$

optimized with GRPO/PPO-style policy updates. Experimentally, the RL-trained Qwen2.5-1.5B-Instruct model gains roughly 20 percentage points absolute on both Compilation Success Rate and Genuine Fix Rate over its base version and performs comparably to Qwen2.5-14B-Instruct; the same RL-trained models also improve over base and SFT variants on MBPP and HumanEval (Sun et al., 19 Sep 2025).

Repair-R1 calibrates repair through test generation rather than post hoc semantic judgment. It uses GRPO to jointly optimize test generation and bug repair, with rule-based rewards for format, code repair, and test generation. A generated test is valid iff it passes the ground-truth code $G$ and fails the buggy code $B$,

$V_{t} = f_t(t, G) \cdot (1-f_t(t, B)).$

The pipeline becomes buggy code $(\theta,\gamma)=(0.80,0.80)$0 test generation $(\theta,\gamma)=(0.80,0.80)$1 patch generation conditioned on tests $(\theta,\gamma)=(0.80,0.80)$2 validation. Across four benchmarks—HumanEval, MBPP, CodeForces, and CodeContests—the paper reports improvements over vanilla models of +2.68\% to +48.29\% in repair success rate, +16.38\% to +53.28\% in test generation success rate, and +0.78\% to +53.96\% in test coverage. Joint optimization with RL-Both outperforms the single-objective variants, and RL-Both improves over RL-Repair by 0.24\% to 6.25\% on repair in 11 out of 12 settings (Hu et al., 30 Jul 2025).

SpecTune introduces a different learning-era calibration mechanism based on intermediate behavioral signals. It generates checkpoints $(\theta,\gamma)=(0.80,0.80)$3, candidate postconditions $(\theta,\gamma)=(0.80,0.80)$4, and then filters these postconditions using two execution-derived quantities. The specification validation signal

$(\theta,\gamma)=(0.80,0.80)$5

measures how often a postcondition holds on passing executions that reach checkpoint $(\theta,\gamma)=(0.80,0.80)$6, while the discriminative signal

$(\theta,\gamma)=(0.80,0.80)$7

measures how often the same postcondition is still satisfied by failing executions at that checkpoint. The retained specification set is

$(\theta,\gamma)=(0.80,0.80)$8

On a synthetic APR dataset derived from LiveCodeBench, SpecTune improves Pass@1 from 72.78 to 76.82 for Kimi-K2-Instruct and from 81.90 to 87.73 for DeepSeek-V3.1; it also improves ChatRepair and REx. In ablations on DeepSeek-V3.1, removing $(\theta,\gamma)=(0.80,0.80)$9 reduces Pass@1 to 82.97, removing $R = S_{\text{judge}} + S_{\text{compile}},$0 reduces it to 84.86, and removing both reduces it to 82.04, indicating that inconsistent and trivial specifications are distinct sources of error (Le-Anh et al., 13 Apr 2026).

The Art of Repair studies a further calibration dimension: how a limited patch budget should be allocated across breadth and depth. Under the constraint

$R = S_{\text{judge}} + S_{\text{compile}},$1

it compares seven strategies ranging from A (10×1) to G (1×10). Base models benefit strongly from iteration, especially on the more complex Defects4J benchmark, whereas fine-tuned models often perform best with more outputs early and fewer iterations. The paper’s broader implication is that feedback-guided repair must calibrate not only which signals are used, but also how repair effort is distributed across rounds (Ruiz et al., 5 May 2025).

5. Static, formal, and domain-specific calibration

Calibration-guided repair is not limited to LLM reward design. The static-analysis integration paper treats static bug detection as a source of formalized repair constraints rather than as an external checker. For arithmetic bugs, it synthesizes overflow-detection rules into a repair specification; for termination bugs, it uses composite validation with test cases and provers such as AProVE and 2LS. It further introduces control-variable analysis, monotonic loop statements, conditional mutation rules, slicing to a reduced program $R = S_{\text{judge}} + S_{\text{compile}},$2, and a search process that narrows the candidate patch space and biases search order. In its initial study, 2LS proved termination for almost 80% of examined loop programs, AProVE for about 37%, and together they proved about 84%; around 98% of benchmark loops were monotonic across the combined suites, supporting the monotonic mutation rules (Al-Bataineh et al., 2021).

In formal specification repair, AuRUS extends calibration from program behavior to semantic proximity between specifications. It repairs unrealisable LTL specifications with a genetic algorithm over candidate repairs $R = S_{\text{judge}} + S_{\text{compile}},$3, using the fitness

$R = S_{\text{judge}} + S_{\text{compile}},$4

where $R = S_{\text{judge}} + S_{\text{compile}},$5 rewards satisfiable and realisable specifications, $R = S_{\text{judge}} + S_{\text{compile}},$6 measures subformula overlap, and $R = S_{\text{judge}} + S_{\text{compile}},$7 measures behavioral overlap via bounded model counting. Because exact LTL model counting is expensive, the paper approximates counts through an automaton transfer matrix and computes accepted paths of length $R = S_{\text{judge}} + S_{\text{compile}},$8 as

$R = S_{\text{judge}} + S_{\text{compile}},$9

Across 26 unrealisable specifications, AuRUS succeeded in generating satisfiable and realisable repairs in 100% of runs. Compared with a random generator using the same mutation operator, it produced on average about 23× more repairs, while the approximate model-counting method matched the exact ranking in 9/10 random formula sets and misclassified only 2/50 formulas in the one failing set (Brizzio et al., 2021).

In vulnerability repair, VulKey calibrates generation with structured security knowledge rather than generic prompts. It organizes historical repair knowledge into a three-level hierarchy of CWE type, syntactic action, and semantic key element, producing repair patterns of the form $$S_{\text{judge}} = \begin{cases} 0.5 & \text{if classification is ``Genuine Fix''} \ 0 & \text{otherwise} \end{cases}$$0. A CodeT5p matcher predicts top-10 candidate patterns from $$S_{\text{judge}} = \begin{cases} 0.5 & \text{if classification is ``Genuine Fix''} \ 0 & \text{otherwise} \end{cases}$$1, and a repair model conditions generation on the selected action and key element. On PrimeVul, VulKey achieves 31.5% repair accuracy, surpassing the best baseline by 7.6 percentage points; the paper reports 23.9% for the best fine-tuned StarCoder-based baseline, 22.8% for NTR, 8.7% for VulMaster, 10.6% for GPT-5, and 10.1% for GPT-4.1. In ablations, removing CWE type lowers Exact Match to 26.4%, removing key element to 27.6%, and removing action to 27.1%, indicating that all three layers contribute complementary guidance (Li et al., 3 May 2026).

6. Evaluation practice, misconceptions, and unresolved issues

A central misconception is that calibration-guided repair is synonymous with statistical calibration of model confidence. Several papers use “calibration-guided” or “calibration-like” to describe alignment of repair with stronger behavioral evidence rather than probability calibration. ROSE explicitly fits this pattern: its guidance comes from the developer’s symptom description, debugger state, and execution comparison, and the paper notes that it is not calibration in the machine-learning sense of probability calibration (Reiss et al., 2022).

A second misconception is that better calibration eliminates the need for careful evaluation. CCrepairBench validates its judge against human experts, yet that judge remains a scalable proxy rather than a ground-truth oracle. SpecTune shows that most LLM-generated postconditions are either inconsistent or trivial before filtering. VulKey shows that top-1 matched patterns are much less reliable than top-10 candidate sets, with top-1 action correctness of 43.0% and key-element correctness of 19.0%, versus top-10 action correctness of 80.0%, key-element correctness of 58.0%, and at least one actionable hit in 93% of cases (Sun et al., 19 Sep 2025, Le-Anh et al., 13 Apr 2026, Li et al., 3 May 2026).

A third misconception is that richer signals always imply heavier or less practical workflows. Some evidence points in the opposite direction. ROSE reports quick repair suggestions during debugging, often in seconds. RESTORE improves both effectiveness and efficiency by reusing partial validation for localization. SpecTune reports modest token cost increases, about \$S_{\text{judge}} = \begin{cases} 0.5 & \text{if classification is ``Genuine Fix''} \ 0 & \text{otherwise} \end{cases}$20.0128 for Kimi-K2-Instruct, with somewhat higher costs in refinement settings (Reiss et al., 2022, Xu et al., 2019, Le-Anh et al., 13 Apr 2026).

The unresolved issues are correspondingly clear in the literature. PracAPR remains an envisioned system in several respects, particularly for full multi-location implementation. Static-analysis-guided repair can be limited by prover cost and unknown results on complex loops. Repair-R1 assumes oracle tests for evaluation and reward computation. SpecTune is evaluated on a synthetic dataset designed to reduce contamination and leakage, which the paper notes may not fully represent real-world bugs, languages, or large systems. VulKey’s failure analysis highlights missing cross-function context, user-defined APIs, exact string or constant matching, and complex logical conditions as major remaining barriers (Xin et al., 2024, Al-Bataineh et al., 2021, Hu et al., 30 Jul 2025, Le-Anh et al., 13 Apr 2026, Li et al., 3 May 2026).

The overall trajectory nevertheless indicates a coherent research program. Calibration-guided automated repair increasingly replaces single, coarse success criteria with multi-source evidence: semantic reward gating, pre-repair test discrimination, debugger-centered symptom tracking, static bug patterns, model-counting estimates, intermediate postconditions, or structured repair knowledge. A plausible implication is that future repair systems will be judged less by raw plausible-patch counts and more by how well their training and inference signals are aligned with non-trivial correctness.