Calibration-Guided Repair

• Calibration-Guided Repair is a framework that uses calibrated decision signals to guide selective, iterative repair across diverse technical domains.
• It couples a diagnostic signal with a repair operator to direct, monitor, and weight corrective actions, enhancing process reliability.
• Empirical studies show its strongest impact as a triage and weighting mechanism, despite limitations as a full standalone autonomous repair solution.

Calibration-Guided Repair denotes a family of repair strategies in which a calibrated or calibration-like signal is used to decide whether repair should be attempted, where corrective effort should be concentrated, or how strongly repaired outputs should be trusted in subsequent optimization. Across recent work, the term does not name a single canonical algorithm; rather, it spans several technical regimes: post-hoc remediation of LLM-generated insecure code (Siddiq et al., 30 Jun 2026), Bayesian-workflow-guided repair of probabilistic programs (Xu et al., 30 Jun 2026), confidence-weighted iterative data repair (Chen et al., 16 Jun 2026), calibration-based pruning allocation under lightweight post-repair correction (Zhan et al., 25 May 2026), and user-level bias compensation in quantum annealers (Perdomo-Ortiz et al., 2015). A recurring theme is that calibration is most reliable as a triage, allocation, or weighting mechanism, whereas successful repair still depends on the underlying system’s capacity to execute the required structural transformation.

1. Conceptual scope and historical development

In its most general form, calibration-guided repair couples two components: a diagnostic signal intended to align with true error or recoverability, and a repair operator that consumes that signal to select, prioritize, or modulate corrective actions. The diagnostic may be a probability estimate, a calibrated residual ratio, a posterior-predictive misfit score, or a reliability-weighted pseudo-labeling signal. The repair operator may be iterative code regeneration, channelwise variance correction, compensating control offsets, or weighted learning updates.

An early hardware instance appears in "Determination and correction of persistent biases in quantum annealers" (Perdomo-Ortiz et al., 2015). That work studies persistent, systematic biases between user-specified and realized programmable parameters in D-Wave Two devices and uses thermal-response experiments to infer those biases and then compensate for them in future programming. The repair rule is explicitly iterative: $$h_i^{(p,k)} = h_i^{(p,0)} - \sum_{k'=1}^{k} h_i^{(b,k')}, \qquad J_{ij}^{(p,k)} = J_{ij}^{(p,0)} - \sum_{k'=1}^{k} J_{ij}^{(b,k')}.$$ The paper is not phrased in modern ML calibration terminology, but it is a clear instance in which a statistically estimated calibration signal is fed back into a repair loop (Perdomo-Ortiz et al., 2015).

Recent 2026 work makes the notion explicit. In LLM code security, calibration-guided repair is framed as a closed-loop remediation problem in which calibrated security confidence determines which generations enter repair (Siddiq et al., 30 Jun 2026). In probabilistic programming, the repair signal is a calibration oracle built from posterior predictive checks, simulation-based calibration, sampler diagnostics, and held-out predictive density (Xu et al., 30 Jun 2026). In data repair, LasRepair++ uses column-calibrated model confidence to down-weight unreliable repaired rows during iterative corrector updates (Chen et al., 16 Jun 2026). In pruning, Relative Repairability measures the fraction of layerwise pruning damage that remains after a fixed calibration-based repair operator, and this diagnostic guides sparsity allocation (Zhan et al., 25 May 2026).

This distribution of use suggests that "Calibration-Guided Repair" is best understood as an umbrella concept rather than a narrowly delimited method. A plausible implication is that the common object is not the repair primitive itself, but the decision layer that mediates repair through a reliability-sensitive signal.

2. Formalizations of calibration and repair guidance

The most explicit generic definition in the provided literature appears in code security. "An Empirical Study of Security Calibration in LLMs for Code" (Siddiq et al., 30 Jun 2026) adopts the standard calibration condition

$P(Y' = Y \mid P(Y' \mid X) = p) = p,$

meaning that predictions made with confidence $p$ should be correct about $p$ of the time. In that setting, calibration is applied to different labels rather than defined by different formulas. Functional labels are $y^{\mathrm{func}} \in \{0,1\}$, vulnerability labels are $y^{\mathrm{vuln}} \in \{0,1\}$, and the final security label is

$$y^{\mathrm{sec}} = y^{\mathrm{func}} \wedge \neg y^{\mathrm{vuln}}.$$

The same calibration machinery yields $ECE^s$, $ECE^f$, and related quantities, with comparison metric

$\Delta m = m^{\mathrm{s}} - m^{\mathrm{f}}.$

The paper uses Expected Calibration Error, Brier score, Overconfidence Gap, and False Trust rate, where False Trust captures the operationally dangerous case of high-confidence wrong outputs, especially high-confidence insecure code (Siddiq et al., 30 Jun 2026).

Other domains define guidance signals differently. In Relative Repairability, the calibrated object is not a correctness probability but a normalized residual-distortion ratio: $P(Y' = Y \mid P(Y' \mid X) = p) = p,$0 where $P(Y' = Y \mid P(Y' \mid X) = p) = p,$1 is single-layer pruning distortion on unlabeled calibration data and $P(Y' = Y \mid P(Y' \mid X) = p) = p,$2 is the residual after channelwise variance-matching repair plus BatchNorm recalibration (Zhan et al., 25 May 2026). A low $P(Y' = Y \mid P(Y' \mid X) = p) = p,$3 means that most local damage is repaired; a high value means the damage is repair-resistant.

In LasRepair++, calibration is column-specific and operationalized through first-token temperature-scaled confidences, token-wise thresholds within each target column, and row-level aggregation. Cell confidence is defined as

$P(Y' = Y \mid P(Y' \mid X) = p) = p,$4

with row reliability

$P(Y' = Y \mid P(Y' \mid X) = p) = p,$5

These weights then modulate the corrector update (Chen et al., 16 Jun 2026).

In probabilistic programming, calibration is not reduced to a single scalar. The oracle aggregates executability, $P(Y' = Y \mid P(Y' \mid X) = p) = p,$6, divergence count, posterior predictive tail probabilities, and held-out predictive density: $P(Y' = Y \mid P(Y' \mid X) = p) = p,$7 with $P(Y' = Y \mid P(Y' \mid X) = p) = p,$8 and $P(Y' = Y \mid P(Y' \mid X) = p) = p,$9 nats (Xu et al., 30 Jun 2026).

These formulations differ substantially, but they share a common functional role: the signal is not the repair itself, but a calibrated decision variable controlling repair selection, allocation, or trust.

3. LLM-generated code: triage, iterative repair, and failure modes

The most direct empirical study of calibration-guided automated repair in code security is "An Empirical Study of Security Calibration in LLMs for Code" (Siddiq et al., 30 Jun 2026). The paper studies GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next across temperatures $p$0, using SALLM for self-contained Python security tasks and AICGSecEval for repository-level calibration analysis. Calibration-guided repair itself is evaluated only on SALLM, which contains 100 self-contained Python security tasks covering 45 CWEs and provides functional unit tests together with exploit tests (Siddiq et al., 30 Jun 2026).

The repair pipeline is explicit. SALLM is split by task into 50 training tasks and 50 held-out test tasks using seed=42, yielding a 6,000-sample held-out repair set: $p$1 Post-hoc calibrators are fit on the train split. Functional confidence uses Isotonic Regression; security confidence uses a Logistic Regression pipeline trained on the logit of $p$2 augmented with 16 static sink-pattern counts, including occurrences of pickle, eval, and os.system. The repair loop then proceeds in three stages. In S1, filtering, any sample with calibrated security confidence below $p$3 is flagged for repair. In S2, repair, the flagged sample is resubmitted to the same model with the original task description and its own insecure output, plus a prompt instructing it to fix vulnerabilities while preserving functionality; the loop runs for up to three iterations and stops early if repaired confidence reaches $p$4. In S3, re-verification, each repaired sample is rerun in the same Docker harness and both functional and vulnerability outcomes are recomputed (Siddiq et al., 30 Jun 2026).

The empirical results are uniformly skeptical. On the 6,000-sample held-out SALLM split, GPT-4o-mini drops from 29.49% functional accuracy and 6.88% overall secure rate to 12.35% and 3.73% after calibration-guided repair; ECE rises from 0.469 to 0.569, False Trust falls only from 95.79% to 89.68%, and repair success rate is 1.02% (Siddiq et al., 30 Jun 2026). Gemini-2.0-Flash decreases from 59.00% to 54.72% functional accuracy while overall secure rate remains 15.47%; ECE rises from 0.195 to 0.487, False Trust increases from 61.83% to 82.88%, and repair success on vulnerable samples is 0.00% (Siddiq et al., 30 Jun 2026). Qwen3-Coder-Next falls from 15.41% to 6.62% functional accuracy and from 5.78% to 2.40% overall secure rate; ECE worsens from 0.216 to 0.740, False Trust remains 100.00% to 97.63%, and repair success rate is 2.39% but is accompanied by 69.78% functional breakage (Siddiq et al., 30 Jun 2026).

The paper’s SR/CE split clarifies why. Sink-Replacement vulnerabilities require replacing an insecure API or sink with a secure alternative, such as pickle -> json or removing eval or os.system. Constraint-Enforcement vulnerabilities can be addressed by adding validation or sanitization logic while leaving the sink in place. Under the baseline calibration-guided loop, SR repair success is 0.00% for all three models. CE repair success is nonzero but small—1.40% for GPT, 0.00% for Gemini, 3.20% for Qwen—and often paired with severe breakage (Siddiq et al., 30 Jun 2026). The authors’ interpretation is explicit: the bottleneck lies in the repair mechanism, not in the calibration signal itself (Siddiq et al., 30 Jun 2026).

The broader deployment implication is equally explicit. Functional calibration is consistently worse than security calibration, apparently because functional correctness depends on execution brittleness, imports, framework setup, and environment assumptions, whereas many security flaws are visible from static code patterns (Siddiq et al., 30 Jun 2026). This suggests that calibration can identify risky generations, but does not turn current LLMs into reliable automatic security repair systems.

4. Probabilistic programs: calibration as the verifier and repair oracle

"Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by LLMs" (Xu et al., 30 Jun 2026) redefines correctness for probabilistic code. In this setting, a program can compile, run, return finite posterior samples, and pass every unit test while still being statistically wrong. The paper therefore treats the Bayesian workflow itself as the repair oracle: posterior predictive checks, simulation-based calibration, sampler diagnostics, and held-out predictive density (Xu et al., 30 Jun 2026).

The main detection benchmark covers 14 misspecification types across 10 model families and 200 instances. The reference-based calibration oracle achieves AUC 0.97 and detects 88% of bugs at 2% false positive rate when given the correct reference program as an upper bound. A fully reference-free version reaches 62%--78%, whereas the unit-test oracle detects 0% (Xu et al., 30 Jun 2026). Component ablations show that held-out lppd only detects 59% of bugs with 0% false positives, PPC only detects 56% with 0% false positives, sampler diagnostics only 6% with 2% false positives, and SBC only 5% with 5% false positives (Xu et al., 30 Jun 2026).

The repair loop is an iterative writer-oracle-writer cycle. Starting from a writer $p$5, the system fits a candidate program $p$6, computes $p$7, and, if the oracle fails, passes a natural-language interpretation of the diagnostics back to the model for revision, up to budget $p$8 (Xu et al., 30 Jun 2026). Three feedback regimes are compared: none, test, and diag. The most striking result is that calibration feedback significantly outperforms unit-test feedback, while unit-test feedback is itself significantly worse than no feedback at all (Xu et al., 30 Jun 2026).

On code-invisible injected bugs, pooled fix rates are 46% for diag, 36% for none, and 27% for test, with paired exact McNemar tests showing diag vs none $p$9, diag vs test $p$0, and none vs test $p$1 over $p$2 paired trajectories (Xu et al., 30 Jun 2026). Model-specific improvements include GPT-5.1 from 33% to 92%, Claude Sonnet 4.6 from 75% to 100%, and Gemini-3.1-pro from 75% to 100% under diagnostic feedback (Xu et al., 30 Jun 2026).

The from-scratch generation setting generalizes the finding. Across nine writer models, 15%--47% of runnable programs are statistically misspecified, and unit tests catch none (Xu et al., 30 Jun 2026). On misspecified programs from capable writers, pooled repair success is 84% for calibration diagnostics, 71% for LLM-as-judge review, 63% for a Bayesian checklist, 46% for data summary self-debug, 46% for none, and 36% for test, with diag significantly outperforming every baseline (Xu et al., 30 Jun 2026).

This body of results establishes a strong domain-specific lesson: when the object being repaired is itself probabilistic, calibration is not merely a confidence layer on top of ordinary repair. It becomes the correctness criterion that determines what counts as a successful repair.

5. Confidence-weighted and calibration-conditioned repair outside code generation

"Collaborative Large and Small LLMs for Accurate and Scalable Data Repair" (Chen et al., 16 Jun 2026) studies relational data repair under detected error cells $p$3. LasRepair uses an LLM instructor to select a global repair context and an SLM corrector to repair erroneous cells. LasRepair+ turns this into an EM-style alternating optimization, and LasRepair++ adds the calibration-guided step: estimate confidence from top-$p$4 first-token logits, temperature-scale them, calibrate within each target column using token-specific thresholds, aggregate accepted confidences to row reliability weights, and use those weights to down-weight unreliable repaired rows in the next corrector update (Chen et al., 16 Jun 2026). Theorem 2 states

$p$5

under a heteroscedastic-noise model for row gradients. Empirically, LasRepair++ achieves an average F1-score improvement of 18.1% over the strongest baseline, and removing confidence-based weighting reduces average F1 from 0.8104 to 0.7314 (Chen et al., 16 Jun 2026).

"Relative Repairability: A Calibration-Based Diagnostic for High-Sparsity Post-Pruning Allocation" (Zhan et al., 25 May 2026) moves the repair signal upstream into pruning allocation. The fixed repair operator is channelwise variance-matching correction followed by BatchNorm recalibration, abbreviated CR+BN. RR is measured using only 128 unlabeled calibration images and guides a shared greedy allocator over sparsity grid $p$6 (Zhan et al., 25 May 2026). The diagnostic is not universally dominant, but it is strongest near an architecture-dependent recoverability transition. On CIFAR100 ResNet18 at 95% sparsity, RR improves over ERK from 9.38 to 11.86, and the fine-grained sweep shows improvements over ERK of $p$7 at 94.0%, $p$8 at 94.5%, $p$9 at 95.0%, and $y^{\mathrm{func}} \in \{0,1\}$0 at 95.5% (Zhan et al., 25 May 2026). The broader claim is that high-sparsity pruning should allocate not only retained weights, but also repairable damage (Zhan et al., 25 May 2026).

The quantum-annealer study (Perdomo-Ortiz et al., 2015) provides a physically grounded version of the same pattern. By programming sufficiently small $y^{\mathrm{func}} \in \{0,1\}$1 and $y^{\mathrm{func}} \in \{0,1\}$2 values, measuring output statistics, fitting a simple thermal model, and compensating inferred offsets, the authors narrow the post-correction bias distribution and improve benchmark performance. On random Ising instances, field correction outperforms the uncorrected device on fractions $y^{\mathrm{func}} \in \{0,1\}$3 under the greedy metric and $y^{\mathrm{func}} \in \{0,1\}$4 under elite mean for $y^{\mathrm{func}} \in \{0,1\}$5, respectively (Perdomo-Ortiz et al., 2015).

These systems differ in task, model class, and calibration object, yet they share a structural property: calibration affects how repair updates are weighted or allocated, not merely how errors are reported.

6. Related guided-repair paradigms and boundary cases

Several adjacent papers do not perform calibration-guided repair in the strict probabilistic sense, but they help delimit the concept. "Loc2Repair" (Awad et al., 29 Jun 2026) shows that repository-level repair should be decomposed into localization and repair under a shared runtime and evaluation harness. Using three repair backbones on SWE-bench Verified, pooled resolved rate rises from 44.7% for baseline repair to 48.9% and 49.1% with predicted localization, and to 52.4% with gold localization (Awad et al., 29 Jun 2026). The framework does not calibrate localization scores, but it exposes file-level issue localization as an explicit intervention variable. This suggests that calibrated confidence over candidate files could be a natural future policy input (Awad et al., 29 Jun 2026).

"Enhancing Program Repair with Specification Guidance and Intermediate Behavioral Signals" (Le-Anh et al., 13 Apr 2026) introduces SpecTune, in which LLM-generated checkpoint-level postconditions are filtered by two execution-derived signals: $y^{\mathrm{func}} \in \{0,1\}$6, a consistency rate over passing tests, and $y^{\mathrm{func}} \in \{0,1\}$7, a discriminative rate over failing tests. A specification is retained only if

$y^{\mathrm{func}} \in \{0,1\}$8

The paper does not present these as calibrated probabilities, but $y^{\mathrm{func}} \in \{0,1\}$9 functions as an empirical trustworthiness estimate and $y^{\mathrm{vuln}} \in \{0,1\}$0 as an informativeness estimate. Only 11.56% of generated specifications fall into the acceptable region at $y^{\mathrm{vuln}} \in \{0,1\}$1, while 45.05% are trivial and 43.39% non-consistent (Le-Anh et al., 13 Apr 2026). This is a reliability-filtered guidance mechanism rather than calibration in the strict sense.

"Guided Game Level Repair via Explainable AI" (Bazzaz et al., 2024) uses SHAP or Integrated Gradients attribution maps from a solvability classifier to select the top 20% attribution tiles, extract the largest connected component, assign weight 1 inside that region and weight 10 elsewhere, and then run weighted constraint-based repair (Bazzaz et al., 2024). The method improves repair time on harder domains such as Mario and Supercat, but the paper explicitly characterizes the signal as explainability-based localization rather than calibrated uncertainty (Bazzaz et al., 2024).

"Counterexample-Guided Repair of Reinforcement Learning Systems Using Safety Critics" (Boetius et al., 2024) similarly uses a learned safety critic and counterexample states as a guide for policy repair. The guide model is repaired jointly with the policy, but the paper does not discuss calibration, expected calibration error, or uncertainty estimation (Boetius et al., 2024).

These cases mark an important conceptual boundary. Calibration-guided repair, narrowly defined, uses a signal intended to align with empirical correctness, misfit, or repairability. Signal-guided repair more broadly includes attribution maps, safety critics, localization hints, or counterexamples even when they are not calibrated in that sense. The distinction matters because the guarantees, failure modes, and evaluation metrics differ.

7. Limits, misconceptions, and research directions

A central misconception is that better calibration automatically yields better repair. The code-security study rejects this directly: calibration-guided filtering can identify low-confidence risky samples, but iterative repair rarely turns them into secure, working code, and often worsens calibration or breaks functionality (Siddiq et al., 30 Jun 2026). The pruning paper makes the same point in a different form: RR is useful only near a recoverability transition and is explicitly not a universal replacement for ERK or LAMP (Zhan et al., 25 May 2026). In probabilistic programming, the calibration oracle can diagnose a likelihood mismatch or pathological parameterization, but weak models may still fail to synthesize the correct structural fix (Xu et al., 30 Jun 2026).

A second misconception is that calibration is equivalent to generic confidence. Several systems rely on confidence-like signals that are not formally calibrated: repair-engine scores in Guided Data Repair (Yakout et al., 2011), attribution intensities in game-level repair (Bazzaz et al., 2024), or safety critic outputs in RL repair (Boetius et al., 2024). Those are adjacent but not identical. A plausible implication is that future work should distinguish confidence estimation, calibration, and decision-theoretic use of uncertainty more sharply than many current pipelines do.

A third recurring lesson is that benchmark conditions can overstate repair feasibility. In code security, calibration deteriorates sharply from SALLM to AICGSecEval, with average ECE rising from 0.411 to 0.697 for GPT, 0.161 to 0.721 for Gemini, and 0.241 to 0.734 for Qwen, while False Trust rises to 70–83% on average (Siddiq et al., 30 Jun 2026). Repair was not attempted in that harder repository-level setting, so the self-contained repair results are already a best-case scenario (Siddiq et al., 30 Jun 2026). In probabilistic programming, some harder model classes were excluded because NUTS fails even on correct references, and predictive calibration remains distinct from causal or structural correctness (Xu et al., 30 Jun 2026).

Across the literature, the most stable use cases for calibration-guided repair are therefore selective and conditional. Calibration is effective as a triage signal for human review (Siddiq et al., 30 Jun 2026), as a diagnostic oracle for model misspecification (Xu et al., 30 Jun 2026), as a weighting mechanism for pseudo-repair trust (Chen et al., 16 Jun 2026), and as an allocation criterion for where repairable damage should be placed (Zhan et al., 25 May 2026). It is much less reliable as a standalone basis for autonomous end-to-end remediation. The current evidence suggests that calibration can prioritize, filter, and budget repair effort; it does not by itself create the missing edit capability, semantic integration ability, or structural modeling competence required to complete the repair successfully.