RePaCA: Reasoning Patch Correctness Assessment

• RePaCA is a static automated patch correctness assessment technique that combines chain-of-thought reasoning with reinforcement learning finetuning.
• It employs a structured multi-part prompt to generate human-readable analysis and binary verdicts, enhancing transparency in APR patch evaluation.
• RePaCA outperforms prior methods by achieving up to 83.1% accuracy and strong generalization on diverse datasets.

RePaCA (Reasoning Patch Correctness Assessment) is a static automated patch correctness assessment (APCA) technique that leverages LLMs specialized in reasoning tasks to address fundamental limitations in assessing the correctness of automated program repair (APR) patches without dynamic execution. RePaCA introduces a chain-of-thought (CoT) driven framework, accompanied by reinforcement learning-based finetuning, to classify candidate code patches as "correct" or "overfitting," advancing the state of the art in accuracy, generalization, and explainability in static APCA (Fuster-Pena et al., 30 Jul 2025).

1. Problem Definition and Objectives

Automated Program Repair (APR) seeks to generate fixes for software bugs without human intervention. A significant challenge in APR is the tendency to produce plausible patches that pass a given test suite but do not truly address the underlying defect—these are termed overfitting patches. Automated Patch Correctness Assessment (APCA) focuses on discriminating between correct and overfitting patches.

Static APCA methods, by definition, only use source code—the buggy version $c_{\text{buggy}}$ and patched version $c_{\text{patched}}$—without relying on execution traces or external information. The task requires high reliability (accuracy), flexibility (generalization to new datasets and patch generators), and transparency (explainable reasoning about patch validity).

2. RePaCA Model Architecture and Prompting

RePaCA operates by prompting an LLM with both the buggy and patched code snippets, supplemented by a carefully designed multi-part prompt template:

• Brief task description.
• Explicit analysis requirements, including code comparison and evaluation of root-cause addressing.
• Output format specification: tag-delimited chain-of-thought (> ...) for the internal reasoning trace and final answer in <answer> correct | overfitting </answer>.
• An in-context one-shot example.
• The concrete buggy and patched code, ending with "Begin your reasoning."

The model’s CoT generation within <think> systematically covers:

1. Describing the original (buggy) behavior.
2. Summarizing the change introduced by the patch.
3. Judging whether the patch addresses the root cause or simply circumvents specific test cases.
4. Citing any missing edge cases, then outputting the binary verdict.

No auxiliary features—such as runtime traces, external specifications, or handcrafted code analysis—are required.

3. Group Relative Policy Optimization (GRPO) for LLM Fine-tuning

To enhance reasoning quality and APCA-specific accuracy, RePaCA finetunes the LLM with Group Relative Policy Optimization (GRPO), a policy-gradient reinforcement learning (RL) paradigm. GRPO optimizes the joint generation of reasoning traces and classification labels.

Formulation

• The LLM defines a policy $\pi_\theta(y\mid x)$ over outputs $y$ (comprising both CoT and label) conditioned on input $x$ (prompt and code).
• For each prompt, $K$ samples $y_1,\dots,y_K$ are generated with corresponding rewards $r_k$.
• The total reward $r_k = R_f(y_k) + R_a(y_k)$ combines:
  • Format reward $R_f$ ($c_{\text{patched}}$0): structure compliance (+0.5 for <think>, +0.25 for <answer>, +0.25 for valid label).
  • Accuracy reward $c_{\text{patched}}$1 ($c_{\text{patched}}$2): 2 for correct "overfitting", 1 for correct "correct", 0 for incorrect.
• The group-average reward $c_{\text{patched}}$3 establishes the baseline, and the advantage for each sample is $c_{\text{patched}}$4.
• The GRPO gradient is:

$c_{\text{patched}}$5

Optionally using clipped importance-sampling ratios for stability.

4. Training Regimen and Evaluation Datasets

RePaCA is initialized from the Qwen2.5-Coder LLM (3B parameters). Training involves:

• Two primary datasets:
  • Small, Defects4J-derived: 1,183 balanced patches.
  • Large, heterogeneous: 49,694 patches (ManySStuBs4J correct, RepairThemAll overfitting).
• Five-fold cross-validation on the small set for principal evaluation.
• RL hyperparameters:
  • 20 epochs, learning rate $c_{\text{patched}}$6, batch size 8 prompts, 8 samples per prompt, gradient accumulation steps = 2.
  • Generation temperature 0.6, max token length 2,048.
  • Hardware: NVIDIA H100 NVL, TRL + PyTorch + Unsloth.
• Metrics: accuracy, precision, recall, F1, and AUC (with overfitting as positive class).

Quantitative Performance

Model	Acc.	Precision	Recall	F1	AUC
ODS	62.3%	–	–	69.1%	62.5%
Quatrain	69.4%	–	–	77.8%	64.5%
CACHE	75.4%	79.5%	76.5%	78.0%	80.3%
APPT	79.7%	80.8%	83.2%	81.8%	82.5%
RePaCA	83.1%	84.0%	85.7%	84.8%	82.8%

Generalization: When trained on the small dataset and tested on the large set, RePaCA sustains 72.7% accuracy and 75.4% F1, substantially outperforming APPT (60.5% and 71.4%).

5. Explainability via Structured Chain-of-Thought

Unlike prior APCA frameworks, RePaCA mandates self-interpretable outputs through explicit CoT traces:

• Overfitting example: The model identifies a code modification that inserts a specific guard (e.g., if (scale.length!=maxstep)…) but fails to resolve the true bug (e.g., division-by-zero). The CoT reveals the superficiality of the patch and correctly classifies it as overfitting:

$c_{\text{patched}}$7

• Correct example: A patch that introduces a NaN check in Complex.add(...), addressing all undefined behaviors for NaN inputs, is explained as a fix that generalizes:

$c_{\text{patched}}$8

This transparency is not present in static discriminative baselines and enhances developer trust and patch review.

6. Comparative Analysis and Significance

Quantitatively, RePaCA surpasses previous static APCA methods—including ODS, Quatrain, CACHE, and APPT—on all core metrics in five-fold cross-validation, and demonstrates markedly superior generalization on cross-dataset transfer. Qualitatively, it is the only approach explicitly producing human-readable justification alongside the verdict. The model’s CoT not only increases explainability but also potentially improves reliability by structuring and surfacing error analysis that could otherwise be opaque.

7. Limitations and Directions for Further Research

RePaCA exhibits some limitations:

• Occasional reasoning failures, e.g., hallucinated explanations or mischaracterization of code changes.
• Approximately 1% malformed output tags due to generative errors.
• Computationally intensive RL training, given the need for multiple generations per prompt.

Possible improvements include dataset augmentation with richer static and dynamic contexts (e.g., ASTs, bug reports, partial execution traces), incorporation of an explicit “reasoning quality” reward for CoT coherence, and adoption of larger or more reasoning-specialized LLMs. Integration of RePaCA as a corrective filter in end-to-end APR pipelines and exploration of hybrid static+dynamic APCA are suggested as future avenues (Fuster-Pena et al., 30 Jul 2025).