Repair-R1: Test-Driven LLM Program Repair

• The paper introduces a paradigm shift by integrating pre-repair test generation to localize faults before proposing patches.
• It leverages joint reinforcement learning to co-optimize discriminative test generation and patch synthesis for improved bugfix success.
• Empirical evaluations across benchmarks show substantial gains in test coverage and patch correctness compared to traditional methods.

Repair-R1: Integrating Pre-Repair Test Generation into LLM-Based Automated Program Repair

The Repair-R1 approach introduces a paradigm shift in Automated Program Repair (APR) by requiring LLMs to explicitly generate discriminative test cases before proposing repairs. This framework, involving joint reinforcement learning for test generation and patch creation, is designed to teach LLMs to localize and understand defects more effectively. Repair-R1 is distinct from prior LLM-based APR systems that treat tests solely as post hoc validators at inference time. Empirical results across several benchmarks indicate substantial gains over baseline and traditional supervised finetuning methods, particularly in test generation capability, test coverage, and ultimate patch correctness (Hu et al., 30 Jul 2025).

1. Motivation and Conceptual Foundation

Conventional LLM-based APR methods are characterized by two main limitations: over-reliance on historical code-patch patterns and neglect of test data during training. Typically, such models attempt to generate repairs by leveraging code knowledge acquired during pre-training or fine-tuning and validate these proposed patches only after inference using test cases. This approach fails to incorporate the diagnostic power of unit tests at training time or to connect the repair process with the specific defective behavior. The Repair-R1 framework addresses these deficiencies by making test generation a first-class, model-internal operation, with the explicit goal of producing tests that pass on ground-truth code but fail on the buggy implementation. This serves as a form of internal fault localization, enriching the model's input space and guiding patch synthesis toward the actual root cause of failures.

2. Architectural Overview and Formalization

Repair-R1 comprises a two-stage, LLM-driven workflow:

1. Discriminative Test Generation. Given a buggy program $b$, the model is tasked with producing a set of candidate test cases $z = \{t_1, \ldots, t_n\}$ such that each $t_i$ passes the ground-truth implementation $G$ and fails on $b$.
2. Patch Generation Conditioned on Tests. The model subsequently generates a patch $p$ using both $b$ and the synthesized tests $z$ as input context.

Formally, the complete model factors as a joint latent-variable model: $$\pi(p, b) = \int \pi(p \mid z, b) \cdot \pi(z \mid b) \, dz$$ with the two phases explicitly separated. The reinforcement learning (RL) objective maximizes expected reward as a sum of (i) format validity, (ii) discriminative test success, and (iii) patch correctness over an augmented oracle test suite.

During RL training, the objective is: $$\max_{\theta} \, \mathbb{E}_{(b, G)} \left[ R_t(b; \theta) + \lambda R_r(b, G; \theta) \right]$$ where $z = \{t_1, \ldots, t_n\}$0 is the test generation reward (fraction of discriminative tests), $z = \{t_1, \ldots, t_n\}$1 is the patch correctness reward (fraction passing oracle tests), and $z = \{t_1, \ldots, t_n\}$2 balances the two.

3. Reinforcement Learning Co-Optimization and Policy Structure

The RL optimization in Repair-R1 employs Group Relative Policy Optimization (GRPO), targeting both sub-tasks jointly. The policy outputs a combined response that consists of test definitions and the patch proposal, with samples drawn via a group policy for computing relative advantages.

Reward Structure

• Format Reward ($z = \{t_1, \ldots, t_n\}$3). Enforces output structure and syntactic validity.
• Test Generation Reward ($z = \{t_1, \ldots, t_n\}$4). Fraction of generated tests where $z = \{t_1, \ldots, t_n\}$5 and $z = \{t_1, \ldots, t_n\}$6.
• Patch Repair Reward ($z = \{t_1, \ldots, t_n\}$7). Average success across an augmented oracle test set.

The total reward for a trajectory is: $z = \{t_1, \ldots, t_n\}$8 The policy update clips the surrogate loss using the KL-divergence to the reference model and maximizes expected rewards over the training batch.

4. Discriminative Test Case Semantics and Criteria

Discriminative tests serve to isolate concrete behaviors that are correct in the ground-truth but fail in the buggy implementation. The model is required to output, for each test case $z = \{t_1, \ldots, t_n\}$9:

• Passing on ground-truth: $t_i$0
• Failing on buggy: $t_i$1

Each test is structured as a JSON-style list (input, expected output, assertion). During training, the test set is augmented and filtered to enforce discrimination and maximize information content for the repair task. These generated tests are provided as explicit context for patch generation, grounding the LLM's repair proposal in specific failing behaviors, rather than in generic or memorized fix patterns.

5. Empirical Evaluation and Quantitative Findings

Repair-R1 has been evaluated using three backbones—Qwen2.5-Coder-1.5B-Instruct, Qwen2.5-Coder-3B-Instruct, and Qwen3-4B-RL—on four benchmarks: HumanEval, MBPP, CodeForces, and CodeContests (Hu et al., 30 Jul 2025). Key metrics and improvements observed include:

Metric	Vanilla Baseline	Repair-R1 (RL-Both)	Improvement
Bugfix Success	Variable	+2.68% – 48.29%	Up to +48.3 ppt
Test Gen Success	Variable	+16.38% – 53.28%	Up to +53.3 ppt
Test Coverage	Variable	+0.78% – 53.96%	Up to +54.0 ppt

Notably, only the RL-Both (joint test+patch) configuration yields significant simultaneous gains on both patch correctness and test generation. RL-Test alone only modestly improves bugfixing; RL-Repair without test co-optimization improves test metrics, but not patch quality to the same degree.

Ablation studies confirm that training solely for repair or test generation leaves significant headroom unexploited, and models trained with purely supervised finetuning suffer from catastrophic forgetting on less-represented benchmarks.

6. Impact, Insights, and Limitations

The “test-before-repair” protocol enforced by Repair-R1 leads to several key findings:

• Explicit Fault Localization. By compelling the model to produce failing/passing discriminative tests, the model's repair operation is effectively grounded in localized behavioral evidence of the bug.
• Improved Patch Validity. The semantic link between generated tests and the subsequent repair produces more targeted and correct patch code as compared to models that operate on patched code alone.
• Alignment with Developer Practice. Repair-R1’s sequencing mirrors real-world development workflows, moving model-based repair pipelines closer to practical adoption.

Primary limitations observed include increased training cost due to RL, occasional reward sparsity (particularly on compound or context-dependent bugs), and the challenge of producing fully discriminative and valid tests in cases with complex control or multi-file dependencies.

7. Future Directions and Broader Implications

Potential extensions of Repair-R1 include:

• Integrating Static Analysis. Leveraging static analysis or symbolic execution during test synthesis could strengthen discriminative test coverage, especially when the LLM cannot self-elicit sufficient test diversity.
• Enhancing Reward Design. Learning richer critic models or adaptive reweighting of test/repair objectives may mitigate RL reward sparsity or convergence challenges.
• Scaling to Multi-File and Real-World Systems. Adapting the framework to large-scale, multi-module codebases while preserving interpretability and efficiency remains ongoing work.

Repair-R1 establishes a new baseline for LLM-driven APR, demonstrating that joint RL optimization of pre-repair test generation and patch synthesis leads to marked gains in both test informativeness and bug fixing, and suggests a general movement toward process-cognizant AI repair workflows (Hu et al., 30 Jul 2025).