RepoReflectionCoder: Repository Code Reflection
- RepoReflectionCoder is a repository-oriented code language model designed for multi-file tasks that require iterative error-driven repair.
- It uses a two-turn interaction where initial code generation is refined based on execution feedback such as compilation and runtime errors.
- The associated LiveRepoReflection benchmark spans six programming languages, showcasing significant improvements in code repair and format compliance.
RepoReflectionCoder is a repository-oriented code LLM introduced for repository-based code reflection, a setting in which a model must modify or generate code inside a structured, multi-file repository and then improve that code after receiving execution feedback such as compilation errors, runtime failures, or unit-test failures. In the paper’s formulation, the model is trained and evaluated not on isolated function synthesis, but on repository-style tasks with problem definitions, code signatures, unit tests, and environment support files, using a bounded two-turn interaction of initial generation followed by error-driven repair (Zhang et al., 14 Jul 2025).
1. Scope and conceptual definition
Repository-based code reflection is defined as the ability of a model to “examine and modify their previous responses” in a repository context rather than in a standalone coding prompt. RepoReflectionCoder therefore targets a problem setting that differs from standard code-generation benchmarks such as HumanEval, MBPP, MultiPL-E, LiveCodeBench, McEval, and BigCodeBench, because those benchmarks do not directly model the scenario of modifying code in repositories. The paper also distinguishes this setting from simpler single-file editing: repository tasks require reasoning across multiple files, interfaces, dependencies, tests, and environment configuration (Zhang et al., 14 Jul 2025).
The motivating claim is that multi-file repository contexts are intrinsically harder. The benchmark introduced with RepoReflectionCoder is explicitly described as richer than Aider Polyglot, with average repository statistics of 8.02 files versus 5.97, 2.00 solution files versus 1.26, 1.06 test files versus 1.01, and 2.01 example files versus 1.16. The paper further argues that static public benchmarks are vulnerable to contamination and “data hacking,” and frames its own benchmark as dynamically generated from repositories “never-before-seen” through May 2025 (Zhang et al., 14 Jul 2025).
In this framework, reflection is operationalized as a tightly bounded repair regime. Evaluation gives one initial attempt and, if that attempt fails, one additional attempt after the model can view its previous code together with the resulting error messages. A plausible implication is that the system is designed less as an open-ended agentic debugger than as a supervised repository-editing model with an explicit post-failure repair step (Zhang et al., 14 Jul 2025).
2. LiveRepoReflection benchmark
The benchmark paired with RepoReflectionCoder is LiveRepoReflection, a dynamic, executable, multi-file repository benchmark containing 1,888 manually finalized test cases across 6 programming languages: Python, Java, Go, Rust, C++, and JavaScript. Appendix counts are 820 Python, 403 Go, 250 Java, 212 Rust, 143 C++, and 60 JavaScript (Zhang et al., 14 Jul 2025).
Each benchmark instance is organized into a standardized repository structure with five components: problem definition, reference answer, code signature, unit test, and unit test environment support file. The construction pipeline begins from 702 coding problems from Exercism for the six languages, removes 225 problems overlapping with Aider Polyglot tests, removes erroneous data, and retains about 473 coding problems mainly to define and standardize repository structure. The broader pipeline then collects seed code data from GitHub, Hugging Face, and Reddit; the framework figure also mentions Stack Overflow, which the paper notes inconsistently relative to the body text (Zhang et al., 14 Jul 2025).
The generation stage produces new coding program repositories using a “creative” LLM for topics and definitions and multiple “reasoning” LLMs for unit tests and reference answers. The automated pipeline first creates 100K coding program repository cases, filters them by successful sandbox execution and difficulty, and retains 10K high-difficulty, high-correctness cases. A subsequent difficulty-selection stage using 10 selected mainstream strong reasoning LLMs reduces this set to 2300 hard cases. Final curation is performed by 8 graduate students, each reviewing nearly 300 cases, yielding the final 1,888-case benchmark (Zhang et al., 14 Jul 2025).
Executability is treated as part of the benchmark design rather than as ancillary metadata. The appendix specifies unit-test commands for all six languages, including pytest for Python, cargo test -- --include-ignored for Rust, go test ./... for Go, ./npm-test.sh for JavaScript, ./cpp-test.sh for C++, and ./gradlew test --no-daemon for Java. Environment notes include Node.js v16.20.2 for JavaScript and rustc 1.75.0 (82e1608df 2023-12-21) with edition <= 2021 for Rust (Zhang et al., 14 Jul 2025).
3. RepoReflection-Instruct dataset and supervision design
The training corpus for RepoReflectionCoder is RepoReflection-Instruct, which the paper describes as a large-scale instruction-tuning dataset for repository-aware code generation, code modification, execution-guided repair, style standardization, and multi-turn coding interaction. It collects approximately 500,000 code examples, including 400,000 newly automatically and dynamically generated ones, and uses retained Exercism-style repository structures as exemplars of repository organization (Zhang et al., 14 Jul 2025).
The dataset is filtered by five explicit repository-validity criteria: there must be at least one unit test file, at least one reference answer file, the number of code signature files matches the number of reference answer files, environment configuration files correspond with the declared programming language, and file naming and extensions are standardized and free of anomalies. After rejection sampling, the paper applies LLM-as-a-judge plus a weighted quality score
with
where the five component scores are executability, novelty, difficulty, code style, and inverse perplexity on reference answers. From the top 10,000 ranked code problems, decontamination against LiveRepoReflection yields 8,702 high-quality training code program cases (Zhang et al., 14 Jul 2025).
The supervision signal is strongly dialogue-oriented. The paper states that 840,839 multi-turn coding dialogues were simulated using four top models—claude-3-7-sonnet-20250219, qwen3-235b-a22b-instruct, o1-mini, and o4-mini—each run four times and in two formats, namely full-file generation and patch-based edits. Examples are then distributed across four categories: Direct generation (40%), Error-driven repair (40%), Style standardization (10%), and Dialogue summarization (10%) (Zhang et al., 14 Jul 2025).
Although the paper often uses the phrase “multi-turn,” the core supervision pattern is effectively a two-turn coding dialogue. The first turn performs initial repository-level generation or modification; the second turn performs repair after observing compiler, runtime, or unit-test failures. This two-turn structure is the principal mechanism by which repository-based reflection is converted into supervised instruction tuning (Zhang et al., 14 Jul 2025).
4. Model, training pipeline, and implementation characteristics
RepoReflectionCoder is fine-tuned from Qwen2.5-Coder-32B-Instruct. The training procedure has two stages. In the first stage, the model is instruction-tuned on RepoReflection-Instruct plus roughly 500K repositories. In the second stage, it is further trained on 150,000 simulated multi-turn dialogue trajectories covering code synthesis, iterative debugging, and style standardization (Zhang et al., 14 Jul 2025).
The implementation details reported for stage one are specific: training uses 128 NVIDIA H800-80GB GPUs, the Adam optimizer, cosine-decay scheduling, 100 warmup steps, peak learning rate 5e-5, global batch size 1,024, and maximum input length 32,768 tokens. The paper does not describe a novel optimization objective beyond supervised fine-tuning, and explicitly notes in its limitations discussion that future work may explore RLHF methods such as DPO, implying that current RepoReflectionCoder does not use them (Zhang et al., 14 Jul 2025).
The representation of repository context is standardized at the data level but not fully formalized at the prompt-serialization level. The appendix shows file-path-based formatting with full file contents under code fences, and the benchmark structure always contains problem definitions, signatures, reference answers, tests, and support files. However, the paper does not provide a single canonical serialization format for repository context at training or inference time, nor does it describe architecture modifications beyond instruction tuning (Zhang et al., 14 Jul 2025).
The paper also contains an internal inconsistency that matters for reproducibility. Its implementation section and results tables clearly identify Qwen2.5-Coder-32B as the base model, whereas the limitations section says the model is “mainly supervised fine-tuned on the 7B open-source base LLMs.” The 32B description is the one supported by the implementation and leaderboard tables, but the mismatch remains a documented caveat (Zhang et al., 14 Jul 2025).
5. Evaluation protocol and empirical performance
LiveRepoReflection evaluates both full-file code generation and patch-based incremental edits. The four reported metrics are Pass@1, Pass@2, Well Format (WF), and Fix Weight (FW). Pass@1 is the proportion of tasks solved on the first attempt; Pass@2 is the proportion solved after allowing a second attempt with feedback; WF measures adherence to the required edit format; and FW is intended to measure the contribution of second-attempt repair. The paper prints FW as
$\mathrm{FW} \;=\;\frac{\mathrm{Pass@2} - \mathrm{Pass@1}}{\mathrm{Pass@2}$
which appears to contain a typesetting error and is presented as such in the source (Zhang et al., 14 Jul 2025).
The reported overall results are summarized below in the order Pass@1 / Pass@2 / WF / FW.
| Mode | RepoReflectionCoder | Qwen2.5-Coder-32B-Instruct |
|---|---|---|
| Full-file generation | 9.0 / 28.2 / 99.8 / 68.0 | 3.9 / 14.9 / 97.6 / 74.0 |
| Patch-based edits | 10.5 / 27.0 / 94.7 / 61.0 | 3.9 / 12.9 / 74.6 / 69.7 |
In full-file generation, RepoReflectionCoder reaches Pass@1 = 9.0 and Pass@2 = 28.2, compared with 3.9 and 14.9 for its base model. In patch mode, it reaches Pass@1 = 10.5 and Pass@2 = 27.0, with WF = 94.7, compared with 3.9, 12.9, and 74.6 for the base model. The strongest relative gain in patch mode is therefore format compliance, which rises from 74.6 to 94.7 (Zhang et al., 14 Jul 2025).
Per-language results show marked heterogeneity. In full-file mode, RepoReflectionCoder attains 13.5 / 40.4 on Python, 8.4 / 22.0 on Java, 5.6 / 15.4 on C++, 1.4 / 3.3 on Rust, 4.5 / 17.4 on Go, and 15.0 / 78.3 on JavaScript. In patch mode, the corresponding figures are 15.7 / 37.3, 12.0 / 22.0, 4.9 / 10.5, 2.8 / 4.7, 5.2 / 18.1, and 10.0 / 85.0. The paper characterizes Python as the easiest language and C++ and Rust as the hardest, while noting that second-attempt repair contributes substantially across models, with FW often in the 60–80% range (Zhang et al., 14 Jul 2025).
RepoReflectionCoder is one of the strongest open-source models on the benchmark, but it remains below the best closed-source systems. For example, on full-file generation, o4-mini-2025-04-16 reports 14.0 / 40.5, o3-mini-2025-01-31 reports 11.9 / 36.1, and Claude-3-7-Sonnet-20250219 reports 11.8 / 37.1, all above RepoReflectionCoder’s 9.0 / 28.2. The paper also reports that pass@k curves for selected models are strictly increasing from to $10$, with the largest gain from , after which improvements diminish (Zhang et al., 14 Jul 2025).
6. Position in the literature and documented limitations
RepoReflectionCoder occupies an intersection between repository-aware code modeling and reflection-based code repair. On the repository side, earlier systems such as RepoCoder introduced iterative retrieval and generation for repository-level completion (Zhang et al., 2023), RepoFusion trained seq2seq models to consume multiple repository contexts jointly (Shrivastava et al., 2023), GraphCoder used statement-level code-context graphs with coarse-to-fine retrieval (Liu et al., 2024), RLCoder trained retrievers from weighted-perplexity feedback without labeled retrieval pairs (Wang et al., 2024), and RepoGenReflex added a Reflector, Evaluator, and Experience cache to iteratively refine retrieval targets (Wang et al., 2024). Those works primarily target repository-aware completion and retrieval, whereas RepoReflectionCoder is centered on repository modification plus post-failure repair (Zhang et al., 14 Jul 2025).
On the reflection side, ReflectionCoder learns from execution-grounded reflection sequences to improve one-off code generation (Ren et al., 2024), and ReflexiCoder trains a structured generate→reflect→correct trajectory with RL-zero and reflection-aware reward shaping (Jiang et al., 6 Mar 2026). Both demonstrate that reflection can be internalized into model behavior, but both are evaluated mainly on algorithmic or standalone code-generation benchmarks rather than multi-file repository modification. RepoReflectionCoder can therefore be situated as a repository-level specialization of the broader shift from single-pass generation toward explicit reflection and self-correction (Zhang et al., 14 Jul 2025).
The paper is also explicit about several limitations. It does not provide a canonical prompt serialization for final supervised fine-tuning, exact inference prompt templates for RepoReflectionCoder itself, or conventional ablation studies isolating the effects of data sources, two-turn supervision, or second-stage trajectory training. The benchmark allows only one repair attempt after failure, so long-horizon iterative debugging is not evaluated. Release details are partially ambiguous: the paper points to https://LiveRepoReflection.github.io and a dynamic leaderboard, but does not clearly enumerate whether full benchmark files, full training data, and checkpoints are all publicly downloadable. These omissions make RepoReflectionCoder a substantial benchmark-and-model contribution, but not a fully specified end-to-end reproduction recipe (Zhang et al., 14 Jul 2025).
A plausible summary is that RepoReflectionCoder converts repository-level code modification and repair into a supervised, executable, contamination-aware learning problem. Its distinctive contribution lies less in a new architecture than in the combination of LiveRepoReflection, RepoReflection-Instruct, and a two-turn error-driven training regime that measurably improves repository-scale repair behavior while still leaving a considerable gap to the strongest closed-source systems (Zhang et al., 14 Jul 2025).