RepoReflection-Instruct: Code Reflection Dataset

• RepoReflection-Instruct is a large-scale instruction-tuning dataset that trains models to generate and repair code within multi-file repositories.
• It employs a two-turn reflection loop where initial code generation is followed by testing, error feedback, and iterative debugging.
• The dataset is built through a multi-stage synthetic pipeline, ensuring high-quality, diverse examples across six programming languages.

Searching arXiv for RepoReflection-Instruct and closely related reflection papers. arxiv_search(query="RepoReflection-Instruct repository-based code reflection", max_results=10, sort_by="relevance") RepoReflection-Instruct is a large-scale instruction-tuning dataset for repository-based code reflection: it is designed to train a code model to generate code inside a multi-file repository, encounter compiler, runtime, or unit-test failures, and then repair the code in response to that feedback. Within the broader framework introduced alongside LiveRepoReflection and RepoReflectionCoder, its defining property is that “reflection” is operationalized as an error-driven generation-and-repair loop in realistic repository contexts rather than as single-function synthesis from a natural-language prompt (Zhang et al., 14 Jul 2025).

1. Definition and problem setting

RepoReflection-Instruct addresses a setting that earlier code benchmarks and instruction datasets left under-specified: modifying code in repositories. The motivating claim is that prior work had not adequately captured repository-based code reflection capability, particularly in multi-file environments where code generation is followed by execution, failure observation, and iterative repair. In this formulation, the target behavior is not merely code completion, but repository-level adaptation under test feedback (Zhang et al., 14 Jul 2025).

The dataset is paired conceptually with LiveRepoReflection, but the two serve different functions. LiveRepoReflection is the benchmark and evaluation set, whereas RepoReflection-Instruct is the training corpus used to fine-tune RepoReflectionCoder. The training set is explicitly decontaminated against the evaluation set through MinHash plus LSH over character 5-grams followed by exact matching, leaving 8,702 high-quality training code program cases decontaminated from 1,888 LiveRepoReflection test cases (Zhang et al., 14 Jul 2025).

A central design premise is that the repository context should resemble developer workflow. This means tasks are constructed so that a model must operate over repository structure, produce complete code artifacts, run into errors, and revise its own implementation. This suggests that the dataset is intended to teach not only synthesis, but also a procedural debugging competence anchored in executable repository state.

2. Data sources and repository standardization

The construction process begins from a large multi-source pool comprising about 500,000 code examples, including 400,000 newly automatically and dynamically generated examples, together with code harvested from GitHub, Hugging Face, and Stack Overflow. Repository and file-structure guidance is derived from Exercism repositories. The repository-code pipeline spans six programming languages: Python, Java, Go, Rust, C++, and JavaScript (Zhang et al., 14 Jul 2025).

A specific Exercism-derived subset provides structural guidance rather than direct task content. The pipeline collects 702 coding problems from Exercism in the six languages above, removes the 225 Aider Polyglot overlapping problems and some erroneous data, and retains about 473 Exercism-derived problems for file-structure guidance only. This subset does not supply the substantive content of the newly generated problems; instead, it regularizes repository organization (Zhang et al., 14 Jul 2025).

The standardized repository layout has five parts:

Component	Role
problem definition	task specification
reference answer	canonical solution
code signature	interface or required code skeleton
unit test	executable correctness check
unit-test environment support files	runnable environment scaffolding

This standardization is used for both LiveRepoReflection and RepoReflection-Instruct. Its significance lies in constraining synthetic generation to a reproducible repository format, which in turn supports execution, verification, and repair.

3. Synthetic generation and quality-control pipeline

RepoReflection-Instruct is generated through a sequential synthetic process. The system generates program topic, problem definition, unit tests, and reference answers in order, with each stage conditioned on the previous ones for consistency. A “creative” LLM is randomly selected from a mixed LLM stream to produce topics and problem definitions, and one or more “reasoning” LLMs are randomly selected to generate unit tests and reference solutions (Zhang et al., 14 Jul 2025).

For each problem, the pipeline generates multiple unit tests and multiple reference answers, then cross-executes unit-test/reference-answer pairs in a sandbox environment. Anomalous samples are discarded. For each coding problem, the retained pair is the unit test with the lowest pass rate and the reference answer with the highest quality / pass performance, while 0%-pass cases are manually inspected. This design is intended to keep tasks difficult while preserving validity (Zhang et al., 14 Jul 2025).

RepoReflection-Instruct applies an additional strict reject-sampling filter. A repository is kept only if it satisfies all five of the following conditions:

1. at least one unit test file
2. at least one reference answer file
3. the number of code-signature files matches the number of reference-answer files
4. environment configuration files match the declared programming language
5. file names and extensions are standardized and contain no anomalies

The broader selection pipeline also uses a composite quality score

$S(p)=\sum_{i\in\{1,2,3,4,5\} w_i S_i$

with weights

$w=(0.3, 0.2, 0.2, 0.15, 0.15).$

The five components are $S_{exec}$ for executability and unit-test behavior, $S_{nov}$ for novelty and semantic diversity, $S_{dif}$ for difficulty, $S_{style}$ for style quality, and $S_{ppl}$ for inverse perplexity on reference answers (Zhang et al., 14 Jul 2025).

4. Instruction format and the two-turn reflection loop

The defining instructional form of RepoReflection-Instruct is a two-turn dialogue. In Turn 1, the model receives a coding task prompt and generates a complete solution. In Turn 2, that generated code is tested; if it fails, the model is shown compiler/runtime/test error feedback and asked to revise the code. This is the dataset’s core notion of reflection: the model must inspect its own failed output and fix it (Zhang et al., 14 Jul 2025).

The multi-turn dialogues may also include code synthesis, iterative debugging, style standardization, and sometimes dialogue summarization, but the generation-and-repair loop is the central mechanism. This makes RepoReflection-Instruct structurally different from static instruction corpora in which the response is a one-shot answer detached from executable feedback.

The appendix-level prompt templates impose concrete formatting constraints. The topic prompt requests a short snake_case project name. The problem-definition prompt asks for a hard coding problem description and explicitly excludes project name, code signature, answer, and unit test. Separate prompts generate unit tests and reference answers while respecting repository structure. A file-format constraint requires each file to be emitted as a path followed by a code block, with no ellipses, no skipping content, and only standard libraries. The runnable environment is language-specific; for example, JavaScript uses Node.js v16.20.2, and Rust uses rustc 1.75.0 with edition $\le 2021$ (Zhang et al., 14 Jul 2025).

A plausible implication is that the dataset teaches not only semantic repair under failure, but also adherence to repository formatting and environment constraints, both of which are necessary for executable repository-level generation.

5. Scale, composition, and role in training RepoReflectionCoder

After filtering and decontamination, RepoReflection-Instruct retains 8,702 high-quality training code program cases for the final training set used to train RepoReflectionCoder. This final set is distilled from the much larger approximately 500,000-example corpus described above and preserves the six-language coverage of the benchmark: Python, Java, C++, Rust, Go, and JavaScript (Zhang et al., 14 Jul 2025).

RepoReflectionCoder is fine-tuned in two stages. In Stage 1, Qwen2.5-Coder-32B is fine-tuned on RepoReflection-Instruct and roughly 500k repositories/examples using Adam, a cosine-decay scheduler, 100 warmup steps, peak learning rate 5e-5, global batch size 1024, input length truncated to 32,768 tokens, and 128 NVIDIA H800-80GB GPUs. In Stage 2, the model is further trained on 150,000 simulated multi-turn dialogue trajectories covering code synthesis, iterative debugging, and style standardization (Zhang et al., 14 Jul 2025).

The paper does not provide a clean ablation isolating RepoReflection-Instruct alone. It does, however, report that RepoReflectionCoder clearly outperforms its Qwen2.5-Coder base on LiveRepoReflection, performs strongly on both full-file code generation and patch-based incremental edits, and is especially strong on JavaScript. The benchmark also uses Pass@1, Pass@2, FW (Fix Weight), and WF (Well Formed) to characterize first-attempt success, improvement after feedback, error-fixing contribution, and format adherence, respectively (Zhang et al., 14 Jul 2025).

6. Conceptual significance and relation to other reflection paradigms

Within contemporary work on model reflection, RepoReflection-Instruct exemplifies a specifically repository-centered and execution-grounded form of retrospective reflection. Its core pattern is: generate code, observe failure, and repair. That differs from the formulation in PreFlect, which shifts from retrospective reflection to prospective reflection by criticizing and refining plans before execution and supplementing that process with dynamic re-planning during execution (Wang et al., 6 Feb 2026).

It also differs from ReflectEvo, where reflection learning is organized around multi-domain failed reasoning traces and self-generated reflections that verify failure, locate errors, diagnose causes, and produce corrected answers; ReflectEvo-460k contains about 460k reflection samples and is not repository-specific (Li et al., 22 May 2025). RepoReflection-Instruct, by contrast, is explicitly tied to multi-file repository environments, standardized file structures, runnable unit-test setups, and code repair after compiler/runtime/test feedback (Zhang et al., 14 Jul 2025).

These contrasts clarify the meaning of “reflection” in RepoReflection-Instruct. Here, reflection is neither compile-time program introspection nor plan-level foresight. It is a supervised training signal built around repository execution, failure evidence, and corrective revision. The broader significance is that repository-based code generation is treated as an iterative systems problem rather than a single-turn text generation problem, with decontamination, execution-based filtering, and repository structure serving as integral parts of the dataset design.