---
title: 'RepoReflection-Instruct: Code Reflection Dataset'
url: https://www.emergentmind.com/topics/reporeflection-instruct
type: topic
---

# RepoReflection-Instruct: Code Reflection Dataset

Searching arXiv for RepoReflection-Instruct and closely related reflection papers.
arxiv_search(query="RepoReflection-Instruct repository-based code reflection", max_results=10, sort_by="relevance")
RepoReflection-Instruct is a large-scale instruction-tuning dataset for repository-based code reflection: it is designed to train a code model to generate code inside a multi-file repository, encounter compiler, runtime, or unit-test failures, and then repair the code in response to that feedback. Within the broader framework introduced alongside LiveRepoReflection and RepoReflectionCoder, its defining property is that “reflection” is operationalized as an error-driven generation-and-repair loop in realistic repository contexts rather than as single-function synthesis from a natural-language prompt [2507.09866].

## 1. Definition and problem setting

RepoReflection-Instruct addresses a setting that earlier code benchmarks and instruction datasets left under-specified: modifying code in repositories. The motivating claim is that prior work had not adequately captured repository-based code reflection capability, particularly in multi-file environments where code generation is followed by execution, failure observation, and iterative repair. In this formulation, the target behavior is not merely code completion, but repository-level adaptation under test feedback [2507.09866].

The dataset is paired conceptually with LiveRepoReflection, but the two serve different functions. LiveRepoReflection is the benchmark and evaluation set, whereas RepoReflection-Instruct is the training corpus used to fine-tune RepoReflectionCoder. The training set is explicitly decontaminated against the evaluation set through MinHash plus LSH over character 5-grams followed by exact matching, leaving **8,702 high-quality training code program cases** decontaminated from **1,888** LiveRepoReflection test cases [2507.09866].

A central design premise is that the repository context should resemble developer workflow. This means tasks are constructed so that a model must operate over repository structure, produce complete code artifacts, run into errors, and revise its own implementation. This suggests that the dataset is intended to teach not only synthesis, but also a procedural debugging competence anchored in executable repository state.

## 2. Data sources and repository standardization

The construction process begins from a large multi-source pool comprising **about 500,000 code examples**, including **400,000 newly automatically and dynamically generated examples**, together with code harvested from **GitHub, Hugging Face, and Stack Overflow**. Repository and file-structure guidance is derived from **Exercism** repositories. The repository-code pipeline spans **six programming languages**: **Python**, **Java**, **Go**, **Rust**, **C++**, and **JavaScript** [2507.09866].

A specific Exercism-derived subset provides structural guidance rather than direct task content. The pipeline collects **702 coding problems from Exercism** in the six languages above, removes the **225 Aider Polyglot** overlapping problems and some erroneous data, and retains about **473** Exercism-derived problems for **file-structure guidance only**. This subset does not supply the substantive content of the newly generated problems; instead, it regularizes repository organization [2507.09866].

The standardized repository layout has five parts:

| Component | Role |
|---|---|
| problem definition | task specification |
| reference answer | canonical solution |
| code signature | interface or required code skeleton |
| unit test | executable correctness check |
| unit-test environment support files | runnable environment scaffolding |

This standardization is used for both LiveRepoReflection and RepoReflection-Instruct. Its significance lies in constraining synthetic generation to a reproducible repository format, which in turn supports execution, verification, and repair.

## 3. Synthetic generation and quality-control pipeline

RepoReflection-Instruct is generated through a sequential synthetic process. The system generates **program topic**, **problem definition**, **unit tests**, and **reference answers** in order, with each stage conditioned on the previous ones for consistency. A “creative” LLM is randomly selected from a mixed LLM stream to produce topics and problem definitions, and one or more “reasoning” LLMs are randomly selected to generate unit tests and reference solutions [2507.09866].

For each problem, the pipeline generates multiple unit tests and multiple reference answers, then cross-executes unit-test/reference-answer pairs in a sandbox environment. Anomalous samples are discarded. For each coding problem, the retained pair is the **unit test with the lowest pass rate** and the **reference answer with the highest quality / pass performance**, while **0%-pass cases** are manually inspected. This design is intended to keep tasks difficult while preserving validity [2507.09866].

RepoReflection-Instruct applies an additional strict reject-sampling filter. A repository is kept only if it satisfies all five of the following conditions:

1. at least one unit test file  
2. at least one reference answer file  
3. the number of code-signature files matches the number of reference-answer files  
4. environment configuration files match the declared programming language  
5. file names and extensions are standardized and contain no anomalies  

The broader selection pipeline also uses a composite quality score
\[
S(p)=\sum_{i\in\{1,2,3,4,5\} w_i S_i
\]
with weights
\[
w=(0.3, 0.2, 0.2, 0.15, 0.15).
\]
The five components are \(S_{exec}\) for executability and unit-test behavior, \(S_{nov}\) for novelty and semantic diversity, \(S_{dif}\) for difficulty, \(S_{style}\) for style quality, and \(S_{ppl}\) for inverse perplexity on reference answers [2507.09866].

## 4. Instruction format and the two-turn reflection loop

The defining instructional form of RepoReflection-Instruct is a **two-turn dialogue**. In **Turn 1**, the model receives a coding task prompt and generates a **complete solution**. In **Turn 2**, that generated code is tested; if it fails, the model is shown **compiler/runtime/test error feedback** and asked to revise the code. This is the dataset’s core notion of reflection: the model must inspect its own failed output and fix it [2507.09866].

The multi-turn dialogues may also include **code synthesis**, **iterative debugging**, **style standardization**, and sometimes **dialogue summarization**, but the generation-and-repair loop is the central mechanism. This makes RepoReflection-Instruct structurally different from static instruction corpora in which the response is a one-shot answer detached from executable feedback.

The appendix-level prompt templates impose concrete formatting constraints. The topic prompt requests a short snake_case project name. The problem-definition prompt asks for a hard coding problem description and explicitly excludes project name, code signature, answer, and unit test. Separate prompts generate unit tests and reference answers while respecting repository structure. A file-format constraint requires each file to be emitted as a path followed by a code block, with **no ellipses**, **no skipping content**, and **only standard libraries**. The runnable environment is language-specific; for example, **JavaScript uses Node.js v16.20.2**, and **Rust uses rustc 1.75.0** with edition \(\le 2021\) [2507.09866].

A plausible implication is that the dataset teaches not only semantic repair under failure, but also adherence to repository formatting and environment constraints, both of which are necessary for executable repository-level generation.

## 5. Scale, composition, and role in training RepoReflectionCoder

After filtering and decontamination, RepoReflection-Instruct retains **8,702** high-quality training code program cases for the final training set used to train RepoReflectionCoder. This final set is distilled from the much larger approximately **500,000**-example corpus described above and preserves the six-language coverage of the benchmark: **Python**, **Java**, **C++**, **Rust**, **Go**, and **JavaScript** [2507.09866].

RepoReflectionCoder is fine-tuned in two stages. In **Stage 1**, **Qwen2.5-Coder-32B** is fine-tuned on RepoReflection-Instruct and roughly **500k repositories/examples** using **Adam**, a **cosine-decay scheduler**, **100 warmup steps**, peak learning rate **5e-5**, global batch size **1024**, input length truncated to **32,768 tokens**, and **128 NVIDIA H800-80GB GPUs**. In **Stage 2**, the model is further trained on **150,000 simulated multi-turn dialogue trajectories** covering **code synthesis**, **iterative debugging**, and **style standardization** [2507.09866].

The paper does not provide a clean ablation isolating RepoReflection-Instruct alone. It does, however, report that RepoReflectionCoder **clearly outperforms its Qwen2.5-Coder base** on LiveRepoReflection, performs strongly on both **full-file code generation** and **patch-based incremental edits**, and is especially strong on **JavaScript**. The benchmark also uses **Pass@1**, **Pass@2**, **FW (Fix Weight)**, and **WF (Well Formed)** to characterize first-attempt success, improvement after feedback, error-fixing contribution, and format adherence, respectively [2507.09866].

## 6. Conceptual significance and relation to other reflection paradigms

Within contemporary work on model reflection, RepoReflection-Instruct exemplifies a specifically repository-centered and execution-grounded form of retrospective reflection. Its core pattern is: generate code, observe failure, and repair. That differs from the formulation in PreFlect, which shifts from retrospective reflection to **prospective reflection** by criticizing and refining plans **before execution** and supplementing that process with dynamic re-planning during execution [2602.07187].

It also differs from ReflectEvo, where reflection learning is organized around multi-domain failed reasoning traces and self-generated reflections that verify failure, locate errors, diagnose causes, and produce corrected answers; ReflectEvo-460k contains about **460k reflection samples** and is not repository-specific [2505.16475]. RepoReflection-Instruct, by contrast, is explicitly tied to **multi-file repository environments**, standardized file structures, runnable unit-test setups, and code repair after compiler/runtime/test feedback [2507.09866].

These contrasts clarify the meaning of “reflection” in RepoReflection-Instruct. Here, reflection is neither compile-time program introspection nor plan-level foresight. It is a supervised training signal built around repository execution, failure evidence, and corrective revision. The broader significance is that repository-based code generation is treated as an iterative systems problem rather than a single-turn text generation problem, with decontamination, execution-based filtering, and repository structure serving as integral parts of the dataset design.

Source: https://www.emergentmind.com/topics/reporeflection-instruct