LiveRepoReflection: Repository Code Reflection Benchmark

• LiveRepoReflection is a benchmark for repository-based code reflection, measuring multi-file reasoning and iterative repair with compiler and test feedback.
• It standardizes repository tasks by integrating project structure, environment configuration, automated unit tests, and a two-turn evaluation process.
• It uses a rigorous multi-stage pipeline to curate diverse, high-difficulty cases across six programming languages while minimizing contamination.

Searching arXiv for the benchmark and directly related work to ground the article. Use the arXiv search tool to find papers related to "LiveRepoReflection" and repository-based code reflection. LiveRepoReflection is a benchmark for repository-based code reflection, defined as the ability of code LLMs to examine and modify their own prior outputs in the context of an entire multi-file repository rather than a single file or function. In this setting, a model must navigate project structure, cross-file dependencies, build and test harnesses, and environment configuration, then apply iterative fixes using feedback from compilers and unit tests. The benchmark contains 1,888 rigorously filtered test cases across six programming languages and is paired with RepoReflection-Instruct, a repository-centric instruction-tuning corpus, and RepoReflectionCoder, a 32B model trained for two-turn code generation and error-driven repair (Zhang et al., 14 Jul 2025).

1. Conceptual scope and problem setting

LiveRepoReflection addresses a regime that differs materially from single-function and single-file benchmarks. HumanEval and MBPP evaluate localized code generation, while LiveCodeBench targets larger real-world tasks; LiveRepoReflection instead standardizes end-to-end repository tasks with multi-file layouts, unit tests, and environment files. The benchmark is motivated by the observation that modifying code in repositories requires cross-file reasoning, build-system awareness, and iterative repair under execution feedback, all of which exceed the scope of isolated function synthesis (Zhang et al., 14 Jul 2025).

The benchmark frames repository-based code reflection as a two-stage capability. First, a model must generate edits in either a full-file or a patch-based format. Second, after execution in a sandbox, it must interpret compiler, runtime, or unit-test failures and attempt a second repair. This makes reflection operational rather than rhetorical: the model’s prior output, the repository state, and the observed error trace become the inputs to the next action.

LiveRepoReflection was also designed to reduce contamination risk. Its construction pipeline is automatic and dynamic, dedupes and decontaminates against existing public benchmarks including Aider Polyglot, and retains only tasks that stump strong LLMs. This suggests that the benchmark is intended not merely as a static test suite, but as an evolving evaluation surface for repository-level repair under realistic constraints (Zhang et al., 14 Jul 2025).

2. Repository format, task types, and execution model

Each LiveRepoReflection repository is standardized into five parts: problem definition, reference answer, code signature, unit test, and unit test environment support files. Visibility flags determine which files the model may read or modify during evaluation. The design is inspired by Aider’s code editing benchmarks and Exercism repositories, but it is streamlined for consistency and minimal size while preserving reproducibility (Zhang et al., 14 Jul 2025).

Task types simulate maintenance workflows rather than pure greenfield synthesis. They include bug fixing, feature updates, refactoring or style changes, and iterative repair. The last category is central: after a first attempt fails, the model receives abbreviated error logs and its previous changes, then produces a second attempt. LiveRepoReflection evaluates both full-file code generation and patch-based incremental edits. Full-file generation is easier to use but token-expensive; patch-based edits are closer to code review flows and are more efficient.

Execution is sandboxed with language-specific environment support files. A submission passes only if all unit tests run to completion without compilation or runtime errors and the required assertions hold. Tasks exceeding 180 seconds runtime are discarded to maintain stability and practicality.

Language	Unit test command
Python	pytest
Rust	cargo test -- --include-ignored
Go	go test ./...
JavaScript	./npm-test.sh
C++	./cpp-test.sh
Java	./gradlew test --no-daemon

This execution model makes repository reflection inseparable from build and test infrastructure. A correct local edit is insufficient if it violates repository structure, environment constraints, or required output formatting.

3. Construction pipeline, curation, and decontamination

The benchmark is produced by a multi-stage generation and filtering pipeline. Seed code data come from GitHub, Hugging Face, Stack Overflow, and Reddit across Python, Java, Go, Rust, C++, and JavaScript. File-structure guidance is derived from 702 Exercism problems across six language repositories and from 225 problems in Aider Polyglot, with broken samples removed and about 473 retained to guide structure only rather than content (Zhang et al., 14 Jul 2025).

Generation proceeds through multi-turn dialogue synthesis. A randomly sampled “creative” LLM produces topics and definitions, while multiple “reasoning” LLMs generate unit tests and reference answers. These artifacts are cross-executed in a sandbox; anomalous samples are dropped; the unit test with the lowest pass rate and the reference solution with the highest pass rate are retained. From this process, 100K repository cases are generated and executed. Easy cases, slow cases, and heavily overlapping cases are removed, leaving 10K high-difficulty, high-correctness cases.

Difficulty stratification is then performed with 10 strong reasoning LLMs, each allowed one modification after a failed first attempt. Cases are categorized as “success”, “failure-success”, or “failure-failure”. The retained subset emphasizes tasks where more than half of models remain in “failure-failure” or where many exhibit “failure-success”, thereby selecting for tasks that are either hard or diagnostically informative. After this stage, about 2,300 high-quality, high-difficulty, high-diversity cases remain. Human annotation by eight graduate students, each inspecting about 300 cases for code rationality, environment files, file structure, reference answers, and unit tests, yields the final 1,888-case benchmark (Zhang et al., 14 Jul 2025).

Decontamination is performed with character 5-grams, MinHash signatures, and locality-sensitive hashing. Candidate overlaps with Jaccard similarity greater than 0.8 are filtered and then verified by exact matching. The result is 8,702 high-quality training programs strictly decontaminated against the 1,888-case test set.

Language	Test cases
Python	820
Go	403
Java	250
Rust	212
C++	143
JavaScript	60

The scale is substantially larger than Aider Polyglot. LiveRepoReflection contains 1,888 problems versus 225, with an average of 8.02 files per repository versus 5.97, 2.00 solution files versus 1.26, and longer example contexts and instructions. These choices are consistent with the benchmark’s aim to stress multi-file reasoning rather than local edit generation.

4. Evaluation protocol and metrics

LiveRepoReflection uses a two-turn evaluation protocol. On the first attempt, a model receives the editable files and the instruction and must respond in the required edit format. The proposed changes are applied and executed in the sandbox. If the repository passes all tests, the task counts toward first-attempt success. Otherwise, the model receives abbreviated error logs and its previous changes and is allowed one repair attempt. This explicitly measures whether compiler and unit-test feedback can be converted into a successful repository-level correction (Zhang et al., 14 Jul 2025).

The principal metrics are Pass@1, Pass@2, Fix Weight, and Well Format. With $N$ total tasks and $C_k$ tasks passed within $k$ attempts, the benchmark defines

$$\mathrm{Pass@k}=\frac{C_k}{N}, \quad k\in\{1,2,\dots\}.$$

Fix Weight is defined as

$$\mathrm{FW}=\frac{\mathrm{Pass@2}-\mathrm{Pass@1}}{\mathrm{Pass@2}},$$

that is, the fraction of successful second-attempt fixes among all second-attempt successes. Well Format measures the proportion of outputs that strictly follow the specified full-file or patch-based edit format.

These metrics separate several distinct behaviors. Pass@1 measures direct repository completion. Pass@2 measures overall completion under one repair cycle. Fix Weight isolates how much of second-attempt success is attributable to genuine reflective repair. Well Format evaluates whether a model can adhere to repository-edit protocols rather than merely produce semantically plausible code.

The evaluation also supports a formal comparison between LiveRepoReflection and Aider Polyglot. The same models attain consistently lower absolute pass rates on LiveRepoReflection, and pass@1/pass@2 correlations are only moderate, with $R^2 \approx 0.65/0.78$ and most points below $y=x$. This indicates that repository-based code reflection is not reducible to existing repository-edit benchmarks and imposes a harder distribution over build, dependency, and repair behavior (Zhang et al., 14 Jul 2025).

5. RepoReflection-Instruct and RepoReflectionCoder

The benchmark is paired with RepoReflection-Instruct, a large-scale instruction-tuning corpus designed for repository-centric generation and repair. It is built from about 500,000 code examples, 400,000 of which are newly and dynamically generated. Strict reject sampling retains only repositories with at least one unit test file, at least one reference answer file, matched counts of code signature and reference answer files, environment configuration aligned with language, and standardized file names and extensions (Zhang et al., 14 Jul 2025).

Quality scoring for source programs is given by

$S(p)=\sum_{i\in\{1,2,3,4,5\}} w_i S_i,$

with weights $w=(0.3,0.2,0.2,0.15,0.15)$. The components are executability, novelty or diversity, difficulty, style, and inverse perplexity on reference answers. The resulting corpus is further expanded into 840,839 multi-turn coding dialogues generated with four top models and two edit formats. Task type distribution is 40% direct generation, 40% error-driven repair, 10% style standardization, and 10% dialogue summarization.

RepoReflectionCoder is initialized from Qwen2.5-Coder-32B. In stage 1, it is fine-tuned on RepoReflection-Instruct and about 500K repositories with Adam, cosine decay, 100 warm-up steps, peak learning rate $5\times10^{-5}$, global batch size 1,024, input truncation 32,768 tokens, and 128 NVIDIA H800-80GB GPUs. In stage 2, it is additionally trained on 150,000 simulated multi-turn dialogue trajectories to strengthen code synthesis, iterative debugging, and style standardization. At inference, the model mirrors benchmark evaluation by producing a first attempt and then a second, error-driven repair if needed (Zhang et al., 14 Jul 2025).

On the full benchmark, RepoReflectionCoder improves substantially over its Qwen2.5-Coder base. Under full-file generation it reaches Pass@1 9.0, Pass@2 28.2, Well Format 99.8, and Fix Weight 68.0; under patch-based editing it reaches Pass@1 10.5, Pass@2 27.0, Well Format 94.7, and Fix Weight 61.0. These numbers remain below the strongest closed-source systems, but they show that repository-focused multi-turn tuning materially improves reflective repair behavior (Zhang et al., 14 Jul 2025).

6. Empirical findings, limitations, and significance

The public leaderboard evaluates more than 40 LLMs under uniform hyperparameters. Several empirical regularities emerge. Python tasks are the easiest, while C++ and Rust are the hardest. Format compliance is very high, with nearly every model exceeding 90% Well Format. Fix Weight typically falls in the 60–80% range, indicating that test feedback reliably drives successful second-attempt repair. Patch-based editing reduces token cost and often narrows the gap between first and second passes relative to full-file generation (Zhang et al., 14 Jul 2025).

Among closed-source models, the strongest reported systems attain substantially higher Pass@2 than open-source alternatives. Representative full-file aggregates include o4-mini-2025-04-16 at 40.5 Pass@2, Claude-3-7-Sonnet-20250219 at 37.1, and GPT-4.1-2025-04-14 at 30.9. Pass@k curves are strictly increasing with diminishing returns after $C_k$0, suggesting that the benchmark’s two-turn protocol captures most of the practically accessible repair signal while avoiding excessive interaction depth.

Several limitations are explicit. Repository-level multilingual scenarios are not fully explored. Fine-tuning is supervised rather than RLHF- or DPO-based, which the paper identifies as future work. Reflection across very large repositories and diverse build systems remains difficult. These limitations are structural rather than incidental: they arise from the complexity of scaling repair-oriented evaluation beyond curated, medium-sized repositories (Zhang et al., 14 Jul 2025).

LiveRepoReflection is significant because it turns repository repair into a measurable reflective process. A model is not evaluated solely on its ability to emit code, but on whether it can interpret failure, localize repository-level causes, and produce a corrected edit under real execution constraints. RepoReflection-Instruct and RepoReflectionCoder extend that same premise into training. A plausible implication is that LiveRepoReflection formalizes a transition from static code generation benchmarks toward dynamic, execution-grounded evaluation of repository maintenance, where multi-file structure, environment management, and second-attempt repair are first-class concerns.