RepoBench: Repository-Level Code Evaluation

• RepoBench is a repository-level benchmark that evaluates code auto-completion systems using multi-file, real-world software projects.
• It implements three tasks—retrieval, code completion, and pipeline—to measure cross-file context integration in languages like Python and Java.
• The benchmark uses metrics such as exact-match and edit similarity to assess the impact of retrieval strategies and prompt engineering on LLM performance.

RepoBench is a repository-level benchmark explicitly designed to evaluate the capacity of code auto-completion systems—especially those based on LLMs—to operate effectively in the context of multi-file, real-world programming projects. Unlike benchmarks confined to single-file or local contexts, RepoBench models the authentic software engineering environment, capturing long-range, cross-file dependencies, and evaluating code models under realistic, multi-source information conditions. The benchmark is implemented for both Python and Java and is publicly available, providing a structured, multi-part evaluation to measure retrieval, context integration, and completion quality at the repository scale (Liu et al., 2023).

1. Task Structure and Evaluation Protocol

RepoBench consists of three interrelated tasks, each targeting a distinct repository-scale code modeling ability:

1. RepoBench-R (Retrieval): Evaluates the system’s capacity to retrieve the most relevant cross-file code snippet given in-file context and a set of candidates. The core challenge lies in locating, from among numerous imported or contextually linked snippets, the one that maximally aids next-line prediction.
2. RepoBench-C (Code Completion): Assesses next-line code prediction when models are furnished with an augmented prompt comprising both in-file and retrieved cross-file context. Several settings are considered, including:
   • XF-F: Masking the first usage of a cross-file snippet.
   • XF-R: Masking a random non-first cross-file usage.
   • IF: Masking an in-file line lacking cross-file dependencies.

The completion is measured under differing prompt lengths (e.g., 2K vs. 8K tokens), directly testing token-window efficiency and the model’s ability to maintain context.

1. RepoBench-P (Pipeline): Simulates end-to-end code assistance scenarios. The system must first retrieve relevant cross-file context and then utilize it in a next-line completion task, reflecting an actual developer workflow.

The evaluation across tasks relies on exact-match (EM) and edit similarity (ES) metrics for code prediction, and Accuracy@k (acc@1, acc@3, etc.) for retrieval, with mathematical definitions as follows:

• Retrieval ranking: $$\operatorname*{argmax}_{i \in \{1,\ldots,n\}} f(C[-m:], S_i)$$
• Autoregressive code completion: $$P(Y) = \prod_{i=1}^{n} P(y_i \mid y_{<i}, C_x, C_\mathrm{in})$$

2. Methodological Innovations

RepoBench’s novelty lies in its repository-level focus, which shifts away from the myopia of file- and function-level benchmarks. Key technical features include:

• Integration of Cross-File Context: Tasks are constructed to require retrieval and use of code elements spanning multiple files (e.g., function definitions used indirectly through imports), emulating the navigation required in real software development.
• Prompt Engineering and Ablation: Multiple ablation studies are conducted to discern the impact of cross-file context (XFC), import statements (IS), and in-file context (IFC) on completion accuracy, exposing sensitivity to context design and prompt length.
• Lexical vs. Semantic Retrieval: The retrieval task explores both lexical (Jaccard, edit similarity) and semantic (CodeBERT, UniXcoder embeddings with cosine similarity) ranking. In certain scenarios, simple lexical retrieval demonstrates surprisingly strong performance, highlighting that LLM improvements must be paired with robust retrieval strategies.

3. Technical Formulation and Cross-File Dependency Modeling

The benchmark establishes rigorous mathematical formulations for critical tasks:

• Retrieval: Ranking among $n$ candidates $S_i$ for code context $C[-m:]$: $$\operatorname*{argmax}_{i \in \{1,\ldots,n\}} f(C[-m:], S_i)$$ where $f$ may be a lexical or semantic similarity function.
• Completion with Integrated Context:

$$P(Y) = \prod_{i=1}^n P(y_i | y_{<i}, S_1, ..., S_k, C_{\mathrm{in}})$$ expresses full pipeline modeling, where $S_1,...,S_k$ are retrieved cross-file snippets.

RepoBench further analyzes the effect of candidate pool size (“easy” 5–9, “hard” 10+) and prompt length (e.g., 2K vs. 8K tokens), probing LLMs’ capacity for long-context retrieval and next-token synthesis.

4. Comparison with Contemporaneous Benchmarks

RepoBench addresses deficiencies in previously dominant benchmarks:

Benchmark	Context Span	Retrieval Requirement	Cross-file Evaluation
RepoBench	Multi-file, repository	Yes	Yes
CodeXGLUE/PY150	Single file/function	No	No
CrossCodeEval	File-level, partial	Limited	Partial

By prescribing explicit cross-file retrieval and context integration, RepoBench elicits model behaviors critical for real-world assistance, setting a research direction for advanced code intelligence systems.

5. Integration in Recent Research and Extensions

RepoBench is actively adopted as an evaluation backbone in follow-up research:

• PIE (Positional Integrity Encoding) (He et al., 3 Jul 2024): PIE leverages RepoBench-C-8k to demonstrate that efficient rotary positional cache updates in LLMs support real-time code editing with $>$85% reduction in computational overhead, validating PIE’s practical utility in repository-scale completion.
• Graph-based Dependency Retrieval: RANGER (Shah et al., 27 Sep 2025) directly targets cross-file retrieval as posed by RepoBench, constructing a persistent knowledge graph to deliver sub-file-granularity context with high accuracy (Accuracy@5 up to 0.5471), highlighting the benchmark’s role in motivating advanced retrieval architectures.
• Application in Verilog RTL (RTL-Repo) (Allam et al., 27 May 2024): Adaptations of the RepoBench formulation facilitate RTL-level multi-file code completion, revealing sharply declining LLM performance as token context windows increase, indicative of generalization challenges in real project-scale tasks.
• Literate Programming for LLMs (Zhang et al., 25 Dec 2024): ILP structures, when evaluated on RepoBench, yield measurable gains in code synthesis consistency, especially for documentation-centric project styles and in languages underrepresented in LLM pretraining corpora.

6. Significance, Challenges, and Community Impact

RepoBench’s repository-level approach exposes several practical and technical challenges:

• Long-Range Contextualization: As prompt length increases, model performance declines, revealing limits in transformer-based model attention span and cache efficiency. This observation has driven work such as PIE (He et al., 3 Jul 2024).
• Retrieval Quality’s Effect on Completion: Direct correlations are demonstrated between the retrieval stage’s output and final completion accuracy, substantiating the importance of robust, context-sensitive retrieval engines.
• Benchmark as Research Driver: RepoBench’s evaluation setup has become a reference standard, with new systems and algorithms targeting its specific cross-file pipelines and dependency chaining demands.

Its public availability accelerates reproducibility and allows direct, head-to-head comparison of LLM-based code assistants, informing both research benchmarks and industrial tools (e.g., Copilot-like systems).

7. Future Directions

Ongoing extensions and critiques suggest the following avenues:

• Expansion to New Languages and Project Types: There is movement to adapt RepoBench’s methodology to additional programming paradigms (e.g., hardware description languages, HPC frameworks), pairing code completion with full-stack dependency management.
• Holistic Functional Evaluation: Suggestions from related work (e.g., RepoTransBench (Wang et al., 23 Dec 2024)) advocate for integrating automated build and test pipelines into benchmarks, shifting evaluation from static similarity metrics to real-world execution and correctness criteria.
• Complex Prompt and Retrieval Structures: As evidenced by recent ablations, devising “smarter” prompt construction and integrating advanced graph-based or agentic retrieval mechanisms could be pivotal in driving further LLM advancements.

RepoBench stands as a canonical repository-level benchmark, providing rigorous, reproducible assessment for the rapidly evolving landscape of code auto-completion and retrieval systems. By foregrounding multi-file context dependencies and clear evaluation metrics, it has set a definitive baseline for both academic paper and practical LLM deployment in large-scale software engineering.