---
title: 'CrossCodeEval: Cross-File Code Evaluation'
url: https://www.emergentmind.com/topics/crosscodeeval
type: topic
---

# CrossCodeEval: Cross-File Code Evaluation

CrossCodeEval is a multilingual benchmark designed to rigorously evaluate the capability of code completion models to resolve cross-file dependencies in realistic, repository-scale software engineering scenarios. Unlike previous benchmarks constrained to in-file completion, CrossCodeEval enforces strict requirements such that each test example demands retrieval and semantic reasoning over code dispersed across multiple files. Its construction leverages static analysis to pinpoint locations where code completion explicitly depends on external, intra-repository symbols and definitions, thereby simulating authentic challenges faced in large-scale software development and pushing the boundaries of context-aware code generation.

## 1. Benchmark Construction and Characteristics

**CrossCodeEval** was introduced by Ding et al. as the first benchmark strictly focused on repository-level, cross-file code generation. The benchmark comprises four major programming languages: Python, Java, TypeScript, and C♯. The datasets consist of real-world, open-source, permissively-licensed repositories, selected with stringent filters (creation in Mar–Jun 2023, star ≥3, size <1MB, 10–50 source files, exclusion from LLM pretraining datasets) to ensure diversity and minimize test-train leakage [2310.11248].

For each repository, static analysis is used to identify completion targets that necessarily require cross-file context—e.g., statements involving classes, functions, or constants defined only in external files. Each instance is then formulated as a code-completion "hole": given all tokens up to the missing snippet in one file (local context) and the ability to retrieve from the rest of the repository (global context), the model must generate exactly the reference code (typically a single line or a concise statement). In all languages, pre-processing steps filter out trivial, auto-solvable, or overly generic completions.

**Dataset statistics (Python, Java as typical):**
| Language   | #Repos | #Files | #Examples | Avg. Target Tokens |
|------------|--------|--------|-----------|-------------------|
| Python     | 471    | 1,368  | 2,665     | 14.45             |
| Java       | 239    | 745    | 2,139     | 16.76             |

No standard training/validation/test splits are provided; CrossCodeEval is widely treated as a zero-shot, evaluation-only test set [2310.11248, 2407.19487, 2406.10018].

## 2. Task Formulation and Evaluation Protocol

Each task instance consists of a prompt (all code up to a cursor position in a given file) and a ground-truth reference snippet (the missing line or statement). The core requirement is that the missing statement depends on symbols—such as imported classes, called functions, or constants—not declared in the current file but available elsewhere in the same repository [2310.11248, 2512.04538].

To succeed, a model must ingest the prompt, optionally incorporate retrieved cross-file context (using retrieval-augmented generation or static analysis), and emit code that matches the reference under multiple evaluation criteria. Common retrieval approaches include BM25, dense vector search, dynamic graph traversal, hybrid retrieval, and static analysis–guided augmentation [2407.19487, 2509.25257, 2601.19697].

An illustrative instance: given a Django management command in Python requesting to complete a function using a helper `process_data` imported from another file, the model must correctly synthesize the invocation based on available context and project structure [2512.04538].

## 3. Evaluation Metrics

CrossCodeEval employs a strict and multifaceted evaluation protocol, with four principal metrics—computed per sample and aggregated across the dataset [2310.11248, 2407.19487, 2402.14323, 2601.23254, 2507.20888]:

- **Exact Match (EM):**
  $$
  \mathrm{EM} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat y_i = y_i]
  $$
  The prediction must be byte-for-byte identical to the reference, including whitespace and symbols.

- **Edit Similarity (ES):**
  $$
  \mathrm{ES} = \frac{1}{N}\sum_{i=1}^N \left(1 - \frac{\mathrm{LevDist}(\hat y_i,\,y_i)}{\max(|\hat y_i|,\,|y_i|)}\right)
  $$
  Normalized Levenshtein distance reflects partially correct outputs.

- **Identifier Exact Match (ID-EM):**
  $$
  \mathrm{ID\_EM} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[\operatorname{id}(\hat y_i) = \operatorname{id}(y_i)]
  $$
  Requires all predicted identifiers (variable/class/function names) to exactly match ground truth.

- **Identifier F1 (ID-F1):**
  $$
  \operatorname{Prec}_i = \frac{|\operatorname{id}(\hat y_i) \cap \operatorname{id}(y_i)|}{|\operatorname{id}(\hat y_i)|},\quad
  \operatorname{Rec}_i = \frac{|\operatorname{id}(\hat y_i) \cap \operatorname{id}(y_i)|}{|\operatorname{id}(y_i)|}
  $$
  $$
  \mathrm{ID\_F1} = \frac{1}{N}\sum_{i=1}^N 2\cdot\frac{\operatorname{Prec}_i \cdot \operatorname{Rec}_i}{\operatorname{Prec}_i + \operatorname{Rec}_i}
  $$

Extended studies also report recall, pass@k, and context retrieval precision, but EM and ES remain central [2310.11248, 2601.23254].

## 4. Comparative Context and Distinctive Features

CrossCodeEval is distinguished from earlier benchmarks by several design principles [2310.11248, 2406.10018, 2512.04538]:

- **Forced cross-file dependencies:** All test instances mandate reference to code outside the immediate file. Simple in-file completions are excluded.
- **Multilinguality:** Four languages (Python, Java, TypeScript, C♯), facilitating broad generalization studies and robust evaluation [2310.11248].
- **Real-world scale:** Hundreds of repositories per language, spanning 1,000s of files.
- **High rigor in sample curation:** Use of static analysis and AST-level checking prevents trivial and duplicative solutions.
- **Strict evaluation:** EM is notably unforgiving, and identifier-based metrics penalize partial or semantically incorrect completions.

In contrast, *RepoEval* focuses on in-file or limited cross-file completions in a smaller set of large repositories, while *ProjBench* alters import masking strategies and is focused on internal API representation. CrossCodeEval is considered the reference standard for any method claiming cross-file code understanding [2407.19487, 2507.20888, 2508.10068].

## 5. Retrieval, Static Analysis, and System Performance

A broad array of retrieval and static analysis enhancements have been benchmarked on CrossCodeEval:

| Method           | Retrieval Type                          | Key Innovation                                 | Max EM (Python/Java) |
|------------------|----------------------------------------|------------------------------------------------|----------------------|
| BM25             | Lexical (BM25)                         | Sparse lexical ranking                         | ~21%                |
| RLCoder          | RL policy, learned retrieval            | Perplexity-based RL/sampling                   | ~36–40%              |
| RANGER           | Graph-based + BM25                     | Cypher on repo KG + lexical fusion             | 36.3%                |
| RepoFuse         | Fusion (analogy + rationale context)    | Rank-truncated prompt, context condensation    | 28–30%               |
| SaraCoder        | Hierarchical semantic & structural      | Entropy-pruned retrieval, identifier disambig. | up to 40%            |
| AlignCoder       | Query enhancement + RL retriever        | Sampled completions for query, RL refinement   | 34%                  |
| STALL⁺           | Static analysis integration             | Prompt, decode, post-phase dependency cues     | 28–46% (Java), 29% (Python); >50% when combined with RAG |
| GrepRAG          | Index-free lexical + re-ranking         | ripgrep-command generation, identifier refinement | 42–44%               |
| CodexGraph       | LLM-driven, graph database (Neo4j)      | Iterative Cypher query, structure-based selection| 27.9% (GPT-4o)      |
| CoCo             | Multi-granular static extraction        | Project/file/function-level static context     | up to +20.2% EM over baseline |

Across all methods, combining static analysis with RAG or graph/lexical retrieval yields the highest gains, particularly in static languages (e.g. Java) [2406.10018, 2601.23254]. Methods based purely on BM25 retrieval are consistently outperformed by approaches that inject semantic, structural, or dependency-aware context [2407.19487, 2509.25257, 2512.04538]. 

Current SOTA for 7B–16B LLMs with advanced retrieval and static analysis achieves EM up to 44% (Java; GrepRAG) and ~35% (Python; STALL⁺ + RAG) [2601.23254, 2406.10018], but absolute performance remains well below single-file tasks.

## 6. Evaluation Procedure and Statistical Analysis

Recent work also addresses multi-metric and statistically robust evaluation across CrossCodeEval’s multi-lingual splits [2501.18243]. For a given system and dataset ($D_j$), metrics are paired per sample; paired t-tests and Cohen’s $d$ are the default tools for testing significance. Aggregate system rankings employ within-language score standardization and across-language harmonic mean p-value combination.

Visualization includes boxplots for value and rank, with statistical cliques for indistinguishable systems. Leading LLMs such as CodeLlama-13B-hf, CodeGemma-7B, and Granite-34B have been empirically ranked under this protocol, revealing significant and robust performance differences between contemporary models [2501.18243].

## 7. Limitations, Applications, and Future Directions

While CrossCodeEval has become the reference evaluation suite for cross-file code completion, several limitations remain explicit in the literature:

- The benchmark is by construction zero-shot; no public training split is offered [2310.11248, 2407.19487, 2512.04538].
- Average snippet and prompt lengths are moderate (~15 tokens/line; prompts ~900–1,000 tokens), which stresses context retrieval but not very long-form generation [2310.11248, 2406.10018].
- Most methods report results only on Python and Java; TypeScript and C♯ have less extensive published results [2310.11248].
- Masking policies (e.g., on imports) affect challenge severity and comparability for certain retrieval baselines [2507.20888].
- Identifier-based metrics, while informative, do not fully guarantee semantic correctness; execution-based or functional correctness measures (e.g., unit-test pass rate) are rare [2403.10059].

Applications of CrossCodeEval span leaderboard evaluations, retrieval algorithm benchmarking, and ablation studies in code LLM and RAG systems. Future work cited in the literature converges on enhancing retrieval efficiency (hybrid lexical/semantic/graph retrievers), integrating program structure at finer granularity, maintaining up-to-date indices for evolving codebases, and improving static analysis for dynamic languages [2509.25257, 2512.04538, 2406.10018].

CrossCodeEval remains a critical testbed for measuring progress in context-aware, repository-level code understanding and completion, establishing a high bar for both retrieval and generation components in modern code language models.

Source: https://www.emergentmind.com/topics/crosscodeeval