DEPENDEVAL: Code Dependency Benchmark

• DEPENDEVAL is a benchmark for repository-level code dependency reasoning, addressing gaps left by function- or file-level evaluations.
• It features a three-tier task hierarchy—Dependency Recognition, Repository Construction, and Multi-file Editing—that challenges LLMs from syntactic extraction to architectural reasoning.
• Empirical results show that while LLM performance improves on basic dependency tasks with scale, multi-file editing remains challenging, underlining the need for repository-aware training.

DEPENDEVAL is a benchmark and systematic protocol for evaluating the capacity of LLMs for repository-level code dependency reasoning, in direct contrast to earlier function-level or file-level evaluation suites. Designed around a carefully curated multilingual dataset and a hierarchy of real-world programming tasks, it enables both precise quantitative assessment and detailed qualitative insight into the challenges of automated codebase understanding at scale (Du et al., 9 Mar 2025).

1. Motivation and Benchmark Positioning

The emergence of LLMs capable of code generation prompted new questions about their readiness for complex software engineering workflows. Real-world development requires manipulating and understanding dependencies across large, multi-file, and multilingual code repositories. Preceding benchmarks, such as HumanEval, RepoBench, and CrossCodeEval, only captured function-level or single-file competencies, omitting the central role of cross-file invocation graphs, project layouts, and coordinated multi-file refactoring found in practice (Du et al., 9 Mar 2025). DependEval was developed explicitly to fill this methodological gap.

2. Hierarchical Task Design

DependEval consists of a three-task hierarchy, each escalating in complexity:

1. Dependency Recognition (DR):
   • Input: Set of files $F = \{f_1, ..., f_n\}$
   • Objective: Infer the directed invocation graph $D = \{ (f_i, f_j) \mid f_i$ directly invokes $f_j \}$
   • Output: An ordered file chain $P = [p_1, ..., p_n]$ such that for all $(f_i, f_j) \in D$, $T^{-1}(f_i) < T^{-1}(f_j)$ (i.e., called files come after callers).
2. Repository Construction (RC):
   • Input: Natural-language repository requirement $R$ and per-file descriptions $\{(f_i, d_i)\}$
   • Objective: Recover all invocation chains $\{P_1, ..., P_m\}$ consistent with $R$, where each chain specifies a partial ordering of function or module calls across files.
3. Multi-file Editing (ME):
   • Input: An invocation chain $D = \{ (f_i, f_j) \mid f_i$0 and a new feature description $D = \{ (f_i, f_j) \mid f_i$1
   • Objective: Propose coordinated modifications (edits or additions) such that the original dependency ordering is preserved, across both in-place (updates within $D = \{ (f_i, f_j) \mid f_i$2) and expansion (adding new files/features) scenarios (Du et al., 9 Mar 2025).

This hierarchical organization stresses abilities ranging from syntactic dependency extraction to architectural reasoning and distributed editing across diverse codebase layouts.

3. Dataset Collection and Statistics

DependEval leverages 15,576 high-quality repositories from public sources (notably Github) representing eight major programming languages (C, C++, C#, Java, JavaScript, TypeScript, PHP, Python). Repository filtering ensured recency, moderate size, a minimal activity and documentation standard, and linguistic focus.

• File pre-selection: Criteria included average and maximal line lengths, alphabetic density, and the presence of project-relevant documentation.
• Test sample curation: After parsing imports and extracting call chains (length 2–4), human annotation refined a test set to 2,683 multi-language instances spanning the three tasks.
• Sample/task/language distribution: Each language is represented with 245–460 samples per task. Inputs averaged 520–675 tokens; outputs ranged from 30 (DR) to 300 (ME-expansion) when using Qwen2.5-Coder's tokenizer (Du et al., 9 Mar 2025).

4. Evaluation Strategy and Performance Metrics

The benchmark specifies both prompting protocols and metrics for each task:

• Prompting modes: DR and RC use one-shot prompting, while ME uses zero-shot.
• Metrics:
  • DR: Exact Match Rate (EMR): $D = \{ (f_i, f_j) \mid f_i$3
  • RC: Directed graph-based node and edge F1-scores combined as $D = \{ (f_i, f_j) \mid f_i$4
  • ME: Human/LLM-judged scores on Correctness, Purpose Alignment, Functionality Accuracy, Completeness, and Code Quality, averaged with specified weights

25+ state-of-the-art LLMs—including code-specialized (DeepSeek, Qwen-Coder, StarCoder) and large general models (Llama3, Claude-3.5, GPT-4o-mini, etc)—were evaluated identically on a single held-out test set, with no additional training or filtering (Du et al., 9 Mar 2025).

5. Empirical Results and Diagnostic Insights

Summary of findings:

• Task hardness: DR and RC see monotonic improvement with scale, but ME shows a pronounced plateau, with even 70B parameter models and closed APIs rarely exceeding 47–48% accuracy.
• Language differences: Python and JavaScript yield higher accuracy (due to explicit imports and dynamic typing); C/C++ challenges are mostly due to pointer/header dependency analysis; C# and PHP also lag, especially on ME.
• Code-specialization vs. size: Models such as DeepSeek-V3 and Qwen2.5-32B-Instruct consistently outperform much larger general-purpose models in DR and RC. Closed-source APIs (such as Claude-3.5 and GPT-4o-mini) deliver more stable performance on ME.
• Scalability limitations: Pure scale delivers diminishing returns: RC→ME accuracy stagnation suggests the need for inductive architectural bias, repository-aware pretraining, and specialist fine-tuning (Du et al., 9 Mar 2025).

Key challenges for LLMs:

• Dependency parsing: Many models falter on indirect/transitive dependencies.
• Architectural reasoning: Natural-language to multi-file call chain extraction is especially error-prone without architectural priors.
• Cross-file consistency: Simultaneous edits across multiple files quickly degrade in quality and consistency.
• Memory/context limitations: Context window bottlenecks limit multi-module coherence.

6. Implications and Recommendations for Future Work

DependEval validates the pressing need to move beyond single-file code benchmarks and develop models explicitly attuned to real-world repository challenges. The reported failure modes and plateau effects highlight:

• The inadequacy of pure scale for structured, repository-level code understanding
• The requirement for enhanced training signals explicitly targeting cross-file, architectural, and semantic dependency tracking
• A potential role for structured retrieval, repository graph input representations, and cross-file consistency objectives in next generation LLMs
• The necessity to evaluate the fidelity of proposed dependency chains and edits not just via textual similarity, but through invocation-graph and functionality-preservation metrics

DependEval therefore serves as both a public, reproducible benchmark and as a strategic research agenda for the machine learning and programming languages communities, focusing efforts on the unsolved problems of repository-level reasoning in code LLMs (Du et al., 9 Mar 2025).