---
title: 'DEPENDEVAL: Code Dependency Benchmark'
url: https://www.emergentmind.com/topics/dependeval
type: topic
---

# DEPENDEVAL: Code Dependency Benchmark

DEPENDEVAL is a benchmark and systematic protocol for evaluating the capacity of large language models (LLMs) for repository-level code dependency reasoning, in direct contrast to earlier function-level or file-level evaluation suites. Designed around a carefully curated multilingual dataset and a hierarchy of real-world programming tasks, it enables both precise quantitative assessment and detailed qualitative insight into the challenges of automated codebase understanding at scale [2503.06689].

## 1. Motivation and Benchmark Positioning

The emergence of LLMs capable of code generation prompted new questions about their readiness for complex software engineering workflows. Real-world development requires manipulating and understanding dependencies across large, multi-file, and multilingual code repositories. Preceding benchmarks, such as HumanEval, RepoBench, and CrossCodeEval, only captured function-level or single-file competencies, omitting the central role of cross-file invocation graphs, project layouts, and coordinated multi-file refactoring found in practice [2503.06689]. DependEval was developed explicitly to fill this methodological gap.

## 2. Hierarchical Task Design

DependEval consists of a three-task hierarchy, each escalating in complexity:

1. **Dependency Recognition (DR):**
   - *Input:* Set of files \( F = \{f_1, ..., f_n\} \)
   - *Objective:* Infer the directed invocation graph \( D = \{ (f_i, f_j) \mid f_i \) directly invokes \( f_j \} \)
   - *Output:* An ordered file chain \( P = [p_1, ..., p_n] \) such that for all \( (f_i, f_j) \in D \), \( T^{-1}(f_i) < T^{-1}(f_j) \) (i.e., called files come after callers).

2. **Repository Construction (RC):**
   - *Input:* Natural-language repository requirement \( R \) and per-file descriptions \( \{(f_i, d_i)\} \)
   - *Objective:* Recover all invocation chains \( \{P_1, ..., P_m\} \) consistent with \( R \), where each chain specifies a partial ordering of function or module calls across files.

3. **Multi-file Editing (ME):**
   - *Input:* An invocation chain \( C = \{f_1, ..., f_k\} \) and a new feature description \( R \)
   - *Objective:* Propose coordinated modifications (edits or additions) such that the original dependency ordering is preserved, across both in-place (updates within \( C \)) and expansion (adding new files/features) scenarios [2503.06689].

This hierarchical organization stresses abilities ranging from syntactic dependency extraction to architectural reasoning and distributed editing across diverse codebase layouts.

## 3. Dataset Collection and Statistics

DependEval leverages 15,576 high-quality repositories from public sources (notably Github) representing eight major programming languages (C, C++, C#, Java, JavaScript, TypeScript, PHP, Python). Repository filtering ensured recency, moderate size, a minimal activity and documentation standard, and linguistic focus. 

- **File pre-selection:** Criteria included average and maximal line lengths, alphabetic density, and the presence of project-relevant documentation.
- **Test sample curation:** After parsing imports and extracting call chains (length 2–4), human annotation refined a test set to 2,683 multi-language instances spanning the three tasks.
- **Sample/task/language distribution:** Each language is represented with 245–460 samples per task. Inputs averaged 520–675 tokens; outputs ranged from 30 (DR) to 300 (ME-expansion) when using Qwen2.5-Coder's tokenizer [2503.06689].

## 4. Evaluation Strategy and Performance Metrics

The benchmark specifies both prompting protocols and metrics for each task:

- **Prompting modes:** DR and RC use one-shot prompting, while ME uses zero-shot.
- **Metrics:**
  - *DR*: Exact Match Rate (EMR): \(\frac{1}{N}\sum_{i=1}^N \mathbf{1}[P_i = G_i]\)
  - *RC*: Directed graph-based node and edge F1-scores combined as \(\mathrm{F1}_{\mathrm{com}} = 0.15\,\mathrm{F1}_n + 0.85\,\mathrm{F1}_e\)
  - *ME*: Human/LLM-judged scores on Correctness, Purpose Alignment, Functionality Accuracy, Completeness, and Code Quality, averaged with specified weights

25+ state-of-the-art LLMs—including code-specialized (DeepSeek, Qwen-Coder, StarCoder) and large general models (Llama3, Claude-3.5, GPT-4o-mini, etc)—were evaluated identically on a single held-out test set, with no additional training or filtering [2503.06689].

## 5. Empirical Results and Diagnostic Insights

**Summary of findings:**

- *Task hardness:* DR and RC see monotonic improvement with scale, but ME shows a pronounced plateau, with even 70B parameter models and closed APIs rarely exceeding 47–48% accuracy.
- *Language differences:* Python and JavaScript yield higher accuracy (due to explicit imports and dynamic typing); C/C++ challenges are mostly due to pointer/header dependency analysis; C# and PHP also lag, especially on ME.
- *Code-specialization vs. size:* Models such as DeepSeek-V3 and Qwen2.5-32B-Instruct consistently outperform much larger general-purpose models in DR and RC. Closed-source APIs (such as Claude-3.5 and GPT-4o-mini) deliver more stable performance on ME.
- *Scalability limitations:* Pure scale delivers diminishing returns: RC→ME accuracy stagnation suggests the need for inductive architectural bias, repository-aware pretraining, and specialist fine-tuning [2503.06689].

**Key challenges for LLMs:**

- *Dependency parsing:* Many models falter on indirect/transitive dependencies.
- *Architectural reasoning:* Natural-language to multi-file call chain extraction is especially error-prone without architectural priors.
- *Cross-file consistency:* Simultaneous edits across multiple files quickly degrade in quality and consistency.
- *Memory/context limitations:* Context window bottlenecks limit multi-module coherence.

## 6. Implications and Recommendations for Future Work

DependEval validates the pressing need to move beyond single-file code benchmarks and develop models explicitly attuned to real-world repository challenges. The reported failure modes and plateau effects highlight:

- *The inadequacy of pure scale for structured, repository-level code understanding*
- *The requirement for enhanced training signals explicitly targeting cross-file, architectural, and semantic dependency tracking*
- *A potential role for structured retrieval, repository graph input representations, and cross-file consistency objectives in next generation LLMs*
- *The necessity to evaluate the fidelity of proposed dependency chains and edits not just via textual similarity, but through invocation-graph and functionality-preservation metrics*

DependEval therefore serves as both a public, reproducible benchmark and as a strategic research agenda for the machine learning and programming languages communities, focusing efforts on the unsolved problems of repository-level reasoning in code LLMs [2503.06689].

Source: https://www.emergentmind.com/topics/dependeval