--- title: 'DI-BENCH: LLM Dependency Inference Benchmark' url: https://www.emergentmind.com/topics/di-bench type: topic --- # DI-BENCH: LLM Dependency Inference Benchmark DI-BENCH is a comprehensive benchmark for evaluating large language models (LLMs) on the task of dependency inference—specifically, reconstructing all internal and external package dependencies required to build and test real-world software repositories. This problem has emerged as fundamental for automated code generation, with dependency inference errors accounting for over 40% of observed runtime failures in LLM-generated repositories. DI-BENCH combines scale, diversity, and execution-based assessment, uniquely measuring whether a candidate dependency list permits full, automated verification via each repository’s original continuous integration (CI) environment [2501.13699]. ## 1. Benchmark Composition and Dataset Construction DI-BENCH is constructed from a curated collection of 581 open-source software repositories spanning four major programming languages: Python, C#, Rust, and JavaScript. Repository selection emphasizes both real-world relevance and CI-testability. The inclusion criteria are as follows: - Minimum 100 GitHub stars; - Repository size ≤ 10 MB; - Presence of a `.github/workflows` directory for GitHub Actions support. The dataset is stratified into two size tiers: | Subset | Number of repos | Avg. files/repo | Avg. dependencies | Avg. tokens/repo | |----------|----------------|-----------------|-------------------|------------------| | Regular | 387 | 33.9 | 11.9 | 29 K | | Large | 194 | 218.0 | 29.7 | 385 K | Repositories were selected to reflect a long-tail real-world distribution. Python and JavaScript dominate the small-to-medium regime, while Rust and C# represent more complex, dependency-heavy codebases. Each repository is rendered testable by leveraging its native CI pipeline. An automated harness uses the open-source `act` runner to execute the original GitHub Actions test jobs, ensuring no manual reconfiguration or environment hacking. A semi-automated pipeline first locates repo test jobs using LLMs, then locally validates that these jobs run in a sandboxed environment, only retaining successful runs within the benchmark. Dependency masking is performed by removing or blanking out dependency-declaration sections (e.g., `[project.dependencies]` in `pyproject.toml`, `` in `.csproj` files). The dependency inference task is then defined as reconstructing these masked sections such that the repository passes all tests under its original CI workflow [2501.13699]. ## 2. Dependency Inference Task Formalization Given as input a software repository $R$ and its build configuration files $\{b_i^m\}$ with dependency-declaration regions masked, the dependency inference problem is to output the set $\{b_i\}$, where each $b_i$ restores the correct dependency definitions: \[ \mathcal{F} \colon (R, \{b_i^m\}_{i=1}^k) \rightarrow \{b_i\}_{i=1}^k \] Here, each $b_i^m$ (e.g., `pyproject.toml` with dependencies removed) must be reconstructed so that inserting the predicted dependencies yields a fully functional build, passing all repository tests. The ground-truth is precisely the dependency list that enables successful test completion in the original environment. This formulation captures all sources of dependency error: missing/extraneous packages, incorrect version constraints, local/internal modules, extras/features, and ecosystem consistency. ## 3. Evaluation Protocol and Performance Metrics DI-BENCH applies both textual and execution-based evaluation. The primary axes are: - **Precision, Recall, F₁:** These are computed per repository by comparing the set of predicted dependencies $D_\text{pred}$ to the ground-truth $D_\text{gt}$: \[ \text{Precision} = \frac{|D_\text{pred} \cap D_\text{gt}|}{|D_\text{pred}|} \] \[ \text{Recall} = \frac{|D_\text{pred} \cap D_\text{gt}|}{|D_\text{gt}|} \] \[ F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} \] - **Execution (Executability) Rate:** The proportion of instances where, after inserting $D_\text{pred}$ into the masked build files, all CI tests pass successfully. This metric captures holistic correctness: any omission, version error, hallucination (“fake” dependency), or misspecification results in failure. - **Fake Rate:** The fraction of generated dependencies not found in either the relevant package ecosystem or the local project tree. Execution-based metrics are prioritized, as successful repository build and test serves as the definitive end-to-end verification [2501.13699]. ## 4. Experimental Setup and Prompting Paradigms DI-BENCH supports several prompting and inference strategies for LLM evaluation: 1. **All-In-One:** Concatenate all source files plus masked build files into a single input prompt. The model predicts all dependency sections in one forward pass. 2. **File-Iterate:** For each source file, prompt the model for required dependencies, aggregate results, then prompt for the merged final build. 3. **Imports-Only:** Extract only import/use statements (via `tree-sitter`), provide these and the masked build files as model context. A diverse set of models was benchmarked, including proprietary (GPT-4o, GPT-4o-mini) and open-source (Llama-3.1-8B-Instruct, DeepSeek-Coder-V2-Lite-Instruct, Qwen2.5-Coder-Instruct) architectures, all supporting context windows up to 128 K tokens on 4 × A100 GPUs. Ablations include oracle substitution of model-predicted versions with ground-truth constraints (quantifying the specific impact of version errors), and hallucination filtering (removal of fake dependencies before test execution) [2501.13699]. ## 5. Core Results and Observations The state-of-the-art GPT-4o achieves only a 42.9% execution pass rate on the regular Python subset (≤120 K tokens) under the All-In-One strategy; all models perform substantially worse on larger repositories, with frequent context-window overflows or severe performance degradation. Selected cross-model, cross-language results (All-In-One, regular repos): | Model | Python Exec | Rust Exec | JS Exec | Precision (Py) | Recall (Py) | F₁ (Py) | Fake Rate (Py) | |-------------------|-------------|-----------|---------|---------------|-------------|---------|----------------| | GPT-4o | 42.9% | 11.2% | 43.2% | 61.8% | 73.6% | 67.2% | 2.8% | | GPT-4o-mini | 24.5% | 4.7% | 24.9% | 56.5% | 57.5% | 57.0% | 2.0% | | Llama-3.1-8B | 13.3% | 5.0% | 21.9% | 28.8% | 38.4% | 32.9% | 4.3% | | DeepSeek-16B (MoE)| 17.3% | 8.9% | 26.8% | 48.0% | 48.6% | 48.3% | 18.4% | | Qwen2.5-7B | 22.4% | 6.5% | 16.7% | 55.4% | 44.7% | 49.5% | 5.3% | For the large subset, the execution rate is even lower; most All-In-One methods exceed available context memory, and modular strategies (e.g., File-Iterate, Imports-Only) incur compound inference errors, with execution rates frequently below 10% for all languages except JavaScript [2501.13699]. The dominant failure mode is missing dependencies, followed by erroneous or hallucinated package names and incorrect version specifications. Versioning errors are especially damaging—substituting ground-truth constraints (“oracle metadata”) boosts Python execution from 42.9% to 55.1%, and Rust from 11.2% to 38.8%. Hallucination removal provides only incremental improvement, suggesting that most failures arise from omission rather than commission. All models display negative scalability: as repository size (tokens, files, dependencies) grows, executable pass rates drop monotonically, exposing current LLMs’ limitations in long-context reasoning and holistic codebase understanding. ## 6. Challenges Identified and Prospective Solutions DI-BENCH reveals critical gaps in current LLMs for end-to-end software synthesis: - **Long-context fidelity:** All-In-One methods fail outright on projects with token counts near current context limits (≥120K tokens), while modularized splitting (File-Iterate or Imports-Only) fails to preserve global dependency integrity. - **Metadata and version constraint reasoning:** Precise versions, extras, features, and inter-package conditions remain a severe bottleneck. - **Hallucination and ecosystem grounding:** Even low fake rates (2–6%) materially impact real-world CI performance; models lack robust external validation mechanisms. - **Open-source vs. proprietary performance disparity:** Proprietary models (e.g., GPT-4o) continue to outperform open-source LLMs by large margins, particularly on execution-critical tasks. Suggested improvement avenues include hybrid pipelines that combine static analysis (e.g., AST import resolution) with LLM completion, dynamic retrieval of registry metadata, retrieval-augmented and multi-agent inference procedures for iterative constraint refinement, and continued scaling of LLM context capability [2501.13699]. ## 7. Significance and Impact DI-BENCH is the first benchmark to isolate dependency inference as a stand-alone, execution-verified capability in LLM-based code generation. By combining a large, multi-language corpus with stringent, reproducible CI-based assessment, DI-BENCH sets a clear standard against which future model improvements can be measured. The observed maximum execution pass rate of 43% (for “small” Python repos) underscores that reliable automated software construction remains unsolved. Given the centrality of dependency inference to real-world repository usability, DI-BENCH positions this problem as a core blocking challenge for robust, end-to-end generative software engineering [2501.13699]. Source: https://www.emergentmind.com/topics/di-bench