REQBench: Python Upgrade Compatibility Benchmark
- REQBench is an evaluation benchmark designed to assess Python third-party library upgrades by measuring both version compatibility issues (VCIs) and code compatibility issues (CCIs) through real project execution.
- It employs a systematic methodology using isolated conda environments and targeted upgrade procedures, ensuring that installation success alone does not mask underlying runtime failures.
- The benchmark provides detailed diagnostic annotations—differentiating between direct Project‑TPL and indirect TPL‑TPL failures—thereby enabling precise comparisons between dependency-solving tools and runtime behavior.
Searching arXiv for the specified paper to ground the article in the source record. REQBench is an evaluation benchmark introduced in the PCREQ study to test automatic inference of compatible requirements for Python third-party library upgrades under realistic upgrade conditions. It is designed around the observation that upgrade failures are not exhausted by dependency-resolution conflicts: a target environment may install successfully and still fail at runtime because of code-level incompatibilities. Accordingly, REQBench explicitly covers both version compatibility issues (VCIs) and code compatibility issues (CCIs), including direct Project‑TPL failures and indirect TPL‑TPL failures, and uses actual project execution rather than dependency solving alone as the criterion for success (Lei et al., 4 Aug 2025).
1. Problem setting and motivation
The benchmark is grounded in a specific upgrade scenario: a project has an existing requirements.txt, one designated target TPL is upgraded to a requested version, and the resulting environment is evaluated for both installability and runtime correctness. Within this setting, the PCREQ paper distinguishes two forms of incompatibility. VCIs occur when dependency constraints among packages become unsatisfiable and typically appear as pip dependency-resolution or installation failures. CCIs occur when dependency constraints are satisfied, or installation still proceeds despite pip errors, but the program crashes at runtime because of changes in modules, API names, parameters, or API behavior (Lei et al., 4 Aug 2025).
REQBench is motivated by a study of 2,095 real upgrade attempts across 34 real-world projects and 20 TPLs. In that study, pip installs succeeded in 1,955 of 2,095 cases, or 93.32%, yet 368 of those successful installs still crashed at runtime, constituting CCIs. Pip reported install errors in 140 cases, or 6.68%; among those, 38 also crashed when the project was executed, whereas 102 ran normally despite the install error. Across the 406 total runtime crash cases, 211 were Project‑TPL and 195 were TPL‑TPL. This distribution motivates a benchmark in which dependency solving alone is insufficient, and actual execution is required to assess whether an upgrade is compatible (Lei et al., 4 Aug 2025).
The benchmark’s problem framing also reflects the behavior of pip during direct upgrades. When a single TPL is upgraded directly, pip may still install the user-requested version even if its resolver reports a conflict. For REQBench, such cases are not treated as resolved merely because installation completed; any unresolved conflict invalidates an inferred requirements.txt, and any runtime crash is labeled as a CCI. This makes the benchmark stricter than workflows that stop at installation.
2. Construction methodology
REQBench was built from 34 GitHub projects selected from a public dataset attributed to Lei et al., ISSRE’23. The selection criteria were that a project provide a clear requirements.txt specifying TPLs and versions, and that the project actually use APIs of those TPLs in project code. From these projects, the authors extracted direct dependencies and verified source-level usage, producing 103 project–library pairs spanning 20 widely used TPLs, including torch, torchvision, tensorflow, numpy, pillow, scipy, scikit-learn, torchaudio, transformers, pytorch-lightning, opencv-python, librosa, tensorboardX, nltk, pandas, tqdm, seqeval, scikit-image, and matplotlib (Lei et al., 4 Aug 2025).
For each project–library pair, the benchmark construction pipeline followed a fixed procedure. A fresh conda virtual environment was created for each target version using Anaconda 23.5.2 in order to avoid cross-contamination. The project’s original requirements.txt was installed via pip. Exactly one target TPL was then upgraded to each successive version from the next version after the starting version up to the latest stable version as of 2024‑07‑23 using pip install --upgrade <library>==<version>. The project’s run script was executed, and installation output, runtime behavior, and stack traces were logged. CUDA and cuDNN mismatches were normalized away so that labels focused on dependency and code compatibility rather than GPU packaging idiosyncrasies (Lei et al., 4 Aug 2025).
Each upgrade attempt became a test case. Cases were labeled as Pip Solved if the environment remained functional after the targeted upgrade and project execution, and Pip Unsolved if the environment no longer functioned correctly after the upgrade. In this benchmark terminology, Pip Unsolved includes both fatal dependency-resolution failures and runtime CCIs. The benchmark includes all 406 cases described as “unsolvable by pip” in this sense.
3. Annotation model and test-case contents
REQBench records both coarse outcome labels and finer diagnostic annotations. A VCI is recorded when pip’s resolver reports a dependency conflict on upgrade; the paper gives the example that torchvision==0.7.0 requires torch==1.6.0, while the user requests torch==1.9.0. A CCI is recorded when the upgraded environment crashes at runtime because of code-level changes, even when dependency constraints installed successfully or when pip errors did not prevent execution (Lei et al., 4 Aug 2025).
CCIs are annotated along two axes. The first is level: Project‑TPL denotes a case where the project directly calls a changed TPL API or module, whereas TPL‑TPL denotes indirect use through another dependency. The second is fine-grained type: module removed or relocated, API name removed or renamed, API parameter changed, and API behavior or body changed. This diagnostic scheme is central to REQBench’s role as a testbed for systems that must reason beyond dependency metadata.
Each REQBench test case bundles the project, identified either by path or repository commit; the starting requirements.txt; the Python version used in the study, namely Python 3.9 with conda per-version isolation; the target TPL name; its starting version in requirements.txt; the target version to upgrade to; the observed outcome label; and, for Unsolved cases, a diagnosis tag indicating VCI presence and/or CCI type and level. Logs and stack traces gathered during installation and execution are included as part of the case record (Lei et al., 4 Aug 2025).
A concise summary of the benchmark’s annotated structure is given below.
| Component | Content |
|---|---|
| Starting environment | Project, requirements.txt, Python 3.9, isolated conda environment |
| Upgrade specification | Target TPL name, starting version, requested upgrade version |
| Outcome record | Pip Solved or Pip Unsolved, plus logs and stack traces |
| Diagnostic labels for Unsolved cases | VCI presence and/or CCI level and type |
The benchmark assumes deterministic isolation per attempt and explicitly excludes GPU runtime quirks such as CUDA and cuDNN packaging gaps from labels.
4. Coverage and empirical profile
REQBench comprises 2,095 upgrade test cases, of which 1,689 are Pip Solved and 406 are Pip Unsolved. Among all 2,095 runs, pip reported installation errors in 140 cases, or 6.68%; 38 of those also crashed at runtime, while 102 ran successfully despite the pip resolver warning. Among the 406 runtime crash cases, the level distribution is 211 Project‑TPL cases, or 52.0%, and 195 TPL‑TPL cases, or 48.0% (Lei et al., 4 Aug 2025).
The fine-grained CCI distribution is also reported. Module removals account for 55 cases, split into 33 Project‑TPL and 22 TPL‑TPL. API name removals or renames account for 247 cases, split into 136 Project‑TPL and 111 TPL‑TPL. API parameter changes account for 21 cases, split into 20 Project‑TPL and 1 TPL‑TPL. API body or behavior changes account for 83 cases, split into 22 Project‑TPL and 61 TPL‑TPL.
| CCI type | Total cases | Project‑TPL / TPL‑TPL |
|---|---|---|
| Module removals | 55 | 33 / 22 |
| API name removals/renames | 247 | 136 / 111 |
| API parameter changes | 21 | 20 / 1 |
| API body/behavior changes | 83 | 22 / 61 |
The benchmark spans 20 libraries across ML/DL, data, vision, and NLP stacks; 34 projects; and 103 project–library pairs. The version trajectories include long upgrade sequences, such as torch 1.3.0→1.13.1. The paper identifies this breadth as part of the benchmark’s realism: it captures both familiar resolver conflicts, such as tight coupling between torchvision and torch, and subtler code-level breaks that propagate through import chains and transitive calls, including numpy.testing.nosetester removal affecting SciPy imports and PIL.PILLOW_VERSION removal affecting torchvision (Lei et al., 4 Aug 2025).
Concrete examples from REQBench illustrate these phenomena. In a TPL‑TPL import-chain break, scikit-learn 0.18.1 calls from scipy.misc import comb; SciPy 1.3.0 removed comb, so scikit-learn fails after upgrading SciPy. In another case, numpy.float is removed in 1.24.0 but is used indirectly through torchvision transforms in siamese-pytorch, producing AttributeError: module 'numpy' has no attribute 'float'. These examples show why REQBench labels indirect TPL‑TPL incompatibilities explicitly rather than collapsing them into generic runtime failures.
5. Evaluation protocol and benchmark semantics
REQBench defines success operationally. An approach solves a case if, given the initial requirements.txt and a designated target TPL@version upgrade, it outputs a requirements.txt that is both dependency-consistent for pip install -r and executable without a crash when the project is run (Lei et al., 4 Aug 2025).
The benchmark adopts two metrics. The first is Inference success rate, defined textually as . The second is Inference time (per case), defined as . Timing is measured on isolated conda environments, and averages are also reported, with outlier removal in analysis (Lei et al., 4 Aug 2025).
This protocol makes REQBench substantively different from benchmarks centered only on constraint satisfaction. In REQBench, a candidate environment that satisfies version constraints but fails at runtime does not count as solved. Conversely, the benchmark also captures cases in which pip reports an installation error but the executed project still runs normally, demonstrating that resolver output and runtime validity are not equivalent observables. A plausible implication is that REQBench is designed to stress the gap between declarative dependency metadata and effective runtime compatibility.
The benchmark also interfaces directly with PCREQ’s end-to-end pipeline. REQBench supplies, per case, the project, initial requirements.txt, target TPL and versions, and ground-truth labels. PCREQ then executes six modules: knowledge acquisition, version compatibility assessment, invoked APIs and modules extraction, code compatibility assessment, version change, and missing TPL completion. Within that process, version compatibility assessment encodes candidate versions and constraints into an SMT formula solved with Z3 and optimizes according to
where is the version in the starting requirements for that TPL. Although this optimization belongs to PCREQ rather than REQBench, it clarifies the kind of system behavior that the benchmark is intended to evaluate (Lei et al., 4 Aug 2025).
6. Comparative role, limitations, and reproducibility
REQBench is used in the PCREQ paper to compare PCREQ with pip, PyEGo, ReadPyE, and LLM-based repair-loop approaches using GPT‑4o, DeepSeek V3, and DeepSeek R1. On the full 2,095-case benchmark, PCREQ solves 1,970 cases for a 94.03% success rate, with an average time of 60.79 seconds per case. The pip baseline achieves 80.62% success. By category, PCREQ solves 1,677 of 1,689 Pip Solved cases, or 99.29%, and 293 of 406 Pip Unsolved cases, or 72.17% (Lei et al., 4 Aug 2025).
On the subset covered by PyEGo’s knowledge graph, denoted REQBench‑PyEGo and comprising 1,210 cases, PCREQ achieves 98.26% overall, compared with 37.02% for PyEGo, an improvement of 61.24 percentage points. On the subset covered by ReadPyE’s knowledge graph, denoted REQBench‑ReadPyE and comprising 1,846 cases, PCREQ achieves 95.18% overall, compared with 37.16% for ReadPyE, an improvement of 58.02 percentage points. On the full benchmark, GPT‑4o achieves 75.27%, DeepSeek V3 74.03%, and DeepSeek R1 75.42%; PCREQ is reported to outperform them by 18–20 percentage points.
These results situate REQBench relative to prior benchmarks and tools. The paper states that earlier systems such as Watchman, SmartPip, HELP, and LooCo mainly focus on dependency conflicts, while DockerizeMe, V2, and SnifferDog are file- or notebook-centric and do not analyze cross-TPL code interactions. PyEGo and ReadPyE extend dependency inference with knowledge graphs, but still under-represent code-level incompatibilities, especially TPL‑TPL interactions. REQBench differs by explicitly targeting both VCIs and CCIs, including 406 unsolved-by-pip cases, and by requiring that successful inference survive actual execution rather than merely dependency solving (Lei et al., 4 Aug 2025).
The paper also identifies threats to validity. These include generalization beyond the chosen projects and stacks, classification accuracy for CCIs, and runtime environment idiosyncrasies. The reported mitigations are combined automated and manual validation for classification and normalization away from CUDA/cuDNN issues. A further nuance appears in the error analysis: static matching can misjudge compatibility when an API is retained through internal redirection, as in scikit-learn retaining sklearn.metrics.classification.accuracy_score via an internal import of _classification. This explains a small number of pip-solved yet PCREQ-unsolved cases, reported as 12 total. This suggests that REQBench not only measures compatibility inference but also exposes the limits of purely syntactic API-diff reasoning.
Reproducibility is treated as part of the benchmark artifact. The paper states that all data and tools are publicly available, that REQBench accompanies the evaluation artifact with 2,095 labeled cases and scripts for environment setup and execution, and that experiments use Anaconda 23.5.2, Python 3.9, isolated conda environments, and pip. Per-case logs and labels are retained to enable outcome reproduction, and licensing follows the repository’s terms (Lei et al., 4 Aug 2025).