---
title: 'LibEvoBench: Versioned API Benchmark'
url: https://www.emergentmind.com/topics/libevobench
type: topic
---

# LibEvoBench: Versioned API Benchmark

LibEvoBench is a multi-task, version-indexed benchmark for evaluating whether code generation models can use, identify, and parameterize APIs correctly for a specified library version rather than for whichever version is most common in training data. It is built around the claim that contemporary code LLMs exhibit weak **temporal knowledge stratification**: they compress code and documentation from many releases into a single parametric representation and therefore make **anachronistic errors**, namely predictions that correspond to a real API or parameter from the wrong point in a library’s history. The benchmark’s motivating example is `torch.optim.lr_scheduler.StepLR` in PyTorch 2.7, where adding `verbose=False` is plausible but temporally incorrect because that parameter was deprecated by v2.2 [2606.25402].

## 1. Conceptual basis

LibEvoBench addresses a software engineering setting in which projects pin dependencies to exact releases, often older ones. In that setting, correctness is version-conditional: whether a symbol exists, whether it is public, and what signature it exposes all depend on the target version. The benchmark therefore treats version-aware API use as a distinct capability, not as a minor variant of ordinary code completion. Its central failure mode is not generic hallucination, but temporal misalignment within a real library history. An anachronistic prediction is one that existed somewhere in the library’s evolution but not in the requested target version; by contrast, a hallucinated symbol may never have existed at all [2606.25402].

The benchmark is organized around the distinction between **stable APIs** and **evolving APIs**. Stable APIs are present with an identical signature across all covered versions. Evolving APIs are introduced, removed, deprecated, or signature-modified over the covered period. This separation is not merely descriptive. It is the basis for LibEvoBench’s evaluation logic: good performance on stable APIs is necessary but insufficient, because the core target is whether a model can preserve accuracy when the API surface changes over time [2606.25402].

## 2. Corpus design and benchmark construction

LibEvoBench focuses on the Python ecosystem and covers **PyTorch**, **NumPy**, and **SciPy**. It spans **29 library versions**—**10 PyTorch versions**, **10 NumPy versions**, and **9 SciPy versions**—and the main text describes **over 125k evaluation instances** across **more than 16k APIs across 29 versions**. Table 1 reports post-sampling task sizes of **29,667** API-C samples covering **3,567 APIs**, **39,665** API-I samples covering **5,288 APIs**, and **55,094** SR samples covering **7,324 APIs**, for a total of **124,426** examples [2606.25402].

Library and version selection are usage-driven. The authors crawled about **70k `requirements.txt` files** from public GitHub repositories, filtered for permissive license, at least 10 stars, and deduplication, then selected PyTorch, NumPy, and SciPy from the dominant scientific Python cluster. They extracted exact `==` specifiers, built empirical version-usage distributions, and retained the **top-10 most-used versions per library** with support from at least **100 repositories**. SciPy 1.5 was excluded because documentation inventory parsing and runtime introspection failed for that version [2606.25402].

LibEvoBench combines two kinds of evidence for each versioned API surface. First, it uses official Sphinx `objects.inv` files as a documentation inventory of public symbols. Second, it installs each library version in a dedicated environment and performs runtime introspection. An API is treated as present only when both sources agree. This yields an API compatibility matrix
$$
A \in \{0,1,NA\}^{|F|\times|V|}
$$
and a signature compatibility matrix
$$
S \in \{0,1,NA\}^{|P|\times|V|}.
$$
Cells are `NA` when documentation and runtime disagree irreconcilably. Those `NA` cells account for only **3.4%** of all matrix cells overall, under **1%** on SciPy and around **5%** on PyTorch [2606.25402].

The API-C corpus is mined from real repositories pinned to exact library versions. For each target library, the pipeline identifies repositories whose `requirements.txt` pins that library, clones the repository at the manifest commit, scans Python files with static analysis, tracks imports, assignments, and alias propagation, and resolves call sites to fully qualified public API names using **Jedi** inside an isolated environment created with `uv venv`. API-I and SR are then derived from the compatibility matrices and documentation dictionary. Sampling is deterministic and coverage-oriented: evolving APIs are retained in full, while stable APIs are subsampled with parameters up to **\(C = 1{,}000\)** APIs per library, **\(m = 5\)** versions per API, and **\(n = 3\)** snippets per cell for API-C. Documentation text is filtered by signature stripping, section truncation, directive truncation, length capping at 600 chars, name redaction, collision deduplication, and version scrubbing; a manual audit of **500** retained short descriptions found **98%** preserved informative, discriminative semantics [2606.25402].

## 3. Tasks and prompting protocol

LibEvoBench evaluates three complementary tasks: contextual code completion, API name identification, and exact signature recall. Together they separate practical code use from declarative API recognition and fine-grained parameter memory. All tasks are version-indexed, but explicit version visibility varies by task and prompt level [2606.25402].

| Task | Input | Scoring |
|---|---|---|
| **API-C** | Python snippet with `<FILL_HERE>`; four prompt levels L0–L3 | Exact match on API name at L0/L1/L2; parameter-focused comparison at L3 |
| **API-I** | Library, target version, module prefix, redacted description | Exact match on final API component |
| **SR** | Fully qualified API name, library, target version | Parameter F1 |

**API Calling (API-C)** is contextual code completion derived from real repository use sites. It has four prompt levels. **L0** uses raw code only. **L1** adds a redacted docstring describing the target API without naming it. **L2** adds an explicit version constraint such as `# requires: lib == ver`. **L3** gives the API name and version, shifting the problem from retrieval to parameter prediction. The benchmark describes these as noise-reduction levels: L0 measures pure contextual completion, L1 factors out retrieval difficulty, L2 tests whether explicit version information helps, and L3 focuses on signature recall once the API is already known [2606.25402].

**API Identification (API-I)** gives the library name, target version, module path prefix, and a redacted description, then requires only the final API component name. **Signature Recall (SR)** gives a fully qualified API name, library, and target version, and asks for the complete function or method signature. Response extraction is deterministic: API-C strips fences and echoes before resolving the extracted call, API-I strips to a legal Python identifier, and SR normalizes signatures by removing `def`, return annotations, and type annotations before parameter extraction. Table 14 reports that API-I and SR failures are almost zero, API-C failures are usually in the low single digits, and Claude Sonnet 4.6 shows somewhat higher API-C failure due to “thinking-token leakage” [2606.25402].

The evaluated models comprise **13 models across four families**: OpenAI, Google, Anthropic, and Qwen3.5. All are run with **zero temperature** where supported, **no thinking budget**, and greedy decoding for open models, so the benchmark isolates stored versioned knowledge rather than long-horizon reasoning [2606.25402].

## 4. Error taxonomy and the Software Evolution Understanding Score

A methodological contribution of LibEvoBench is a deterministic error taxonomy derived from the compatibility matrices. At API level, predictions are classified as **Correct**, **Invalid**, **Anachronistic**, or **Unknown**. “Invalid” means a real public symbol in the target version, but not the intended one. “Anachronistic” means present somewhere in the library history but absent in the target version. “Unknown” means never observed in any covered version. At parameter level, the taxonomy becomes **Correct**, **Anachronistic**, and **Unknown** [2606.25402].

The benchmark’s aggregate metric is the **Software Evolution Understanding Score (SEUS)**. For each version \(v\), let \(S_v\) denote the score on stable APIs and \(E_v\) the score on evolving APIs. LibEvoBench combines them with a harmonic mean,
$$
B_v=\frac{2S_vE_v}{S_v+E_v},
$$
so that strong stable-API performance cannot compensate for weak evolving-API performance. The final SEUS then averages across library-task pairs and subtracts two penalties: a **stability penalty** based on version-to-version variability in evolving-API performance, and an **anachronism penalty** based on the anachronistic error rate on evolving APIs. The default weights are \(\lambda=\gamma=0.5\). The benchmark also reports **retention** \(E_v/S_v\) as a diagnostic of how much stable-API performance is preserved on evolving APIs [2606.25402].

This design makes SEUS more restrictive than ordinary average accuracy. It rewards high performance on both stable and evolving APIs, stable behavior across versions, and low temporal confusion. The benchmark explicitly notes that retention alone can be misleading for weak models, because a low-over-low ratio can still appear numerically respectable [2606.25402].

## 5. Empirical findings

The central empirical result is that current models are largely **version-oblivious**. Across model families, performance on stable APIs stays roughly flat across the version timeline, whereas performance on evolving APIs declines toward more recent versions. Even the strongest models show a persistent **7–10 point accuracy gap** between stable and evolving APIs. Within Qwen3.5, scaling from **35B** to **122B** to **397B** raises the level but leaves the degradation shape largely unchanged, which the paper interprets as evidence that the issue is structural rather than merely a matter of model size [2606.25402].

Documentation materially helps, but explicit version tags do not. In API-C, moving from **L0** to **L1**—adding a redacted API description—produces a consistent **10–20 point** gain in exact match across models. By contrast, moving from **L1** to **L2**, which adds an explicit version constraint, yields **no measurable improvement**. This is one of the benchmark’s strongest findings: models react to the version cue, but accuracy remains essentially unchanged, suggesting that the version string does not activate sufficiently stratified internal knowledge [2606.25402].

The SEUS leaderboard places **GPT-5.4** first at **86.0**, followed by **GPT-5.5** at **85.1**, **Sonnet 4.6** at **81.0**, **Gemini-3-flash** at **77.3**, and **Qwen3.5 397B** at **67.9**. GPT-5.4 ranks above GPT-5.5 despite slightly lower stable performance because it is more consistent across versions, which SEUS explicitly rewards. Per-task results show that the stable–evolving gap is especially visible on **Signature Recall**, and particularly pronounced in libraries with stronger evolution pressure such as PyTorch [2606.25402].

Error analysis further refines the picture. At API-selection level, anachronistic predictions are usually negligible; most wrong answers are **Invalid** or **Unknown**. The paper warns that this should not be mistaken for genuine version awareness, because models often back off to high-frequency APIs that remain valid across many versions. Temporal confusion surfaces more clearly at parameter level. On **SR**, anachronistic parameter rates generally range **6–14%**, and unknown parameters range from **7–42%** depending on model and library. Signature Recall therefore acts as a forcing function: unlike code completion, which allows models to hedge by emitting only high-confidence arguments, SR requires commitment to the full parameter list and exposes hidden version confusion [2606.25402].

## 6. Position in the benchmark literature

LibEvoBench sits within a broader line of work on temporally grounded and evolving benchmarks, but its target is more specific: version-indexed library API knowledge in code generation. "LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation" studies version-specific in-line code completion across **eight libraries**—`torch`, `torchvision`, `scipy`, `pil`, `tqdm`, `pyyaml`, `matplotlib`, and `pandas`—and reports that **version-aware RAG** yields substantially larger gains than explicit version comments alone [2412.04478]. "RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation" shifts the same general problem to Rust, synthesizing **588 API changes** from the Rust standard library and **15 third-party crates**, with four evolution categories—**Stabilizations**, **Signature Changes**, **Behavioral Changes**, and **Deprecations**—and showing that **Behavioral Changes** are the hardest category, at **38.0%** average success versus **65.8%** on **Stabilizations** [2503.16922]. At a more general methodological level, "Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM" argues for evolving coding benchmarks rather than static ones, expanding HumanEval into **828 problems** across semantic-altering and semantic-preserving transformations [2403.19114].

Within that landscape, LibEvoBench is narrower in domain than generic evolving-benchmark frameworks and narrower in language coverage than multilingual coding benchmarks, but more explicit about **temporal knowledge stratification**, **version-conditioned API surfaces**, and the distinction between **anachronism** and other error types. Its main contribution is therefore not benchmark freshness alone, but a formalization of version-aware API reasoning as a measurable capability gap [2606.25402].

## 7. Limitations and implications

LibEvoBench is deliberately scoped. It covers only **Python** and only **three scientific libraries**: PyTorch, NumPy, and SciPy. All models are evaluated with **zero thinking budget**, because the target is stored version-specific knowledge rather than extended reasoning. Some cells in the compatibility matrices remain `NA`, though the fraction is limited to **3.4%** overall. The benchmark relies primarily on repository code, documentation inventories, and introspected runtime surfaces; it does not directly use migration guides, issue discussions, or bug reports. Contamination cannot be fully ruled out, particularly for API-C, because the underlying repositories are public GitHub artifacts, although the paper argues against contamination being the dominant explanation for the observed version-correlated degradation [2606.25402].

The benchmark’s practical conclusion is correspondingly specific. Modern code LLMs are strong on general API knowledge and on stable APIs, but weak on **version-specific** API knowledge, especially for evolving signatures and parameters. Merely specifying the target version is ineffective; supplying **relevant documentation** is materially helpful. For deployment, this points toward retrieval of version-matched documentation conditioned on the project’s dependency manifest rather than reliance on parametric memory alone. For research, it establishes temporal knowledge stratification as a concrete benchmark dimension for code generation and provides publicly released artifacts through a Hugging Face dataset and a Zenodo-hosted code and pipeline release [2606.25402].

Source: https://www.emergentmind.com/topics/libevobench