LibEvoBench: Versioned API Benchmark

Updated 5 July 2026

LibEvoBench is a multi-task benchmark that measures LLMs' ability to correctly use version-specific APIs, highlighting errors when using outdated API parameters.
It employs dual evidence from Sphinx documentation and runtime introspection to verify API presence across 29 library versions in PyTorch, NumPy, and SciPy.
The benchmark demonstrates that current models perform strongly on stable APIs but struggle with evolving ones, revealing significant temporal knowledge gaps.

LibEvoBench is a multi-task, version-indexed benchmark for evaluating whether code generation models can use, identify, and parameterize APIs correctly for a specified library version rather than for whichever version is most common in training data. It is built around the claim that contemporary code LLMs exhibit weak temporal knowledge stratification: they compress code and documentation from many releases into a single parametric representation and therefore make anachronistic errors, namely predictions that correspond to a real API or parameter from the wrong point in a library’s history. The benchmark’s motivating example is torch.optim.lr_scheduler.StepLR in PyTorch 2.7, where adding verbose=False is plausible but temporally incorrect because that parameter was deprecated by v2.2 (Cipollone et al., 24 Jun 2026).

1. Conceptual basis

LibEvoBench addresses a software engineering setting in which projects pin dependencies to exact releases, often older ones. In that setting, correctness is version-conditional: whether a symbol exists, whether it is public, and what signature it exposes all depend on the target version. The benchmark therefore treats version-aware API use as a distinct capability, not as a minor variant of ordinary code completion. Its central failure mode is not generic hallucination, but temporal misalignment within a real library history. An anachronistic prediction is one that existed somewhere in the library’s evolution but not in the requested target version; by contrast, a hallucinated symbol may never have existed at all (Cipollone et al., 24 Jun 2026).

The benchmark is organized around the distinction between stable APIs and evolving APIs. Stable APIs are present with an identical signature across all covered versions. Evolving APIs are introduced, removed, deprecated, or signature-modified over the covered period. This separation is not merely descriptive. It is the basis for LibEvoBench’s evaluation logic: good performance on stable APIs is necessary but insufficient, because the core target is whether a model can preserve accuracy when the API surface changes over time (Cipollone et al., 24 Jun 2026).

2. Corpus design and benchmark construction

LibEvoBench focuses on the Python ecosystem and covers PyTorch, NumPy, and SciPy. It spans 29 library versions—10 PyTorch versions, 10 NumPy versions, and 9 SciPy versions—and the main text describes over 125k evaluation instances across more than 16k APIs across 29 versions. Table 1 reports post-sampling task sizes of 29,667 API-C samples covering 3,567 APIs, 39,665 API-I samples covering 5,288 APIs, and 55,094 SR samples covering 7,324 APIs, for a total of 124,426 examples (Cipollone et al., 24 Jun 2026).

Library and version selection are usage-driven. The authors crawled about 70k requirements.txt files from public GitHub repositories, filtered for permissive license, at least 10 stars, and deduplication, then selected PyTorch, NumPy, and SciPy from the dominant scientific Python cluster. They extracted exact == specifiers, built empirical version-usage distributions, and retained the top-10 most-used versions per library with support from at least 100 repositories. SciPy 1.5 was excluded because documentation inventory parsing and runtime introspection failed for that version (Cipollone et al., 24 Jun 2026).

LibEvoBench combines two kinds of evidence for each versioned API surface. First, it uses official Sphinx objects.inv files as a documentation inventory of public symbols. Second, it installs each library version in a dedicated environment and performs runtime introspection. An API is treated as present only when both sources agree. This yields an API compatibility matrix

$A \in \{0,1,NA\}^{|F|\times|V|}$

and a signature compatibility matrix

$S \in \{0,1,NA\}^{|P|\times|V|}.$

Cells are NA when documentation and runtime disagree irreconcilably. Those NA cells account for only 3.4% of all matrix cells overall, under 1% on SciPy and around 5% on PyTorch (Cipollone et al., 24 Jun 2026).

The API-C corpus is mined from real repositories pinned to exact library versions. For each target library, the pipeline identifies repositories whose requirements.txt pins that library, clones the repository at the manifest commit, scans Python files with static analysis, tracks imports, assignments, and alias propagation, and resolves call sites to fully qualified public API names using Jedi inside an isolated environment created with uv venv. API-I and SR are then derived from the compatibility matrices and documentation dictionary. Sampling is deterministic and coverage-oriented: evolving APIs are retained in full, while stable APIs are subsampled with parameters up to $C = 1{,}000$ APIs per library, $m = 5$ versions per API, and $n = 3$ snippets per cell for API-C. Documentation text is filtered by signature stripping, section truncation, directive truncation, length capping at 600 chars, name redaction, collision deduplication, and version scrubbing; a manual audit of 500 retained short descriptions found 98% preserved informative, discriminative semantics (Cipollone et al., 24 Jun 2026).

3. Tasks and prompting protocol

LibEvoBench evaluates three complementary tasks: contextual code completion, API name identification, and exact signature recall. Together they separate practical code use from declarative API recognition and fine-grained parameter memory. All tasks are version-indexed, but explicit version visibility varies by task and prompt level (Cipollone et al., 24 Jun 2026).

Task	Input	Scoring
API-C	Python snippet with `<FILL_HERE>`; four prompt levels L0–L3	Exact match on API name at L0/L1/L2; parameter-focused comparison at L3
API-I	Library, target version, module prefix, redacted description	Exact match on final API component
SR	Fully qualified API name, library, target version	Parameter F1

API Calling (API-C) is contextual code completion derived from real repository use sites. It has four prompt levels. L0 uses raw code only. L1 adds a redacted docstring describing the target API without naming it. L2 adds an explicit version constraint such as # requires: lib == ver. L3 gives the API name and version, shifting the problem from retrieval to parameter prediction. The benchmark describes these as noise-reduction levels: L0 measures pure contextual completion, L1 factors out retrieval difficulty, L2 tests whether explicit version information helps, and L3 focuses on signature recall once the API is already known (Cipollone et al., 24 Jun 2026).

API Identification (API-I) gives the library name, target version, module path prefix, and a redacted description, then requires only the final API component name. Signature Recall (SR) gives a fully qualified API name, library, and target version, and asks for the complete function or method signature. Response extraction is deterministic: API-C strips fences and echoes before resolving the extracted call, API-I strips to a legal Python identifier, and SR normalizes signatures by removing def, return annotations, and type annotations before parameter extraction. Table 14 reports that API-I and SR failures are almost zero, API-C failures are usually in the low single digits, and Claude Sonnet 4.6 shows somewhat higher API-C failure due to “thinking-token leakage” (Cipollone et al., 24 Jun 2026).

The evaluated models comprise 13 models across four families: OpenAI, Google, Anthropic, and Qwen3.5. All are run with zero temperature where supported, no thinking budget, and greedy decoding for open models, so the benchmark isolates stored versioned knowledge rather than long-horizon reasoning (Cipollone et al., 24 Jun 2026).

4. Error taxonomy and the Software Evolution Understanding Score

A methodological contribution of LibEvoBench is a deterministic error taxonomy derived from the compatibility matrices. At API level, predictions are classified as Correct, Invalid, Anachronistic, or Unknown. “Invalid” means a real public symbol in the target version, but not the intended one. “Anachronistic” means present somewhere in the library history but absent in the target version. “Unknown” means never observed in any covered version. At parameter level, the taxonomy becomes Correct, Anachronistic, and Unknown (Cipollone et al., 24 Jun 2026).

The benchmark’s aggregate metric is the Software Evolution Understanding Score (SEUS). For each version $v$ , let $S_v$ denote the score on stable APIs and $E_v$ the score on evolving APIs. LibEvoBench combines them with a harmonic mean,

$B_v=\frac{2S_vE_v}{S_v+E_v},$

so that strong stable-API performance cannot compensate for weak evolving-API performance. The final SEUS then averages across library-task pairs and subtracts two penalties: a stability penalty based on version-to-version variability in evolving-API performance, and an anachronism penalty based on the anachronistic error rate on evolving APIs. The default weights are $\lambda=\gamma=0.5$ . The benchmark also reports retention $S \in \{0,1,NA\}^{|P|\times|V|}.$ 0 as a diagnostic of how much stable-API performance is preserved on evolving APIs (Cipollone et al., 24 Jun 2026).

This design makes SEUS more restrictive than ordinary average accuracy. It rewards high performance on both stable and evolving APIs, stable behavior across versions, and low temporal confusion. The benchmark explicitly notes that retention alone can be misleading for weak models, because a low-over-low ratio can still appear numerically respectable (Cipollone et al., 24 Jun 2026).

5. Empirical findings

The central empirical result is that current models are largely version-oblivious. Across model families, performance on stable APIs stays roughly flat across the version timeline, whereas performance on evolving APIs declines toward more recent versions. Even the strongest models show a persistent 7–10 point accuracy gap between stable and evolving APIs. Within Qwen3.5, scaling from 35B to 122B to 397B raises the level but leaves the degradation shape largely unchanged, which the paper interprets as evidence that the issue is structural rather than merely a matter of model size (Cipollone et al., 24 Jun 2026).

Documentation materially helps, but explicit version tags do not. In API-C, moving from L0 to L1—adding a redacted API description—produces a consistent 10–20 point gain in exact match across models. By contrast, moving from L1 to L2, which adds an explicit version constraint, yields no measurable improvement. This is one of the benchmark’s strongest findings: models react to the version cue, but accuracy remains essentially unchanged, suggesting that the version string does not activate sufficiently stratified internal knowledge (Cipollone et al., 24 Jun 2026).

The SEUS leaderboard places GPT-5.4 first at 86.0, followed by GPT-5.5 at 85.1, Sonnet 4.6 at 81.0, Gemini-3-flash at 77.3, and Qwen3.5 397B at 67.9. GPT-5.4 ranks above GPT-5.5 despite slightly lower stable performance because it is more consistent across versions, which SEUS explicitly rewards. Per-task results show that the stable–evolving gap is especially visible on Signature Recall, and particularly pronounced in libraries with stronger evolution pressure such as PyTorch (Cipollone et al., 24 Jun 2026).

Error analysis further refines the picture. At API-selection level, anachronistic predictions are usually negligible; most wrong answers are Invalid or Unknown. The paper warns that this should not be mistaken for genuine version awareness, because models often back off to high-frequency APIs that remain valid across many versions. Temporal confusion surfaces more clearly at parameter level. On SR, anachronistic parameter rates generally range 6–14%, and unknown parameters range from 7–42% depending on model and library. Signature Recall therefore acts as a forcing function: unlike code completion, which allows models to hedge by emitting only high-confidence arguments, SR requires commitment to the full parameter list and exposes hidden version confusion (Cipollone et al., 24 Jun 2026).

6. Position in the benchmark literature

LibEvoBench sits within a broader line of work on temporally grounded and evolving benchmarks, but its target is more specific: version-indexed library API knowledge in code generation. "LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation" studies version-specific in-line code completion across eight libraries—torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas—and reports that version-aware RAG yields substantially larger gains than explicit version comments alone (Kuhar et al., 2024). "RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation" shifts the same general problem to Rust, synthesizing 588 API changes from the Rust standard library and 15 third-party crates, with four evolution categories—Stabilizations, Signature Changes, Behavioral Changes, and Deprecations—and showing that Behavioral Changes are the hardest category, at 38.0% average success versus 65.8% on Stabilizations (Liang et al., 21 Mar 2025). At a more general methodological level, "Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM" argues for evolving coding benchmarks rather than static ones, expanding HumanEval into 828 problems across semantic-altering and semantic-preserving transformations (Xia et al., 2024).

Within that landscape, LibEvoBench is narrower in domain than generic evolving-benchmark frameworks and narrower in language coverage than multilingual coding benchmarks, but more explicit about temporal knowledge stratification, version-conditioned API surfaces, and the distinction between anachronism and other error types. Its main contribution is therefore not benchmark freshness alone, but a formalization of version-aware API reasoning as a measurable capability gap (Cipollone et al., 24 Jun 2026).

7. Limitations and implications

LibEvoBench is deliberately scoped. It covers only Python and only three scientific libraries: PyTorch, NumPy, and SciPy. All models are evaluated with zero thinking budget, because the target is stored version-specific knowledge rather than extended reasoning. Some cells in the compatibility matrices remain NA, though the fraction is limited to 3.4% overall. The benchmark relies primarily on repository code, documentation inventories, and introspected runtime surfaces; it does not directly use migration guides, issue discussions, or bug reports. Contamination cannot be fully ruled out, particularly for API-C, because the underlying repositories are public GitHub artifacts, although the paper argues against contamination being the dominant explanation for the observed version-correlated degradation (Cipollone et al., 24 Jun 2026).

The benchmark’s practical conclusion is correspondingly specific. Modern code LLMs are strong on general API knowledge and on stable APIs, but weak on version-specific API knowledge, especially for evolving signatures and parameters. Merely specifying the target version is ineffective; supplying relevant documentation is materially helpful. For deployment, this points toward retrieval of version-matched documentation conditioned on the project’s dependency manifest rather than reliance on parametric memory alone. For research, it establishes temporal knowledge stratification as a concrete benchmark dimension for code generation and provides publicly released artifacts through a Hugging Face dataset and a Zenodo-hosted code and pipeline release (Cipollone et al., 24 Jun 2026).