LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models

Abstract: Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one. However, current LLMs are trained on temporally mixed corpora and lack explicit mechanisms for such version-specific reasoning, leading to anachronistic errors - calling APIs as they exist in a different library version. To systematically evaluate this phenomenon, we introduce LibEvoBench, a multi-task benchmark spanning multiple versions of widely used Python libraries, along with a new metric, the Software Evolution Understanding Score (SEUS), to measure models' consistency when working with evolving APIs. Our results show that state-of-the-art models are largely version-oblivious: performance degrades for evolving APIs, while for stable APIs it remains the same across versions. Moreover, simply specifying the target version provides no benefit, while relevant documentation significantly boosts models' accuracy. These findings highlight a systematic limitation of current training paradigms and motivate new approaches for temporally grounded knowledge in code generation.

• The paper introduces LibEvoBench to evaluate version-aware code generation with tasks like API calling, identification, and signature recall.
• It employs a novel SEUS metric to measure performance differences between stable and evolving APIs, noting a 5–10 point accuracy drop for evolving ones.
• Empirical results show that even state-of-the-art LLMs struggle with temporal knowledge loss, despite improvements from documentation-augmented prompts.

Authoritative Summary of "LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models" (2606.25402)

Motivation and Problem Definition

The paper addresses a persistent challenge in code generation by LLMs: the requirement to generate code using APIs from specific library versions, as opposed to a generic or most recent version. This is highly relevant in practical software engineering, where large-scale projects often pin dependencies to historical library releases due to compatibility, stability, and migration constraints. The lack of temporally stratified API knowledge in current LLMs leads to anachronistic code generation and version confusion, particularly when APIs evolve or deprecate over time.

Benchmark Design: LibEvoBench and SEUS Metric

LibEvoBench is introduced as a comprehensive multi-task benchmark to quantify model performance in version-aware code completion, API identification, and signature recall across major releases of PyTorch, NumPy, and SciPy.

1. Evaluation Tasks:
   • API Calling (API-C): challenges the model to synthesize a contextually correct API call from version-pinned code snippets, under varying augmentation levels (code-only to explicit version specification and documentation).
   • API Identification (API-I): tests ability to select the appropriate API based on redacted descriptions, decoupling retrieval from code context.
   • Signature Recall (SR): requires the model to reproduce the exact function/method signature for a given API and version.
2. Compatibility Matrices & API Stratification:
   • Deterministic matrices built from dual-sourced documentation and runtime introspection track API presence, signature evolution, and deprecations, partitioning APIs into stable (unchanged across versions) and evolving (added, removed, or modified).
3. Hallucination Taxonomy:
   • Errors are rigorously classified at API and parameter granularity: Correct, Invalid, Anachronistic (valid in another version but not the target), Unknown (non-existent symbols). This enables fine-grained analysis of temporal confusion versus fabrication.
4. Software Evolution Understanding Score (SEUS):
   • SEUS aggregates task-level metrics for stable and evolving APIs, penalizing instability (inconsistent performance across versions) and anachronistic errors (temporal confusion), ensuring high scores reflect genuine version-aware capability.

Empirical Findings

Evaluation covers 13 models spanning four model families (GPT-4.1—5.5, Gemini 2.0—3, Sonnet 4—4.6, Qwen3.5 35B—397B), using zero-temperature, greedy decoding, and standardized prompt templates.

• Version-Obliviousness: Frontier models consistently degrade on evolving APIs, even when provided explicit version constraints. Accuracy for stable APIs is flat across versions, while performance for evolving APIs drops, particularly in recent releases.
• Context Sensitivity: Adding relevant documentation to prompts improves API-C accuracy by 10–20 points, but explicit version tags do not yield measurable gains.
• Scale and Recency Limitations: State-of-the-art, large models (GPT-5.x, Sonnet 4.6, Gemini-3) exhibit only partial recovery; performance gap endures, suggesting structural loss of version-differentiated knowledge during training.
• Structural Limitation: Increasing model scale or updating pretraining data does not rectify temporal stratification deficiencies. Models fail primarily by retrieving plausible, valid APIs from a different version, not by hallucination.
• Strong Numerical Results:
  • Top SEUS scores: GPT-5.4 (86.0), GPT-5.5 (85.1), Sonnet 4.6 (81.0), Gemini-3-flash (77.3), Qwen3.5 397B (67.9).
  • Even top models suffer 5–10 point accuracy loss on evolving APIs relative to stable ones. Retention metric (Evolving/Stable) shows >93% for best models but mid-tier models drop below 86%.
  • Anachronistic parameter errors in SR task reach 6–14%, indicating persistent temporal confusion.

Theoretical and Practical Implications

The findings empirically demonstrate a structural bottleneck in current LLM training paradigms: temporally mixed corpora collapse version-specific facts, leaving models incapable of fine-grained version-grounded reasoning. This insight extends prior work on factual knowledge evolution in NLP and deprecated API usage (Kuhar et al., 2024, Wang et al., 2024), grounding it at scale in applied code generation scenarios.

Practical Impacts:

• Model-assisted coding tools will continue to produce anachronistic or invalid code unless version-awareness is explicitly engineered.
• Documentation retrieval notably ameliorates API confusion but does not solve the underlying temporal stratification gap.
• Fine-grained version-specific benchmarks, such as LibEvoBench, are essential for advancing continual and temporal learning capabilities in code models.

Theoretical Impacts:

• Highlights the need to revisit training objectives: compression of temporally heterogeneous corpora without stratification results in a loss of version-differentiated knowledge.
• Evidences that temporal alignment, continual learning, and version conditioning require novel architectural innovations and dataset curation strategies.

Future Directions

The systematic limitation revealed by LibEvoBench motivates future research along several axes:

• Continual and temporal learning: Integrate version-indexed retrieval, temporal alignment modules, and continual updating to reduce anachronistic errors.
• Prompt-augmented retrieval: Blend in-context retrieval with external version documentation and temporal signals.
• Benchmarking and evaluation: Expand evaluation to broader libraries, programming languages, and real-world engineering scenarios; develop contamination-controlled datasets.

Conclusion

LibEvoBench and SEUS establish the definitive evaluation paradigm for version-sensitive code generation. Existing LLMs—including leading proprietary and foundation models—are fundamentally version-oblivious and unable to reliably distinguish evolving API surfaces. The structural degradation persists across model scale and pretraining recency. Documentation aids retrieval but does not cure temporal ignorance. Addressing this requires a paradigm shift toward temporally stratified, version-tethered knowledge representations and training protocols. Future advances in AI for software engineering will hinge on closing this foundational gap.