SEUS: Software Evolution Understanding Score

• SEUS is a composite metric designed to assess code models’ capacity to navigate API evolution by measuring version-sensitive competence.
• It uses a harmonic mean to combine stable API accuracy with evolving API performance while penalizing instability and anachronistic errors.
• SEUS, implemented in LibEvoBench, offers a rigorous framework for benchmarking models on version-specific API behavior with clear diagnostic insights.

Software Evolution Understanding Score (SEUS) is a composite metric introduced in "LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models" to measure whether a code model truly understands that software APIs change over time (Cipollone et al., 24 Jun 2026). It is motivated by a specific failure mode in code generation over evolving libraries: a model may appear accurate in aggregate while remaining version-oblivious, because it succeeds on stable APIs yet fails on APIs that were added, removed, or modified across releases. In projects that pin exact library versions, the correct API call depends on the requested version rather than on the library in the abstract. SEUS was therefore designed to capture version-sensitive competence and to penalize anachronistic mistakes, namely predictions that are real somewhere in a library’s history but wrong for the target version.

1. Origin and problem setting

SEUS emerges from the observation that large software projects often depend on older versions of libraries even as APIs continue to evolve across releases. LibEvoBench frames this as a challenge for LLMs trained on temporally mixed corpora without explicit mechanisms for version-specific reasoning: such models can confuse historical and current interfaces and generate calls that belong to a different release lineage. The benchmark is explicitly concerned with this temporal mismatch and with the practical reality that developers work against pinned dependencies rather than an abstract, timeless API surface (Cipollone et al., 24 Jun 2026).

Within that setting, ordinary accuracy is insufficient. A model can obtain strong overall scores by performing well on APIs whose signatures remain unchanged, while still failing precisely where temporal knowledge matters. SEUS addresses that limitation by separating stable APIs from evolving APIs and by treating temporally confused outputs as a distinct error mode. A plausible implication is that SEUS operationalizes software evolution understanding not as generic code competence, but as competence under versioned API ecosystems.

2. Mathematical construction

The benchmark first partitions the API surface into stable APIs and evolving APIs. Stable APIs are those whose signatures remain identical across the tracked versions, whereas evolving APIs are those that are introduced, removed, or change signature over time. For a given model, library $\ell$, task $t$, and version $v$, the paper defines $S_v$ as the model’s score on stable APIs and $E_v$ as its score on evolving APIs. These are computed by averaging across four task-level metrics: API-C@L2 exact match, API-C@L3 parameter recall, API-I exact match, and SR parameter F1 (Cipollone et al., 24 Jun 2026).

The two version-level quantities are combined through the harmonic mean

$B_v = \frac{2 S_v E_v}{S_v + E_v}.$

This choice is central to the metric’s design. Because the harmonic mean is high only when both terms are high, strong stable-API performance cannot compensate for weak version-aware behavior on evolving APIs.

SEUS then aggregates over all library–task pairs and versions:

$$\mathrm{SEUS} = \frac{1}{N} \sum_{(\ell,t)\in\mathcal{L}\times\mathcal{T}} \left( \mathbb{E}_v[B_v] - \lambda \cdot \mathrm{Std}_v(E_v) - \gamma \cdot A_{\ell,t} \right),$$

where $N = |\mathcal{L}\times\mathcal{T}|$, $A_{\ell,t}$ is the anachronistic error rate on evolving APIs, and the paper uses $\lambda = \gamma = 0.5$ (Cipollone et al., 24 Jun 2026). The first term rewards average balanced performance across versions, the second is a stability penalty that discourages large swings in performance across versions on evolving APIs, and the third is an anachronism penalty that directly punishes temporally confused predictions.

The appendix restates the same definition and identifies “retention,” $t$0, as a useful auxiliary diagnostic. Retention expresses how much performance on evolving APIs is preserved relative to stable ones. This suggests that SEUS is not only a ranking score but also a decomposition framework for diagnosing whether failure arises from low evolving-API accuracy, instability across versions, or specifically anachronistic confusion.

3. Error taxonomy and semantic interpretation

SEUS is tightly coupled to LibEvoBench’s error taxonomy. The benchmark classifies predictions as correct, invalid, unknown, or anachronistic, depending on whether they match the target version, exist somewhere in the library, or are entirely fabricated (Cipollone et al., 24 Jun 2026).

Category	Brief characterization
Correct	Matches the target version
Invalid	Does not match the valid target behavior
Unknown	Entirely fabricated
Anachronistic	Real somewhere in history, but wrong for this version

The distinction between unknown and anachronistic is especially important. An anachronistic prediction is not a standard hallucination; it is “real but wrong-for-this-version.” That means the model has confused library history rather than simply inventing a nonexistent symbol. SEUS explicitly counts that as a separate failure mode.

Stable APIs, by definition, should not vary across versions, so version-to-version variation there is not especially informative. The real signal comes from evolving APIs. If a model is consistent across versions and genuinely version-aware, it should perform similarly on the correct release-specific form of an evolving API and should not drift toward historical or future alternatives. In that sense, SEUS measures temporal robustness rather than raw API recall alone.

4. Function within LibEvoBench

Within LibEvoBench, SEUS serves as the benchmark’s top-line score for ranking models in a multi-task, multi-version setting. LibEvoBench evaluates three tasks: API Calling (API-C), API Identification (API-I), and Signature Recall (SR). SEUS is computed from these tasks at the versioned API-surface level, with particular emphasis on the API-C@L2, API-C@L3, API-I, and SR variants that expose version knowledge and signature recall (Cipollone et al., 24 Jun 2026).

In the experimental setup, the authors evaluate frontier proprietary models and Qwen3.5 variants across PyTorch, NumPy, and SciPy and then compute SEUS per model. The leaderboard reports stable performance, evolving performance, retention, and the two penalties alongside SEUS. This organization makes the score interpretable: it does not merely state whether a model is capable, but whether that capability survives under API evolution.

SEUS is therefore more informative than ordinary accuracy or version-specific evaluation alone. Ordinary accuracy collapses all versions together and can hide brittleness on evolving APIs. Version-specific accuracy can expose some of that brittleness, but it does not summarize whether a model is simultaneously accurate and consistent across a full version history, nor does it separate fabricated outputs from historically real but version-inappropriate ones. SEUS combines balanced performance, consistency, and anachronism into a single metric targeted at understanding software evolution (Cipollone et al., 24 Jun 2026).

5. Empirical behavior and robustness

The reported results show that SEUS differentiates models in ways that ordinary accuracy does not. GPT-5.4 ranks first with SEUS 86.0, slightly ahead of GPT-5.5 at 85.1, even though GPT-5.5 has marginally higher stable-API accuracy. The paper attributes this ordering to GPT-5.4 being more consistent across versions and incurring a smaller stability penalty, so SEUS rewards it over a model with similar raw accuracy but more variation across releases (Cipollone et al., 24 Jun 2026).

More broadly, the experiments show that state-of-the-art models are largely version-oblivious. Performance degrades for evolving APIs, while for stable APIs it remains the same across versions. The paper reports that top models still show a nontrivial gap between stable and evolving APIs, typically around 7–10 points, and that weaker models not only score lower but also show larger penalties and more anachronistic confusion. These results are consistent with the claim that current training paradigms do not produce temporally grounded API knowledge.

The paper also reports two intervention findings with direct practical relevance. Simply specifying the target version provides no benefit, whereas relevant documentation significantly boosts models’ accuracy. This suggests that the failure is not merely one of prompt formatting; the models do not reliably maintain version-stratified knowledge internally, but can benefit from external grounding when documentation supplies the missing temporal context (Cipollone et al., 24 Jun 2026).

The appendix further reports that SEUS rankings are robust to penalty calibration. Sweeping $t$1 leaves Spearman rank correlation with the default setting at at least 0.995. The paper interprets this as evidence that model ordering is driven by genuine evolution-aware behavior rather than by an arbitrary weighting choice.

6. Relation to adjacent notions of software evolution understanding

SEUS is a specific metric defined in LibEvoBench, not a generic label already standardized across the literature. Nearby work addresses software evolution understanding in different senses. "Comparing Constraints Mined From Execution Logs to Understand Software Evolution" studies software evolution at the behavior level by comparing run-time constraints mined from execution logs recorded before and after a change, classifying differences as Identical, DifferentVal, Different, Missing, and New (Krismayer et al., 2020). That work contributes to understanding changed behavior after maintenance or feature updates, but it does not define SEUS as a version-aware code-generation metric.

"SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution" also does not define SEUS, but it operationalizes software evolution understanding as the ability to recover a developer’s reasoning processes and solutions from issue text, code context, and an oracle patch, using a rule-based reward built from patch similarity (Wei et al., 25 Feb 2025). This is related in spirit, since both settings treat software evolution as a structured source of supervision, but the target capability differs: SWE-RL focuses on issue-to-patch reasoning over project lifecycle data, whereas SEUS focuses on temporal knowledge of evolving library APIs.

"Quantifying Process Quality: The Role of Effective Organizational Learning in Software Evolution" does not explicitly define SEUS either. Instead, it proposes score-based, context-aware quantification of software evolution and process quality using source code density, maintenance-activity classifiers, activity-based process models, and Automatic Calibration (Hönel, 2023). That dissertation is relevant as a broader quantitative framework for making software evolution interpretable, but it addresses organizational learning and process deviation rather than version-sensitive API competence.

"The genome is software and evolution is a software developer" is more distant still. It does not use the term SEUS and offers theoretical and metaphorical context around software evolution, genetic programming, bugs, half-cooked products, and falsifiability (Rodriguez, 2010). Its connection to SEUS is therefore indirect and conceptual rather than methodological.

A common misconception is to treat any metric or framework about software change as equivalent to SEUS. The available evidence does not support that. SEUS refers specifically to the LibEvoBench composite score for balanced performance on stable and evolving APIs, penalized by instability across versions and anachronistic error rate (Cipollone et al., 24 Jun 2026).