CodeUpdateArena: Benchmarking Code LLM Knowledge Updates

• CodeUpdateArena is a benchmark evaluating large language models' ability to acquire and apply knowledge of evolving code APIs in downstream program synthesis tasks.
• The benchmark features 161 synthetic atomic API updates across 7 Python packages and 670 program synthesis tasks requiring the updated functionality, evaluated via unit tests.
• Experimental results indicate that current LLM knowledge editing techniques, including retrieval and fine-tuning, largely fail to deeply propagate new code logic required for robust application.

The CodeUpdateArena Task is a benchmark and evaluation framework targeting the fundamental challenge of updating LLMs' (LLMs) knowledge about code APIs as libraries evolve. It specifically focuses on whether and how LLMs can acquire and propagate the semantics of API changes for use in downstream program synthesis, without retraining from scratch. This is a problem that is far more complex than factual knowledge editing in natural language LLMs, requiring mechanistic understanding and propagation of updated program logic within the model.

1. Problem Scope and Benchmark Design

CodeUpdateArena is designed to evaluate LLMs' abilities to incorporate atomic API updates and apply the new or modified functionality in practical code synthesis. Each benchmark instance consists of:

• A synthetic API function update: An atomic change to a function’s interface or behavior (e.g., new argument, signature modification) drawn from realistic patterns but avoiding contamination from pre-existing training data.
• Program synthesis examples: For each update, at least three synthesis tasks that can only be correctly solved by making use of the updated API, with unit tests specifying the required behavior.

The dataset encompasses 161 function updates across 54 functions from seven diverse Python packages (itertools, math, numpy, pandas, re, sympy, torch), totaling 670 synthesis examples. Update types are taxonomized along “action” (add, modify), “locus” (function, argument, output), and “aspect” (e.g., name, data type, supported value).

Synthetic updates generated via GPT-4 are manually filtered for validity, and program synthesis tasks are created to require the newly introduced or changed functionality, with strict pass/fail unit tests. This synthetic approach prevents contamination by pretraining data and ensures forward-looking generalization.

2. Methodological Approach and Data Pipeline

The dataset construction pipeline involves:

1. API Update Synthesis:
   • For a target function $f$, GPT-4 is prompted to generate an atomic update description, a new docstring (documenting the update), and the modified Python implementation.
   • The update must pass at least 70% of a generated suite of 10 diverse unit tests, ensuring it is executable and meaningful.
2. Program Synthesis Scenario Generation:
   • Each update is paired with several program synthesis prompts where the correct solution necessarily involves the updated API.
   • For each, GPT-4 generates unit tests (≥3 per update) and a reference solution using the update.
3. Manual Curation:
   • Trivial or duplicate updates/synthesis tasks are removed using deduplication and edit-distance filtering.

The benchmark thus reflects plausible, real-world API churn while remaining agnostic to potential future overlaps with model training corpora.

3. Challenges of Knowledge Editing in Code LLMs

CodeUpdateArena exposes several challenges unique to code-domain knowledge editing:

• Deep mechanistic propagation: Unlike factual editing, updating an API's logic requires LLMs to apply new computation in generation, not just recall text or docstrings.
• Compositional usage: APIs are embedded in diverse calling contexts; updating knowledge must trigger correct application and parameter use in many scenarios, not just in isolated “factoid” form.
• Downstream task measurement: Success is measured not by reciting the updated docstring but by solving code generation problems where the update is required for correctness and passing all unit tests.
• Regression and specificity: Evaluation must also check that the edited model does not degrade in unrelated code generation domains (“specificity,” as reflected in SPass@k scores on HumanEval).

This constitutes a stricter and more comprehensive test than traditional knowledge editing for facts in text models, because the knowledge must be internalized as functional code logic and reliably propagated to new generation contexts.

4. Experimental Findings and Baseline Results

Extensive experiments on CodeUpdateArena show:

• Prepending documentation at inference time (retrieval-augmented generation) allows GPT-4 to solve many tasks (pass@5 up to 83% in some packages), but smaller open-source models like CodeLlama and DeepSeek show little to no improvement using this strategy.
• Fine-tuning on update documentation alone does not impart the required new logic, and often harms unrelated code generation (“catastrophic forgetting” as measured on HumanEval).
• Fine-tuning on usage exemplars marginally boosts performance, but ablation shows that the gains may reflect format rather than real propagation of semantics. Models fine-tuned on unrelated updates perform nearly as well as those fine-tuned on the actual update, underscoring that parameter editing does not reliably “implant” new code behavior.
• Combined strategies (fine-tuning on both doc and examples) offer little further improvement without substantial risk of degrading generalization.
• Resurfacing: Even after parametric editing, prepending the docstring at inference still helps (or is even necessary), showing that current methods do not deeply implant new function knowledge.

The main evaluation metric, UPass@k, measures the probability that at least one of $k$ generated samples both (a) passes all test cases and (b) uses the updated API. Sample formula:

$$\text{UPass@k} = \frac{1}{D} \sum_{i=1}^{D} \left[1 - \frac{\binom{n-c_i}{k}}{\binom{n}{k}} \right]$$

where $c_i$ is the number of correct samples per task.

Model	Base	Prepend	FT(U)	FT(PS)	FT(U+PS)
CodeLlama-7B	17%	21%	18%	21%	21%
DeepSeek-7B	20%	29%	21%	26%	27%
GPT-4	33%	64%	--	--	--

Performance of SOTA open-source models remains substantially below the level required for robust, automated API update propagation; fine-tuning approaches risk overfitting and regression.

5. Impact, Evaluation, and Future Directions

CodeUpdateArena sets a new bar for evaluating LLMs' adaptation to dynamic code knowledge, exposing the acute difficulty of updating code semantics:

• Parameter editing techniques (gradient- or example-based) are currently insufficient for code: they often fail to propagate new knowledge predictably, require further retrieval at inference, and degrade on unrelated code tasks.
• Robust evaluation requires temporally-constructed, cross-project, and complex code update scenarios: simple random splits or artificially localized tasks significantly overstate progress.
• Suitable metrics (UPass@k, SPass@k) must jointly capture task-specific success and preservation of unrelated capabilities.
• Research directions matter: Improved structure-aware knowledge editing, causal-tracing to localize API logic within the model, and editing methods that guarantee downstream propagation across diverse usage contexts are necessary.
• Broad applicability: Adoption of similar benchmarks for other programming languages (Java, C++) and APIs would enable comprehensive insights into model robustness under practical evolution.

Relevant benchmarks, data, and code are made available at https://github.com/leo-liuzy/CodeUpdateArena.

6. Technical Illustration

Example API update and synthesis instance:

• Update: Add inverse boolean to math.pow(), computing $1/(x^y)$ if inverse=True
• Problem prompt: Implement function compute_present_value(r: float, n: int) -> float returning $1/(1 + r)^n$
• Reference solution:

def compute_present_value(r: float, n: int) -> float: return math.pow(1.0 + r, n, inverse=True)

7. Summary Table: Key Aspects

Attribute	Value/Methodology	Significance
Update Scope	161 atomic API updates	Future-proof, fine grain
Evaluation Format	Program synthesis + unit testing	Enforces functional understanding
Metrics	UPass@k, SPass@k	Precision and robustness
Core Finding	Model updates rarely propagate correctly	Reveals deep technical challenge
Dataset Availability	Public: https://github.com/leo-liuzy/CodeUpdateArena	For reproducible research

This benchmark establishes a rigorous, realistic baseline for measuring and advancing the ability of LLMs to adjust code understanding as APIs evolve—a capability essential for trustworthy code automation tools in rapidly changing software ecosystems.