GitChameleon Benchmark

• GitChameleon is a curated benchmark that evaluates AI systems' ability to generate Python code meeting explicit version constraints across evolving library APIs.
• It consists of 328 version-conditioned tasks covering 26 libraries, using execution-based evaluation and comprehensive unit tests to ensure real-world relevance.
• Empirical results highlight significant challenges in version awareness for current models, pointing to the need for advanced, version-specific AI architectures.

GitChameleon is a benchmark and curated dataset designed to rigorously evaluate the capacity of modern AI code generation systems to produce functionally correct Python code in environments where library versions are explicitly constrained. The GitChameleon benchmark, also referred to as "violet," addresses a critical blind spot for LLMs and AI-powered code assistants: the ability to perform version-conditioned code generation amid the rapidly evolving Python ecosystem, in which APIs experience frequent and often breaking changes. Unlike traditional code generation benchmarks, GitChameleon emphasizes execution-based evaluation across diverse library versions, providing a defensible measure of real-world capability and adaptability for AI-assisted programming (Misra et al., 16 Jul 2025, Islah et al., 2024).

1. Motivation and Conceptual Framework

Version-conditioned code generation (VCG) is defined as the problem of producing code $c$ in response to a prompt $p$ and an explicit version constraint $v$ for one or more libraries, such that $c$ both implements the specified functionality and executes without error when the environment enforces $v$. The evolution of libraries in the Python ecosystem leads to renaming of functions, changes in arguments and behaviors, and introduction or deprecation of modules. As a result, code that was previously correct can break with upgrades or downgrades, presenting a challenge for LLMs trained on heterogeneous and often conflicting codebases. Most previous benchmarks evaluate models assuming the "latest library" setup, neglecting the real-world need for legacy and pinned-version support (Misra et al., 16 Jul 2025, Islah et al., 2024). GitChameleon fills this deficit by enforcing version specificity as an explicit requirement.

2. Dataset Composition and Scope

GitChameleon comprises hand-curated code completion tasks, each explicitly conditioned on a real, historically documented breaking change in a Python library. The most recent version of the benchmark contains 328 problems from 26 libraries (e.g., NumPy, PyTorch, Pandas, SciPy, Matplotlib, Flask, Django) with versions spanning 2014–2023 (Misra et al., 16 Jul 2025). Each sample in the dataset is formalized as $(p_i, v_i, T_i)$, where $p_i$ is a natural-language prompt (often with starter code), $v_i$ specifies an exact version constraint (e.g., pandas==1.2.0), and $T_i$ consists of a comprehensive suite of hidden, assertion-based unit tests, achieving 96.5% code coverage. Problems are annotated according to change type, including argument/attribute changes, function-name substitutions, semantic/behavioral shifts, and new-feature or dependency changes. A smaller, earlier release comprises 116 problems covering 11 libraries (Islah et al., 2024).

Each task is accompanied by both visible and hidden tests. The visible test provides feedback for iterative generation (self-debugging), while the hidden test is used to compute official success metrics, ensuring unbiased and execution-based assessment. All unit tests are validated in isolated, version-pinned Docker environments to guarantee reproducibility and fidelity to historical behavior (Misra et al., 16 Jul 2025, Islah et al., 2024).

3. Evaluation Methodology

GitChameleon uses an execution-based approach for model evaluation. For each code-generation method $M$ and sample $p$0, the score is binary: whether $p$1's completion passes all assertions in $p$2 under the library version $p$3. The central metric is the hidden success rate,

$p$4

where $p$5 is the number of benchmark problems.

In addition to core "greedy" decoding, the evaluation spans:

• Chain-of-thought (CoT) prompting (zero-shot)
• Self-debugging (using visible test failures as feedback)
• Retrieval-augmented generation (RAG), e.g., DocPrompting with version-aligned API documentation
• Tool-calling agents with external search/sandboxing abilities
• Code assistant products in CLI and IDE configurations.

For models that support sampling multiple completions, pass@$p$6 statistics are used, computed according to established unbiased estimators (see Narasimhan et al.),

$p$7

where $p$8 is the number of samples and $p$9 the number of correct ones (Islah et al., 2024). Error-feedback prompting is also included: upon test failure, runtime error traces are prepended to the prompt, and the model is re-invoked to regenerate code.

4. Empirical Results

Evaluations demonstrate that current LLMs and code generation systems face significant challenges with version-conditioned code generation. The following table summarizes key performance metrics as reported for core LLMs under greedy decoding and with enhancements:

Model	Greedy Success Rate	Self-Debug (pp gain)	RAG (k=3) Success Rate	Pass@10 Baseline (116-task variant)
GPT-4.1	48.5% ±2.8%	+10–20 pp	58.5%	–
GPT-4o	49.1% ±2.8%	+10–20 pp	–	39.9%
Claude 3.7 Sonnet	48.8% ±2.8%	–	59.4%	–
Gemini 2.5 Pro	50.0% ±2.8%	–	56.7%	41.4%
Llama 4 Maverick	40.8% ±2.7%	–	~49%	–

For the 116-task version, pass@10 for GPT-4o is 39.9%, increasing to 43.7% with error-feedback prompting. Other models, such as Gemini-1.5 and Phi-3.5-MoE, show similar performance. Success rates on the main benchmark (328 tasks) range from 40.8% to 51.2% for enterprise LLMs under baseline settings. Self-debugging leads to absolute gains of 10–20 percentage points. RAG improves results by 5–10 percentage points. Tool-using agents and IDE assistants demonstrate further, though variable, improvements—sandbox-enabled agents yield gains of 10–20 percentage points, and presenting the problem statement to code assistants increases their effectiveness by 12–35 percentage points (Misra et al., 16 Jul 2025, Islah et al., 2024).

Detailed analysis by change type reveals that semantic changes are the easiest for self-debugging strategies (60–80% success), whereas argument/attribute changes and "new-feature" additions are most difficult, with greedy success rates of 25–50% (50–65% post-debug). Model performance is systematically worse on tasks based on library releases from 2022–2023.

5. Failure Modes and Analytical Insights

Qualitative inspection underscores that version gaps are the main source of model failures. LLMs frequently invoke deprecated APIs, use obsolete parameters, or call methods that do not exist in the specified version; most errors constitute trivial API mismatches rather than deep semantic or logical bugs. This phenomenon roots in the nature of pretraining, which homogenizes code examples from all versions without explicit version tags, causing the model to default to the most common (typically latest) API calls. As a result, even when prompted for a legacy version, models often fail to override their learned priors in the absence of strong external signals (Misra et al., 16 Jul 2025, Islah et al., 2024).

Statistical analysis shows moderate to high correlation between model size and performance, with Spearman’s ρ=0.82 for pass@1 and 0.69 for pass@10 in base models. The effect is attenuated (ρ≈0.45) for instruct-tuned variants. Model performance on GitChameleon is only weakly correlated with other code benchmarks (e.g., ρ=0.55 with SWE-Bench, ρ=0.21 with LiveCodeBench). This suggests that GitChameleon exposes a distinct capability axis not captured by traditional static-API or latest-version code generation tasks (Misra et al., 16 Jul 2025, Islah et al., 2024).

6. Implications, Limitations, and Future Directions

GitChameleon establishes an execution-based, version-aware standard for benchmarking code generation systems. Its findings indicate that current models and even commercial code assistants are not robust on version-specific tasks, achieving only around 40–51% hidden success rates under best-case automated regimes. A plausible implication is that version-aware architectures—e.g., explicit version embeddings, fine-tuning on changelogs, continual learning of library evolution, or advanced RAG pipelines indexed on versioned documentation—are necessary to close the gap (Misra et al., 16 Jul 2025, Islah et al., 2024).

Current limitations include restriction to Python and 26 libraries, absence of in-distribution migration tasks (only fresh code completions are tested), and lack of a human-performance baseline. No multi-step cross-library migrations or dependency chains are explored; RAG and fine-tuning are not exhaustively evaluated on all variants. Expansion to other languages (Rust, Java, JavaScript), the automation of changelog mining for large-scale dataset construction, and integration with multi-version context and API repair feedback loops are proposed future directions (Misra et al., 16 Jul 2025, Islah et al., 2024).