---
title: 'TimeMachine-bench-Verified: Python Migration Benchmark'
url: https://www.emergentmind.com/topics/timemachine-bench-verified
type: topic
---

# TimeMachine-bench-Verified: Python Migration Benchmark

TimeMachine-bench-Verified is a rigorously human-curated subset of the TimeMachine-bench framework, designed for evaluating model capabilities in migration tasks over real-world, repository-level Python projects. Uniquely, it offers ground-truth–verified tasks that enforce minimal, code-only fixes for dependency-induced breakages under precisely controlled environment timelines. Its standardized construction and evaluation protocols enable granular assessment of both LLM-driven and agent-based migration strategies, furnishing robust empirical insights into sufficiency and necessity of candidate repairs [2601.22597].

## 1. Benchmark Construction and Filtering Pipeline

The TimeMachine-bench construction process is strictly end-to-end automated for large-scale reproducibility, with manual curation integrated downstream to ensure feasibility and quality:

- **Pre-Execution Filtering:** Extraction starts from The Stack v2 GitHub snapshot, retaining Python repositories with permissive licenses, at least one recognized environment file or testing framework, and public recognition (≥1 GitHub star), narrowing to approximately 45,000 candidates.
- **Environment Reproduction:** For each repo, two environments are recreated in Docker: the origin environment (commit-date dependencies) and a target environment (fixed future cutoff: 2025-07-31), with package versions and install semantics resolved via LLM prompts or conservative heuristics.
- **Test-Driven Extraction:** Candidate repositories are retained if tests pass under the origin setup yet fail in the target, after package release masking and excluding cases where failures arise in third-party (site-packages) code or hit timeouts, yielding 1,145 repositories for the TimeMachine-bench-Full set.
- **Human Verification (‘Verified’ Subset):** From this set, 196 repositories are sampled. A Python expert is tasked with producing the minimal patch to restore all tests, adhering to strict rules: only *.py files may be changed (no dependency downgrades or test file edits), with a maximum allowed annotation time of two hours per repository. Tasks not solvable within these constraints or requiring only test-edit fixes are excluded. The result is 100 repositories, labeled by repair difficulty (Easy, Medium, Hard), denoted TimeMachine-bench-Verified.

Pseudocode for this pipeline reflects step-wise environment testing, filtering, manual patching under timing/difficulty constraints, and consistent human-in-the-loop validation.

## 2. Human Verification Criteria and Task Annotation

Verification in TimeMachine-bench-Verified is performed by a single high-competency annotator (Python, 8+ years). The protocol mandates:

- Manual validation of every candidate patch, with guidance permitting consulting any resource (official docs, LLMs), but requiring all final code changes to be explicitly reviewed and authored by the annotator.
- Strict “minimal-edit” criteria: Only the smallest possible code change that causes all tests to pass is accepted, minimizing spurious or extraneous edits.
- Edits involving downgrading dependencies or altering only test files are explicitly disallowed.
- Each verification task is annotated with time-to-fix and assigned a difficulty label:
  - Easy (<15 minutes)
  - Medium (15–60 minutes)
  - Hard (<2 hours)
- Quantitative summary: Of 100 verified repositories, 64 tasks are Easy, 30 Medium, 6 Hard. The distribution of unique libraries triggering errors is broad—NumPy and pandas are most frequent, with 35 others occurring singularly; the median number of gold lines to edit is 2, maximum is 54.

Reproducibility is ensured by archiving the full set of patches and difficulty labels. However, the single-annotator design precludes inter-annotator agreement statistics.

## 3. Evaluation Metrics and Formal Definitions

Two orthogonal axes—sufficiency and necessity—are systematically quantified:

- **Sufficiency (\(\mathrm{pass@1}(n,m)\)):**
  - For task \(i\) in the verified set \(D = \{D_1, \ldots, D_{100}\}\), success \(S_i(n,m)=1\) requires all tests pass, with at most \(n\) LLM/tool calls and \(m\) test executions. Formally,
    $$
    \mathrm{pass@1}(n,m) = \frac{1}{N} \sum_{i=1}^N S_i(n,m)
    $$
    with the primary reporting regime using \(n=100\), \(m=10\).
- **Necessity (\(\mathrm{prec@1}(n,m)\)):**
  - For successful solutions (\(D^+\)), precision on code changes is computed as fraction of gold-patch lines present in model patch (strict line-level matching):
    $$
    \mathrm{prec@1}(n,m) = \frac{1}{N}\sum_{i=1}^N N_i(n,m)
    $$
    where \(N_i(n,m) = p_i\) if instance \(i\) is solved (\(S_i=1\)), else 0.
- **Auxiliary metrics:** Levenshtein edit distance to gold; unnecessary-change rate (\(1 - \mathrm{prec@1}\)).

This rigorous structure penalizes “spurious” fixes, incentivizing minimal, effective code changes rather than overfitting to low-coverage tests.

## 4. Baseline Results for LLM- and Agent-based Approaches

A ReAct-style agent, instrumented with 10 repair tools, is evaluated using 11 models (GPT-5, GPT-4o, several Claude, Qwen, Llama, DeepSeek models, including high-parameter open-weight options), under identical call/test budgets (\(n=100\), \(m=10\)). Headline results:

| Model                  | pass@1(100,10) | prec@1(100,10) | Easy/Med/Hard solved (n/64/30/6) |
|------------------------|----------------|----------------|-------------------------------|
| Claude Sonnet 4        | 99.0%          | 78.0%          | 64 / 30 / 5                   |
| Qwen3-480B             | 90.0%          | 70.1%          | 62 / 26 / 2                   |
| GPT-5                  | 91.0%          | 54.2%          | 62 / 27 / 2                   |
| GPT-4o                 | 76.0%          | 61.4%          | 57 / 19 / 0                   |
| Llama-3.3              | 52.0%          | 44.0%          | 40 / 12 / 0                   |

Key error sources identified:
- Spurious solutions exploiting insufficient test coverage (e.g., introducing dummy constants).
- Unnecessary edits: 20–50% of candidate patches include extraneous, non-contributory code changes. This aligns with observed nec@1 figures and reflects a need for improved minimality constraints in automated patching protocols.
- Agent operational patterns such as excessive file access (GPT-5) and off-by-one line errors further impact result quality.

Difficulty remains a limiting factor: “Hard” tasks are solved by no model at >83.3%; for most, hard-tier success remains under 50%.

## 5. Key Empirical Findings and Reliability Analysis

The primary empirical insights are:

- **Task sufficiency and minimality are distinct dimensions:**
  - Many models achieve high pass@1 but with low prec@1 (e.g., GPT-5 at 91% pass@1, only 54.2% prec@1), indicating that brute-force success alone is not diagnostic of true migration competency.
- **Open-weight LLMs are competitive with proprietary models:**
  - Qwen3-480B and Qwen3-235B close the gap, particularly for task sufficiency and minimizing unnecessary code changes.
- **Complexity and difficulty correlate strongly with model performance:**
  - Easy tasks are nearly solved by all high-capacity models, but none can consistently handle the most complex migrations.
- **Test coverage bottleneck:**
  - The current benchmark framework, reflecting test-driven definition of “success,” allows some models to “game” success metrics by inserting generic pass-through fixes; this compromises guarantees beyond the monitored code surface.
- **Library- and tool-specific nuances:**
  - Most common faults relate to high-velocity libraries (e.g., NumPy, pandas). Many errors are model hallucinations regarding library or version details, revealing gaps in semantic dependency understanding.

## 6. Limitations and Recommendations

Current limitations include:

- **Test-set dependency:** Results are only as reliable as the encapsulating test modules; untested code regions remain outside the measurement envelope.
- **Human curation biases:** Verified subset omits complex, multi-file, or time-intensive migrations (>2h or requiring test code edits), thereby under-representing the full distribution of practical migration complexity.
- **Lack of inter-annotator metrics:** Single-annotator design, while high-expertise, precludes formal measurement of annotation reliability.
- **Python-only focus:** While date-based environment control is general, the present benchmark is not validated on other language ecosystems.

Recommended future directions:

- Automated test generation to expand validated code coverage, reducing unintentional “gaming.”
- Introspective agent refinements capable of reverting or pruning unnecessary code edits, thus directly addressing the minimality axis.
- Expansion to other language ecosystems (e.g., npm, Maven) for ecosystem-general claims.
- Scalable, multi-annotator verification to quantify and improve patch reproducibility and reliability.

## 7. Significance and Position in Software Evaluation Landscape

TimeMachine-bench-Verified uniquely positions itself at the intersection of automated software engineering, LLM-based agent assessment, and rigorous, temporal-consistent repository analysis. Its combination of strict construction protocols, ground-truth–verified tasks, and orthogonally measured sufficiency–necessity serves as a replicable foundation for isolating genuine progress—and persistent shortcomings—in automated migration capabilities [2601.22597]. This resource enables fine-grained dissection of both the upper bounds of current model-driven repair primitives and the operational bottlenecks constraining scalable deployment in real-world scenarios.

Source: https://www.emergentmind.com/topics/timemachine-bench-verified