TimeMachine-bench-Verified: Python Migration Benchmark

• TimeMachine-bench-Verified is a rigorously curated benchmark offering ground-truth verified minimal code patches for dependency-induced breakages in Python repositories.
• It employs an automated filtering pipeline with Docker-recreated environments and test-driven extraction to ensure reproducibility and high-quality migration tasks.
• Evaluation using sufficiency and necessity metrics reveals practical challenges and performance variations across LLM-driven and agent-based migration strategies.

TimeMachine-bench-Verified is a rigorously human-curated subset of the TimeMachine-bench framework, designed for evaluating model capabilities in migration tasks over real-world, repository-level Python projects. Uniquely, it offers ground-truth–verified tasks that enforce minimal, code-only fixes for dependency-induced breakages under precisely controlled environment timelines. Its standardized construction and evaluation protocols enable granular assessment of both LLM-driven and agent-based migration strategies, furnishing robust empirical insights into sufficiency and necessity of candidate repairs (Fujii et al., 30 Jan 2026).

1. Benchmark Construction and Filtering Pipeline

The TimeMachine-bench construction process is strictly end-to-end automated for large-scale reproducibility, with manual curation integrated downstream to ensure feasibility and quality:

• Pre-Execution Filtering: Extraction starts from The Stack v2 GitHub snapshot, retaining Python repositories with permissive licenses, at least one recognized environment file or testing framework, and public recognition (≥1 GitHub star), narrowing to approximately 45,000 candidates.
• Environment Reproduction: For each repo, two environments are recreated in Docker: the origin environment (commit-date dependencies) and a target environment (fixed future cutoff: 2025-07-31), with package versions and install semantics resolved via LLM prompts or conservative heuristics.
• Test-Driven Extraction: Candidate repositories are retained if tests pass under the origin setup yet fail in the target, after package release masking and excluding cases where failures arise in third-party (site-packages) code or hit timeouts, yielding 1,145 repositories for the TimeMachine-bench-Full set.
• Human Verification (‘Verified’ Subset): From this set, 196 repositories are sampled. A Python expert is tasked with producing the minimal patch to restore all tests, adhering to strict rules: only *.py files may be changed (no dependency downgrades or test file edits), with a maximum allowed annotation time of two hours per repository. Tasks not solvable within these constraints or requiring only test-edit fixes are excluded. The result is 100 repositories, labeled by repair difficulty (Easy, Medium, Hard), denoted TimeMachine-bench-Verified.

Pseudocode for this pipeline reflects step-wise environment testing, filtering, manual patching under timing/difficulty constraints, and consistent human-in-the-loop validation.

2. Human Verification Criteria and Task Annotation

Verification in TimeMachine-bench-Verified is performed by a single high-competency annotator (Python, 8+ years). The protocol mandates:

• Manual validation of every candidate patch, with guidance permitting consulting any resource (official docs, LLMs), but requiring all final code changes to be explicitly reviewed and authored by the annotator.
• Strict “minimal-edit” criteria: Only the smallest possible code change that causes all tests to pass is accepted, minimizing spurious or extraneous edits.
• Edits involving downgrading dependencies or altering only test files are explicitly disallowed.
• Each verification task is annotated with time-to-fix and assigned a difficulty label:
  • Easy (<15 minutes)
  • Medium (15–60 minutes)
  • Hard (<2 hours)
• Quantitative summary: Of 100 verified repositories, 64 tasks are Easy, 30 Medium, 6 Hard. The distribution of unique libraries triggering errors is broad—NumPy and pandas are most frequent, with 35 others occurring singularly; the median number of gold lines to edit is 2, maximum is 54.

Reproducibility is ensured by archiving the full set of patches and difficulty labels. However, the single-annotator design precludes inter-annotator agreement statistics.

3. Evaluation Metrics and Formal Definitions

Two orthogonal axes—sufficiency and necessity—are systematically quantified:

• Sufficiency ($\mathrm{pass@1}(n,m)$):
  • For task $i$ in the verified set $D = \{D_1, \ldots, D_{100}\}$, success $S_i(n,m)=1$ requires all tests pass, with at most $n$ LLM/tool calls and $m$ test executions. Formally,

  $$\mathrm{pass@1}(n,m) = \frac{1}{N} \sum_{i=1}^N S_i(n,m)$$

  with the primary reporting regime using $n=100$, $m=10$.

• Necessity ($\mathrm{prec@1}(n,m)$):

  • For successful solutions ($i$0), precision on code changes is computed as fraction of gold-patch lines present in model patch (strict line-level matching):

  $i$1

  where $i$2 if instance $i$3 is solved ($i$4), else 0.

• Auxiliary metrics: Levenshtein edit distance to gold; unnecessary-change rate ($i$5).

This rigorous structure penalizes “spurious” fixes, incentivizing minimal, effective code changes rather than overfitting to low-coverage tests.

4. Baseline Results for LLM- and Agent-based Approaches

A ReAct-style agent, instrumented with 10 repair tools, is evaluated using 11 models (GPT-5, GPT-4o, several Claude, Qwen, Llama, DeepSeek models, including high-parameter open-weight options), under identical call/test budgets ($i$6, $i$7). Headline results:

Model	pass@1(100,10)	prec@1(100,10)	Easy/Med/Hard solved (n/64/30/6)
Claude Sonnet 4	99.0%	78.0%	64 / 30 / 5
Qwen3-480B	90.0%	70.1%	62 / 26 / 2
GPT-5	91.0%	54.2%	62 / 27 / 2
GPT-4o	76.0%	61.4%	57 / 19 / 0
Llama-3.3	52.0%	44.0%	40 / 12 / 0

Key error sources identified:

• Spurious solutions exploiting insufficient test coverage (e.g., introducing dummy constants).

• Unnecessary edits: 20–50% of candidate patches include extraneous, non-contributory code changes. This aligns with observed nec@1 figures and reflects a need for improved minimality constraints in automated patching protocols.

• Agent operational patterns such as excessive file access (GPT-5) and off-by-one line errors further impact result quality.

Difficulty remains a limiting factor: “Hard” tasks are solved by no model at >83.3%; for most, hard-tier success remains under 50%.

5. Key Empirical Findings and Reliability Analysis

The primary empirical insights are:

• Task sufficiency and minimality are distinct dimensions:

  • Many models achieve high pass@1 but with low prec@1 (e.g., GPT-5 at 91% pass@1, only 54.2% prec@1), indicating that brute-force success alone is not diagnostic of true migration competency.
• Open-weight LLMs are competitive with proprietary models:
  • Qwen3-480B and Qwen3-235B close the gap, particularly for task sufficiency and minimizing unnecessary code changes.
• Complexity and difficulty correlate strongly with model performance:
  • Easy tasks are nearly solved by all high-capacity models, but none can consistently handle the most complex migrations.
• Test coverage bottleneck:
  • The current benchmark framework, reflecting test-driven definition of “success,” allows some models to “game” success metrics by inserting generic pass-through fixes; this compromises guarantees beyond the monitored code surface.
• Library- and tool-specific nuances:
  • Most common faults relate to high-velocity libraries (e.g., NumPy, pandas). Many errors are model hallucinations regarding library or version details, revealing gaps in semantic dependency understanding.

6. Limitations and Recommendations

Current limitations include:

• Test-set dependency: Results are only as reliable as the encapsulating test modules; untested code regions remain outside the measurement envelope.
• Human curation biases: Verified subset omits complex, multi-file, or time-intensive migrations (>2h or requiring test code edits), thereby under-representing the full distribution of practical migration complexity.
• Lack of inter-annotator metrics: Single-annotator design, while high-expertise, precludes formal measurement of annotation reliability.
• Python-only focus: While date-based environment control is general, the present benchmark is not validated on other language ecosystems.

Recommended future directions:

• Automated test generation to expand validated code coverage, reducing unintentional “gaming.”
• Introspective agent refinements capable of reverting or pruning unnecessary code edits, thus directly addressing the minimality axis.
• Expansion to other language ecosystems (e.g., npm, Maven) for ecosystem-general claims.
• Scalable, multi-annotator verification to quantify and improve patch reproducibility and reliability.

7. Significance and Position in Software Evaluation Landscape

TimeMachine-bench-Verified uniquely positions itself at the intersection of automated software engineering, LLM-based agent assessment, and rigorous, temporal-consistent repository analysis. Its combination of strict construction protocols, ground-truth–verified tasks, and orthogonally measured sufficiency–necessity serves as a replicable foundation for isolating genuine progress—and persistent shortcomings—in automated migration capabilities (Fujii et al., 30 Jan 2026). This resource enables fine-grained dissection of both the upper bounds of current model-driven repair primitives and the operational bottlenecks constraining scalable deployment in real-world scenarios.