DeployBench: Research Artifact Deployment Benchmark

• DeployBench is a benchmark that evaluates automated research-artifact deployment using native environment setup from published papers and code repositories.
• It tests agents’ ability to configure complex multi-language and system-level dependencies, including GPU/CUDA and legacy compatibility, on fresh Linux VMs.
• The benchmark comprises 51 tasks across diverse domains, using a hidden two-layer verification process to measure successful execution of paper-designated experiments.

DeployBench is a benchmark for research-artifact deployment: the task of taking a published paper and its associated code repository, starting from a fresh machine, and producing a native environment that can run a paper-designated experiment. It is explicitly framed as a benchmark for a bottleneck in research automation rather than for generic repository setup, because many research artifacts require multi-language toolchains, system-level dependencies beyond containers, legacy compatibility repair, GPU/CUDA installation, or kernel and QEMU configuration before any downstream debugging, replication, or extension can begin (Wang et al., 3 Jun 2026).

1. Problem formulation and benchmark objective

DeployBench targets a narrower and more operational capability than standard software-engineering or environment-setup benchmarks. The benchmark objective is not merely to make a repository install, pass unit tests, or build in Docker, but to set up a native host environment sufficient for a lightweight version of the actual experiment the paper claims to support. Each task gives an agent a paper, the associated code repository, and a fresh Linux VM; the agent must bridge the gap from “paper + repo” to a runnable scientific environment (Wang et al., 3 Jun 2026).

This formulation matters because research-artifact deployment is often broader and messier than ordinary repository setup. The agent must infer requirements jointly from the paper and repository rather than from a README alone. The benchmark therefore includes cases that extend beyond containerized Python workflows, such as multi-language pipelines, host-level dependencies, GPU/CUDA installation on machines with no preinstalled drivers, kernel and VM setup, and old artifacts that must be adapted to modern compilers and APIs. The paper’s central claim is that a runnable environment is frequently the gating factor before any broader research-agent workflow can begin (Wang et al., 3 Jun 2026).

A plausible implication is that DeployBench operationalizes a distinct layer of scientific automation: not scientific reasoning or reproduction in the full methodological sense, but the infrastructural precondition for those activities.

2. Corpus construction and task composition

DeployBench contains 51 deployment tasks drawn from 49 unique source artifacts, because two papers each contribute both CPU and GPU variants. The tasks span 25 AI/ML tasks, 19 computer-systems tasks, and 7 scientific-computing tasks. The source papers range from 2008 to 2025 and include artifacts from venues such as NeurIPS, ICML, ICLR, CVPR, ECCV, RSS, CoRL, OSDI, SOSP, EuroSys, ATC, USENIX Security, JOSS, and Cell (Wang et al., 3 Jun 2026).

The benchmark’s language coverage is deliberately broad: 11 ecosystems are represented, including Python, C/C++, Rust, Fortran, Julia, R, Haskell, Lua, Racket, JavaScript, and Java. Some tasks require multiple languages in one pipeline. The task set is also intentionally difficult along several dimensions. Twenty-two tasks require GPU execution, and the benchmark starts from a VM with no CUDA drivers installed. Five tasks require building a custom Linux kernel and booting it inside a QEMU virtual machine, which is why Docker is explicitly disallowed. Ten tasks are legacy artifacts from 2011–2018 that had not been updated since 2020 and require compatibility repair against a modern OS, compiler, or library stack (Wang et al., 3 Jun 2026).

Construction emphasizes feasibility and realism. The authors selected papers from top venues, preferred widely used public repositories, and, for systems and security papers, additionally preferred artifacts that had passed artifact evaluation when such evidence existed. Candidate repositories were manually inspected and tested; artifacts that were clearly broken, incomplete, or unreproducible were excluded. Every included task was first validated through a reference deployment on a clean VM, and that reference setup both established feasibility and informed verifier design. The authors report that the reference setup script passes the verifier from scratch on all 51 tasks (Wang et al., 3 Jun 2026).

The benchmark also annotates task difficulty as 12 easy, 20 medium, and 19 hard. More generally, the task set includes native compilation, dataset or model downloads, library-path issues, service startup, and host/guest interactions that cannot be reduced to building a container image (Wang et al., 3 Jun 2026).

3. Task semantics, hidden verification, and scoring

A defining feature of DeployBench is that every task is tied directly to a paper-designated experiment. For each artifact, the verification target is derived from the paper’s original evaluation and is chosen to be lightweight enough for practical benchmarking while still representative enough to exercise the main pipeline and key dependencies. A typical instance is running one epoch of training rather than a full benchmark. Success is therefore not awarded for successful installation, import, or repository-provided unit tests if those do not demonstrate readiness for the designated experiment (Wang et al., 3 Jun 2026).

Evaluation uses a hidden two-layer verification pipeline. After the agent finishes, times out, or reaches the step limit, the final VM state is checked as follows. Layer 1 is a global rule-based parser over logs that catches common execution failures such as missing dependencies, compilation errors, runtime crashes, and out-of-memory conditions. If no Layer-1 failure is found, Layer 2 performs task-specific checking by rerunning the designated lightweight experiment and checking for required outputs, artifacts, runtime evidence, or service availability. A task counts as solved only if it passes both layers, and the hidden verifier scripts are never shown to the agent (Wang et al., 3 Jun 2026).

The primary metric is task success rate, defined as the fraction of tasks whose hidden verification pipeline returns a passing verdict. The paper does not provide a nontrivial scoring equation beyond that verbal definition, and it explicitly notes that there are no substantive benchmark equations to reproduce in LaTeX (Wang et al., 3 Jun 2026).

This hidden, paper-specific evaluation is crucial to the benchmark’s design. More than half of failed runs pass generic “looks runnable” checks and are rejected only by the task-specific verifier, which means the benchmark is deliberately resistant to superficial success proxies.

4. Experimental protocol and empirical performance

The reported evaluation uses OpenHands as the common scaffold and tests four frontier LLMs: GPT-5.3-Codex, Gemini-3.1-Pro, Grok-4.20, and GPT-5.4-Mini. All runs use fresh Google Cloud Ubuntu 22.04 VMs with 16 vCPUs and 64 GB RAM. GPU tasks use one NVIDIA L4 GPU with 24 GB memory and no preinstalled CUDA drivers. Docker is forbidden. Each task is run on a fresh VM with an individually assigned time budget between 10 and 60 minutes (Wang et al., 3 Jun 2026).

Overall performance shows that current agents remain far from autonomous deployment. The reported pass rates are 51.0% for GPT-5.3-Codex, 27.5% for Gemini-3.1-Pro, 11.8% for Grok-4.20, and 7.8% for GPT-5.4-Mini (Wang et al., 3 Jun 2026).

Performance varies by domain. On AI/ML tasks, GPT-5.3-Codex reaches 52.0%, Gemini 16.0%, Grok 8.0%, and GPT-5.4-Mini 0.0%. On scientific computing, Gemini is strongest at 57.1%, ahead of GPT-5.3-Codex at 42.9%, with Grok at 14.3% and Mini at 0.0%. On systems tasks, GPT-5.3-Codex scores 52.6%, Gemini 31.6%, Grok 15.8%, and Mini 21.1%. Difficulty matters as well: Codex reaches 66.7% on easy tasks, 45.0% on medium tasks, and 47.4% on hard tasks, whereas Gemini drops from 33.3% on easy and 40.0% on medium to 10.5% on hard (Wang et al., 3 Jun 2026).

Task characteristics expose additional fault lines. For legacy artifacts, Codex scores 40.0% versus 53.7% on recent tasks, indicating compatibility drift. In AI/ML, 88% of tasks require GPU-capable environments, making CUDA/GPU setup a major difficulty source. In systems, the five QEMU/kernel tasks average only 20.0% success across models, underscoring the difficulty of VM- and kernel-level deployment (Wang et al., 3 Jun 2026).

Runtime and token usage add nuance. GPT-5.3-Codex averages 44.5 agent steps, 2.15M total tokens, and 18.1 minutes per task; Gemini averages 57.8 steps, 2.55M tokens, and 15.9 minutes; Grok averages 15.2 steps, 0.75M tokens, and 14.0 minutes; GPT-5.4-Mini averages 21.7 steps, 0.65M tokens, and 6.1 minutes. More tokens do not imply better performance, and failed runs consume more tokens than successful ones for every model, consistent with agents becoming trapped in retry loops rather than converging on a correct setup (Wang et al., 3 Jun 2026).

5. Failure modes and representative case studies

DeployBench’s most important analytic result is the dominance of a completion-judgment problem. Out of 154 failures across all model-task runs, 97 are agent-terminated self-stops: the agent explicitly chose to finish rather than timing out or hitting a step cap, yet the hidden verifier judged the environment incorrect. This means many failures are not simple inability-to-progress failures; they are cases where the agent believes it has finished successfully, or sufficiently, while missing the actual paper-specific target (Wang et al., 3 Jun 2026).

The failure-location breakdown supports that interpretation. Of the 154 failed runs, 71 are caught by Layer 1: 39 runtime crashes, 27 dependency errors, and 5 compilation errors. The remaining 83 failures pass the global checks and are rejected only by Layer 2. More than half of failures therefore look runnable in generic terms but do not satisfy the designated experiment (Wang et al., 3 Jun 2026).

The self-stop statistics are especially striking. Self-stop failures are 12 of 25 failures for GPT-5.3-Codex, 15 of 37 for Gemini-3.1-Pro, 28 of 45 for Grok-4.20, and 42 of 47 for GPT-5.4-Mini. Among the 97 self-stopped failures, 87 show RUNBOOK/verifier mismatch: 55 runs report SUCCESS in the required RUNBOOK.MD despite failing verification, and 32 report PARTIAL but still terminate unsuccessfully. The paper calls the broader phenomenon “self-validation drift”: pre-finish checks are meaningful in a local sense but weaker than the actual target, aimed at the wrong script, or missing a paper-specific dependency (Wang et al., 3 Jun 2026).

The AdaLoRA case illustrates this directly. GPT-5.3-Codex installed a CUDA-enabled PyTorch stack, installed the local loralib package, and wrote a toy smoke test around a small SVDLinear model. The smoke test passed, so the agent wrote RUNBOOK = SUCCESS and stopped. The hidden verifier, however, ran the paper-specific target: a lightweight GLUE CoLA fine-tuning procedure through run_glue.py using DeBERTa-v3-base and AdaLoRA modules. That target depended on the HuggingFace evaluation stack, which the smoke test never exercised; verification failed with ModuleNotFoundError: huggingface_hub (Wang et al., 3 Jun 2026).

The SILT case illustrates the benchmark’s legacy-compatibility dimension. GPT-5.4-Mini patched include paths and type substitutions while trying to build the 2011 SILT key-value store, but could not bridge a deeper API gap. A successful deployment required new compatibility code. GPT-5.3-Codex succeeded by writing a 52-line shim in fawnds/tbb/atomic.h and modifying eight additional files so that SILT’s legacy TBB atomic interface mapped onto modern std::atomic (Wang et al., 3 Jun 2026).

Across the benchmark, the recurring root-cause patterns are dependency and package resolution, GPU/CUDA setup, native-build and toolchain failures, artifact and output mismatch, and system- or VM-level execution failures. These frequently co-occur. Among the 20 tasks that all four models fail, every one involves at least two patterns, and 14 involve three or more, with an average of 3.4 patterns per task. Mixed-outcome tasks average only 2.47 patterns (Wang et al., 3 Jun 2026).

Trajectory analysis reinforces that simply granting more budget is insufficient. After failed shell commands, successful runs use “Inspect” plus “Adapt” 48.5% of the time versus 32.7% in failed runs, while failed runs exhibit “Loop” behavior 60.3% of the time versus 44.3% in successful runs. This suggests that better diagnosis and recovery, rather than merely longer trajectories, are central missing capabilities (Wang et al., 3 Jun 2026).

6. Relation to prior benchmarks, scope boundaries, and implications

DeployBench is positioned as broader and more infrastructure-realistic than prior environment-setup benchmarks such as Installamatic, ExecutionAgent, EnvBench, SetupBench, and Repo2Run. Relative to these, it is multi-domain rather than open-source-repository-only, uses per-task verifiers tied to designated paper experiments rather than generic install/build/test signals, starts from a fresh VM rather than a containerized environment, includes system-level setup such as kernel and QEMU tasks, and includes legacy artifacts requiring compatibility repair (Wang et al., 3 Jun 2026).

It is also distinct from earlier research-repository deployment benchmarks. CSR-Bench evaluates deployment of 100 GitHub repositories from CS research, stages the task into setup, download, training, evaluation, and inference, and measures command completion behavior in standardized Docker environments (Xiao et al., 10 Feb 2025). DeployBench, by contrast, explicitly targets native host deployment from raw infrastructure, disallows Docker, couples success to hidden paper-specific experiment verification, and expands beyond recent CS repositories into systems and scientific-computing artifacts that may require GPU driver installation, VM boot paths, or modern compatibility shims (Wang et al., 3 Jun 2026).

The benchmark is clear about its scope. It focuses on deployment, not the full scientific workflow. The goal is to make an environment runnable for a designated experiment, not to debug arbitrary hypotheses, modify algorithms, conduct full reproduction studies, or perform new scientific discovery. The verifier runs a lightweight target rather than the full benchmark suite. The benchmark is also necessarily selective: tasks are manually curated and candidates whose datasets or model weights are unavailable are excluded, which improves feasibility but may bias the benchmark away from the worst forms of artifact rot (Wang et al., 3 Jun 2026).

The broader implication is that current LLM agents remain far from autonomous deployment of real research artifacts, and that the hardest missing capabilities are not only dependency repair or system setup in isolation. DeployBench suggests three salient gaps: dependency repair, system-level setup, and strong completion checks. The last is particularly important: self-designed smoke tests and runbook self-assessments are not reliable substitutes for a lightweight paper-specific target (Wang et al., 3 Jun 2026).

A plausible implication is that future progress on scientific research agents will require tighter integration of paper understanding, repository inspection, verifier-like target inference, and completion-detection mechanisms that can avoid self-validation drift. DeployBench’s contribution is to make that requirement measurable.