DeployBench: Benchmarking LLM Agents for Research Artifact Deployment

Abstract: LLM agents have made rapid progress on software engineering and ML research tasks, but these advances often assume access to a working runnable environment. For research artifacts released alongside published papers, setting up such an environment from a fresh machine remains a major bottleneck. Existing environment setup benchmarks do not cover the full scope of research artifact deployment, which involves multi-language toolchains, system-level dependencies beyond containers (e.g. GPU/CUDA and kernel configurations), and legacy artifact compatibility. We introduce DeployBench, a multi-domain benchmark of 51 research-artifact deployment tasks spanning AI/ML, computer systems, and scientific computing, covering all these dimensions. Each task is verified by a hidden pipeline that executes the paper's designated experiment and checks its outputs. Evaluating four state-of-the-art LLMs with OpenHands yields pass-rates from 7.8% - 51.0% . Failures are dominated by a completion-judgment problem: 97 of 154 are agent-terminated self-stops, where the agent's pre-finish checks validate a different or weaker target than the paper-specific task requires. DeployBench highlights the gap between current agents and autonomous deployment, and offers a realistic testbed for scientific research agents.

• The paper introduces DeployBench, a benchmark that evaluates LLM agents’ ability to autonomously deploy research artifacts from raw infrastructures using native VM setups.
• It demonstrates notable performance gaps among top LLMs, highlighting challenges in dependency repair and system-level configuration across multi-language and legacy codebases.
• The study emphasizes the need for integrated verifier mechanisms and enhanced recovery frameworks to ensure accurate, end-to-end deployment in realistic research settings.

DeployBench: A Multi-Domain Benchmark for Autonomous LLM Research Artifact Deployment

Motivation and Background

While LLM-based agents have achieved marked progress on software engineering and research-code interaction tasks, evaluations to date typically operate under the assumption of a pre-configured, runnable environment. In realistic research workflows, deploying published artifacts from scratch—especially those with complex, multi-language, system-level, and legacy dependencies—remains a high-friction barrier hindering reproducibility and downstream research. Existing benchmarks inadequately probe these deployment steps, generally focusing on package installation within containers, Python-only repositories, or evaluation via generic signals such as import checks or test suite completion.

DeployBench explicitly targets this capability gap by providing a comprehensive, multi-domain benchmark for evaluating LLM agents’ proficiency in fully autonomous research artifact deployment from raw infrastructure, spanning AI/ML, computer systems, and scientific computing contexts.

Figure 1: The DeployBench evaluation pipeline spans task selection from top-venue artifacts across diverse domains, and performs end-to-end native environment construction starting from a blank VM.

Benchmark Design and Task Specification

DeployBench comprises 51 tasks drawn from papers published between 2008 and 2025, sampling three main research domains: AI/ML (25 tasks), computer systems (19), and scientific computing (7). Tasks are selected for ecological breadth—covering 11 primary programming language ecosystems (Python, C/C++, Rust, Fortran, Julia, R, Haskell, Lua, Racket, JavaScript, Java)—and for realistic deployment complexity. Artifacts range from recent, actively maintained repositories to legacy codebases requiring extensive compatibility repair (e.g., adapting to modern compilers and library APIs).

Each DeployBench task grants the agent only the paper, the raw code repository, and an unconfigured cloud VM. Agents are explicitly forbidden from using Docker or containerized runners: all dependency resolution, system configuration (including GPU/CUDA driver stacks, kernel compilation, etc.), and artifact fixes must occur natively to reflect realistic operational demands.

Task success is determined by a hidden two-stage verifier, where the first layer detects generic failures (e.g., missing dependencies, compilation/runtime errors) and the second executes a paper-specific minimal experiment, checking for correct outputs and side effects that validate full-stack deployment for the intended research outcome. This stringent validation protocol enforces fidelity to the original artifact intention, beyond superficial successful builds.

Figure 2: The DeployBench evaluation pipeline applies a global log parser and then a paper-derived task-specific check to rigorously assess deployment success.

Experimental Evaluation and Model Analysis

The benchmark evaluates four state-of-the-art LLMs (GPT-5.3-Codex, Gemini-3.1-Pro, Grok-4.20, GPT-5.4-Mini) embedded in the OpenHands agent scaffold. Results show considerable performance variability: GPT-5.3-Codex achieves 51.0% pass rate, but other models fail to cross even the 30% success threshold (Gemini-3.1-Pro at 27.5%, Grok-4.20 at 11.8%, GPT-5.4-Mini at 7.8%). Task difficulty correlates with both the need for legacy code repair and system-level setup (e.g., GPU driver and kernel stack resolution). Notably, only 40% of legacy artifacts (<2020) are solved by the best model, versus 53.7% for recent ones.

Further, detailed trajectory analysis demonstrates that high token utilization or longer command traces do not predict success—solved runs are actually shorter and feature more targeted log inspection and adaptation steps, whereas failed runs cluster around repeated, unproductive retries or premature self-termination.

Failure Analysis and Completion-Judgment Problem

Out of 154 failed deployments, 97 are self-terminated by the agent via a "completion judgment" mechanism, i.e., the agent's own pre-finish checks misidentify a weaker or unrelated smoke test as sufficient, instead of the task-specific execution required for success. Even for models that perform relatively robust validation (e.g., GPT-5.3-Codex and Gemini-3.1-Pro), failures frequently stem from agents enacting generic or incomplete checks drawn from their inferred task descriptions, rather than the actual paper-specified target.

Other significant root causes are:

• Dependency/pipeline breakage across multi-language stacks, often compounded by version drift;
• GPU/CUDA stack misalignment, especially when native driver installation is required (vs. containerized environments);
• Native build failures for non-Python artifacts (e.g., kernel builds);
• Artifact/output mismatches (unexpected artifact layouts, mismatches in binary and script runs);
• System- or VM-level runtime errors, most acute in QEMU/kernel-boot tasks.

Notably, generic, shallow checks (e.g., import/test command completion or shell success) fail to capture 53.9% of crucial failure cases, reiterating the necessity of per-task paper-derived verifiers.

Comparative Positioning and Implications for Agent Design

Relative to prior benchmarks (e.g., EnvBench (Eliseeva et al., 18 Mar 2025), CSR-Bench [Xiao2025CSRBenchBL], ResearchEnvBench (Wang et al., 6 Mar 2026)), DeployBench uniquely combines:

• Support for multiple language/toolchain ecosystems per task,
• Per-paper task validation via actual experimental execution,
• Inclusion of true system-level (host/VM) setup, and
• Explicit coverage of legacy (pre-2020) artifacts demanding compatibility repair.

This multidimensional scope exposes limits in current agent scaffolds. Specifically, the results highlight that present-day LLM agents lack robust dependency repair, system-level orchestration, and—most critically—a mechanism for extracting (or learning) verifiable completion targets from semi-structured scientific artifacts in a way that aligns to intended, paper-defined behaviors.

Forward-Looking Considerations

Practical progress in fully autonomous agentic research demands:

• Frameworks for extracting task-specific verifier targets directly from paper text and code structure,
• Enhanced log/error signal integration for adaptive recovery after failures,
• Tightly integrated, cross-language dependency resolution and repair,
• More principled completion judgment, potentially via learned or modularized verifier-synthesis agents, avoiding reliance on heuristic smoke tests.

As LLM agents advance, DeployBench provides a realistic and discriminative testbed for measuring progress toward reliable, reproducible, end-to-end deployment in real-world research settings.

Conclusion

DeployBench establishes a complex, multi-domain benchmark to stress-test LLM agents on realistic research artifact deployment. Evaluation of multiple top-tier LLMs reveals a steep challenge: the most capable model solves only about half the tasks, and failure analysis reveals that erroneous self-diagnosis of deployment success is the dominant limiting factor. These results directly inform requirements for future agents: deeply integrated dependency repair and system-level management, coupled with verifier-like completion checks anchored to the true semantics of scientific artifacts.