TerminalBench (TB): CLI Agent Benchmark

• TerminalBench (TB) is a benchmark that evaluates terminal agents using realistic, Dockerized CLI tasks with deterministic, outcome-based verification.
• TB distinguishes its releases—TB 1.0 and TB 2.0—by curating tasks with unique environments, comprehensive tests, and explicit time limits.
• The benchmark emphasizes real execution, tool discovery, and error recovery, offering a practical measure for agent performance on long-horizon terminal tasks.

TerminalBench (TB) is a benchmark for evaluating terminal agents on hard, realistic tasks in command line interfaces. Across the papers that use it, TB is defined by real terminal environments, containerized task packaging, and executable verification of the final environment state rather than text-only judging. The literature distinguishes an original benchmark used as TB 1.0 and the later release “Terminal-Bench 2.0”; the earlier benchmark is described as 200 curated Dockerized tasks with deterministic verification scripts, while TB 2.0 contains 89 tasks with unique environments, human-written solutions, comprehensive tests, and explicit time limits (Merrill et al., 17 Jan 2026, Zhu et al., 6 Feb 2026).

1. Purpose, motivation, and benchmark lineage

TB was introduced to measure whether an agent can complete valuable, long-horizon tasks in diverse domains through direct interaction with a live terminal. The command line is treated as a ubiquitous, high-leverage interface for software engineering, scientific computing, cybersecurity, data engineering, and ML/AI workflows, and the benchmark is positioned against narrower alternatives such as code-generation-only tasks, natural-language-to-Bash translation, and GUI- or web-oriented evaluations. Its central target is outcome-driven competence in realistic sandboxed OS environments, where agents must discover tools, issue commands, handle failures, and modify system state (Merrill et al., 17 Jan 2026).

The original benchmark used as TB 1.0 is repeatedly described in later work as costly to construct: it “required extensive manual curation to produce merely 200 tasks.” TB 2.0 formalized a later release process, containing 89 tasks curated from 229 contributed candidates and selected through multi-stage auditing by three experienced reviewers per task. This progression suggests continuity in evaluation philosophy—real execution, containerization, and test-based verification—combined with a more explicit release and auditing pipeline in TB 2.0 (Merrill et al., 17 Jan 2026, Zhu et al., 6 Feb 2026).

Release	Scale	Packaging and verification
TB 1.0	200 curated tasks	Expert-authored Dockerized environment with deterministic verification script
TB 2.0	89 tasks from 229 candidates	Unique environment, comprehensive tests, oracle solution script, time limit

2. Task model, environments, and verification semantics

TB tasks are terminal-native. In the TB 1.0 usage described by TermiGen, each task is provisioned as a self-contained Dockerized environment with a deterministic verification harness, and the agent interacts only via a bash shell. An episode is counted as a pass if and only if the agent’s final state passes all cases in the verification script, typically test.sh; that harness may call unit tests such as pytest, diff expected artifacts, or validate invariants, but it remains executable and deterministic within the container (Zhu et al., 6 Feb 2026).

TB 2.0 generalizes the same idea with a broader task interface. Each task provides an instruction, a Docker image or Dockerfile defining a unique environment, comprehensive tests, an oracle solution script, and a time limit. The agent may issue shell commands, edit files, run scripts, install packages, start servers or VMs, and use any tools available or install new ones; there is no benchmark-level restriction to Bash-only unless the scaffold enforces it. Verification is outcome-driven: tests inspect the final container state rather than the action sequence, which permits multiple valid solution strategies and reduces dependence on a single canonical trace (Merrill et al., 17 Jan 2026).

The benchmark’s realism derives from its insistence on grounded execution. Tasks may require environment setup, dependency management, debugging, data wrangling, training, system and network configuration, cryptanalysis, or forensics. At the same time, TB 2.0 does not claim absolute determinism in every circumstance: although authors and CI tools check solvability by running oracle solutions, some inherent nondeterminism may remain due to external resources, hardware, or long-running tasks. Internet access is allowed for package installation or artifact retrieval, and the paper notes a theoretical risk of “cheating” by finding oracle solutions online, though no such behavior was observed in tens of thousands of trials (Merrill et al., 17 Jan 2026).

3. Metrics, harnesses, and evaluation protocol

TB 1.0 is commonly reported with pass rate over the benchmark task set,

$$\mathrm{PassRate}=\frac{\#\text{ tasks passed}}{\#\text{ tasks total}},$$

and later work using TB 1.0 often reports Average Pass@1 by averaging over three independent runs. In the TermiGen evaluation, the model temperature is set to 0.6, no auxiliary tools such as browser or RAG are used, and the agent acts through a raw bash shell inside an isolated Docker container; success is the deterministic result of executing test.sh in that container (Zhu et al., 6 Feb 2026).

TB 2.0 formalizes aggregate success as

$$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$

with reported 95% confidence intervals over trials. The paper recommends multiple runs per agent–model pair, and its own experiments use at least five. Later studies on TB 2.0 also report Pass@k, defined conventionally as the probability that at least one of the $k$ generated solutions is correct, with $k\in\{1,2,4,8,16\}$ in TerminalTraj and Pass@1/Pass@3 in Terminal-World (Merrill et al., 17 Jan 2026, Wu et al., 1 Feb 2026, Cheng et al., 20 May 2026).

The benchmark ecosystem includes a standardized harness. All TB 2.0 tasks use the Harbor format, and the Harbor harness integrates multiple agent scaffolds, including Claude Code, Codex CLI, Gemini CLI, OpenHands, Mini‑SWE‑Agent, and the neutral baseline Terminus 2. TB 2.0 is distributed through the Harbor registry and can be launched with:

$33.3\%\leq$7

Terminus 2 is a minimal scaffold that uses only a headless terminal (tmux) to send keystrokes and execute Bash commands, while still supporting interactive TUI use, scrolling, multiple shells, and a context summarization module (Merrill et al., 17 Jan 2026).

4. Task domains, difficulty structure, and relation to other benchmarks

TB spans heterogeneous technical workflows. TB 2.0 lists domains including system operations and configuration, software engineering and debugging, data wrangling and processing, package management and environment setup, Git and release workflows, ML/AI, security/forensics/crypto, and text processing and scripting. The earlier TB 1.0 description in TermiGen does not enumerate official categories, but it consistently characterizes tasks as realistic terminal workflows involving programming and compilation, system administration and DevOps, data processing and ML/MLOps, networking/security/forensics, and general file or text operations (Merrill et al., 17 Jan 2026, Zhu et al., 6 Feb 2026).

These tasks are intentionally long-horizon and interdependent. Many require sequencing dozens of actions across shell use, editors, scripts, library installation, compiles, tests, validators, servers, and VMs. In TB 2.0, author-supplied time estimates indicate that for experts most tasks are at most one day, while for juniors many are one day to one week, with a few exceeding one week. This is consistent with the benchmark’s role as a realism-oriented measure rather than a short-context coding suite (Merrill et al., 17 Jan 2026).

TB 2.0 also introduces empirical difficulty labels based on Terminus 2 pass rates across selected frontier models: a task is Easy if pass rate is at least $66.7\%$, Medium if $33.3\%\leq$ pass rate $<66.7\%$, and Hard if pass rate is below $33.3\%$. Correlation with human labels is positive $(r=0.436,\ p<0.001)$, and $93.3\%$ of human-hard tasks are also empirically hard. This gives TB a dual characterization: it is curated by human judgment but also calibrated by observed agent behavior (Merrill et al., 17 Jan 2026).

Relative to adjacent benchmarks, TB is distinguished by its emphasis on end-to-end terminal workflows in live OS environments. SWE‑Bench and related software engineering benchmarks focus more narrowly on repository-level issue resolution or code synthesis and repair; WebArena, Visual Web Arena, OS World, and AppWorld focus on browser or desktop interaction; τ‑Bench and BFCL emphasize function-calling correctness; MLGym and ReplicationBench overlap in realism and long-horizon structure but are centered on scientific or ML settings. TB’s differentiating property is CLI-first, execution-grounded evaluation across many technical domains (Merrill et al., 17 Jan 2026).

5. Reported performance and what TB discriminates

TB is deliberately difficult. In the TB 2.0 benchmark paper, frontier models and agents score under $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$0, with the top reported result at $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$1 for Codex CLI paired with GPT‑5.2. Other reported results include $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$2 for Terminus 2 + Claude Opus 4.5, $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$3 for Terminus 2 + Gemini 3 Pro, and an open-weight best of approximately $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$4 for Terminus 2 + Kimi K2 Thinking. Some hard tasks remain unsolved by any model–agent pair, including examples involving complex system configuration, kernel driver compilation, and database migration (Merrill et al., 17 Jan 2026).

The earlier TB 1.0 evaluations in follow-on training papers show similarly strong discrimination among open-weight terminal agents. In the TermiGen comparison on TB 1.0, proprietary systems range from $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$5 for Apex2 (Claude‑4.5‑Sonnet) to $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$6 for o4‑mini, general-purpose open-weight bases range from $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$7 for GPT‑OSS‑120B to $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$8 for Qwen‑2.5‑Coder‑32B‑Instruct, and terminal-specific fine-tuned baselines such as Reptile, LiteCoder‑30a3b, and TerminalAgent‑32B remain below $$S=\frac{1}{|T|}\sum_{i=1}^{|T|} X_i,\qquad X_i\in\{0,1\},$$9. TermiGen‑Qwen‑2.5‑Coder‑32B reaches $k$0 Average Pass@1, which the paper describes as a new open-weights state of the art in its comparison and as $k$1 percentage points above o4‑mini (Zhu et al., 6 Feb 2026).

Later work extends these observations across both TB versions. TerminalTraj reports $k$2 on TB 1.0 and $k$3 on TB 2.0 for TerminalTraj‑32B, with consistent gains over Qwen2.5‑Coder backbones and strong Pass@k scaling up to pass@16 of $k$4 in the main text and $k$5 in the conclusion (Wu et al., 1 Feb 2026). Terminal‑World reports $k$6 Pass@1 and $k$7 Pass@3 on TB 2.0 for Terminal‑World‑32B, exceeding Nemotron‑Terminal‑32B by $k$8 Pass@1 and $k$9 Pass@3 while using only $k\in\{1,2,4,8,16\}$0 of Nemotron‑Terminal’s training trajectories (Cheng et al., 20 May 2026). TACO, a compression adapter rather than a new terminal policy, improves many strong models on TB; for example, on TB 2.0 it raises DeepSeek‑V3.2 from $k\in\{1,2,4,8,16\}$1 to $k\in\{1,2,4,8,16\}$2, Qwen3‑Coder‑480B from $k\in\{1,2,4,8,16\}$3 to $k\in\{1,2,4,8,16\}$4, and MiniMax‑M2.5 from $k\in\{1,2,4,8,16\}$5 to $k\in\{1,2,4,8,16\}$6 while reducing token overhead in several settings (Ren et al., 21 Apr 2026).

Taken together, these results show that TB discriminates not only among base models but also among data-generation pipelines, agent scaffolds, test-time sampling strategies, and long-context efficiency methods. A plausible implication is that TB’s difficulty comes from several interacting demands—tool discovery, environment assembly, verification discipline, and recovery from runtime errors—rather than from any single subskill.

6. Methodological influence on terminal-agent research

TB has become a primary external evaluation target for methods that generate training environments and trajectories. TermiGen is explicit about this dependency: it is introduced to synthesize the kind of high-fidelity environments and resilient expert trajectories that TB requires. Its ablations on TB 1.0 isolate three effects. First, training on verifiable, Docker-executed observations yields $k\in\{1,2,4,8,16\}$7 Pass@1 versus $k\in\{1,2,4,8,16\}$8 for simulated observations, and an audit finds that $k\in\{1,2,4,8,16\}$9 of simulated data contain observation errors. Second, active error-injection training reaches $66.7\%$0 versus $66.7\%$1 for standard “golden” trajectories without injection. Third, keeping imperfect attempts helps substantially: using only perfect runs gives $66.7\%$2, while including all $66.7\%$3 trajectories raises performance to $66.7\%$4. These findings imply that TB rewards grounded execution signals and explicit supervision for diagnosis and recovery, not just idealized successful traces (Zhu et al., 6 Feb 2026).

TerminalTraj frames the same issue in terms of executability and verifiability. It constructs Dockerized environments at scale, synthesizes executable validation code, and retains only trajectories that pass instance-specific validation. On TB, the resulting models improve Qwen2.5‑Coder backbones by up to $66.7\%$5 on TB 1.0 and $66.7\%$6 on TB 2.0, and domain ablations show that removing Web Service or QEMU produces the largest drops, up to $66.7\%$7 on TB 1.0 and $66.7\%$8 on TB 2.0. This suggests that TB is especially sensitive to system-level Linux skills and tool-centric interaction patterns (Wu et al., 1 Feb 2026).

Terminal‑World approaches the same benchmark through skill-centered synthesis. Each skill encodes what to accomplish, when it applies, and how to execute; the method then co-derives task instructions, environment blueprints, evaluation criteria, and execution guidelines, and builds environments through an iterative generate–verify–repair process. On TB 2.0, the paper attributes its gains partly to tighter task–environment–verifier alignment and partly to the retention of failure trajectories, noting that $66.7\%$9 of verifier-failed trajectories were judged semantically complete by a four-judge majority vote. This indicates that TB can reward training corpora that preserve locally correct but globally incomplete behaviors, because such traces still contain useful supervision for long-horizon repair (Cheng et al., 20 May 2026).

TB has also motivated research on agent efficiency rather than only correctness. TACO treats TB 1.0 and TB 2.0 as prime examples of long-horizon terminal tasks with heavy, noisy stdout/stderr streams. It shows that full-history retention induces cumulative token growth of

$33.3\%\leq$0

and uses evolving compression rules that preserve critical error traces while removing repetitive progress noise. On TB, this yields consistent gains of $33.3\%\leq$1 across strong agentic models and around $33.3\%\leq$2 under the same token budget. The benchmark therefore probes not only task-solving ability but also the ability to allocate context efficiently under terminal-specific observation distributions (Ren et al., 21 Apr 2026).

7. Limitations, reproducibility, and open issues

TB’s principal limitation is curation cost relative to scale. The original benchmark is described as requiring extensive manual curation for 200 tasks, and TB 2.0 reports approximately three reviewer-hours per included task after selection. This high labor burden explains why later work increasingly uses TB as held-out evaluation rather than as direct training data, and why synthetic environment-generation pipelines have become central in the surrounding literature (Zhu et al., 6 Feb 2026, Merrill et al., 17 Jan 2026).

A second limitation is benchmark finiteness and public availability. TB 2.0 is openly distributed, includes a canary string to aid decontamination, and does not provide a private held-out set in the release described. The paper explicitly notes contamination risk, and later training-data papers filter repositories created after the TerminalBench release or filter files that match TB test solutions. This suggests that TB is now influential enough that benchmark hygiene is an active methodological concern rather than a peripheral one (Merrill et al., 17 Jan 2026, Wu et al., 1 Feb 2026).

Reproducibility is comparatively strong by benchmark standards. TB 2.0 provides a website at https://www.tbench.ai/, Harbor distribution, experiment configurations, comprehensive CI checks, oracle-solution solvability checks, contributor checklists, LLM-backed quality checks, and an adversarial exploit agent used to search for shortcuts or reward hacking. The project reports an overall ABC score of approximately $33.3\%\leq$3, with Outcome Validity $33.3\%\leq$4, Task Validity $33.3\%\leq$5, and Benchmark Reporting $33.3\%\leq$6 (Merrill et al., 17 Jan 2026).

Even so, later papers identify persistent failure modes on TB tasks: long-horizon brittleness, cascading mistakes, insufficient global planning, context overflow, execution deadlocks, premature completion, and task substitution. Some of these errors arise from the benchmark’s realism rather than from flaws in specification. A plausible implication is that TB’s lasting value lies not in being fully solved or perfectly closed-world, but in exposing the interaction between reasoning, environment manipulation, verification, and recovery in a setting where success is mechanically checkable and operationally meaningful (Cheng et al., 20 May 2026, Zhu et al., 6 Feb 2026).