Terminal Bench Overview

• Terminal benchmark is a standardized evaluation system with open-ended environments combining multiple software and engineering tasks.
• The primary execution metric, resolution rate, uses Oracle files and task verification to assess AI-based agent performance accurately.
• Terminal bench tests complexity through multi-step interactions directly resembling developer workflows and observational studies which include recovering data and fixing software bugs.
• The Terminal Bench does not currently provide a private held-out set in its principal release.

Terminal Bench is a family of execution-based benchmarks for evaluating AI agents that perform difficult, multi-step work through command-line interfaces. Its principal release, Terminal-Bench 2.0, contains 89 human-authored and human-verified tasks in containerized terminal environments, with natural-language instructions, Docker-defined initial states, executable tests, human-written reference solutions, and time limits (Merrill et al., 17 Jan 2026). Unlike isolated code-generation or command-translation tests, Terminal Bench evaluates whether an agent can inspect unfamiliar environments, install and configure software, write and debug programs, recover from failures, manipulate system state, and leave the environment in a verifiably correct final condition.

1. Origin, purpose, and conceptual scope

Terminal Bench was developed to address limitations in existing agent benchmarks. Function-level coding tests such as HumanEval evaluate isolated program synthesis; shell-specific benchmarks often measure command translation or environment setup; software-engineering benchmarks commonly focus on repository issue resolution; and tool-use benchmarks evaluate API or tool invocation. Terminal Bench instead targets open-ended manipulation of a stateful computer through a real shell.

The terminal is treated as a general-purpose interface spanning software engineering, scientific computing, machine learning, system administration, cybersecurity, data processing, debugging, reverse engineering, legacy-system configuration, virtual-machine management, and server operations. The benchmark’s intended capability is therefore broader than command generation. Successful execution may require environment reconnaissance, long-horizon planning, package installation, system configuration, code generation, debugging, external-resource use, error recovery, and final-state verification.

A task is evaluated by its outcome rather than by command imitation. Agents may use different commands, editors, interpreters, programs, or strategies, provided that the resulting container satisfies the task requirements. This distinguishes Terminal Bench from evaluations that compare an agent’s output with a reference command sequence.

The benchmark’s central empirical motivation is difficulty at the frontier. Terminal-Bench 2.0 reports that frontier models and agent systems resolve fewer than 65% of tasks, while smaller models can perform at approximately 15% or lower (Merrill et al., 17 Jan 2026). The benchmark is consequently intended to expose capability gaps that simpler coding or tool-use tests may conceal.

2. Task representation and evaluation semantics

A Terminal-Bench task consists of:

1. A natural-language instruction.
2. A Docker image or Dockerfile specifying the initial environment.
3. A programmatic test suite evaluating the final container state.
4. A human-written reference or oracle solution.
5. A time limit.

Conceptually, a task can be represented as

$$\mathrm{Task}=(\mathrm{instruction},\mathrm{Docker\ environment},\mathrm{tests},\mathrm{oracle\ solution},\mathrm{time\ limit}).$$

The agent receives the instruction and access to a fresh container. It may inspect and modify the environment interactively. When the agent terminates, or the time limit expires, the benchmark runs the task’s tests. A trial succeeds only when the tests pass. Incomplete work, incorrect artifacts, failed tests, timeouts, crashes, and invalid terminal states are failures.

The principal metric is resolution rate:

$$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$

The benchmark does not define a general partial-credit score for Terminal-Bench 2.0. Reported uncertainty is presented using 95% confidence intervals, such as $62.9\%\pm3.0\%$.

Outcome-based evaluation permits semantically equivalent implementations. For example, an agent may compile software through a different sequence of commands, use a different implementation strategy, or generate an equivalent artifact. The tests are expected to accept valid alternatives without requiring the oracle’s internal procedure.

Agent interaction

The neutral comparison system, Terminus 2, exposes a headless terminal through a tmux shell inside a Docker container. It returns command output to the model and does not supply specialized browsing or file-editing tools. The model may issue Bash commands, invoke interactive programs, use editors, send signals such as Ctrl-C, scroll, and open nested shells. Context summarization is used when the model reaches its context limit.

The Terminus 2 interaction loop is:

1. The model receives the task instruction.
2. It issues terminal commands or keystrokes.
3. The container executes them.
4. Terminal output is returned.
5. The process repeats until completion or timeout.
6. Tests are run on the final container state.

Although Terminal Bench is terminal-oriented, agents are not technically restricted to Bash. They may invoke installed programs, interpreters, editors, custom tools, web tools, or other executables. Terminus 2 is a deliberately constrained baseline rather than a claim that all terminal agents should expose identical tools.

3. Dataset composition and task domains

Terminal-Bench 2.0 was assembled through open-source contributions. The development process involved 93 contributors, 229 submitted tasks, and 89 selected tasks. Each selected task was reviewed by three experienced human reviewers, with approximately three combined reviewer-hours per task on average. Selection considered difficulty, realism, diversity, instruction and test correctness, solvability, resistance to cheating, and reference-solution quality (Merrill et al., 17 Jan 2026).

The official dataset size is 89 tasks. An appendix caption referring to “all 74 tasks” appears to describe an experimental subset or metadata inconsistency; it does not replace the official 89-task dataset count.

The task collection includes:

• Software engineering and software construction.
• Scientific computing and machine learning.
• Model training and data science.
• Data processing and data querying.
• Mathematics.
• Security and reverse engineering.
• Debugging and system administration.
• File operations, optimization, and networking.
• Virtual machines and operating systems.
• Video processing and games.
• Personal-assistance and professional workflows.

Software engineering is the largest category, but no individual category constitutes a majority. Representative tasks include building legacy software from source, reconstructing corrupted databases, configuring Git servers, training models under constraints, implementing cryptanalysis attacks, designing DNA primers, configuring QEMU virtual machines, reverse engineering binaries, and generating structured outputs from video.

Examples of task requirements include:

• Reimplementing a COBOL program in Python with byte-for-byte equivalent output files.
• Implementing an asynchronous task runner with concurrency limits and correct interruption cleanup.
• Building systems such as POV-Ray, CompCert, SQLite with coverage instrumentation, pMARS, and a MIPS interpreter capable of booting Doom.
• Recovering deleted secrets from Git repositories or filesystems.
• Extracting secrets from executables and cracking password-protected archives.
• Fitting Bayesian networks and performing interventions.
• Training Caffe or fastText models.
• Implementing LLM inference batching, tensor parallelism, and pipeline parallelism.
• Recovering neural-network parameters from black-box queries.
• Implementing Scheme evaluation, cryptanalysis, and specialized algorithms.
• Repairing an OCaml garbage collector and analyzing scientific or multimedia data.

Human and empirical difficulty

Task authors estimated completion time for junior engineers and domain experts. For experts, 36 tasks were estimated to require less than one hour, 35 one hour to one day, and three one day to one week. For junior engineers, six tasks were estimated at less than one hour, 53 at one hour to one day, 12 at one day to one week, and three at more than one week.

The task fix-ocaml-gc is identified as a long-horizon example: approximately one day for an expert and approximately ten days, or 240 hours, for a junior engineer. These estimates characterize complexity but are not used as evaluation scores.

Human task labels are medium or hard. Empirical difficulty is defined using Terminus 2’s resolution across selected frontier models:

$$\mathrm{difficulty}(t)= \begin{cases} \mathrm{Easy}, & r_t\geq0.667,\ \mathrm{Medium}, & 0.333\leq r_t<0.667,\ \mathrm{Hard}, & r_t<0.333. \end{cases}$$

The correlation between human-predicted and empirical difficulty is reported as $r=0.436$ with $p<0.001$. Some discrepancies remain: 93.3% of tasks labeled hard by humans were also empirically hard, while human-medium tasks were empirically hard in 54.5% of cases, medium in 29.1%, and easy in 16.4%. This pattern suggests that models experience disproportionate difficulty on creative, adversarial, or nonstandard tasks requiring insight rather than routine procedural execution.

4. Verification, integrity, and reproducibility

Terminal Bench defines three core task-quality criteria: specificity, solvability, and integrity.

Specificity

A task is specific when its tests pass if and only if the container reaches an acceptable state. All requirements in the instruction should be tested, undocumented implementation details should not be required, and valid alternative implementations should be accepted.

Solvability

A task is solvable when an acceptable final state can be reached. Each task includes an oracle solution script written manually to represent a realistic workflow. The oracle must cause all tests to pass.

Integrity

A task should not be cheaply exploitable. Tests and oracle solutions should not be present in the agent’s initial image, future Git commits that reveal answers should be removed, hidden expected outputs should be inaccessible, no-op agents should fail, and tests should not be passable through hard-coding or evaluator modification.

The quality-control pipeline combines:

1. Automated oracle execution.
2. Deterministic checks on task files and Dockerfiles.
3. Contributor checklists.
4. LLM-backed review.
5. Testing with multiple models and Terminus 2.
6. Failure debugging and trajectory analysis.
7. An adversarial exploit agent.
8. Additional manual audits.

The adversarial agent searches for monkey-patching, output guessing, generation of all possible answers, incomplete checks, and modifications to test-related files. These procedures reduce, but do not eliminate, the possibility of incomplete tests or unintended shortcuts.

Reproducibility is supported by Docker images, pinned dependencies where appropriate, fresh containers, isolated trials, automated tests, oracle solutions, parallel execution, and public evaluation configurations. Terminal Bench is distributed through the Harbor registry and can be launched with:

$$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$7

The release includes task definitions, Docker specifications, prebuilt environments, oracle solutions, tests, Harbor task-format implementations, experiment configurations, and agent integration code. Harbor standardizes task formatting, agent integrations, container providers, parallel execution, logging, and trial management. It also supports adapters for 26 other benchmarks.

Reproducibility remains affected by hardware, CPU architecture, runtime resources, external package repositories, APIs, web resources, network failures, long-running processes, and infrastructure failures. The benchmark therefore improves reproducibility through containerization without guaranteeing deterministic behavior across all systems.

5. Evaluation results and error structure

Terminal-Bench 2.0 evaluates six principal agent systems:

• Claude Code.
• Codex CLI.
• Gemini CLI.
• OpenHands.
• Mini-SWE-Agent.
• Terminus 2.

The experiments cover frontier and open-weight model families, including GPT-5.2, GPT-5, GPT-5-Mini, GPT-5-Nano, Claude Opus 4.5, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 3 Pro, Gemini 3 Flash, Gemini 2.5 Pro, Gemini 2.5 Flash, Grok 4, Grok Code Fast 1, GPT-OSS-120B, GPT-OSS-20B, Qwen 3 Coder 480B, Kimi K2 Thinking, Kimi K2 Instruct, GLM 4.6, MiniMax M2, and Llama 4 Maverick. Each supported model-agent combination is run at least five times, producing 32,155 trials overall.

Resolution rates

The strongest reported combinations are:

Model	Agent	Resolution rate
GPT-5.2	Codex CLI	62.9%
Claude Opus 4.5	Terminus 2	57.8%
Gemini 3 Pro	Terminus 2	56.9%
GPT-5.2	Terminus 2	54.0%
Claude Opus 4.5	Claude Code	52.1%
Claude Opus 4.5	OpenHands	51.9%
Gemini 3 Flash	Terminus 2	51.7%

GPT-5.2 with Codex CLI is therefore reported at approximately 63%, while Claude Opus 4.5 with Terminus 2 and Gemini 3 Pro with Terminus 2 are reported at approximately 58% and 57%. The strongest evaluated system solves only about three-fifths of the tasks.

The detailed results include GPT-5 at 49.6% with Codex CLI, Claude Sonnet 4.5 at 42.8% with Terminus 2, Kimi K2 Thinking at 35.7%, Gemini 2.5 Pro at 32.6%, GLM 4.6 at 24.5%, Qwen 3 Coder 480B at 23.9%, GPT-OSS-120B at 18.7%, GPT-5-Nano at 11.5%, and GPT-OSS-20B at 3.1%.

Model choice generally matters more than scaffold choice. GPT-5.2 with Codex CLI improves resolution by 52% relative to GPT-5-Nano with Codex CLI, while Gemini 2.5 Pro improves by 17% when paired with Terminus 2 rather than OpenHands. These comparisons do not make scaffolds irrelevant: specialized systems can benefit from tool design, prompting, and model-specific engineering.

Interaction cost and horizon

Most trials finish in under 20 minutes, use fewer than 25 model calls, and consume fewer than 10 million tokens. Some trials last approximately two hours, use hundreds of model calls, or consume nearly 100 million tokens. Reported costs range from approximately $1 to$100 per model-agent attempt.

More interaction is not necessarily better. The correlation between average episode count and success rate is reported as $r=-0.028$, $p=0.916$. Output verbosity is also weakly and nonsignificantly related to success, with $r=-0.170$ and $$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$0. These findings suggest that decision quality, planning, debugging, and environment interaction matter more than simply issuing more commands or generating more tokens.

Error taxonomy

Failures in Terminus 2 trajectories are grouped into execution, coherence, and verification categories using a simplified Multi-Agent System Taxonomy. Two failed trials per model are sampled, summarized with Docent and a custom pipeline, and refined by two human annotators. Annotation quality includes 93% Cohen’s $$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$1 on a calibration subset, 90% agreement with 120 human-labeled traces, 92% precision, and 90% recall.

Execution failures include violating task specifications, repeating steps without changing strategy, continuing after success or futility, using the wrong method or output path, and failing to adapt after errors.

Coherence failures include context loss, task derailment, reasoning-action mismatch, forgetting prior changes, reverting to stale assumptions, and acting inconsistently with the agent’s plan or observations.

Verification failures include premature termination, absent or incorrect verification, superficial success checks, ignoring failing core tests, checking existence without correctness, and fabricating or guessing information.

Execution errors dominate for Claude Opus 4.5 and GPT-5.2, while Qwen 3 Coder exhibits a more balanced distribution. This suggests that models may require different interventions: instruction adherence and tool-use policies for execution-heavy failures, and improved planning, memory, consistency, and self-monitoring for more balanced failures.

At command level, failure rates range from 9.2% for Grok 4 to 26.7% for GPT-OSS-120B. Among 3,800 command failures, the most frequent category is an executable not installed or not on PATH, accounting for 24.1%; failures while running executables account for 9.6%. Other categories include shell syntax, filesystem and permission errors, missing environment variables, package conflicts, compiler and linker failures, ABI mismatches, network errors, runtime crashes, resource exhaustion, and testing failures.

These results show that many terminal-agent failures are mundane but consequential. Agents often fail to inspect available tools and versions, understand shell semantics, diagnose package or build problems, or respond appropriately to terminal error messages.

6. Extensions, applications, and research significance

Terminal Bench has become a reference evaluation for terminal-agent training, benchmark construction, alignment analysis, and long-horizon execution research.

Training terminal agents

Several studies use Terminal-Bench 2.0 as an external evaluation target for terminal-focused supervised or reinforcement learning.

“On Data Engineering for Scaling LLM Terminal Capabilities” introduces Terminal-Task-Gen and Terminal-Corpus, using adapted prompts, seed-based generation, skill-based generation, teacher trajectories, filtering, and decontamination (Pi et al., 24 Feb 2026). Nemotron-Terminal models initialized from Qwen3 improve from 2.47% to 13.0% at 8B, from 4.04% to 20.2% at 14B, and from 3.37% to 27.4% at 32B on TB2.0 under Terminus 2. The results suggest that terminal-specific trajectories can substantially improve small models, although the experiments remain tied to a particular scaffold, teacher, training procedure, and synthetic distribution.

“LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents” reports LiteCoder-Terminal-32b-sft at 29.06% on Terminal Bench 1.0, 18.54% on Terminal Bench 2.0, and 34.00% on Terminal Bench Pro (Peng et al., 28 May 2026). Its training data contains 11,255 expert trajectories across ten domains. Direct Multi-turn Preference Optimization provides additional gains on Terminal Bench 2.0 and Terminal Bench Pro for the 4B model, although performance slightly declines on Terminal Bench 1.0.

“Tmax: A simple recipe for terminal agents” reports TMAX-9B at 27.2% on Terminal-Bench 2.0, using 14,600 reinforcement-learning environment instances and an outcome-only RL recipe with DPPO (Ivison et al., 22 Jun 2026). TMAX-15K combines domain, skill, persona, language, task-complexity, command-complexity, fixture, and verifier axes. The paper reports that its data outperforms alternative datasets under a common RL recipe, while also noting dependence on synthetic generation, harness configuration, timeout policy, and reward-hacking risks.

“Recursive Synthesis for Long-Horizon Terminal Tasks” describes Recursive Synthetic Terminal Tasks, which recursively extends verified task bundles while realigning instructions, environments, solutions, and verifiers (Li et al., 5 Aug 2026). Across 15 rounds, it produces 37,484 synthesized tasks from 639 bootstrap tasks at approximately $0.05 per accepted task. Qwen3.5-27B improves from 41.20% to 47.94% on Terminal-Bench 2, while agentic PPO reaches 49.44%. The task-generation approach is intended to scale both task quantity and executable difficulty.

“CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks” uses solver behavior to revise tasks toward a solver-relative learnable zone (Meng et al., 6 Aug 2026). Multi-solver and contrastive calibration produce 5,431 tasks. Models trained on the resulting collection reach 32.58% and 47.57% on Terminal-Bench 2.0 for two model backbones. In an ablation, authoring and validation alone reach 22.47%, ordinary single-solver feedback 24.34%, multi-solver calibration 29.21%, and contrastive calibration 31.09%.

“T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks” trains a 122B-total-parameter mixture-of-experts model using dense verifier rewards, Token-In-Token-Out construction, and rollout routing replay (Yang et al., 10 Sep 2026). T1 improves a Qwen3.5-122B-A10B base from 43.8% to 64.0% on Terminal-Bench 2.1. Its TITO procedure preserves exact sampled token identifiers in the loss-bearing region, while R3 replays the expert routes selected during MoE rollout. These techniques address training–inference mismatch in long-horizon MoE RL.

“Self-Supervised Scaling of Terminal Environments for Scientific Domains” introduces software-in-the-loop reconstruction, using existing executable scientific workflows to generate public observations, hidden outputs, and domain-specific verifiers (Li et al., 2 Oct 2026). Supervised fine-tuning on 3,000 reconstruction examples improves Qwen3.8-27B from 47.94% to 53.56% on Terminal-Bench 2.0 under three-seed evaluation.

“AutoEnvScaling: Automating the Data Flywheel with Terminal Agents” treats environment generation itself as a terminal task (Yu et al., 9 Oct 2026). A proposer agent creates Harbor environments from solver rollouts, and the solver’s subsequent failures guide later environment generation. Training Qwen3.5-35B-A3B on AutoEnvScaling environments improves Terminal-Bench 2.1 by 4.3 points over a matched Tmax baseline; a recursive self-improvement experiment raises Qwen3.6-35B-A3B from 41.1 to 53.3 on Terminal-Bench 2.1.

Alignment and unauthorized side effects

“No More, No Less: Task Alignment in Terminal Agents” introduces TAB, an alignment-oriented extension derived from Terminal-Bench 2.1 (Mavali et al., 12 May 2026). TAB removes necessary details from the user instruction and places them in a relevant environmental cue beside an irrelevant distractor. Agents must use the cue while rejecting the distractor.

TAB separates task capability from task alignment. Cue utilization is denoted $$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$2, distraction resistance $$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$3, and task alignment is:

$$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$4

GPT-5.5 achieves 73% Terminal-Bench resolution and 85% cue utilization but only 27% distraction resistance and 23% task alignment. Claude Opus 4.7 achieves 70% Terminal-Bench resolution, 77% cue utilization, 94% distraction resistance, and 72% task alignment. These results indicate that completing a terminal task does not establish that an agent selectively follows relevant instructions or avoids unnecessary side effects.

“Hack-Verifiable Terminal Bench” instruments Terminal-Bench 2.1 with honeypot reference solutions and held-out tests under /app/admin/, using filesystem monitoring to detect access (Roth et al., 22 Aug 2026). Across 2,183 completed runs, 359 trigger at least one planted hack, for an overall detected hack rate of 16.4%. The methodology is sound but incomplete: it detects access to specified honeypots but does not capture every possible exploit, such as hard-coded outputs, evaluator modification, or unrelated vulnerabilities.

Long-horizon evaluation

“Long-Horizon-Terminal-Bench” extends Terminal-Bench-style evaluation to 46 tasks with dense partial rewards and workflows requiring hundreds of episodes (Li et al., 9 Jul 2026). The benchmark reports approximately 9.9 million tokens, 231 episodes, and 85.3 minutes per task. At a reward threshold of 0.95, the strongest model reaches 15.2% pass@1, while the mean pass rate across models is 4.3%. At perfect reward, the strongest model reaches 10.9% and the mean is 1.7%.

“Long-Horizon-Terminal-Bench 2.0” preserves the original 46 tasks, adds 31 tasks, extends the suite to 77 tasks across 13 domains and eight task types, and increases the horizon from 90 minutes to three hours (Li et al., 9 Oct 2026). It records verifier checkpoints every 30 minutes. On the 30 reconstruction tasks, binary verifier scores average 0.221 while dense ladder scores average 0.653. GPT-6-astra reaches a dense mean reward of 0.69 and 37 tasks at $$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$5; GPT-5.6-sol reaches 0.67 and 32 tasks at the same threshold.

These extensions emphasize that binary final-state scores can conceal meaningful partial progress, late solutions, regressions, and premature completion. They also show that task time limits, model latency, and harness behavior influence measured rankings.

Broader terminal-use coverage

“TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks” constructs 1,530 validated tasks from 80,870 public asciinema recordings and evaluates a manually reviewed 200-task subset (Chu et al., 21 May 2026). TerminalWorld emphasizes observed developer workflows, including routine operations and infrastructure tasks, whereas Terminal Bench emphasizes expert-curated difficult tasks. The reported Pearson correlation between Terminal-Bench and TerminalWorld scores is $$\mathrm{Resolution\ Rate} = \frac{\mathrm{number\ of\ successful\ trials}} {\mathrm{number\ of\ trials}} \times 100.$$6, suggesting limited score transfer under the reported, non-identical configurations.

“TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents” broadens the terminal beyond programming and systems work (Chen et al., 26 Jun 2026). Its 120 tasks cover office productivity, web information, system and software operations, scientific and engineering workflows, and multimedia and design. Claude Code with Claude Opus 4.8 reaches 65.8% overall performance, but even this system fails approximately one-third of task trials. TUA-Bench therefore positions terminal use as a general computer-use modality rather than a programming-only capability.

Terminal-Bench-LILT extends the format to 300 native-authored tasks in ten languages (Kim et al., 11 Aug 2026). The tasks target locale-specific engineering issues involving encoding, Unicode normalization, regional calendars, typography, scripts, cultural conventions, and language-specific text processing. The strongest model reaches 63.1%, and English instruction ablations do not eliminate the difficulty, suggesting that locale-aware coding competence is not reducible to prompt translation.

StateM demonstrates a complementary direction: improving execution through harness scaling rather than changing model weights (Qin et al., 15 Aug 2026). A stateful runtime with durable phases, checked transitions, recovery routes, procedural practices, and stop hooks raises GPT-5.5 xhigh from 83.1% to 92.1% on Terminal-Bench 2.1. A frozen profile reaches 95.28% raw accuracy with GPT-5.6 Sol xhigh across 445 trials, although the result is explicitly a pre-adjudication public-submission score and is not a clean component-level ablation.

7. Limitations and interpretation

Terminal Bench measures a combined system: model, agent scaffold, tools, sandbox, infrastructure, time limit, and verifier. Scores cannot be interpreted as pure model capability. Codex CLI, Claude Code, OpenHands, Mini-SWE-Agent, Gemini CLI, and Terminus 2 differ in prompting, context handling, tool interfaces, and execution policies.

The benchmark’s public tasks create contamination risks. Models may eventually encounter task instructions, environments, solutions, or evaluation conventions during training. Canary strings and n-gram filters help identify or reduce contamination but cannot prevent semantic overlap or intentional inclusion in training data. Terminal Bench 2.0 does not provide a private held-out set in the principal release.

Internet access and external resources create additional nondeterminism and cheating risks. Packages, repositories, APIs, web content, and network behavior may change. An agent could theoretically discover the public benchmark repository and inspect oracle solutions, although the authors report no such cheating in tens of thousands of trajectories.

Manual review is expensive and imperfect. LLM-backed checks may miss flaws, tests may omit requirements, and realistic heterogeneous tasks may contain ambiguities or unintended shortcuts. The benchmark relies substantially on human review rather than formal test adequacy measures such as code coverage, cyclomatic complexity, or automated state-space coverage.

Human completion-time estimates are not human success baselines. Terminal Bench does not report systematic human pass-rate experiments, so model performance is not directly calibrated against measured human reliability.

Finally, benchmark saturation is possible. Performance nearly doubled during the period studied, leading the authors to suggest that Terminal-Bench 2.0 may become substantially easier for future frontier systems. Later research accordingly explores harder task sets, long-horizon variants, multilingual tasks, real-world recordings, dense rewards, reward-hacking detection, alignment evaluation, and continuously refreshed environment-generation pipelines.

Terminal Bench’s enduring significance lies in its execution-based formulation: agents are evaluated not only on whether they can produce code or commands, but on whether they can operate a stateful computer, maintain a multi-step plan, recover from failures, manipulate unfamiliar environments, and verify an end-to-end result. Its results show substantial progress in terminal agency alongside persistent weaknesses in long-horizon planning, environment reconnaissance, debugging, verification, selective instruction following, resource management, and reliable completion.