Terminal-Bench-2.0 Benchmark

Updated 23 June 2026

Terminal-Bench-2.0 is a rigorously curated benchmark that evaluates autonomous LLM agents on long-horizon Unix tasks using isolated Docker environments.
It comprises 89 human-verified tasks spanning software engineering, system administration, data science, debugging, and package management with structured verification protocols.
The benchmark drives advances in synthetic data pipelines, reward modeling, and context management by providing precise performance metrics and error analysis.

Terminal-Bench-2.0 is a rigorously curated, outcome-driven benchmark designed to evaluate autonomous LLM agents on realistic, long-horizon tasks in Unix-like command-line environments. Building upon its predecessor, it measures key competencies in software engineering, system administration, data science, debugging, package management, and multi-step tool use. By providing standardized, executable Docker environments and programmatic success criteria, Terminal-Bench-2.0 has become the de facto testbed for LLM-based terminal agents, underpinning advances in synthetic data pipelines, trajectory engineering, reward modeling, and context management.

1. Benchmark Construction and Task Scope

Terminal-Bench-2.0 comprises 89 human-verified tasks, selected from 229 community-sourced proposals using a multi-stage filtering process involving automated CI checks and manual audits for specificity, solvability, and integrity (Merrill et al., 17 Jan 2026). Each task is encapsulated in an isolated Docker container, providing a consistent Ubuntu- or Debian-based environment with baseline UNIX tools (bash, coreutils, grep, sed, awk, git, make, tar, gzip) but no pre-installed domain-specific libraries (Zhang et al., 31 Jan 2026). The typical directory structure includes:

instruction.md: Natural-language goal with explicit I/O schema.
environment/: Dockerfile and input artifacts.
solution/: Oracle solution script (not visible to agents).
tests/: Pytest or shell-based verification suite.
task.toml: Metadata, including resource quotas and timeouts.

Task domains span nine to fifteen categories, including software engineering, system administration, security, data processing, mathematical computing, debugging, and games (Pi et al., 24 Feb 2026, Wu et al., 1 Feb 2026, Hua et al., 22 Jun 2026). Each task instance requires multi-turn interaction: the agent receives a fresh environment and must issue a sequence of shell commands to transform the initial state into one that passes all programmatic tests (Merrill et al., 17 Jan 2026, Ren et al., 21 Apr 2026).

2. Evaluation Protocols and Metrics

Terminal-Bench-2.0 is evaluated using execution-grounded protocols centered on outcome-based verification. The central framework, Terminus-2, manages a structured inspect–act–verify loop: at each turn, the agent emits structured JSON specifying analyses, plans, and shell commands, which are executed in the Docker sandbox; terminal outputs (stdout, stderr) are returned; the session ends when the agent signals task_complete:true in two consecutive turns (Yang et al., 2 Jun 2026). On completion, a built-in verifier script (often pytest-based) validates whether the agent's sequence has solved the task.

The primary metrics are:

Success Rate (pass@1): Fraction of tasks solved in a single rollout:

$\mathrm{Success\,Rate} = \frac{\# \text{Solved Instances}}{\#\text{Total Instances}}$

pass@k: Fraction of tasks solved in up to $k$ independent rollouts per task:

$\mathrm{pass@}k = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\Bigl(\max_{1 \leq j \leq k}\text{success}_{i,j}\Bigr)$

Finite-sampling bias can be corrected using the unbiased estimator:

$\frac{1}{|T|} \sum_{t=1}^{|T|} \left( 1 - \frac{\binom{N-c_t}{k}}{\binom{N}{k}} \right)$

where $c_t$ is the number of successful rollouts for task $t$ (Cheng et al., 20 May 2026).

Additional metrics include Fail-to-Pass Rate (instances initially failed but eventually solved), Commit Rate (final committed actions leading to task success), token-budget curves, and wall-clock performance (Zhang et al., 31 Jan 2026, Ren et al., 21 Apr 2026).

3. Task Complexity, Domain Balance, and Verification

Tasks are empirically hard, with frontier models (e.g., GPT-5.2, Claude Opus-4.5) achieving less than 65% pass@1 and open-weight models generally below 35%, despite rapid progress (Merrill et al., 17 Jan 2026, Ivison et al., 22 Jun 2026). Difficulty is not explicitly labeled, but is implicit in multi-step workflows (10–60 commands) and encoded in timeouts. Domains are broadly represented and no single category dominates, mitigating benchmark saturation and overfitting (Pi et al., 24 Feb 2026, Merrill et al., 17 Jan 2026).

A key feature is programmatic, outcome-driven verification rather than output string-matching. Each task's verifier asserts environment invariants through pytest scripts or shell checks, making partial solutions and shortcut exploits ineffective (Merrill et al., 17 Jan 2026). Docker-based round-trip build and test validation ensures each task is constructible, solvable by its reference solution, and free of leakage.

Scoring is strictly binary (0/1) per task in public leaderboards, though some synthetic data pipes (e.g., Nemotron-Terminal) employ partial-credit weighted scores for internal ablations (Pi et al., 24 Feb 2026). In all leaderboards, aggregate scores are computed as (mean)/(±2σ confidence intervals) over all 89 (occasionally more, if augmented) tasks.

4. Role in Data Synthesis, Training, and Pipeline Design

Terminal-Bench-2.0 has become the gold-standard evaluation suite for pipelines generating synthetic training data, RL environments, and expert agentic trajectories:

Synthetic Data Pipelines: Terminal-Task-Gen, Terminal-Lego, Terminal-World, CLI-Universe, LiteCoder-Terminal-Gen, and TMAX generate large-scale, programmatically verified CLI tasks, each designed to mimic the characteristics of TB-2.0, using controlled taxonomies for domain, skill, and complexity (Pi et al., 24 Feb 2026, Yang et al., 2 Jun 2026, Cheng et al., 20 May 2026, Hua et al., 22 Jun 2026, Peng et al., 28 May 2026, Ivison et al., 22 Jun 2026).
Data Efficiency and Scaling: Smaller, highly-structured datasets (e.g., CLI-Universe-6K, Terminal-World-5.7K, Terminal-Lego-15.3K) are shown to outperform or match much larger scraped repositories (TerminalTraj-50K, Nex-N1-69K, Nemotron-Terminal-490K) when distilled via high-fidelity blueprints and environment-grounded supervision (Hua et al., 22 Jun 2026, Cheng et al., 20 May 2026, Yang et al., 2 Jun 2026).
RL and Preference Optimization: Advanced RL objectives (Direct Multi-turn Preference Optimization, ECHO) and dense supervision from environment feedback drive further gains without reliance on large offline expert demonstration sets (Peng et al., 28 May 2026, Shrivastava et al., 23 May 2026).
Context Compression: Observation compression techniques such as TACO improve both performance (1–4% gains) and computational efficiency by adaptively summarizing or filtering token-intensive terminal outputs (Ren et al., 21 Apr 2026).
Harness Engineering: The design and transparency of the agent-environment interaction loop (e.g., explicit inspect–act–verify) have emerged as critical to data efficiency and reproducibility (Yang et al., 2 Jun 2026).

5. Empirical Trends, Baseline Performance, and Comparisons

The following table aggregates representative results at 32B-parameter scale across major open-source approaches as reported in respective works, all evaluated on Terminal-Bench-2.0's 89-task suite:

Model / Pipeline	pass@1 (%)	Data Volume	Methodology
CLI-Universe-32B (Hua et al., 22 Jun 2026)	33.4	6K curated trajectories	Structured blueprint, rubric-gated synthesis, evidence-grounded filtering
Terminal-World-32B (Cheng et al., 20 May 2026)	31.5	5.7K skill-synthesized	Skill-grounded, graph-based, failure-inclusive SFT
Nemotron-Terminal-32B (Pi et al., 24 Feb 2026)	27.4	490K (mixed adapters + synthetic)	Skill-based and seed-based SFT, internal filtering
TerminalTraj-32B (Wu et al., 1 Feb 2026)	22.0	50.7K repo-derived trajectories	Real-repo execution, Docker alignment
LiteCoder-Terminal-32B (Peng et al., 28 May 2026)	18.5	11.2K synthetic tasks	Fully synthetic, 10-domain expert supervision
DockSmith-30B (Zhang et al., 31 Jan 2026)	14.0	Docker-building + SWE	Agentic env-construction, repair loop learning
TMAX-9B (Ivison et al., 22 Jun 2026)	27.2	14.6K RL envs	Difficulty/persona/verifier diversity RL

Closed-source models (e.g. Claude Opus 4.5, GPT-5.2, Gemini 3 Pro) consistently score higher (up to 62.9%), but CLI-Universe and Terminal-World obtain state-of-the-art results for open-weight models ≤32B (Hua et al., 22 Jun 2026, Merrill et al., 17 Jan 2026).

6. Error Taxonomy, Agent Limitations, and Guidance for Improvement

Fine-grained error analyses across thousands of agent rollouts demonstrate that the dominant failure modes in Terminal-Bench-2.0 are:

Execution Errors: Command invocation failures, missing dependencies, runtime/build errors (50–60% of cases for frontier models) (Merrill et al., 17 Jan 2026).
Coherence Failures: Loss of context, reasoning–action mismatches, context overflows in long-horizon workflows.
Verification Lapses: Omitted or inadequate self-checks, premature task completion signals.

Command-level error rates vary by approach, with the highest rates in invoking non-existent executables or missing path entries (24.1%). Empirical studies confirm that execution failures remain higher for models trained on narrowly-sourced or poorly-structured data, while richer, environment-grounded trajectories (high Targeted Observation Ratio) robustly transfer actionable behavior to student agents (Yang et al., 2 Jun 2026, Hua et al., 22 Jun 2026).

Recommendations for further gains focus on:

Robust, long-horizon tool use and context management (hierarchical memory, summarization).
Explicit self-verification cycles (subgoal-driven plan–execute–verify loops).
Systematic token and interaction compression (to mitigate context window growth) (Ren et al., 21 Apr 2026).
Adversarial task construction and OOD stressors (dynamic envs, flaky tests, multi-agent dependencies).

7. Impact and Future Directions

Terminal-Bench-2.0 has established a high bar for the evaluation of CLI agents, providing an extensible and adversarially resilient suite that has driven major methodological shifts in:

Data-centric RL and SFT for embodied and tool-using agents;
Automated, verifiable task synthesis and outcome validation;
Harness engineering and environment-feedback modeling (ECHO, World Modeling) (Shrivastava et al., 23 May 2026, Yang et al., 2 Jun 2026).

Its outcome-driven, multi-domain structure ensures ongoing relevance as model capabilities approach task saturation. The open-source harnesses, datasets, and leaderboards (tbench.ai) underpin cross-group reproducibility and foster community innovation on realistic agentic intelligence benchmarks. The benchmark’s continued extension with more adversarial, dynamic, and multi-agent task types is identified as essential for preventing metric saturation and driving autonomous agent research past current plateaus.

References: