---
title: Long-Horizon Terminal Bench
url: https://www.emergentmind.com/topics/long-horizon-terminal-bench
type: topic
---

# Long-Horizon Terminal Bench

Searching arXiv for the benchmark and closely related terminal-agent benchmark papers to ground the article in current literature.
arXiv_search(query="Long-Horizon-Terminal-Bench", max_results=5)
Long-Horizon-Terminal-Bench, often abbreviated **LHTB**, is a terminal benchmark for evaluating AI agents on genuinely long-horizon, multi-stage workflows rather than short command-line tasks with binary end-state grading. It was introduced to address a gap in prior terminal-agent evaluation: existing benchmarks often emphasize tasks that finish in a few minutes, require only tens of steps, and are judged almost entirely by whether the final output passes a test suite. LHTB instead combines much harder tasks with deterministic dense reward-based grading, so that evaluation captures not only whether an agent reaches the final goal, but also how far it progresses through an open-ended workflow [2607.08964].

## 1. Origin and problem framing

LHTB is positioned as a direct response to limitations in prior terminal-agent benchmarks such as Terminal-Bench 2.0 and related software-engineering evaluations. In those earlier settings, an agent that completes most of a complex pipeline but fails on a final hidden check is treated the same as one that failed immediately. The benchmark’s central intervention is therefore not only to make tasks harder, but to change the scoring philosophy from sparse outcome-only grading to dense, deterministic partial credit [2607.08964].

Conceptually, the benchmark is designed around long-horizon behaviors that current agents often fail to sustain: planning over hundreds of steps, long-context management, iterative debugging, repeated repair of broken pipelines, recovery from mistakes, verification before termination, and effective time allocation before timeout. The paper contrasts this with short-horizon benchmarks, which mostly test whether an agent can act correctly locally. LHTB is intended to measure whether locally sensible actions accumulate into completed long-running workflows.

A common misconception is that LHTB is merely a larger Terminal-Bench-style task set. The benchmark keeps the Terminal-Bench-style environment model, but changes both the task scale and the evaluation target. This means the benchmark is about long-horizon completion under dense grading, not simply more difficult binary pass/fail terminal tasks.

## 2. Task structure, environment model, and benchmark scope

Every LHTB task is a containerized terminal environment in **Harbor format** with four core components: a natural-language instruction, a Docker image defining the environment, a task configuration file, and an oracle implementation or simulator used for grading [2607.08964]. The agent sees only the top-level instruction and must act entirely through the terminal by issuing shell commands, inspecting files, editing code, running scripts, interpreting outputs, and iteratively debugging.

The Docker image contains all code, data, tools, helper scripts, and assets needed to solve the task. In addition, each task is built as a complete but deliberately broken terminal-only project and includes public asset-generation scripts, a weak baseline implementation, an official gold solution, a multi-step `solve.sh`, and a hidden verifier. The official gold solution must achieve verifier score \(1.0\) on the hidden suite.

The benchmark contains **46 tasks**. The main text states that they span **nine categories**, while the appendix provides a finer-grained taxonomy and notes that the tasks cover **21 high-level domains**. The nine-category taxonomy used throughout the paper includes: Interactive games; Research reproduction / ML; APEX professional workflows; Software / reverse engineering; Multimodal / imaging analysis; Systems / performance / security; Earth, climate / energy; Scientific computing / simulation; and Logic / constraint puzzles.

Representative tasks include 2048, chess-mate, super-mario, snake_maze_campaign, sokoban, rush_hour_campaign, duckdb-optimizer-closure, grammar-fuzz-coverage-hunt, spot-scheduler-traces, vector-db-iterative-build, robotics-slam-benchmark-repair, climate-netcdf-extreme-event-audit, dicom-radiology-audit, document-table-layout-reconstruction, scientific-figure-data-reconstruction, riscv-core-debug, foldseek-paper-reproduction, and unison-paper-reproduction. This breadth is meant to stress open-ended workflows rather than one-shot command generation.

## 3. Dense reward-based grading

The benchmark’s central methodological contribution is its **subtask-based dense grading**. Each task is decomposed into a small set of semantically meaningful subtasks,
\[
\{s_1,\dots,s_K\},
\]
intended to reflect intermediate goals in the intended workflow. For each subtask \(s_k\), the deterministic grader returns a normalized subscore
\[
r_k \in [0,1].
\]
The overall task reward is the weighted average
\[
R \;=\; \frac{\sum_{k=1}^K w_k\, r_k}{\sum_{k=1}^K w_k},
\]
where \(w_k\) are non-negative weights. By default, weights are equal, though the final goal can be given a higher weight when appropriate [2607.08964].

The paper distinguishes three kinds of subtasks. **Binary subtasks** return \(r_k \in \{0,1\}\) from strict programmatic checks such as whether unit tests pass or a service responds on the expected port. **Continuous or thresholded subtasks** assign partial credit for quantitative targets, including metric reproduction error or fraction of held-out examples matching oracle predictions. **Episode-aggregating subtasks** are used for campaign or repeated-trial settings such as games and audits, where the score may be the fraction of episodes or levels solved, or the mean normalized simulator reward across runs.

This reward is computed deterministically from the final container state after a rollout ends. The grader uses objective evidence such as files, generated outputs, test results, or simulator state. A task is counted as solved when
\[
R \ge \tau,
\]
for a success threshold \(\tau\). The benchmark reports pass@1 at reward thresholds \(0.9\), \(0.95\), and \(1.0\), with \(R=1.0\) representing full completion. Mean normalized reward across tasks is also reported to capture progress on unsolved tasks.

A second misconception is that dense grading implies judge-model subjectivity. In LHTB, grading is deterministic and environment-grounded rather than based on model judgment. This makes near-misses, partial progress, and verification failures visible without abandoning programmatic evaluation.

## 4. Benchmark construction and verification regime

The benchmark construction process begins from realistic professional or scientific workflows, including materials phase-diagram auditing, robotics SLAM repair, climate NetCDF extreme-event detection, audio-visual event alignment, scientific figure data reconstruction, satellite flood change detection, microscopy cell-count quality control, and scanned-document table reconstruction [2607.08964]. Tasks are then converted into deliberately broken terminal-only projects.

A central design feature is the **hidden verifier**. Public checks are intentionally low-weight and mostly validate command-line behavior, file formats, and a few simple examples, while most reward comes from hidden stress cases that vary schema or inputs. The paper lists examples such as nested manifests, gzip+base64 wrappers, missing values, renamed fields, injected noise, rotated/cropped images, anomalous frames, and alternate coordinate or time conventions. This is meant to prevent overfitting to visible tests and to force robust implementations.

For difficulty calibration, the authors generated **120 candidate tasks** and filtered them down to the final **46**. They repeatedly ran **DeepSeek V4 Pro** under **1.5-hour budgets** and adjusted task design until tasks were “challenging but still solvable in principle.” All final tasks are implemented in Harbor format with containerized environments and a shared evaluation harness. The benchmark and evaluation harness are released, with a project page at `https://zli12321.github.io/LHTB/`.

The appendix also labels task difficulty from average task reward: tasks with mean reward at least \(0.5\) are labeled **Easy**, otherwise **Hard**. Most tasks are Hard. Easy-labeled examples include apex-ib244-matter, apex-law433-matter, generals-bot-arena, grammar-fuzz-coverage-hunt, nbody-accel-iterative, poc-exploit-craft, riscv-core-debug, and spot-scheduler-traces. Hard examples include most multimodal audits, scientific regressions, puzzle campaigns, experiment reproductions, and game-playing tasks.

## 5. Experimental protocol and headline results

LHTB is evaluated using the **Harbor** framework with the **Terminus-2** agent harness, except **GPT-5.3**, which was evaluated using **Codex** as the harness. The model set contains **15 frontier models**: GPT-5.5, GPT-5.4, GPT-5.3 Codex, DeepSeek V4 Pro, Gemini 3.1 Pro, GLM 5.1, GLM 5.2, Kimi K2.6, Kimi K2.7 Code, MiniMax M3, Qwen3.7 Max, Qwen3.6 Plus, Doubao Seed 2.1 Pro, Hy3, and Grok 4.20 [2607.08964].

The reported metrics are pass@1, mean normalized reward, episodes per task, time per task, token usage per task, and estimated dollar cost. In this paper, pass@1 is the fraction of tasks solved by a single rollout per task, where solved means reward above a threshold such as \(R \ge 0.95\) or \(R=1.0\).

The resource profile is one of the benchmark’s defining properties. Across 15 models, rollouts average about **228–231 episodes per task**, **9.66M–9.9M tokens per task**, and **85.1–85.3 minutes** wall-clock time, under a **90-minute timeout**. This is contrasted explicitly with Terminal-Bench 2.0, where tasks average roughly **20–30 minutes** and **20–30 episodes** per task. The paper summarizes LHTB as “an order of magnitude more demanding than prior terminal-based benchmarks.”

The quantitative results are low across the board. **GPT-5.5** is the strongest model, but reaches only **15.2%** pass@1 at \(R \ge 0.95\), corresponding to **7/46** tasks, and **10.9%** at \(R=1.0\). The mean pass rate across all 15 models is **4.3%** at \(R \ge 0.95\) and **1.7%** at \(R=1.0\). MiniMax M3, Kimi K2.7 Code, and DeepSeek V4 Pro follow at **6.5%**. Kimi K2.6 and Grok 4.20 solve zero tasks at \(R \ge 0.95\), and Grok 4.20 has the lowest mean reward, \(R=0.08\).

A notable result is that at \(R=1.0\), **10 of the 15 models solve zero tasks**. The paper therefore emphasizes mean reward as a secondary signal, and reports only moderate rank correlation between pass rate and mean reward, with Spearman
\[
\rho = 0.56.
\]
This suggests that full completion and sustained partial progress capture different aspects of long-horizon competence.

## 6. Failure modes, reward distributions, and interpretive claims

The reward-distribution analysis motivates the dense-grading design. Across all \(15 \times 46 = 690\) model-task runs, only **30 runs (4.3%)** pass the \(R \ge 0.95\) threshold, while **227 runs (32.9%)** achieve almost no progress with \(R < 0.05\). The remaining **433 runs (62.8%)** achieve nontrivial partial reward but would be treated as failures under binary grading. Moreover, **180 runs (26.1%)** reach \(R \ge 0.5\), and near-miss runs with
\[
0.75 \le R < 0.95
\]
occur **73** times, more than twice the number of passes [2607.08964].

The paper highlights **Kimi K2.6** as an illustrative case: it solves zero tasks at \(R \ge 0.95\), yet has five near-misses and mean reward \(0.25\), including a best run at \(R=0.94\) on grammar-fuzz-coverage-hunt. This is presented as evidence that binary grading collapses meaningful differences in long-horizon progress.

The dominant failure mode is **timeout**. Among unresolved runs with \(R < 0.95\), **79% (518/660)** end because the 90-minute budget expires while the agent is still actively working. Only **19%** are early exits, and **3%** are harness errors. Timed-out runs are usually not almost complete: their mean reward ranges only from **0.10 to 0.35** across models. The paper interprets this to mean that the main bottleneck is not a final small fix, but failure to sustain coherent progress quickly enough.

A second failure mode is **false finish**, caused by weak self-verification. Among **124** runs where the agent voluntarily exited before timeout, the authors identify **14** false finishes with
\[
R \ge 0.75.
\]
Examples include Kimi K2.7 Code stopping on duckdb-optimizer-closure at \(R=0.92\), GLM 5.2 stopping on apex-ib244-matter at \(R=0.90\), and seven models stopping on apex-law433-matter between \(R=0.80\) and \(0.87\), often with about 20 minutes remaining. The benchmark uses dense rewards to expose these as verification failures rather than undifferentiated non-solutions.

The paper also notes that stronger performance is not simply a function of spending more tokens or money. GPT-5.4, for example, is the most expensive model in the cost table yet performs worse than GPT-5.5, partly because it uses many more episodes.

## 7. Position within terminal-agent research

LHTB is best understood as an extension of the **Terminal-Bench** line of evaluation. Terminal-Bench 2.0 defines hard, realistic command-line tasks in Harbor-format containerized environments and scores them by final environment-state correctness under binary outcome-based tests [2601.11868]. LHTB preserves the terminal-only environment model but shifts to much longer tasks and dense intermediate grading, thereby turning partial progress into a measurable quantity rather than a discarded failure state.

Several adjacent works target bottlenecks that LHTB makes especially visible. **“Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents”** reports pass@1 gains of **+8.3 pp** on Terminal-Bench 2.0 for a Sonnet 4.5 action agent by adding a selective memory sidecar, framing the core failure as “behavioral state decay” [2607.08716]. **“A Self-Evolving Framework for Efficient Terminal Agents via Observational Context Compression”** proposes TACO, which improves TerminalBench 1.0 and 2.0 scores while reducing token overhead by around **10%** on many settings [2604.19572]. **“CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents”** trains compacting agent policies and reaches **24.5%** Pass@1 on Terminal-Bench 2.0 with GLM-4.5-Air [2607.05378]. **“LiteCoder-Terminal”** and **“KLong”** address long-horizon training data and curriculum design for terminal-style or coding-heavy agents, with transfer gains on Terminal-Bench 2.0 and Terminal-Bench Hard [2605.29559] [2602.17547].

A plausible implication is that LHTB functions as a stress test for precisely these research directions: long-context management, structured memory, iterative debugging, calibrated stopping, and environment-grounded self-verification. Its main contribution is therefore not only a harder task set, but a measurement regime that exposes how and where current terminal agents fail over extended workflows.

Source: https://www.emergentmind.com/topics/long-horizon-terminal-bench