Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Abstract: AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.

• The paper presents LHTB, a benchmark that decomposes lengthy tasks into fine-grained subtasks with dense reward grading.
• It demonstrates that dense, environment-grounded rewards provide nuanced diagnostics beyond binary pass/fail metrics.
• Empirical results reveal significant limitations in current agents, emphasizing the need for improved planning and self-verification in long-horizon scenarios.

Long-Horizon-Terminal-Bench: A Benchmark for AI Agent Long-Horizon Execution with Dense Subtask-Level Rewarding

Motivation and Benchmark Design

Despite growing autonomy among LLM-based agents, existing terminal-agent benchmarks predominantly evaluate agents on short-horizon, trivially-scoped tasks, using binary outcome grading (pass/fail) and sparse reward signals. Such regimes underestimate the complexity and requirements of real-world, long-horizon workflows, where cumulative partial progress, iterative debugging, and robust end-to-end processes are essential.

Long-Horizon-Terminal-Bench (LHTB) addresses these deficiencies by introducing a suite of 46 containerized terminal tasks, each decomposed into fine-grained, environment-grounded subtasks with deterministic checks for partial credit. The result is a dense, informative reward signal that explicitly quantifies intermediate progress, offers nuanced diagnostics of agent limitations, and measures long-range autonomous operation (Figure 1).

Figure 1: The overall structure of a long-horizon terminal task with dense reward grading—partial rewards quantify fine-grained subgoal attainment throughout extensive workflows.

The LHTB suite encompasses nine major domains, including software engineering, scientific computing, interactive games, reverse engineering, and multimodal analysis (Figure 2). Each task embodies a realistic, professional-level problem, designed to demand multi-stage planning, long-context manipulation, and cumulative artifact construction over hundreds of episodes and up to 1.5 hours of execution per rollout.

Figure 2: Distribution of LHTB's 46 tasks across nine long-horizon scientific and engineering domains.

Dense Subtask-Based Grading: Reward Formulation and Evaluation

Each LHTB task specifies an ordered set of subtasks $\{s_1\,\ldots,s_K\}$, with per-subtask deterministic verification at the end of each rollout. Subtask types include:

• Binary checks (strict Boolean conditions over environment state)
• Continuous metrics (e.g., reproducing target figures, matching regression outputs, or generalizing on hidden schema variations)
• Episode-aggregating rewards (multi-episode campaign tasks)

Global task reward $R$ is computed as a normalized, weighted sum across subtasks:

$R = \frac{\sum_k w_k\, r_k}{\sum_k w_k}$

with $r_k \in [0, 1]$ the normalized score per subgoal. This design ensures evaluative granularity and continuity, enabling discrimination between completely failed, partially completed, and nearly-but-not-fully successful executions—especially critical in high-difficulty, long-horizon settings where binary thresholds collapse the reward distribution (see Figure 3).

Benchmark Statistics, Agent Setup, and Performance

Tasks in LHTB require an order-of-magnitude more effort than prior terminal benchmarks—averaging 231 episodes and 85.3 minutes per task, with 9.9M tokens generated per rollout via a shared agent harness (Terminus-2/Harbor for most models).

Fifteen frontier models (including GPT-5.5, MiniMax M3, DeepSeek V4 Pro, GLM 5.x, Gemini 3.1 Pro, and others) were evaluated. The primary results (Figure 4) highlight persistent limitations in current agentic architectures:

• Pass@1 at partial-reward threshold $R \geq 0.95$: best model (GPT-5.5) resolves only 15.2% of tasks; mean across all models is merely 4.3%
• Full solution rate ($R \geq 1.0$): drops to 10.9% (best model), 1.7% (average)
• Dense reward evaluation: reveals 62.8% of all runs achieve non-trivial partial progress ($R \geq 0.05$) but fall short of nearly complete solutions, whereas binary grading would mark these as indistinguishable failures

  Figure 4: LHTB leaderboard—Pass@1 at multiple reward thresholds and mean normalized reward per model. GPT-5.5 leads but exhibits a low absolute solution rate at stringent thresholds, emphasizing unsolved long-horizon agentic challenges.

  

  Figure 3: Final reward distribution over all model-task runs—dense grading exposes that the majority achieve significant partial progress, which binary metrics would ignore.

Model cost-performance tradeoffs are characterized by a Pareto frontier (Figure 5), with the strongest models incurring significant computational expense (e.g., GPT-5.5: ~$21/task), though no consistently optimal efficiency emerges—performance does not scale linearly with API cost.

Figure 5: Cost–reward Pareto frontier showing cost per run (log scale) vs. partial-completion pass rate. Hy3 sits at the low-cost frontier; top-performing models trade off increased API expense for modest gains in pass rate.

Failure Analysis and Diagnostic Insights

Dense subtask-level grading critically surfaces nuanced failure phenotypes that outcome-only metrics cannot expose:

• Timeout Dominance: 79% of unresolved runs end by exceeding a 90-minute wall-clock budget while still making incremental progress—exposing a horizon bottleneck rather than acute reasoning failure.
• False Finishes and Self-Evaluation Weakness: Many early agent exits occur with high (but sub-threshold) reward, indicating agents insufficiently verify comprehensive task satisfaction and systematically overestimate completion (Figure 6).

  Figure 6: Breakdown of failure modes across models—timeouts overwhelmingly dominate, but premature early exits often reflect misjudged self-verification rather than abrupt breakdowns.

Dense rewards thus reveal the principal challenge is not stepwise correctness, but constructing robust long-range plans and terminating only upon true objective satisfaction. Improvements in planning, context management, persistent memory, and meta-cognitive capabilities (e.g., calibrated self-verification) are urgently needed.

Implications for Agentic Research and Evaluation

LHTB advances agentic evaluation in several key respects:

• Reward density: Fine-grained subtask crediting prevents collapse of the evaluation metric, making progress and stagnation legible and actionable, especially as models approach—but do not achieve—true completion.
• Scaling of agentic difficulty: LHTB identifies and quantifies long-horizon deficiencies not captured by short-form or binary-graded tasks, establishing a clear unsolved frontier for automated, tool-augmented reasoning.
• Evaluation harness separation: Model differences are measured under a shared agent harness, isolating underlying capability from confounding agent design heterogeneity.

Practically, LHTB will inform the design of agents for real-world workflows—where robustness to failure, progress monitoring, and adaptive debugging are paramount—and will drive research toward closing the gap to human-level autonomy in extensive, professional environments.

Future Directions

• Harness optimization: Further disentangling agent harness scaffolding and LLM core competence remains essential for understanding system-level limitations.
• Memory and planning augmentation: Advanced memory architectures, plan-state persistence, and higher-order self-verification (e.g., learned process verifiers) could address primary bottlenecks identified on LHTB.
• Transfer to open-ended, real-user environments: Exploring agent generalization and credible progress assessment in settings without hidden verifiers or stress-case generators remains open.
• Reward shaping for training: Incorporating dense credit assignments into RLHF and in-context RL pipelines may spur capabilities that are underexploited in binary grading settings.

Conclusion

Long-Horizon-Terminal-Bench establishes a challenging and diagnostic regime for benchmarking LLM-based agent autonomy on realistic, long-horizon terminal tasks, leveraging dense, environment-grounded subtask grading. Empirical results demonstrate that current agents perform at a considerable remove from task completion, with primary failures stemming from incomplete long-plan execution and flawed progress estimation. By making partial progress measurable and analyzable, LHTB will catalyze both the development of genuinely robust agentic systems and improvements in evaluation methodology for the next generation of autonomous agents.

Reference: "Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading" (2607.08964)