---
title: 'LongCLI-Bench: Long-Horizon CLI Benchmark'
url: https://www.emergentmind.com/topics/longcli-bench
type: topic
---

# LongCLI-Bench: Long-Horizon CLI Benchmark

LongCLI-Bench is a benchmark for long-horizon agentic programming in command-line interfaces. Introduced as a preliminary benchmark for evaluating realistic software-engineering trajectories, it is designed to test whether an agent can operate inside a repository and shell over many interdependent steps while both satisfying new requirements and preserving existing behavior [2602.14337]. Its central object of measurement is therefore broader than code synthesis: it targets planning, repository navigation, environment grounding, multi-file editing, command execution, debugging, and regression avoidance in sustained CLI workflows.

## 1. Motivation and evaluative scope

LongCLI-Bench is motivated by four deficiencies that its authors identify in prior coding evaluation. First, early benchmarks such as HumanEval and MBPP are described as measuring mostly function-level algorithmic correctness. Second, repository-level benchmarks are treated as a step forward but still narrower than realistic engineering workflows. Third, Terminal-Bench is identified as the closest comparison, yet its tasks are characterized as remaining simple with short horizons and lacking fine-grained evaluation. Fourth, many existing coding and SWE benchmarks are presented as contamination-prone because they are mined from GitHub repositories or scraped issues [2602.14337].

The benchmark’s proposed remedy is to treat software engineering as a sequential, long-horizon activity. In this framing, realistic work involves architecture planning, repository inspection, environment setup, dependency configuration, multi-file edits, command execution, log interpretation, debugging, and consistency maintenance across steps. LongCLI-Bench accordingly asks not merely whether an agent can write correct code, but whether it can sustain coherent execution through a complete requirement chain in which earlier actions condition the feasibility of later ones.

The benchmark also makes evaluation granularity a first-class concern. Rather than relying only on binary pass/fail, it introduces step-level scoring so that early stalls, partial progress, and near-complete but unsuccessful runs can be distinguished. This diagnostic emphasis is integral to the benchmark’s conception of long-horizon failure: a trajectory that breaks in its opening stages is not equivalent to one that completes almost all required work and then fails on a final constraint.

## 2. Construction, sourcing, and contamination control

LongCLI-Bench is organized around five stated design principles: **long-horizon**, **contamination control**, **clear requirements**, **solvability**, and **isolated environments** [2602.14337]. Its source pool comprises **958 CS assignments from 108 courses** and **50 real-world workflows**, yielding an initial pool of over 1,000 candidates. The final released benchmark contains **20 tasks**.

The contamination-control strategy is a defining methodological feature. Rather than mining public GitHub issues directly, the benchmark draws on course assignments and manually constructed workflows. For assignment-derived tasks, requirement descriptions are rewritten by changing specific variable names, function names, file names, and even background stories. This does not amount to a formal impossibility claim about contamination, but it does constitute an explicit attempt to reduce retrieval-style matching against public training corpora.

Task construction is parallelized across environment building, human solving, and test development. The execution environment is built in Docker; the task is then solved by the benchmark builders from a base image; dependencies encountered during solving are recorded; and the final deliverables are separated into a **Dockerfile** and a **solution repository**. This separation serves two purposes: it produces a reproducible runtime and a working reference solution that demonstrates solvability.

Each task package contains the following components: **Initial Repo**, **Task Requirement**, **Environment**, **Solution Repo**, **Tests**, **Scoring Parser**, and **Metadata**. The metadata includes task type, domain, difficulty, and estimated human completion time. The tests are written from the requirement document only, not from the human solution, in order to avoid encoding solution-specific implementation choices into the evaluator.

A task is admitted only if it satisfies the benchmark’s verification condition: **F2P tests fail on the initial repo**, **F2P tests pass on the solution repo**, **P2P tests pass on the initial repo**, and **P2P tests pass on the solution repo**. Construction is iterative: ambiguous documents are revised, tests are rerun on both initial and solution repositories, failures are manually inspected, and any task that still cannot satisfy the condition after **three iterations** is discarded. Final expert review precedes inclusion.

## 3. Task taxonomy and long-horizon characteristics

LongCLI-Bench partitions its tasks into four engineering categories: **From Scratch** \((0 \to 1)\), **Feature Addition** \((N \to N+1)\), **Bug Fix** \((No \to Yes)\), and **Refactor** \((A \to A')\) [2602.14337]. These categories are intended to cover the principal structural modes of repository-level engineering rather than a single issue-fixing format. The benchmark also spans six domains: **System Programming**, **Web Development**, **Data Engineering**, **Machine Learning**, **Applications**, and **DevOps**.

Long-horizon difficulty is defined structurally rather than by token count alone. A task is long-horizon because it requires the agent to inspect a large repository, identify and follow correct entry points, work within dependencies, perform multiple interdependent edits, verify intermediate system state, recover from failures, and preserve prior functionality. The benchmark explicitly states that many tasks contain a “complete requirement chain where the outcome of previous steps determines the feasibility of subsequent ones.”

A quantitative comparison with Terminal-Bench@2 is used to substantiate this claim of scale and dependency:

| Measure | Terminal-Bench@2 | LongCLI-Bench |
|---|---:|---:|
| Tasks | 89 | 20 |
| Files | 0.69 | 104.0 |
| LoC | 227.7 | 15,000+ |
| Expert Time (min) | 206.7 | 1000+ |

These statistics locate LongCLI-Bench at repository scale. On average, each task involves **104.0 source files**, **15,000+ lines of code**, and **1000+ minutes** of expert completion time. The contrast with Terminal-Bench@2 is used to argue that the benchmark targets deeply interdependent engineering settings rather than short terminal episodes.

Step decomposition is also part of the task design. For assignments with built-in grading scripts, outputs are parsed into sub-step scores. For tasks without predefined tests, the requirements are manually segmented into sub-tasks and evaluated with corresponding scripts. This operationalizes long-horizon execution as a sequence of testable subgoals.

## 4. Evaluation protocol and scoring semantics

The benchmark’s evaluation protocol is built on a dual-set test design. **Fail-to-Pass (F2P)** tests measure requirement fulfillment: the initial repository should fail them, while a correct solution should pass them. **Pass-to-Pass (P2P)** tests measure regression avoidance: the initial repository should already pass them, and the modified repository should continue to do so [2602.14337].

Step-level scoring is defined as the **percentage of sub-steps completed**. The paper does not typeset a formal equation for this metric, but it states an exact binary criterion: **a test set is considered passed only when its step score reaches 100\%**. Consequently, **F2P Pass** means F2P step score \(=100\%\), and **P2P Pass** means P2P step score \(=100\%\). Overall task pass is defined as the percentage of tasks for which the agent passes **both** F2P and P2P.

The benchmark reports several aggregate quantities: **Pass Rate (%)**, **Pass@3 (%)**, **F2P Pass (%)**, **F2P Step Score (%)**, **P2P Pass (%)**, **P2P Step Score (%)**, and **Time (minutes)**. The main results are averaged over **three independent runs** in a unified single-turn setting. LongCLI-Bench also supports **multiple attempts** and a **self-correction** mode in which the agent re-executes the task after receiving feedback from the previous turn’s test results.

The interaction model is deliberately constrained. Evaluation proceeds by initializing Docker with the initial repository, allowing the agent to work through the terminal, and then running the harnessed tests when the agent stops or times out. The benchmark text does not provide more detailed formal specifications for timeout policy, command budgets, shell restrictions, or structured tool interfaces beyond stating that agents interact through the terminal in the isolated environment.

## 5. Empirical results, error structure, and human guidance

In the main single-turn evaluation, all tested systems remain below **20%** overall pass rate, establishing the benchmark as difficult even for frontier agentic coding systems [2602.14337]. The benchmark evaluates commercial CLI assistants—Codex variants and Claude Code variants—as well as OpenHands configurations using DeepSeek-V3.1, GLM-4.6, and Qwen3-235B-A22B.

The best single-turn overall results are **Claude-Opus-4.5** at **16.7%** pass and **Claude-Opus-4.6** at **16.7%** pass, with **Claude-Opus-4.6** attaining the highest **Pass@3** at **25.0%**. Among Codex configurations, **GPT-5.3-Codex** reaches **15.0%** pass and **20.0%** Pass@3. OpenHands-based systems perform substantially worse in overall pass, ranging from **5.0%** to **6.7%**.

A consistent asymmetry appears between requirement completion and regression preservation. **F2P step scores** range from **25.3%** to **50.7%**, whereas **P2P step scores** range from **98.4%** to **99.7%**. Yet the corresponding **P2P pass rates** are lower, from **70.0%** to **88.3%**, showing that regressions still occur on a nontrivial fraction of tasks even when average regression sub-step completion appears high. The benchmark interprets this as evidence that agents often preserve most prior behavior while still breaking enough functionality to fail strict pass criteria.

Step-score distribution analysis shows that many trajectories fail very early. The largest share of runs falls into the **F2P step-score range \([0,30)\)**. Representative values include **45.0%** for GPT-5.1-Codex-Max, **41.7%** for GPT-5.3-Codex, **38.3%** for Claude-Opus-4.6, **65.0%** for DeepSeek-V3.1, **63.3%** for GLM-4.6, and **58.3%** for Qwen3-235B-A22B. By contrast, the **\([80,100)\)** bucket is typically only **5.0%–15.0%**, indicating that near-complete-but-unsolved trajectories are relatively uncommon. The benchmark uses this pattern to support its claim that strong sequential dependency makes early prerequisite failures especially costly.

Self-correction improves performance, but only modestly relative to human guidance. The paper reports large gains from the first to the second round and smaller gains thereafter; it also notes that P2P behavior can become unstable as later-round edits expand the change scope. For example, **GPT-5.3-Codex** raises P2P pass to **95% at \(T2\)** and **100% at \(T3\)**, while **GPT-5.1-Codex-Max** rises to **85% at \(T2\)** and then falls to **80% at \(T3\)**. **Claude-Opus-4.6** rises to **95% at \(T2\)** and then falls to **90% at \(T3\)**.

Human-agent collaboration yields much larger improvements. For **GPT-5.3-Codex**, pass rises from **15.0%** in the base setting to **40.0%** with self-correction, **41.7%** with static plan injection, **45.0%** with dynamic interactive guidance, and **50.0%** with combined plan plus interaction. Its **F2P Score** rises from **44.1** to **58.4**, **59.3**, **61.3**, and **62.3**, respectively. For **Claude-Opus-4.6**, pass rises from **16.7%** to **55.0%** with self-correction, **58.3%** with plan injection, **58.3%** with interactive guidance, and **61.7%** with combined assistance; its **F2P Score** rises from **50.7** to **63.6**, **65.4**, **67.4**, and **69.3**. The average number of interventions also falls when plans are pre-injected: from **2.7** to **2.2** for GPT-5.3-Codex and from **2.4** to **2.1** for Claude-Opus-4.6.

Manual inspection of **50 failed trajectories** identifies three dominant failure modes: **repetitive loops from weak strategic adaptation**, **environment grounding and verification gaps**, and **long-term inconsistency and regression**. A notable negative result is that these failures are described as being “rarely caused by local syntax-level coding mistakes.” The benchmark instead localizes the main bottleneck at workflow-level coherence.

## 6. Position within the benchmark landscape and acknowledged limitations

LongCLI-Bench occupies a distinct position relative to several neighboring arXiv benchmarks. Compared with text-centric long-context understanding benchmarks such as "MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models" [2505.19959] and "CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models" [2403.03514], LongCLI-Bench is centered on stateful terminal execution, environment grounding, repository mutation, and regression control rather than on static document understanding. Compared with "CL-bench: A Benchmark for Context Learning" [2602.03587], it is less about learning novel knowledge from provided textual context and more about executing long engineering trajectories in isolated CLI environments. Compared with "Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments" [2606.05661], its emphasis is not online improvement across long schedules of related instances, but long-horizon completion of individual software-engineering tasks. Compared with "WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation" [2605.10912], it is more narrowly focused on repository-scale software engineering, whereas WildClawBench spans 60 native-runtime tasks across six thematic categories, including multimodal and cross-tool workflows.

This comparison suggests that LongCLI-Bench isolates a specific evaluative niche: long-horizon command-line software engineering with explicit separation between **requirement fulfillment** and **regression avoidance**, and with **step-level progress diagnostics** rather than final-answer-only scoring. Its most direct historical contrast is with Terminal-Bench@2, against which it claims much larger repositories, longer expert completion time, more contamination-aware sourcing, and finer-grained evaluation.

The benchmark’s acknowledged limitations are substantial. Its scale is small, at **20 tasks**, because manual construction is expensive; the paper reports that **each task takes an average of 40 hours to complete** [2602.14337]. Step-level scores do not fully capture code quality, efficiency, or broader software-engineering quality criteria. The benchmark text also highlights open challenges exposed by the results: robust long-horizon planning, maintaining context over extended workflows, environment-aware diagnosis, avoiding regressions while making nontrivial edits, escaping repetitive failure loops, and balancing self-correction against change-scope expansion.

Within the current benchmark ecosystem, LongCLI-Bench is therefore best understood not as a generic long-context benchmark, but as a command-line programming benchmark that operationalizes long-horizon capability through repository-scale tasks, dual-set testing, and partial-progress analysis. Its empirical finding is that contemporary agents remain far from reliable autonomous performance in this regime, while its methodological finding is that human strategic guidance can raise outcomes far beyond what autonomous single-turn execution presently achieves.

Source: https://www.emergentmind.com/topics/longcli-bench