LongCLI-Bench: Long-Horizon CLI Benchmark
- LongCLI-Bench is a benchmark for long-horizon agentic programming in CLI, emphasizing sequential planning, multi-file editing, and regression avoidance across repository-scale tasks.
- It employs step-level scoring with dual test sets (F2P/P2P) to diagnose granular progress and early failures in realistic software-engineering workflows.
- The benchmark leverages over 1,000 candidate tasks from courses and real workflows to challenge current autonomous coding systems and highlight improvement opportunities.
LongCLI-Bench is a benchmark for long-horizon agentic programming in command-line interfaces. Introduced as a preliminary benchmark for evaluating realistic software-engineering trajectories, it is designed to test whether an agent can operate inside a repository and shell over many interdependent steps while both satisfying new requirements and preserving existing behavior (Feng et al., 15 Feb 2026). Its central object of measurement is therefore broader than code synthesis: it targets planning, repository navigation, environment grounding, multi-file editing, command execution, debugging, and regression avoidance in sustained CLI workflows.
1. Motivation and evaluative scope
LongCLI-Bench is motivated by four deficiencies that its authors identify in prior coding evaluation. First, early benchmarks such as HumanEval and MBPP are described as measuring mostly function-level algorithmic correctness. Second, repository-level benchmarks are treated as a step forward but still narrower than realistic engineering workflows. Third, Terminal-Bench is identified as the closest comparison, yet its tasks are characterized as remaining simple with short horizons and lacking fine-grained evaluation. Fourth, many existing coding and SWE benchmarks are presented as contamination-prone because they are mined from GitHub repositories or scraped issues (Feng et al., 15 Feb 2026).
The benchmark’s proposed remedy is to treat software engineering as a sequential, long-horizon activity. In this framing, realistic work involves architecture planning, repository inspection, environment setup, dependency configuration, multi-file edits, command execution, log interpretation, debugging, and consistency maintenance across steps. LongCLI-Bench accordingly asks not merely whether an agent can write correct code, but whether it can sustain coherent execution through a complete requirement chain in which earlier actions condition the feasibility of later ones.
The benchmark also makes evaluation granularity a first-class concern. Rather than relying only on binary pass/fail, it introduces step-level scoring so that early stalls, partial progress, and near-complete but unsuccessful runs can be distinguished. This diagnostic emphasis is integral to the benchmark’s conception of long-horizon failure: a trajectory that breaks in its opening stages is not equivalent to one that completes almost all required work and then fails on a final constraint.
2. Construction, sourcing, and contamination control
LongCLI-Bench is organized around five stated design principles: long-horizon, contamination control, clear requirements, solvability, and isolated environments (Feng et al., 15 Feb 2026). Its source pool comprises 958 CS assignments from 108 courses and 50 real-world workflows, yielding an initial pool of over 1,000 candidates. The final released benchmark contains 20 tasks.
The contamination-control strategy is a defining methodological feature. Rather than mining public GitHub issues directly, the benchmark draws on course assignments and manually constructed workflows. For assignment-derived tasks, requirement descriptions are rewritten by changing specific variable names, function names, file names, and even background stories. This does not amount to a formal impossibility claim about contamination, but it does constitute an explicit attempt to reduce retrieval-style matching against public training corpora.
Task construction is parallelized across environment building, human solving, and test development. The execution environment is built in Docker; the task is then solved by the benchmark builders from a base image; dependencies encountered during solving are recorded; and the final deliverables are separated into a Dockerfile and a solution repository. This separation serves two purposes: it produces a reproducible runtime and a working reference solution that demonstrates solvability.
Each task package contains the following components: Initial Repo, Task Requirement, Environment, Solution Repo, Tests, Scoring Parser, and Metadata. The metadata includes task type, domain, difficulty, and estimated human completion time. The tests are written from the requirement document only, not from the human solution, in order to avoid encoding solution-specific implementation choices into the evaluator.
A task is admitted only if it satisfies the benchmark’s verification condition: F2P tests fail on the initial repo, F2P tests pass on the solution repo, P2P tests pass on the initial repo, and P2P tests pass on the solution repo. Construction is iterative: ambiguous documents are revised, tests are rerun on both initial and solution repositories, failures are manually inspected, and any task that still cannot satisfy the condition after three iterations is discarded. Final expert review precedes inclusion.
3. Task taxonomy and long-horizon characteristics
LongCLI-Bench partitions its tasks into four engineering categories: From Scratch , Feature Addition , Bug Fix , and Refactor (Feng et al., 15 Feb 2026). These categories are intended to cover the principal structural modes of repository-level engineering rather than a single issue-fixing format. The benchmark also spans six domains: System Programming, Web Development, Data Engineering, Machine Learning, Applications, and DevOps.
Long-horizon difficulty is defined structurally rather than by token count alone. A task is long-horizon because it requires the agent to inspect a large repository, identify and follow correct entry points, work within dependencies, perform multiple interdependent edits, verify intermediate system state, recover from failures, and preserve prior functionality. The benchmark explicitly states that many tasks contain a “complete requirement chain where the outcome of previous steps determines the feasibility of subsequent ones.”
A quantitative comparison with Terminal-Bench@2 is used to substantiate this claim of scale and dependency:
| Measure | Terminal-Bench@2 | LongCLI-Bench |
|---|---|---|
| Tasks | 89 | 20 |
| Files | 0.69 | 104.0 |
| LoC | 227.7 | 15,000+ |
| Expert Time (min) | 206.7 | 1000+ |
These statistics locate LongCLI-Bench at repository scale. On average, each task involves 104.0 source files, 15,000+ lines of code, and 1000+ minutes of expert completion time. The contrast with Terminal-Bench@2 is used to argue that the benchmark targets deeply interdependent engineering settings rather than short terminal episodes.
Step decomposition is also part of the task design. For assignments with built-in grading scripts, outputs are parsed into sub-step scores. For tasks without predefined tests, the requirements are manually segmented into sub-tasks and evaluated with corresponding scripts. This operationalizes long-horizon execution as a sequence of testable subgoals.
4. Evaluation protocol and scoring semantics
The benchmark’s evaluation protocol is built on a dual-set test design. Fail-to-Pass (F2P) tests measure requirement fulfillment: the initial repository should fail them, while a correct solution should pass them. Pass-to-Pass (P2P) tests measure regression avoidance: the initial repository should already pass them, and the modified repository should continue to do so (Feng et al., 15 Feb 2026).
Step-level scoring is defined as the percentage of sub-steps completed. The paper does not typeset a formal equation for this metric, but it states an exact binary criterion: a test set is considered passed only when its step score reaches 100\%. Consequently, F2P Pass means F2P step score , and P2P Pass means P2P step score . Overall task pass is defined as the percentage of tasks for which the agent passes both F2P and P2P.
The benchmark reports several aggregate quantities: Pass Rate (%), Pass@3 (%), F2P Pass (%), F2P Step Score (%), P2P Pass (%), P2P Step Score (%), and Time (minutes). The main results are averaged over three independent runs in a unified single-turn setting. LongCLI-Bench also supports multiple attempts and a self-correction mode in which the agent re-executes the task after receiving feedback from the previous turn’s test results.
The interaction model is deliberately constrained. Evaluation proceeds by initializing Docker with the initial repository, allowing the agent to work through the terminal, and then running the harnessed tests when the agent stops or times out. The benchmark text does not provide more detailed formal specifications for timeout policy, command budgets, shell restrictions, or structured tool interfaces beyond stating that agents interact through the terminal in the isolated environment.
5. Empirical results, error structure, and human guidance
In the main single-turn evaluation, all tested systems remain below 20% overall pass rate, establishing the benchmark as difficult even for frontier agentic coding systems (Feng et al., 15 Feb 2026). The benchmark evaluates commercial CLI assistants—Codex variants and Claude Code variants—as well as OpenHands configurations using DeepSeek-V3.1, GLM-4.6, and Qwen3-235B-A22B.
The best single-turn overall results are Claude-Opus-4.5 at 16.7% pass and Claude-Opus-4.6 at 16.7% pass, with Claude-Opus-4.6 attaining the highest Pass@3 at 25.0%. Among Codex configurations, GPT-5.3-Codex reaches 15.0% pass and 20.0% Pass@3. OpenHands-based systems perform substantially worse in overall pass, ranging from 5.0% to 6.7%.
A consistent asymmetry appears between requirement completion and regression preservation. F2P step scores range from 25.3% to 50.7%, whereas P2P step scores range from 98.4% to 99.7%. Yet the corresponding P2P pass rates are lower, from 70.0% to 88.3%, showing that regressions still occur on a nontrivial fraction of tasks even when average regression sub-step completion appears high. The benchmark interprets this as evidence that agents often preserve most prior behavior while still breaking enough functionality to fail strict pass criteria.
Step-score distribution analysis shows that many trajectories fail very early. The largest share of runs falls into the F2P step-score range . Representative values include 45.0% for GPT-5.1-Codex-Max, 41.7% for GPT-5.3-Codex, 38.3% for Claude-Opus-4.6, 65.0% for DeepSeek-V3.1, 63.3% for GLM-4.6, and 58.3% for Qwen3-235B-A22B. By contrast, the bucket is typically only 5.0%–15.0%, indicating that near-complete-but-unsolved trajectories are relatively uncommon. The benchmark uses this pattern to support its claim that strong sequential dependency makes early prerequisite failures especially costly.
Self-correction improves performance, but only modestly relative to human guidance. The paper reports large gains from the first to the second round and smaller gains thereafter; it also notes that P2P behavior can become unstable as later-round edits expand the change scope. For example, GPT-5.3-Codex raises P2P pass to 95% at and 100% at , while GPT-5.1-Codex-Max rises to 85% at 0 and then falls to 80% at 1. Claude-Opus-4.6 rises to 95% at 2 and then falls to 90% at 3.
Human-agent collaboration yields much larger improvements. For GPT-5.3-Codex, pass rises from 15.0% in the base setting to 40.0% with self-correction, 41.7% with static plan injection, 45.0% with dynamic interactive guidance, and 50.0% with combined plan plus interaction. Its F2P Score rises from 44.1 to 58.4, 59.3, 61.3, and 62.3, respectively. For Claude-Opus-4.6, pass rises from 16.7% to 55.0% with self-correction, 58.3% with plan injection, 58.3% with interactive guidance, and 61.7% with combined assistance; its F2P Score rises from 50.7 to 63.6, 65.4, 67.4, and 69.3. The average number of interventions also falls when plans are pre-injected: from 2.7 to 2.2 for GPT-5.3-Codex and from 2.4 to 2.1 for Claude-Opus-4.6.
Manual inspection of 50 failed trajectories identifies three dominant failure modes: repetitive loops from weak strategic adaptation, environment grounding and verification gaps, and long-term inconsistency and regression. A notable negative result is that these failures are described as being “rarely caused by local syntax-level coding mistakes.” The benchmark instead localizes the main bottleneck at workflow-level coherence.
6. Position within the benchmark landscape and acknowledged limitations
LongCLI-Bench occupies a distinct position relative to several neighboring arXiv benchmarks. Compared with text-centric long-context understanding benchmarks such as "MiniLongBench: The Low-cost Long Context Understanding Benchmark for LLMs" (Huang et al., 26 May 2025) and "CLongEval: A Chinese Benchmark for Evaluating Long-Context LLMs" (Qiu et al., 2024), LongCLI-Bench is centered on stateful terminal execution, environment grounding, repository mutation, and regression control rather than on static document understanding. Compared with "CL-bench: A Benchmark for Context Learning" (Dou et al., 3 Feb 2026), it is less about learning novel knowledge from provided textual context and more about executing long engineering trajectories in isolated CLI environments. Compared with "Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments" (Asawa et al., 4 Jun 2026), its emphasis is not online improvement across long schedules of related instances, but long-horizon completion of individual software-engineering tasks. Compared with "WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation" (Ding et al., 11 May 2026), it is more narrowly focused on repository-scale software engineering, whereas WildClawBench spans 60 native-runtime tasks across six thematic categories, including multimodal and cross-tool workflows.
This comparison suggests that LongCLI-Bench isolates a specific evaluative niche: long-horizon command-line software engineering with explicit separation between requirement fulfillment and regression avoidance, and with step-level progress diagnostics rather than final-answer-only scoring. Its most direct historical contrast is with Terminal-Bench@2, against which it claims much larger repositories, longer expert completion time, more contamination-aware sourcing, and finer-grained evaluation.
The benchmark’s acknowledged limitations are substantial. Its scale is small, at 20 tasks, because manual construction is expensive; the paper reports that each task takes an average of 40 hours to complete (Feng et al., 15 Feb 2026). Step-level scores do not fully capture code quality, efficiency, or broader software-engineering quality criteria. The benchmark text also highlights open challenges exposed by the results: robust long-horizon planning, maintaining context over extended workflows, environment-aware diagnosis, avoiding regressions while making nontrivial edits, escaping repetitive failure loops, and balancing self-correction against change-scope expansion.
Within the current benchmark ecosystem, LongCLI-Bench is therefore best understood not as a generic long-context benchmark, but as a command-line programming benchmark that operationalizes long-horizon capability through repository-scale tasks, dual-set testing, and partial-progress analysis. Its empirical finding is that contemporary agents remain far from reliable autonomous performance in this regime, while its methodological finding is that human strategic guidance can raise outcomes far beyond what autonomous single-turn execution presently achieves.