---
title: 'CLI-Gym: Scalable CLI Repair Task Pipeline'
url: https://www.emergentmind.com/topics/cli-gym
type: topic
---

# CLI-Gym: Scalable CLI Repair Task Pipeline

CLI-Gym is a pipeline for generating and packaging **environment-intensive** command-line repair tasks at scale. It is designed for agentic coding settings in which the primary difficulty lies not mainly in editing source code, but in diagnosing and repairing a live runtime environment through a CLI: resolving dependency issues, fixing problematic configurations, repairing broken environment variables, and addressing system-level failures. Its central mechanism is **agentic environment inversion**: starting from a healthy Dockerized repository environment with passing tests, an agent deliberately perturbs the environment until tests fail, after which the broken state, failing tests, and synthesized issue description are packaged as a repair task [2602.10999].

## 1. Definition and problem setting

CLI-Gym addresses a gap between **code-intensive** benchmarks and **environment-intensive** tasks. The paper contrasts these two regimes directly: code-intensive datasets can be generated automatically at scales of \(10^3\)–\(10^4\) from repository history, whereas environment-intensive benchmarks such as **Terminal-Bench 1.0** and **Terminal-Bench 2.0** contain only **80** and **89** tasks respectively and rely on human labor [2602.10999]. The target tasks are those in which successful agent behavior depends on interacting with a runtime environment rather than only modifying source files.

The paper identifies three reasons why scalable generation of such tasks is difficult. First, there are **no explicit environment histories** analogous to git history. Second, **environment-intensive issues are rare and hard to identify** among repository issues. Third, **Dockerfiles are static specifications**, not naturally paired with failure contexts and historical traces [2602.10999]. CLI-Gym is proposed as a response to this data bottleneck.

The method is grounded in an analogy between a **Dockerfile** and an **agentic task**. A Dockerfile is treated as a sequence of state-transforming commands over an environment, which suggests that environment failure generation can be formulated as a reverse trajectory from a known-good endpoint. The paper formalizes runnable environment state as
\[
\mathcal{S} = (\mathcal{B}, \mathcal{D}, \mathcal{C}),
\]
where \(\mathcal{B}\) is the base environment, \(\mathcal{D}\) is the Dockerfile, and \(\mathcal{C}\) is the software codebase [2602.10999].

This framing distinguishes CLI-Gym from other Gym-style systems. **CompilerGym** exposes compiler optimization tasks through an OpenAI Gym API [2109.08267], **Ecole** exposes solver-internal decision problems as control environments [2011.06069], and **ns3-gym** turns ns-3 simulations into Gym-compatible networking environments [1810.03943]. CLI-Gym instead targets **CLI/runtime-environment interaction** in software engineering workflows. A plausible implication is that it belongs to a newer class of task-generation systems in which the environment itself, rather than only the code repository, becomes the object of inversion and supervision.

## 2. Agentic environment inversion

The core CLI-Gym mechanism is **environment inversion by agentic Dockerfile coding**. The generation process begins from a **gold instance**: a Dockerized repository environment in which all unit tests pass. The paper states that these gold instances follow the protocol of **SWE-Smith**, and that the final corpus uses **29 open-source repositories**, selected randomly from the curated SWE-Smith repository set [2602.10999].

For each gold repository, the system generates candidate degradation tasks. The appendix reports that it **randomly samples 1–3 intervention directions** and **randomly selects 200 unit tests** from the repository. These selected tests, together with previous task titles, are injected into a prompt for an LLM, with a memory pool of earlier task descriptions used to encourage diversity and reduce repetition [2602.10999]. The resulting sabotage prompt asks for a **Task Name**, **Category**, **Selected UTs**, **Task Description**, **Expected Result**, and **Recovery Strategy**.

The inversion stage is executed with a modified **Terminal-Bench harness** using **OpenHands** as the agent framework. The agent is placed in the healthy environment and instructed to deliberately induce failures while interacting freely with the environment. The available tools include `execute_bash`, `think`, `finish`, `execute_ipython_cell`, `task_tracker`, and `str_replace_editor` [2602.10999]. The harness runs each inversion attempt for approximately **15 minutes** and terminates when the agent emits `final_thought` or `finish`.

Acceptance is determined by test outcomes rather than textual plausibility. After sabotage, the harness runs the selected tests through `run-tests.sh`. If some unit tests fail, the failed tests are recorded as **fail-to-pass** and the successful selected tests as **pass-to-pass**. If the test command fails to execute, all selected unit tests are recorded as fail-to-pass. If all selected tests still pass, the attempt is discarded [2602.10999]. In the paper’s terminology, \(\mathcal{S}_{\text{poor}}\) is an environment state that fails at least one unit test, whereas \(\mathcal{S}_{\text{gold}}\) is a state in which all unit tests pass.

This yields a reproducible packaging workflow. The broken environment is reconstructed as a Dockerfile, the observed failures are converted into a natural-language repair issue, and the environment is bundled with evaluation scripts. The paper states that each execution package contains `docker-compose.yaml`, `Dockerfile`, `run-tests.sh`, and `task.yaml` [2602.10999]. This makes the resulting tasks executable and objectively verifiable.

## 3. Dataset construction and task structure

CLI-Gym produces **1,655 environment-intensive tasks** from **29 Python repositories**, and the paper describes this as the **largest public collection of its kind** [2602.10999]. Appendix statistics report **4,066 task prompts** yielding **1,655 problem instances**, which indicates substantial filtering and execution-based acceptance during generation.

Table 2 in the paper summarizes several corpus-level statistics [2602.10999]:

| Attribute | CLI-Gym |
|---|---:|
| # Instances | 1655 |
| # Images | 29 |
| Issue text length | 159.1 words on average |
| Dockerfile lines | 6.8 on average |
| # Fail-to-pass tests | 20.4 on average |
| # Pass-to-pass tests | 29.6 on average |
| Cost | 2.3B tokens |

Repository contributions are uneven. Examples listed in the appendix include `pandas` with **242** tasks, `gunicorn` with **160**, `cantools` with **120**, `scrapy` with **119**, `tweepy` with **110**, and `ptyprocess` with **109**; at the low end, `click` contributes **2** tasks, `cookiecutter` **4**, `feedparser` **4**, and `modin` **5** [2602.10999]. The paper also states that the tasks span a broad set of application domains, though exact numeric category counts are not given in the text.

Each task instance contains three core elements: an executable broken environment, a natural-language issue description, and a set of unit tests specifying desired behavior [2602.10999]. This makes the tasks more environment-intensive than ordinary coding benchmarks because they may require diagnosing system libraries, dependency mismatches, locale issues, shell or configuration errors, package or environment breakage, file permissions, or binary corruption.

A representative example given in the paper corrupts critical shared libraries in a pandas environment by copying `libsqlite3.so` and `libz.so`, zeroing bytes in their ELF headers with `dd`, and copying the corrupted files back into the runtime library directory [2602.10999]. The purpose of the example is to show that the resulting failure is not a conventional source-level bug. Repair requires system-level diagnosis of import and dynamic-linking failures.

This suggests that CLI-Gym tasks are structurally closer to runtime debugging and environment repair than to repository patch generation. The paper’s use of both **fail-to-pass** and **pass-to-pass** tests further indicates that preservation of unaffected behavior is part of the task contract, not merely restoration of one failing path.

## 4. Training trajectories and LiberCoder

CLI-Gym is also used as a source of successful repair trajectories for supervised fine-tuning. The paper reports **417 successful trajectories** collected by running strong language models as policy rollout agents on the 1,655 generated tasks, and **291 high-quality trajectories** after filtering [2602.10999]. The terminology used is **Raw-Success Traj** for the former and **Filtered-Success Traj** for the latter.

The filtering rules given in the appendix are explicit. First, **trajectories with fewer than 20 steps** are removed because they are considered too trivial. Second, **cheating trajectories** are removed, including those that exploit historical artifacts such as cached Git information or Conda logs [2602.10999]. The paper also notes that it filters out trajectories that exploit shortcuts or correspond to trivially easy problems.

These trajectories are used to train **LiberCoder**, with two reported model variants: **LiberCoder-32B** based on **Qwen3-32B**, and **LiberCoder-235B-A22B** based on **Qwen3-235B-A22B-Instruct** [2602.10999]. Training uses a two-stage recipe. Stage 1 fine-tunes on **48K open-source SWE-style trajectories**, explicitly disjoint from Terminal-Bench, to improve general agentic coding capability. Stage 2 fine-tunes on CLI-Gym’s **Filtered-Success Traj** to specialize the model for terminal-centric environment repair [2602.10999].

For **Qwen3-32B**, the paper reports a learning rate of \(2 \times 10^{-5}\), cosine decay, warmup over the first **5%** of steps, epochs of **10**, **15**, and **20**, batch size **16**, agent template `qwen3-coder`, and maximum sequence length **100k tokens**, extended from native **40k** using **YaRN** with expansion factor **2.5** [2602.10999]. For **Qwen3-235B-A22B-Instruct**, the learning rate is \(1 \times 10^{-5}\), with the same warmup fraction, the same candidate epoch counts, agent template `qwen3-coder`, and maximum sequence length **100k tokens** [2602.10999]. At inference time, evaluation uses **OpenHands**, greedy decoding with \(temperature = 0\), and maximum context length **128k tokens**.

The paper does not provide a formal imitation-learning objective in symbolic form. Still, the described setup is standard supervised fine-tuning over successful agent trajectories. A plausible implication is that the key novelty lies less in the optimizer than in the provenance and structure of the supervision: long-horizon environment-repair traces grounded in real execution and verification.

## 5. Evaluation and empirical findings

The main evaluation benchmarks are **Terminal-Bench 1.0** and **Terminal-Bench 2.0**, with **80** and **89** tasks respectively [2602.10999]. Success is measured by the benchmark’s verification scripts, with **pass@1** defined as the proportion of tasks solved in one attempt and **pass@3** as the proportion solved by at least one successful run among three attempts.

The paper’s headline result is that **LiberCoder-235B-A22B** reaches **46.1% pass@1** on Terminal-Bench 1.0, an **absolute +21.1% improvement** over the base **Qwen3-235B-A22B-Instruct** score of **25.0%** [2602.10999]. On Terminal-Bench 2.0, the same model reaches **31.0%**, an **absolute +12.9%** over the base score of **18.1%**. For the smaller model, **Qwen3-32B** scores **10.3%** on Terminal-Bench 1.0 and **5.7%** on Terminal-Bench 2.0, whereas **LiberCoder-32B** reaches **38.9%** and **19.5%** respectively [2602.10999].

The ablation in Table 4 shows that CLI-Gym data contributes more than SWE-only supervision for these terminal tasks [2602.10999]. On Terminal-Bench 1.0 pass@1:

| Model setting | Qwen3-32B | Qwen3-235B-A22B-Instruct |
|---|---:|---:|
| Base | 10.3 | 25.0 |
| SWE only | 22.1 | 28.6 |
| CLI-Gym only | 32.4 | 39.5 |
| SWE + CLI-Gym | 38.9 | 46.1 |

The paper also studies filtering effects. For Qwen3-32B on Terminal-Bench 1.0, **Filtered-only** yields **32.4**, **Raw-only** yields **33.8**, **SWE + Raw** yields **36.4**, and **SWE + Filtered** yields **38.9** [2602.10999]. The interpretation given is that once basic agentic capability is established, **trajectory quality matters more than quantity**.

Two further findings are emphasized. First, performance improves monotonically when the number of source repositories increases while holding the trajectory budget fixed at **100**, indicating that **environment diversity matters independently of data count** [2602.10999]. Second, performance rises consistently with more of the **291** filtered trajectories but begins to plateau after roughly **200 trajectories**.

These results are particularly notable when placed beside later work on CLI interaction. The matched execution-layer benchmark in **“GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents”** reports that CLI performance can be heavily constrained by interface coverage rather than model capability alone, with verifier-guided skill augmentation raising CLI success from **48.2%** to **69.3%** [2606.24551]. CLI-Gym’s results are consistent with the broader view that CLI competence depends strongly on exposure to environment-grounded execution structure, not only on generic coding skill.

## 6. Relation to adjacent systems, limitations, and significance

CLI-Gym sits alongside several adjacent developments in Gym-style environment design and agent-native interfaces. **CLI-Anything** argues for “agent-native computer use” through structured commands, explicit state, deterministic feedback, and machine-readable protocols, formalizing a harness as \(H=(S,C,I,R,V,D)\) [2606.03854]. CLI-Gym does not present such a general harness contract, but it shares a core commitment to executable, verifiable CLI interaction rather than GUI mediation. **Ecole**, **CompilerGym**, and **Gym-preCICE** show analogous design moves in other domains: expose a complex backend through a standardized control interface while preserving realistic dynamics [2011.06069; 2109.08267; 2305.02033].

The paper also has explicit limits. The current dataset is built from **29 Python repositories** and is demonstrated only in Python/containerized settings [2602.10999]. The pipeline depends on **OpenHands** during both generation and evaluation, and the appendix shows that model performance can vary materially by agent. Although tasks originate from real repositories and test suites, some degradations remain synthetic sabotage rather than naturally occurring failures. The paper attempts to mitigate this through prompt design, filtering, and execution-based validation, but it does not report a human realism study.

Several additional constraints emerge from the reported numbers. From **4,066** task prompts only **1,655** valid problem instances are retained, implying significant attrition during generation [2602.10999]. The generation cost is **2.3B tokens**. The paper also notes that, after training, models may explore more aggressively, which can increase the chance of exceeding the **128k** inference context length. Categories such as **gaming** and **scientific computing** remain challenging and are not addressed well by the current method [2602.10999].

Its broader significance lies in shifting the source of scalable supervision from repository history to **simulated environment-state history**. Earlier Gym-like platforms standardized interaction with compilers, solvers, simulators, or cyber ranges [2109.08267; 2011.06069; 1810.03943; 2109.03331]. CLI-Gym standardizes the derivation of training tasks for CLI-based environment repair by starting from a working environment, inverting it into failure, and packaging the result with executable verification [2602.10999]. This suggests a distinct research direction in which the environment itself becomes the unit of task construction.

In that sense, CLI-Gym is not primarily a benchmark in the style of a fixed hand-authored suite, nor primarily a general-purpose CLI harness platform. It is a **public pipeline for scalable derivation of environment-intensive tasks**, coupled to a trajectory-collection and fine-tuning workflow [2602.10999]. Its central contribution is methodological: it makes CLI/runtime-environment supervision scalable in a way analogous to how repository history previously made code-centric supervision scalable.

Source: https://www.emergentmind.com/topics/cli-gym