CLI-Gym: Scalable CLI Repair Task Pipeline
- CLI-Gym is a pipeline that creates environment-intensive CLI repair tasks by simulating runtime failures in Dockerized repositories.
- It uses agentic environment inversion, where an agent deliberately induces failures to transform a healthy environment into a repair scenario.
- The system enables scalable task generation with execution-based validation, producing extensive datasets for training models like LiberCoder.
CLI-Gym is a pipeline for generating and packaging environment-intensive command-line repair tasks at scale. It is designed for agentic coding settings in which the primary difficulty lies not mainly in editing source code, but in diagnosing and repairing a live runtime environment through a CLI: resolving dependency issues, fixing problematic configurations, repairing broken environment variables, and addressing system-level failures. Its central mechanism is agentic environment inversion: starting from a healthy Dockerized repository environment with passing tests, an agent deliberately perturbs the environment until tests fail, after which the broken state, failing tests, and synthesized issue description are packaged as a repair task (Lin et al., 11 Feb 2026).
1. Definition and problem setting
CLI-Gym addresses a gap between code-intensive benchmarks and environment-intensive tasks. The paper contrasts these two regimes directly: code-intensive datasets can be generated automatically at scales of – from repository history, whereas environment-intensive benchmarks such as Terminal-Bench 1.0 and Terminal-Bench 2.0 contain only 80 and 89 tasks respectively and rely on human labor (Lin et al., 11 Feb 2026). The target tasks are those in which successful agent behavior depends on interacting with a runtime environment rather than only modifying source files.
The paper identifies three reasons why scalable generation of such tasks is difficult. First, there are no explicit environment histories analogous to git history. Second, environment-intensive issues are rare and hard to identify among repository issues. Third, Dockerfiles are static specifications, not naturally paired with failure contexts and historical traces (Lin et al., 11 Feb 2026). CLI-Gym is proposed as a response to this data bottleneck.
The method is grounded in an analogy between a Dockerfile and an agentic task. A Dockerfile is treated as a sequence of state-transforming commands over an environment, which suggests that environment failure generation can be formulated as a reverse trajectory from a known-good endpoint. The paper formalizes runnable environment state as
where is the base environment, is the Dockerfile, and is the software codebase (Lin et al., 11 Feb 2026).
This framing distinguishes CLI-Gym from other Gym-style systems. CompilerGym exposes compiler optimization tasks through an OpenAI Gym API (Cummins et al., 2021), Ecole exposes solver-internal decision problems as control environments (Prouvost et al., 2020), and ns3-gym turns ns-3 simulations into Gym-compatible networking environments (Gawłowicz et al., 2018). CLI-Gym instead targets CLI/runtime-environment interaction in software engineering workflows. A plausible implication is that it belongs to a newer class of task-generation systems in which the environment itself, rather than only the code repository, becomes the object of inversion and supervision.
2. Agentic environment inversion
The core CLI-Gym mechanism is environment inversion by agentic Dockerfile coding. The generation process begins from a gold instance: a Dockerized repository environment in which all unit tests pass. The paper states that these gold instances follow the protocol of SWE-Smith, and that the final corpus uses 29 open-source repositories, selected randomly from the curated SWE-Smith repository set (Lin et al., 11 Feb 2026).
For each gold repository, the system generates candidate degradation tasks. The appendix reports that it randomly samples 1–3 intervention directions and randomly selects 200 unit tests from the repository. These selected tests, together with previous task titles, are injected into a prompt for an LLM, with a memory pool of earlier task descriptions used to encourage diversity and reduce repetition (Lin et al., 11 Feb 2026). The resulting sabotage prompt asks for a Task Name, Category, Selected UTs, Task Description, Expected Result, and Recovery Strategy.
The inversion stage is executed with a modified Terminal-Bench harness using OpenHands as the agent framework. The agent is placed in the healthy environment and instructed to deliberately induce failures while interacting freely with the environment. The available tools include execute_bash, think, finish, execute_ipython_cell, task_tracker, and str_replace_editor (Lin et al., 11 Feb 2026). The harness runs each inversion attempt for approximately 15 minutes and terminates when the agent emits final_thought or finish.
Acceptance is determined by test outcomes rather than textual plausibility. After sabotage, the harness runs the selected tests through run-tests.sh. If some unit tests fail, the failed tests are recorded as fail-to-pass and the successful selected tests as pass-to-pass. If the test command fails to execute, all selected unit tests are recorded as fail-to-pass. If all selected tests still pass, the attempt is discarded (Lin et al., 11 Feb 2026). In the paper’s terminology, is an environment state that fails at least one unit test, whereas is a state in which all unit tests pass.
This yields a reproducible packaging workflow. The broken environment is reconstructed as a Dockerfile, the observed failures are converted into a natural-language repair issue, and the environment is bundled with evaluation scripts. The paper states that each execution package contains docker-compose.yaml, Dockerfile, run-tests.sh, and task.yaml (Lin et al., 11 Feb 2026). This makes the resulting tasks executable and objectively verifiable.
3. Dataset construction and task structure
CLI-Gym produces 1,655 environment-intensive tasks from 29 Python repositories, and the paper describes this as the largest public collection of its kind (Lin et al., 11 Feb 2026). Appendix statistics report 4,066 task prompts yielding 1,655 problem instances, which indicates substantial filtering and execution-based acceptance during generation.
Table 2 in the paper summarizes several corpus-level statistics (Lin et al., 11 Feb 2026):
| Attribute | CLI-Gym |
|---|---|
| # Instances | 1655 |
| # Images | 29 |
| Issue text length | 159.1 words on average |
| Dockerfile lines | 6.8 on average |
| # Fail-to-pass tests | 20.4 on average |
| # Pass-to-pass tests | 29.6 on average |
| Cost | 2.3B tokens |
Repository contributions are uneven. Examples listed in the appendix include pandas with 242 tasks, gunicorn with 160, cantools with 120, scrapy with 119, tweepy with 110, and ptyprocess with 109; at the low end, click contributes 2 tasks, cookiecutter 4, feedparser 4, and modin 5 (Lin et al., 11 Feb 2026). The paper also states that the tasks span a broad set of application domains, though exact numeric category counts are not given in the text.
Each task instance contains three core elements: an executable broken environment, a natural-language issue description, and a set of unit tests specifying desired behavior (Lin et al., 11 Feb 2026). This makes the tasks more environment-intensive than ordinary coding benchmarks because they may require diagnosing system libraries, dependency mismatches, locale issues, shell or configuration errors, package or environment breakage, file permissions, or binary corruption.
A representative example given in the paper corrupts critical shared libraries in a pandas environment by copying libsqlite3.so and libz.so, zeroing bytes in their ELF headers with dd, and copying the corrupted files back into the runtime library directory (Lin et al., 11 Feb 2026). The purpose of the example is to show that the resulting failure is not a conventional source-level bug. Repair requires system-level diagnosis of import and dynamic-linking failures.
This suggests that CLI-Gym tasks are structurally closer to runtime debugging and environment repair than to repository patch generation. The paper’s use of both fail-to-pass and pass-to-pass tests further indicates that preservation of unaffected behavior is part of the task contract, not merely restoration of one failing path.
4. Training trajectories and LiberCoder
CLI-Gym is also used as a source of successful repair trajectories for supervised fine-tuning. The paper reports 417 successful trajectories collected by running strong LLMs as policy rollout agents on the 1,655 generated tasks, and 291 high-quality trajectories after filtering (Lin et al., 11 Feb 2026). The terminology used is Raw-Success Traj for the former and Filtered-Success Traj for the latter.
The filtering rules given in the appendix are explicit. First, trajectories with fewer than 20 steps are removed because they are considered too trivial. Second, cheating trajectories are removed, including those that exploit historical artifacts such as cached Git information or Conda logs (Lin et al., 11 Feb 2026). The paper also notes that it filters out trajectories that exploit shortcuts or correspond to trivially easy problems.
These trajectories are used to train LiberCoder, with two reported model variants: LiberCoder-32B based on Qwen3-32B, and LiberCoder-235B-A22B based on Qwen3-235B-A22B-Instruct (Lin et al., 11 Feb 2026). Training uses a two-stage recipe. Stage 1 fine-tunes on 48K open-source SWE-style trajectories, explicitly disjoint from Terminal-Bench, to improve general agentic coding capability. Stage 2 fine-tunes on CLI-Gym’s Filtered-Success Traj to specialize the model for terminal-centric environment repair (Lin et al., 11 Feb 2026).
For Qwen3-32B, the paper reports a learning rate of , cosine decay, warmup over the first 5% of steps, epochs of 10, 15, and 20, batch size 16, agent template qwen3-coder, and maximum sequence length 100k tokens, extended from native 40k using YaRN with expansion factor 2.5 (Lin et al., 11 Feb 2026). For Qwen3-235B-A22B-Instruct, the learning rate is , with the same warmup fraction, the same candidate epoch counts, agent template qwen3-coder, and maximum sequence length 100k tokens (Lin et al., 11 Feb 2026). At inference time, evaluation uses OpenHands, greedy decoding with 0, and maximum context length 128k tokens.
The paper does not provide a formal imitation-learning objective in symbolic form. Still, the described setup is standard supervised fine-tuning over successful agent trajectories. A plausible implication is that the key novelty lies less in the optimizer than in the provenance and structure of the supervision: long-horizon environment-repair traces grounded in real execution and verification.
5. Evaluation and empirical findings
The main evaluation benchmarks are Terminal-Bench 1.0 and Terminal-Bench 2.0, with 80 and 89 tasks respectively (Lin et al., 11 Feb 2026). Success is measured by the benchmark’s verification scripts, with pass@1 defined as the proportion of tasks solved in one attempt and pass@3 as the proportion solved by at least one successful run among three attempts.
The paper’s headline result is that LiberCoder-235B-A22B reaches 46.1% pass@1 on Terminal-Bench 1.0, an absolute +21.1% improvement over the base Qwen3-235B-A22B-Instruct score of 25.0% (Lin et al., 11 Feb 2026). On Terminal-Bench 2.0, the same model reaches 31.0%, an absolute +12.9% over the base score of 18.1%. For the smaller model, Qwen3-32B scores 10.3% on Terminal-Bench 1.0 and 5.7% on Terminal-Bench 2.0, whereas LiberCoder-32B reaches 38.9% and 19.5% respectively (Lin et al., 11 Feb 2026).
The ablation in Table 4 shows that CLI-Gym data contributes more than SWE-only supervision for these terminal tasks (Lin et al., 11 Feb 2026). On Terminal-Bench 1.0 pass@1:
| Model setting | Qwen3-32B | Qwen3-235B-A22B-Instruct |
|---|---|---|
| Base | 10.3 | 25.0 |
| SWE only | 22.1 | 28.6 |
| CLI-Gym only | 32.4 | 39.5 |
| SWE + CLI-Gym | 38.9 | 46.1 |
The paper also studies filtering effects. For Qwen3-32B on Terminal-Bench 1.0, Filtered-only yields 32.4, Raw-only yields 33.8, SWE + Raw yields 36.4, and SWE + Filtered yields 38.9 (Lin et al., 11 Feb 2026). The interpretation given is that once basic agentic capability is established, trajectory quality matters more than quantity.
Two further findings are emphasized. First, performance improves monotonically when the number of source repositories increases while holding the trajectory budget fixed at 100, indicating that environment diversity matters independently of data count (Lin et al., 11 Feb 2026). Second, performance rises consistently with more of the 291 filtered trajectories but begins to plateau after roughly 200 trajectories.
These results are particularly notable when placed beside later work on CLI interaction. The matched execution-layer benchmark in “GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents” reports that CLI performance can be heavily constrained by interface coverage rather than model capability alone, with verifier-guided skill augmentation raising CLI success from 48.2% to 69.3% (Zhou et al., 22 Jun 2026). CLI-Gym’s results are consistent with the broader view that CLI competence depends strongly on exposure to environment-grounded execution structure, not only on generic coding skill.
6. Relation to adjacent systems, limitations, and significance
CLI-Gym sits alongside several adjacent developments in Gym-style environment design and agent-native interfaces. CLI-Anything argues for “agent-native computer use” through structured commands, explicit state, deterministic feedback, and machine-readable protocols, formalizing a harness as 1 (Yang et al., 2 Jun 2026). CLI-Gym does not present such a general harness contract, but it shares a core commitment to executable, verifiable CLI interaction rather than GUI mediation. Ecole, CompilerGym, and Gym-preCICE show analogous design moves in other domains: expose a complex backend through a standardized control interface while preserving realistic dynamics (Prouvost et al., 2020, Cummins et al., 2021, Shams et al., 2023).
The paper also has explicit limits. The current dataset is built from 29 Python repositories and is demonstrated only in Python/containerized settings (Lin et al., 11 Feb 2026). The pipeline depends on OpenHands during both generation and evaluation, and the appendix shows that model performance can vary materially by agent. Although tasks originate from real repositories and test suites, some degradations remain synthetic sabotage rather than naturally occurring failures. The paper attempts to mitigate this through prompt design, filtering, and execution-based validation, but it does not report a human realism study.
Several additional constraints emerge from the reported numbers. From 4,066 task prompts only 1,655 valid problem instances are retained, implying significant attrition during generation (Lin et al., 11 Feb 2026). The generation cost is 2.3B tokens. The paper also notes that, after training, models may explore more aggressively, which can increase the chance of exceeding the 128k inference context length. Categories such as gaming and scientific computing remain challenging and are not addressed well by the current method (Lin et al., 11 Feb 2026).
Its broader significance lies in shifting the source of scalable supervision from repository history to simulated environment-state history. Earlier Gym-like platforms standardized interaction with compilers, solvers, simulators, or cyber ranges (Cummins et al., 2021, Prouvost et al., 2020, Gawłowicz et al., 2018, Li et al., 2021). CLI-Gym standardizes the derivation of training tasks for CLI-based environment repair by starting from a working environment, inverting it into failure, and packaging the result with executable verification (Lin et al., 11 Feb 2026). This suggests a distinct research direction in which the environment itself becomes the unit of task construction.
In that sense, CLI-Gym is not primarily a benchmark in the style of a fixed hand-authored suite, nor primarily a general-purpose CLI harness platform. It is a public pipeline for scalable derivation of environment-intensive tasks, coupled to a trajectory-collection and fine-tuning workflow (Lin et al., 11 Feb 2026). Its central contribution is methodological: it makes CLI/runtime-environment supervision scalable in a way analogous to how repository history previously made code-centric supervision scalable.