unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning

Abstract: Unix competence is the ability to use shell and operating-system primitives as first-class tools, not merely to write programs through a terminal. Current terminal benchmarks tend to blur this distinction: a solver fluent in Python but weak in Unix can pass a substantial fraction of Terminal-Bench 2.0, while the reverse skill profile is rarely exercised. We make the distinction operational and build a training surface for the Unix component. unix-ctf is a procedural generator of capture-the-flag tasks for shell agents. Each task hides a short token (a flag of the form flag(a3b1c9...)) inside a fresh Linux container using a single Unix feature, and the agent must recover it. Tasks are produced by an LLM-assisted synthesis pipeline that generates candidate hiding techniques, rewrites them into parameterized hide-and-find script pairs, and filters them with a bidirectional contract: the hide script must leave no plaintext trace of the flag on disk, and the find script must recover the flag in a fresh directory. Because the LLM only writes the planting and recovery steps (the container, layout, and grading harness are fixed), the pipeline lands 656 of 750 raw attempts as portable, reusable variants (87.5\%). Our reproduction of Endless Terminals' full-container-generation approach lands only 17.4\% under the same checks. The 656 variants canonicalize to 155 distinct techniques. Fine-tuning Qwen3-8B with LoRA using GRPO on this surface lifts solve rate from 11.6\% to 43.6\% on a 15-skill multi-family holdout (n=225), redistributes which InterCode-CTF tasks the model solves, and produces a +33 pp gain in Forensics while reaching 32/100 on InterCode-CTF. These results suggest that Unix competence is separable, trainable, and best evaluated directly rather than folded into programming-through-a-shell.

• The paper presents a two-layer pipeline that harvests a diverse skill library and generates Unix-competence tasks via mechanical validation and LM-driven exploration.
• It demonstrates significant RL improvements, achieving up to a 43.6% solve rate on out-of-distribution benchmarks through focused Unix-specific challenges like ELF manipulation and IPC.
• The study reveals a trade-off between high training rewards and effective downstream transfer, underscoring the importance of curriculum design in reinforcement learning.

Procedural Generation of Unix-Competence Environments for RL Agents

Motivation and Benchmark Taxonomy

The paper presents unix-ctf, a procedural generation framework for evaluating and training RL agents on Unix-competence skills distinct from general terminal coding. The authors argue most terminal benchmarks conflate shell proficiency (programming via terminal) with Unix-system knowledge (OS, shell, file-format features inaccessible to regular programming paradigms) and demonstrate major benchmarks—Terminal-Bench, Cybench, AgentBench-OS, InterCode—rarely probe Unix-competence directly. Tasks solved by generic glue (awk, grep, find, sed, SQLite) have clean cross-language analogues; tasks that depend on irreducible Unix artifacts (e.g., extended file attributes, ELF internals, IPC primitives) are substantially underrepresented.

This dichotomy is quantitatively analyzed via rubric-driven task taxonomy, revealing Unix-competence tasks represent less than 30% of most benchmarks, and procedural-generation pipelines fail to generate rich shells for skill acquisition.

Figure 1: Distribution of task-types across terminal benchmarks, showing Unix-competence tasks as a minority.

Pipeline Architecture and Skill Library Construction

Unix-ctf constructs a scalable procedural evaluation and training surface via a two-layer pipeline: offline skill library harvest and on-demand environment generation. The library is built by sequential LM-driven exploration, mechanical validation (no plaintext flag on disk, canonical recovery validated), parameterized script synthesis (plant.sh, recovery.sh), portability checks, and template-based deduplication. Harvesting yields 155 canonical techniques spanning 16 Unix-surfaces, aggregated from 441 deduplicated variants; representative categories include configuration files, binary and ELF manipulation, IPC, X.509 certs, filesystem traits, encoding/steganography, and OS process artifacts.

Task-quality audits via three independent LLM judges indicate unix-ctf tasks are superior on input richness, solution non-triviality, format independence, tool diversity, and discovery requirements compared to prior LLM-driven generators, with nearly two points aggregate advantage.

Figure 2: Unix-ctf’s skill library coverage and production pipeline, illustrating high task yield and diversity relative to prior generators.

Training Regimen and Environment Scheduling

For agent fine-tuning, the environment generator samples batches of eight techniques per Docker container, each role-dressed as plausible server types. Flags are planted via canonical plant.sh scripts, deleted post-execution, and agents must find and print them via RL interaction. Reward is densified (multi-flag per episode), with scheduling via a sticky-pool mechanism: high-solve-rate techniques are retired per batch, replaced by low-frequency skills from the library, ensuring broad coverage and curricular exposure.

GRPO (Group Relative Policy Optimization) is employed for RL; Qwen3-8B serves as the base agent, optionally pre-adapted via supervised format pass. Training is treated primarily as a probe of the procedural surface, not a recipe for optimal RL.

Empirical Evaluation and Generalization

Evaluation is performed across three benchmarks: (1) InterCode-CTF (100 tasks), (2) InterCode-Bash filesystem variant (60 tasks), and (3) an internal multi-family holdout (225 tasks across 15 OOD skills). The principal findings:

• On the multi-family holdout, GRPO from SFT init achieves 43.6% solve rate, a 3.8x lift over base; GRPO from base reaches 27.6%.
• On IC-CTF, training redistributes agent capabilities: the largest observed gain is Forensics (+33 pp), indicating transfer of Unix-system knowledge; overall aggregate does not substantially increase.
• There is a decoupling between in-distribution training reward and OOD downstream transfer—GRPO from SFT init peaks in training reward but underperforms in downstream solve rate versus GRPO from base.

  Figure 3: Multi-family holdout results, demonstrating significant performance gains for agents trained with the unix-ctf procedural surface.

  

  Figure 4: IC-CTF capability redistribution: training enhances skill acquisition in Unix-proximal task categories.

Reward Dynamics and Pool Rotation

The decoupling of training reward and downstream transfer further manifests in reward curves; GRPO-from-SFT peaks at much higher batch reward but achieves lower IC-CTF solve rate than GRPO-from-base. This indicates format pre-adaptation aids harness-specific recovery but may impede generalization across distributions.

Figure 5: GRPO training-reward curves over batches, showing reward–transfer inversion between initialization variants.

Theoretical Implications and Prospects

Unix-ctf delineates rigorous evaluation methodology for distinguishing Unix-competence from general coding proficiency and provides a mechanistically validated procedural-generation approach. The multi-step LM harvest, deduplication, and mechanical evaluation contract yield robust task surfaces that probe non-trivial OS knowledge.

Practically, unix-ctf enables more granular RL agent assessment, curriculum generation, and up-skilling for real-world server management and offensive security (red-team/blue-team training). The approach’s limitations reside in reliance on LMs for skill harvest (potential bias toward overrepresented techniques) and the CTF-task-type framing (excluding broader Unix workflows). Directions for further research include curriculum learning, adaptive task difficulty scaling, multi-agent interaction, and transfer studies to real-world system administration settings.

Conclusion

Unix-ctf establishes a procedural pipeline for Unix-competence RL evaluation, separating OS-internal skills from generic shell coding. Empirical evidence underscores that task surfaces constructed via canonicalized library and mechanical contracts significantly enhance agent acquisition of Unix-specific knowledge, and that training on such surfaces produces robust capability transfer—especially in forensics and reverse engineering—across standard benchmarks. The methodology advances procedural-task generation, curriculum structuring, and systematic evaluation for shell-based agent learning.